Active learning & endangered languages

Don’t all AI methods need a bunch of data to work? How could AI help document and revitalize endangered languages with “human-in-the-loop” or “active learning” methods? Sarah Moeller from the University of Florida joins us to discuss those and other related questions. She also shares many of her personal experiences working with languages in low resource settings. Featuring: Sarah Moeller – Website https://sarahrmoeller.github.io/, GitHub https://github.com/sarahrmoeller, LinkedIn https://www.linkedin.com/in/sarahrmoellerChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Sarah’s academic website https://sarahrmoeller.github.io/An article about Sarah’s journey into this work https://www.colorado.edu/linguistics/why-computational-linguisticsUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  That was my first time that I encountered speakers of minority languages and some of the issues that they deal with socially because their language is disappearing. And so I found out there is this thing called language documentation, language description, which is going and documenting and analyzing languages that have never really been studied scientifically. You know, barely anyone maybe even speaks it outside of the native speakers and describing it and getting enough data documented so that if it does disappear, there's enough description, enough data, enough recordings that have been analyzed well enough so that the community can revitalize the language if they want.

• 0:47Speaker 01

  Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join us at practicalai.fm/community and follow the show on Twitter. We're at Practical AI FM. Thank you to our partners at Fastly for shipping our pods super fast all around the world. Check them out at fastly.com.

• 1:21Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International. I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

• 1:35Chris Benson?

  Doing very well, Daniel. It's a beautiful spring day here in the Atlanta area. So we're going to get to talk about some pretty cool stuff today, aren't we?

• 1:42Daniel Whitenack?

  Beautiful as in like crushing heat. That's what I think of when I think of Atlanta.

• 1:51Chris Benson?

  Let me put it this way, it will get much worse. It's not so bad right now. So give it July, August, early September and it's, it's a lot hotter.

• 1:59Daniel Whitenack?

  Yeah.

• 2:00Chris Benson?

  Point. So, so right now, we're very optimistic. Okay.

• 2:03Daniel Whitenack?

  Okay, good, good. We have a sort of close neighbor to you, uh, joining today for a really interesting discussion. Today we have with us, uh, Sarah Moeller, who's an assistant professor in the Department of Linguistics, uh, at the University of Florida. Welcome, Sarah.

• 2:19Speaker 00

  Thank you. It's great to be here.

• 2:21Daniel Whitenack?

  We've known each other not that well, but got to know each other a little bit better recently in a couple of meetings. So I was really fascinated, I think, by partly your, your story and how you, you got into both linguistics and fieldwork through sort of teaching English as a foreign language. But now you're kind of being, you know, dabbling in the machine learning and more than dabbling, doing a lot of amazing work there. So maybe you could just start out by telling us kind of a little bit about your background and how you got into, you know, teaching English as a foreign language and how that led eventually to where you're at now.

• 3:00Speaker 00

  Okay, well, to do that, I have to go back even a little bit further than my job teaching English, which I started doing, good grief, as a teenager, actually. And I started doing that because my family, I'd grown up in the American Midwest, so Iowa, and then mostly in St. Louis, Missouri, which is where most of my family's from, Missouri, Arkansas area. And then when I was 15, uh, my parents moved us to Russia. So moved to Moscow, Russia and had that whole very interesting experience of growing up overseas and learning a new language. And that's where I started teaching English and then just kind of falling in love with the country and the culture and all my friends were there. And so I went back and taught English. And I kind of had this idea that I would teach English, maybe eventually someday I'd come back to America and teach Russian.

• 3:43Speaker 00

  So I decided to go get some education to teach languages and I went to Moscow State University. And I started a master's degree, which I wasn't able to finish, but that was where I got introduced to linguistics was at Moscow State University. And a few years later, after teaching English, realized I don't really like teaching languages. I was always bringing in this linguistic, which is like this theoretical or the structure about the language. And probably boring my students to death because I'm like, isn't this just fascinating how this all connects and all the different parts? And so I realized I should probably go into linguistics. I like languages. Um, maybe I should check that out. So I thought about that for a little while and then kind of as I was kind of making that decision, I also ended up working two summers in Siberia.

• 4:26Speaker 00

  As a translator at some English camps, well, some camps for underprivileged children that would have Americans come over. So English, which is a good job skill for people pretty much all over the world. And that was my first time that I encountered speakers of minority languages. Out there in Siberia and some of the issues that they deal with socially because their language is disappearing. And when your language disappears, often not by your choice. So it's not the same as immigrating to a country and deciding to adapt to the country, countries you live in there. And Russia is really a big empire. It was conquered and there's a lot of big history there and they've maintained some of it, but there's this. So when a loss of language happens fairly quickly, when it's not your choice, these things get correlated, highly correlated with

• 5:11Speaker 00

  high rates of suicide, depression, substance abuse. And that's what I discovered. That summer as well as just meeting the people. And so I found out this was, there is this thing called language documentation, language description, which is going and documenting and analyzing languages that are never really been studied scientifically. Uh, maybe, you know, barely anyone maybe even speaks it outside of the, the native speakers and describing it and getting enough data documented so that if it does disappear, so this happened for a lot of Native American languages in the U S. There's enough description, enough data, enough recordings that have been analyzed well enough so that the community can revitalize the language if they want. Yeah.

• 5:55Chris Benson?

  Can I ask you a quick question there?

• 5:56Speaker 00

  Yeah.

• 5:56Chris Benson?

  You started it as you were talking about that experience in Siberia and then as you moved into the Native American thing, I'm wondering, as that process of a language disappearing is happening, like it's in process, what does that do for the folks who are living that, who maybe started in that language and their language is disappearing? I would guess it dramatically impacts identity. And I'm just, as someone who is not familiar with that, I'm curious what that is like for a real person experiencing that in real time.

• 6:27Speaker 00

  Yeah, it can be very traumatic. There's so much variation of it. It can happen gradually, it can happen by choice, but usually, so usually when we talk about endangered languages, kind of the prototypical situation is where, like I said, the language is disappearing, not because the people have chosen, or they've chosen to give it up. But often what happens is, well, we don't, we're not going to pass it on to our children. So the official definition of endangered language is it's not being passed to the next generation, it's not being taught to the next generation of children. And they might say, well, because they can't get a job in this country with that, you know, they need to learn the major language or the international language, the national language.

• 7:03Chris Benson?

  Is it always either or though? I mean, it's not like, learn this because it's what our family is, but learn this because that's what the job is there, or is it just too hard to do?

• 7:12Speaker 00

  For some communities are really good at doing that, actually. So this is something that's like interesting because often people say, well, there's not very many speakers, so it's endangered. And then scientists, you know, linguists will come and discover that actually that community has maintained a very strong sense of identity and their heritage is very strongly connected to their language. And so they do, so there are communities that do a really good job at preserving the language. And there's others that don't have like a culture that ties their culture identity strongly to the language. So it's, you know, then it's like, as linguists who might come in and want to say, well, we've seen this happen through the history with other communities that were like, whatever. But then their grandchildren were upset. They're like, wait a second, we want to connect with our history. That's less traumatic. What's thing that really gets you is the situations where they might've wanted to preserve the language, like Native American communities or First Nations in Canada.

• 7:57Speaker 00

  And point of history, when it was critical, it was decided by those in power, those who had the say about the education that it was basically bilingualism wasn't a good thing. And everybody needed to learn English and the best way they could get succeed in life and just all the other factors is to take them away from their families, put them in a boarding school. And then every time they speak the language that they know from their parents and all that emotional connection they might have with their family and punish them so that then they have a very negative.

• 8:23Chris Benson?

  It's just horrendously cruel.

• 8:24Speaker 00

  It is. It, those stories just tear you apart. And those are the stories that are very common in North America, I think also in Australia. So those are really, really hard and some have preserved a little bit, some, all they have is some scientist at some point recorded a dictionary and now they're trying to revitalize it. Um, and it's interesting with revitalization. So the other part of that experience is so, you know, there also are these same issues in these, in native communities in North America with substance abuse and health problems. So one exciting thing is that someone actually went and measured and found that at least in one community, I think it was in Canada, a community that had a language learning or language revitalization program actually had

• 9:08Speaker 00

  correlating increase in physical health in the community, like lower rates of diabetes, which is a huge problem. So there's an impact in documenting languages and, and studying them and preserving them that goes beyond just scientific interest.

• 9:21Chris Benson?

  To speculate, I'm just curious, uh, like the diabetes thing, cause that's not obviously connected, I think to most people, at least it would not, prior to this conversation, it would not have been obviously connected to me. Is that once again tied possibly to a sense of identity? And if you don't have one, or if yours is diminishing, then you take less good care of yourself because you don't have a sense of identity. Am I, I'm speculating here, but am I on the right track?

• 9:47Speaker 00

  I really can only speculate too, because it's, you know, I haven't studied that and there's only, I think that one study that I'm familiar with. But it makes sense just with so many things that, that sense of identity. And you think about it, if it happens quickly and it's not just, well, don't speak this language, but also everybody around you is saying, as it will happen in other parts of the world, more common than maybe the boarding school situation, what will happen is though you're, you don't even speak a language. Like I was doing fieldwork and we were, um, working with a couple of communities, preparing some literacy material, which by constitution in their country, they should be able to learn two hours a week in their school. So this is a language that's been studied as exists. It's known to exist. It's unique. It's been around for thousands of years, even has some written history, a little bit from very, very long ago.

• 10:31Speaker 00

  And prepared this primer material, a little reader and some writing exercises to the local ministry of education. And they said, this isn't even a language. It doesn't exist. Why, why are you even bothering us with this? So that sense of you don't matter, who you are doesn't matter. Yeah, it has to connect with that. Like, why should I take care of myself? Why am I important? Another thing I connect with too is a study that was done among children from different socioeconomic backgrounds. That who've gone through trauma and it was a study of what helps them come out and be healthy emotionally and socially. And they found that the one factor that was correlated with being healthy and handling the trauma well later on in life.

• 11:13Speaker 00

  Was not whether they were rich or poor, or highly educated or poorly educated, the one factor that was common for all of those ones that were successful, if you will, was knowing the stories of their family. It was connecting with their history of their family and hearing the stories. And so this idea that I belong to a bigger story. The people, my grandparents, my parents have had stories and I'm a part of that and they've gone through difficult things. And so I'm sitting and I'm listening to that and that's going to help me survive in life. But if you can't even speak the same language that your grandparents speak, that whole connection is lost. So it's very important to our, yeah.

• 11:51Daniel Whitenack?

  It makes me realize how incredibly lucky that I am as I'm listening to this. So.

• 11:55Speaker 00

  Yeah, indeed. So that's what I got into. I got the whole documenting languages and just, yeah, being able to study something that maybe nobody else has ever studied is pretty exciting.

• 12:04Daniel Whitenack?

  Yeah. And maybe you could describe a little bit too, like people are probably not familiar with like, what are the sort of like leaving machine learning and AI aside for a second, like what's kind of in the in the toolkit for people who are doing language documentation? What are some of the practices that people do? What's been shown to work and maybe not work over time? Or yeah, what are the kind of tasks, task related things that go on?

• 12:33Speaker 00

  Okay. So if you start with the kind of the prototypical situation, which would be a language that. Is spoken by a few people, has never been studied or written down before. Then what you would do after you get a grant and some background knowledge of at least related languages is you buy yourself some nice recording equipment and you go to the village or the wherever they are, probably a remote place, and you basically stick a microphone in front of someone and say, "Tell me a story." I mean, you're getting like the whole technique of how to get people to speak the language, to be comfortable, to make it natural, that's part of what you have to learn. But basically, you're just getting people to speak the language and you record it. And so stories, speeches, it might be ceremonial things.

• 13:18Speaker 00

  You might just get a list of, there's a list of words that linguists use that are considered, these concepts probably exist in all cultures, like hands and feet and mother and father, and translate them. You might say, "Okay, I'm going to give you a word." Like, we could try it, like, if I give you a word, "jump," what are all the possible forms of the word that you could think of? Does this word change at all? And if so, what are those forms?

• 13:40Daniel Whitenack?

  So as someone coming into a community and also like building relationships with community members, what is that experience like? Because you don't know the language, right? It's like you can't just like buy the book and yet you're trying to connect with people on this very deep level that means a lot to them. So what is that experience like?

• 14:03Speaker 00

  It varies so much, but building relationships is a really huge thing. And that's something that we try to, people who teach people how to do this, try to emphasize a lot, because it's something we've heard also from a lot of the community members who have had linguist common study. They don't want to feel like they're being used. They actually feel like they're building a personal relationship and that's important to them. That's what they want to keep out of this, sometimes even more than the language. So when I did fieldwork, it was, there was people already there that I knew that I connected with that helped me and introduced me. And I feel like, for me, it's a hard thing. I tend to be, let's just sit and read and let's do technical things and let's nerd out. So one of the best things for me was working with a team member. So we came up with a, some survey questions we wanted to ask these community members.

• 14:47Speaker 00

  What, basically, what part of the language was disappearing, but also what part of their cultures they feel like? We're disappearing, so we could kind of help us focus on what work we needed to do. And so we worked together to create it. We called people up. We said, can we come by and visit? We bought the tickets on the bus and planned out our trip. And we'd go into the people, kind of let them know. We'd go, they'd feed us tea, we'd sit and talk. And then usually the oldest male in the house would say, okay, we hear you have some questions. So, okay, these people are here and I've invited this school teacher. So now we'll sit and talk and we'll do business. And so my friend would ask the questions and he would just have a conversation. It's just kind of what we're doing right now. And I would note down on what they were saying. And then we'd finish up and that usually would lead to all sorts of conversations.

• 15:30Speaker 00

  So we'd just have a conversation in that case, make it really natural and then wrap up and say our goodbyes and head on to the next village.

• 16:05Daniel Whitenack?

  Okay, so I find all of this really fascinating. And I'm wondering at what point in this process, like you're involved with language documentation with the people on the ground, you're having these experiences and thinking about the process, at what point do like computer assisted methods sort of occur to you? And maybe, maybe even before that, like how, how is computing, like maybe even outside of the machine learning stuff, how, how does that impact the language documentation process?

• 16:38Speaker 00

  Well, if you think about the actual documentation is getting that microphone in front of someone and, and letting them speak the language. But to make it useful data for linguistic analysis and for maybe their descendants someday who want to learn that language, it has to get out of the audio. So getting it from the audio to a transcription of some sort. And then, so it's written down. That's the first step. And then after that, then you start breaking it down. You break down the words into their smallest meaningful parts. So jumping has jump, which means to hop and it has ing, which tells you kind of the aspect of movement that you're, how you're doing the jumping or over what course of time you're doing it. And so that's called morphology. Morphology is how words are built.

• 17:23Speaker 00

  So you break down the morphemes, which are the parts of words. You do morphological analysis. You figure out what those parts are. You figure out what they mean. You usually do like a rough translation of each sentence. And that's the minimum that you're going to do. If you can do that minimum, then someone else can come along even a hundred years from now and do something useful, create a language learning material. So that's the minimum that you really need to do. Because if it's just someone speaking and the language disappears and no one speaks it, it's like, we have it, but it's like Egyptian hieroglyphics before they found the Rosetta Stone. We can't make very much sense out of it. So it's those, those steps right there. Transcribing, doing basic word analysis, morphological analysis,

• 18:05Speaker 00

  and a rough translation of each sentence are very important steps. And you can do all of that by hand. And there are software that have been used to help people, at least keep it all in one database. There's some software, there's one that S I L has built. There's another one called Elan where it helps you. It's built to help you do this morphological analysis. And the nice thing is if you do analyze a word, the computer will see. If it sees the word that's exactly the same, then it will present that analysis that you had before. And you can check and oh, yes, that's, that's correct. It may change. Sometimes it, you know, different meanings. It looks the same, but it's a different meaning. So that's the tools that have existed for linguists. Um, I would think since the eighties or early nineties.

• 18:49Speaker 00

  But you're still pretty much doing it all by hand. There's some help there. You can set up some rules. Once you figure out the rules of the language and get a little bit more automated things, but you have to do it. You have to put all the words, all the parts in there. You have to enter it by hand. There's no machine learning in those tools that I've seen so far.

• 19:05Daniel Whitenack?

  Yeah. And what sort of like, just to give a sense, like day one, you're, you're coming into the community. You're putting up your microphones. And like, how long of a process might this be in terms of like, from that point of like, starting to gather recordings and building relationships in the community and building a community to like, having something like you were saying that like, for future generations is useful in terms of the documentation of, of the language. What are we talking about here in terms of like timeframe and the labor that's, that's put into this?

• 19:42Speaker 00

  Let me put it this way. I have a linguist friend who just finished doing all the analysis and translations of her data from a language in the Solomon Islands, which she had approximately, I want to say 30,000 words in that corpus. And so she just finished analyzing and translating all of it so that she could retire because she's been working on that for 20 to 30 years now.

• 20:06Chris Benson?

  Wow. I gotta ask, like, if you're thinking about that level of investment, how many target languages in that, you know, that are in that state might need that level of investment to get to that point? What are we talking about? What's the multiple look like?

• 20:23Speaker 00

  Think about, there's approximately 7,000 languages in the world. There's probably more because we still don't know how many sign languages are in the world, but roughly 7,000. And it's estimated that of those 7,000, anywhere from 30 to 90% of them are in danger of disappearing by the end of the century. I think the, probably the most, the best estimate is around 40%.

• 20:44Chris Benson?

  So we need more linguists, it sounds like.

• 20:46Speaker 00

  We need a lot more linguists.

• 20:47Chris Benson?

  Really fast.

• 20:48Speaker 00

  We need really fast. And, you know, to get, the point is, okay, it took decades to get all of that data processed. However, the other side of that is to get enough data where you can understand the language and write a big thick grammar that describes how it works and pretty much figure out, you know, 90% of it. That might only take you a couple of years. And you could use like 10 texts, you know, maybe 3,000 words to be able to get to that point, plus some extra stuff. You might make, need to make an extra trip back after a couple of years to get the final little pieces together. But that means for the communities and for future research further, deeper in the language, you know, you may have collected in a three month trip or three week trip even, depending how fast and how fast people speak.

• 21:34Speaker 00

  Thirty thousand words. Getting two thousand, three thousand might take you two years, but getting that last bit is a lot more. And that's, that was, that's a huge bottleneck.

• 21:43Chris Benson?

  Let me ask you another question, kind of before we're really, really getting into the machine learning stuff and all. If you're looking at different languages and you've been doing this, you have linguists that are going through this process and they're building that, you know, that, that body of understanding around each language. If they go, if, is there a point where even if you were to go to the next language, and let's say that that language was not terribly similar to some of the ones you've done before, is there anything that you can carry through to make that more efficient in terms of that timeline? You know, assuming that certain sounds are not the same and things like that. So it kind of has its own characteristics. Is there enough alike in the process of doing

• 22:28Chris Benson?

  that where you can look at a dissimilar language and kind of accelerate the, the understanding of that? Is that possible? Or is that, is that not?

• 22:35Speaker 00

  Oh yeah. If you, if you've studied a similar language and if it's quite similar in some languages, you know, that line between dialect and language is quite fuzzy. So it could be very similar. You're going to be a lot faster. You're going to be, you'll probably know, instead of maybe having to take a three week and realize, wow, I only got 50% of what I needed because I didn't know what I was looking for. I'm going to have to plan another trip. You could probably get absolutely everything you need, say, maybe in a three week trip. And because you would know, okay, it's similar enough. I know what the difficulties. I'm going to know what's complicated about this language. Yeah. You can get to figure out how the language works and write a grammar. I think that definitely would speed you up, but it still doesn't help you do the actual annotation of the data of people talking, which might. So there's this thing where like, you can create a linguistic description,

• 23:20Speaker 00

  which traditionally is what linguists did. And then they realized that language, languages are disappearing. Wow, this isn't enough. We need to really focus on this primary data. We need to annotate and have all the information so that it's there, kind of in the hundred years view. If it's disappeared and there's no speaker, do we have enough to keep studying it and let the community revitalize it?

• 23:38Chris Benson?

  Going back and recognizing the trust factor and building those relationships. But recognizing you also gave us a deadline a moment ago, in terms of by the end of the century, you know, X number of languages are lost forever and there's only so many linguists doing this. So is there any opportunity for automatic recording of the life of people in a particular area that are speaking a language where you basically put in the technology, it has a microphone, it may not record the specific words that you were looking for, but it records enough of day to day life to where that at least is getting those words into into a library of some sort.

• 24:14Speaker 00

  Yeah, there's actually, there's a, there's a linguist, Stephen Bird has worked on, he's built a mobile app actually called iKuma with this idea of people being able to speak into it on phones, even in its cheap phones where they don't have electricity, but they have, they can charge the phones and record and even do translations of it and do it, everything orally that could speak. So just get natural speech and then someone else can come along, you train them and say, okay, now take that very natural speech, like I'm kind of talking pretty fast right now. And you respeak it very slowly, very carefully. That helps us figure out what are the actual sounds in the language and then translate it into English or whatever language is a language of wider communication. So yeah, there are, there are people that are working on that. So going back to kind of that trust

• 24:56Speaker 00

  and relationship factor, one of the interesting things, one of the things I love about working with linguistics that you don't always get with tech, you don't get with technology per se, is that people are unpredictable and cultures are unpredictable. So in one culture, it may be, oh yeah, phones, we get that. Okay. Yeah. Oh, and yeah, we get the, yeah, our language disappearing that's important to us. Okay. Yeah. Give it to us and we'll do it. They're motivated. Other cultures are like, technology, hmm, kind of scary to put my talk into a microphone or yeah, technology I'm okay with, but why do I care about my language? Like, it's not important. And then there's factors like communities in Australia, in the southwest of the US where there's like religious and cultural taboos about

• 25:40Speaker 00

  listening to someone or hearing their voice or seeing their picture after they're dead. So then you have all this data, but then what do you do with it? How do you do with that? So the fun thing and the challenging thing is you can come up with some solutions, but you can't necessarily predict unless you have those relationships and you understand the culture, how or whether they're going to work and be accepted by the community. Yeah.

• 26:01Daniel Whitenack?

  And maybe that leads sort of naturally into kind of the computer assisted type of methods that we've kind of been alluding to. But how in your, in your journey, while you're kind of learning about this, you're participating in language documentation, you're involved in academia, at what point do you sort of start to think more about the way in which a computer might augment or assist this process? And when does kind of the light bulb turn on that like maybe more is possible, I guess?

• 26:35Speaker 00

  So after I came back to the States to get a master's degree and I focused on these very things that we were talking about documenting and describing endangered languages. And then after I finished my master's degree, lo and behold, I had debt to pay off. So I needed a job and it's not that easy to get a job as a linguist. There really aren't that many jobs unless you're going to teach English or work in the military or for the government.

• 26:55Daniel Whitenack?

  Work for certain agencies. Yeah.

• 26:58Speaker 00

  Yes, yes, exactly. Except that I, a friend of mine, found a job and then kind of posted to me a job posting for a company called Language Computer Corporation. So it was an NLP company and they needed annotators to, I don't remember what we're doing. We're creating some sort of grammatical templates. So, and then that turned into actually because I spoke Russian, because they happened to be getting money from certain agencies that are interested in languages like Russian and Korea and Chinese, that they hired me because I spoke Russian. So I went back and worked and learned, that's where I got introduced to computational linguistics and AI. Just kind of danced around the domain and discovered that this was something I'd never been aware of, had never heard of. It's really cool.

• 27:40Speaker 00

  And I don't know why I came to this conclusion because we were working with, also with Persian or Farsi, which is, there's not a lot of. Resources available for that language, but I just had it in my mind that this is really cool stuff, but you can't do it with minority languages because you need a lot of data to work on it. So this is really neat. Glad I had this job, but I'm gonna go back and I went, that's when I did my fieldwork in the Caucasus Mountains and did documentation. I was like, well, I'll just leave that behind. It's really cool. I wish I could bring it together, but probably not possible. And it was when I was on the field, working with the software to, well, different things that we were working on, transcribing a little

• 28:21Speaker 00

  bit, translating a little bit, doing some linguistic analysis, like word analysis a little bit, working with the metadata of this data. And I'm like, working with the software that were available and it just, kind of those two, two and a half years I'd worked in that company just kept on coming back to me like, there's, there's, there's something missing here. We're missing here. We, we've got to be able to do this faster. The computers can do something here. There's a gap that's missing. And that's what pushed me to doing computational linguistics. It was like, there is an untapped gap that linguists just don't seem to be aware of, and I, I know I wasn't aware of, between documentary and descriptive linguistics, or general linguistics, and computational linguistics, or NLP. It's a huge gap. It's like a chasm and we need to build bridges. And I kept on trying to convince my friends who were thinking about going back to graduate school,

• 29:06Speaker 00

  who'd done linguistics, you should do this, you should do this, you know some programming, you should do this. Couldn't convince anybody until one day I was thinking about it. I was like, ah, this needs to be done. Someone's got to do it. And it was like, a voice said, well, Sarah, why don't you do it? That's when I was like, I'd already gotten down to like, my list of schools I was going to apply to for a PhD to do documentary linguistics. I was going to write a grammar of one of these languages. I had to throw it all out and start over again and find schools that actually had a program, a PhD program that was interested in endangered languages and computational linguistics, which at that time in particular was actually not, not easy to find.

• 30:01Daniel Whitenack?

  Okay, so we've sort of made our way to the intersection of computational linguistics, NLP, machine learning, and language documentation. You've mentioned a few things that maybe listeners are aware of as NLP or AI tasks, like transcription, or like speech to text, or like, maybe people are familiar with sort of named entity recognition, of finding certain entities within text. Maybe not so much, people might not be as familiar with morphological analysis, but you've kind of talked through this process of transcribing and translating text. As someone kind of interested

• 30:44Daniel Whitenack?

  in this intersection, like, where do you begin? At the time that you started, where was the real opportunity in terms of like, you know, next steps towards computers assisting in this process? And kind of what are the main tasks maybe that you see as greenfield now?

• 31:04Speaker 00

  Well, the first thing for me was to learn how to program. So the problem that was getting at me was this bottleneck of, we have all this data recorded of people speaking, and what so often happens, because you can get a certain amount done, you're motivated, linguists are motivated to get a certain amount done, annotated, transcribed, annotated, translated, to do their dissertation, or to describe the language, so it's there. But then you have another 50,000 that are done, you have another 30,000 that hasn't been done, it's just sitting there. Maybe it's transcribed at most, maybe it's translated, those things can be relatively quick compared to doing linguistic analysis, but they're just there. And so all this data, first of all, it's

• 31:47Speaker 00

  not available for the communities for that purpose of revitalization someday, or learning, or having their stories recorded. For their grandchildren, even if they don't speak the language. But also, this need for data in machine learning, right? And machine learning and like, deep learning is supervised. So you need annotation. There's only so much you can do with unsupervised. So there's two kind of two things. If we can annotate this data, if we can get this basic analysis done, basic translation done, then we can train models on these languages to do more. And the morphological analysis is important, that breaking down into word parts. The reason you don't hear about as much in NLP is because mostly work is done in, in English, European languages, Mandarin Chinese, maybe.

• 32:31Speaker 00

  And Mandarin Chinese and English in particular are what we call morphologically simple languages. Words are very short and you don't add much to it. You add a plural to it, you add a past tense, not much more. I mean, you can have these long words like transubstantiation. And those mostly come from Latin or borrowed from another language. But a lot of languages in the world are much more complex than that. And a lot of the endangered languages in the world are much more complex. So Arapaho, which is a Native American language that I've worked on, basically, almost every word is equivalent to an English sentence. And it's all combined together in a complicated way. So each one is kind of, it's not like, you can't necessarily think, well, there are meaningful parts, but making the division between those meaningful parts,

• 33:13Speaker 00

  it's just so combined in such a complicated way that you really kind of have to view it as one word. So morphological analysis becomes pretty important. So I wanted to break that bottleneck, because that's a huge bottleneck. Doing that all by hand, transcribing, morphological analysis, translation. And I was particularly interested in the morphological analysis. So that was what I wanted to get into. Can we teach the computer for these complicated languages to figure out what are the parts of a word? And do it automatically? First of all, to break that bottleneck, to finish annotating these words so that then we could have more training data for NLP. But there's a second part there of discovery. Because if it can learn from what the human has done from annotated data, it can learn.

• 33:55Speaker 00

  But then it's going to run into something that it hasn't seen before and it's very different. And it'll get very confused. And so those are the things then that we could help linguists focus on. It's a discovery process. Wow, the computer hasn't seen this. So the really interesting things about a language are usually things that you don't see very often. And they're, they're out of, you know, not, not repeated. You're not going to get those statistical models to really. See them enough to learn them, so they get surprised, as it were. The models are surprised. And so then we can direct the linguist's attention to the things that might take them a really long time to discover if they're doing it by hand and have that ability to discover more interesting, unique things about human language.

• 34:36Daniel Whitenack?

  So is this like really an intersection of this kind of whole field of like active learning is probably like a term that like people, people might use where like, or human in the loop type of, of process where yes, we are going to need linguists to like really jump in and in, but we sort of need them more for certain samples versus other samples. And so you can kind of focus in their, their attention kind of in a loop on the, on the harder things as you're, as you're building up a data set. Is that kind of a, a strategy that, that's employed?

• 35:11Speaker 00

  That is the dream.

• 35:12Daniel Whitenack?

  Yeah, okay.

• 35:12Speaker 00

  That is the dream for me, yes. Cause it leads to some interesting conversation because if you say, oh, we can automate this to a linguist, to a native speaker who's working on their language, they often get nervous.

• 35:23Daniel Whitenack?

  I've fallen into that pitfall a few times, so.

• 35:27Speaker 00

  Yeah. This is like, wait, no, this is really good. Like me going to and do this by hand, I understand, I'm learning about the language and you can't have a computer, can't understand things. And like, that's true. So first of all, the human has to come in and learn it. They have to understand it. But there's a some point where language is statistical. That's why NLP works. We repeat a lot of things. You don't want to have to do all that by hand. So at some point, let's, let the computer take what you've done and learn. So it's not automated, but it's automatic assistance, I would call it.

• 35:56Daniel Whitenack?

  Yeah, or augmentation.

• 35:58Speaker 00

  Yeah, augmentation, yeah. To be able to sell it for that way. And then the active learning is so important because you don't, unlike English where you maybe just go and web scrape and get more data. It's expensive, you know, it's, it's hard. You may not have the technology. I mean, that's an issue that comes up with NLP and supervised learning. It's always been an issue. It's expensive if you have an expert, you know, and it like the Wall Street Journal and annotate all that, that took a long time and a lot of money. So. In some ways, really, the problem maybe isn't how do we get machine learning models to do this is how do we, how do we speed up what the humans are doing? So that active learning, yeah, part.

• 36:32Chris Benson?

  Can you give us an example of what like a typical workflow there in a very practical sense would look like? So if you're going to do that, because we're kind of talking conceptually, but also I'm trying to, to kind of tie that into how you're using the technology to achieve this. And so will you kind of take us through kind of a generic workflow of what you might do on a given day to achieve, to achieve that, just to get very hands on?

• 36:54Speaker 00

  Yeah, so this is all abstract and theoretical because it hasn't been done yet. And this is something, this is where I want to go. This is what my next

• 37:01Chris Benson?

  Okay.

• 37:02Speaker 00

  big grant is what I've been talking with people here in the University of Colorado, who also worked on this and were my mentors about this opportunity to make it a human in the loop for linguists. So there's two parts here, I see. One is the fact that you, you need to get the NLP working. You need machine learning models that are good, are relatively good at learning from field data from the linguist. So that's something that I worked on a lot because there's a whole issue there with working with field data of just, it's noisy and how linguists create data is different than how people create gold standard data for a machine learning model. It's, it's noisy, I'll just leave it there. So it's a challenge and finding the right models that work for multiple languages. And how much data should we recommend to the linguists that they bring?

• 37:47Speaker 00

  So then on the other side is then how do we bring this together in a way that linguists can access it? Who don't have any computational backgrounds? So that's going to require an interface. So we're talking about creating a third, so there's, there's these software that, that exists that I've talked about that don't have machine learning and they're, they're popular, they're, they're great tools. So we're talking about creating the third kind of, not to replace them, but to be there where it does machine learning. So they could work a little bit, export and then have this interface where in the background, you're doing NLP and people are doing experiments and trying different ways. But the linguist doesn't have to see the computational stuff going on, like what's the best way to do this? So you run, they've worked by hand in their tool, they've annotated the data by hand, bring it into this

• 38:30Speaker 00

  interface, train a machine learning model, say, to do morphological analysis, break the words into its meaningful parts. And let's say, so I've seen languages, depending on how you approach it, if you have, and there's all sorts of different languages with different kinds of words, but. I've seen a language of 3,000 words and we got like 30% accuracy at doing, breaking a word into its meaningful parts and giving the meaning of those parts based on what the human had done. 30%, not really great. And there was a lot more data, right, that hasn't, the human hasn't done. So the machine learning model's learning, because we know it based on the test is about 30% accuracy. And then we have it predicted over, let's say, another 10,000 words that haven't been annotated. And we know that 70% of them are probably wrong.

• 39:13Speaker 00

  What we don't know, because no human has looked at it, is which, what 70% of it is wrong. So then you want to figure out what's most likely incorrect that the model predicted. And not only that, but if of those incorrect ones, which of those incorrect words, incorrect predict, you know, analyze words, if they were corrected, would be most helpful for the model to learn the patterns and become better, like reach a higher accuracy more quickly. So instead of having to do another 1,000 words, can we find 100 words? That if the human corrected those, the model would suddenly jump from 30%, let's say, to 40 or maybe even 50% if we did a really good job.

• 39:56Speaker 00

  So 3,000 words to get to 30%, but only 100 to say get to 40%. So that challenge right there, some things I'm looking at right now. You can choose randomly, you can choose, we have a colleague of mine who figured out that if you look at the confidence, the probability confidence score on the predictions of the machine learning model, that's better than choosing randomly. So one thing I'm trying to do is incorporate some linguistic knowledge into that. So, okay, low confidence, but if we have, like, if you have a bunch of narratives, for example, you probably, at least in English, you'd end up with a lot of past tense verbs. So you'd see a lot of that and you'd learn pretty quickly probably that ED needs to be separated out and labeled past. What you might not see as much of is ING verbs, just using English, which is.

• 40:40Speaker 00

  It's hard to come up with examples because English is really simple. There's not that many complication, complicated things going on in the word. But let's say you don't have very many ing examples in those stories. So one thing we're trying to figure out, okay, we're hypothesizing if you don't have a lot of ing examples, then probably it's not going to be do as good at analyzing words with ing and recognizing that sometimes ing is part of verbs. Sometimes it's like a word like sing, you don't want to separate it out. So if that turns out to be true, which is our hypothesis, then we could combine these things. We could get choose out of those 70%, all those 10,000 words, the lowest confidence, 100 low confidence scored words, but also make sure that they're distributed so that you have more ing examples, for example.

• 41:24Speaker 00

  So that the model is learning from these incorrect examples, but it's learning in a way that's going to like strengthen its weaknesses more effectively. So then the human, you choose it smartly. The human does just say 100 words. It would be wonderful to see it jump up to like 40%, run it again. Now you've got another, you've got 60%, okay, pick out another 100, jump it up another 10% accuracy. I don't know, we still have to find out how that's going to work and how quickly that will go. But that would be like the general workflow with some abstract things thrown in the background as well.

• 41:54Daniel Whitenack?

  Fascinating. Yeah, I think it's really interesting that you're sort of working at this intersection where like NLP or sort of like modern NLP methods are impacting kind of traditional linguistic processes. But then like the more sort of traditional linguistic analysis can impact things the other way. I don't know if you have any perspective on that because that's not, I think it's a really, really valuable perspective, but it's not one that we hear a lot. We more hear like, like you were talking about scrape the internet, like build your model, like, don't worry about the language, it'll figure it out sort of narrative.

• 42:34Speaker 00

  Oh yeah, I think it's a virtuous cycle. I mean, obviously, first of all, if you have more data that's supervised data, so it's been marked up by linguists, you have more data to train models for these minority languages, which you can then build. Tools that allow them to be on the internet and doing cool stuff and having phones that speak their language, which will encourage children to speak the language and preserve the language and identify with the language, which will help their mental and emotional, physical health. There's all of that. There's all sorts of things where this is a virtuous cycle. So providing more data so that you have enough training data to train some models, say, I don't know, Google Translate, say, for example, I don't know what model, if they use one model or multiple models, but let's say it's one model, the more languages you have, I feel like that's going to be more helpful for knowing what is a good model that can learn from multiple languages. It's a multilingual model.

• 43:19Speaker 00

  You're getting languages that are very different than your typical languages like English or German or Mandarin Chinese. These are very different languages, so you are going to be challenged and your models are going to be challenged in a new way. The other thing is that maybe even outside of the virtuous cycle, kind of a separate thing is these techniques that we're looking at of how do you improve NLP models and techniques for low resource languages can work in any or could be applicable to any low resource context. So even like with English, sure, we've got a lot of data, but if you switch to a certain genre. So I remember hearing a researcher who was at Columbia University, who was working on Twitter posts by identified gang members and trying to do NLP

• 44:03Speaker 00

  for law enforcement to figure out what they were talking about for helpful, good reasons, we hope, and trying to figure out their slang, their use of emoticons, that's a low resource context. So these techniques that we're figuring out for low languages can be used to apply even for high resource languages, but in certain genres or contexts where there's not a lot of data, it's very specialized or yeah.

• 44:25Chris Benson?

  I just want to say, because my wife will shoot me if I don't bring this up. Since you mentioned the gang thing, she likes to, she is English and for fun, we will have Cockney rhyming slang in here. And so I just, we need to, we need to, to figure that one out.

• 44:40Speaker 00

  We need to figure that one out. Well, there you go.

• 44:42Chris Benson?

  If we're going to do the, the gang slang, then we got to do Cockney rhyming as well. So if I didn't say that, I, she'd say, why didn't you bring that up? And I'd be in trouble.

• 44:49Speaker 00

  That's great. I think you could probably teach an NLP model to create its own Cockney rhyming and then

• 44:53Chris Benson?

  That would be pretty cool.

• 44:54Speaker 00

  Trick the speakers.

• 44:55Daniel Whitenack?

  Yeah, I mean, I guess as you're kind of looking towards the future and you you are interacting at this intersection between linguistics and AI, like, what are those things that like make you really excited about that future in terms of the possibilities? And like, you know, maybe you see momentum that's starting to build now that, you know, you hope will continue. Like, what are those things that really that really make you excited as you look towards the future?

• 45:24Speaker 00

  I mean, excited about what I do and I'm excited about the linguistics. But I also get excited on things that aren't directly connected to what I do, but are hopefully consequences or byproducts. So I get excited seeing humanities and linguistic students who now have programming skills, which puts them in a much better position in the job market. I know what a difference it made for me and my prospects. I get excited for people out there, hobbyists, community members, probably people at hackathons. I hope there's more who are building useful technology for minority communities, for them to use their language, to learn their language, or just things that are in their language.

• 46:03Speaker 00

  Just, yeah, just to build that sense of identity and pride and value that they have, I believe, just because they are people God created. Yeah, seeing other people take some of what we're doing or building off of it and building tools that, because what I'm doing is kind of theoretical and scientific. You have to make several jumps to be things that people who aren't interested in this specific area think are cool. But like, language learning apps are spell checkers and Microsoft Word, those sort of things are exciting to see people build. And for momentum, so the momentum, when I first started in this area, there wasn't very much work being done on computational morphology. It was it was starting because most of the work had been done in English and, again, Mandarin

• 46:44Speaker 00

  or European languages, which are not complex morphology, they're fairly simple. So that's grown a lot. And then I think specifically in this intersection of documenting and describing endangered languages with NLP is this realization that we have to do a human in the loop. We can't expect to just get more data. We also can't expect to make, there's a lot of cool techniques about how to augment data, how to hallucinate data in interesting ways that does actually help the models, but they only go so far. So this realization that we we do have to build some way to bring the linguist, hopefully the native speakers, back into the loop. That that's the best way to go and and bring that to some sort of to bring it to a place where they are involved and there are interfaces and it is so the flip side of

• 47:29Speaker 00

  that is it becomes accessible to them. Those tools, the NLP stuff isn't just doesn't remain in the computer science departments or in the big companies, but it comes becomes something that's accessible to speakers and workers in minority communities. That's that's really exciting for me to see. That's a trend. That's kind of what I recognize and a lot more people are realizing, kind of confirming what I thought that this this has to happen.

• 47:51Daniel Whitenack?

  That's awesome. I love that perspective. I think that's a really great thing to leave in people's mind as they move forward. And I think you probably convinced Chris to become a linguist at this point.

• 48:02Chris Benson?

  I'm right there. I'm excited about this.

• 48:04Speaker 00

  Yay, yay.

• 48:06Daniel Whitenack?

  But yeah, thank you so much for joining us, Sarah. I really appreciate your perspective and your work.

• 48:11Speaker 00

  Thank you so much for having me and letting me nerd out a little bit on languages and computers.

• 48:15Chris Benson?

  It was fun. Thank you.

• 48:25Speaker 01

  All right, that is Practical AI for this week. If this is your first time listening, subscribe now at practicalai.fm or just search for Practical AI in your favorite podcast app. We're in there. And if you're a longtime listener, please do share the show with your friends. It is the best way you can help Practical AI succeed. Thanks again to Fastly for shipping our shows super fast all around the world to break master cylinder for the beats and to you for listening. We appreciate you. That's all for this week. We'll talk to you again next time.