25 years of speech technology innovation

To say that Jeff Adams is a trailblazer when it comes to speech technology is an understatement. Along with many other notable accomplishments, his team at Amazon developed the Echo, Dash, and Fire TV changing our perception of how we could interact with devices in our home. Jeff now leads Cobalt Speech and Language, and he was kind enough to join us for a discussion about human computer interaction, multimodal AI tasks, the history of language modeling, and AI for social good. Sponsors: O'Reilly Media https://www.oreilly.com/changelog – Learn by doing — Python, data, AI, machine learning, Kubernetes, Docker, and more. Just open your browser and dive in. Learn more and keep your teams’ skills sharp at oreilly.com/changelog https://www.oreilly.com/changelogSnowplow Analytics https://snowplowanalytics.com/ – The behavioral data management platform powering your data journey. Capture and process high-quality behavioral data from all your platforms and products and deliver that data to your cloud destination of choice. Get started and experience Snowplow data for yourself at snowplowanalytics.com https://snowplowanalytics.comChangelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Featuring: Jeff Adams – LinkedIn https://www.linkedin.com/in/jamesdaff, X https://x.com/jamesdaffChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Cobalt Speech and Language https://www.cobaltspeech.com/Building Blocks of Speech Technology https://www.cobaltspeech.com/coblog/2020/4/16/building-blocks-of-voice-technologyArticle: “BYU students, creator of Alexa develop software to preserve Cambodian stories” https://news.byu.edu/character/byu-students-creator-of-alexa-develop-software-to-preserve-cambodian-storiesPrevious episode with Catherine Breslin (previously with Cobalt) https://changelog.com/practicalai/82Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  When I was at Amazon, leading the Alexa team, we were doing the same. We had a lot of, a lot of effort goes into language modeling to figure out what sequences of words are people likely to say to a smart speaker. And on the acoustic side was, I think, more challenging for Alexa, because Alexa was the first breakthrough product that can handle speech at a distance, so that it had to accommodate all of the reverberations and echoes in a room. When I talk to one of my smart speakers now, my voice is bouncing off of any number of walls, windows, computers, whatever, before it gets to that smart speaker. And, and that can be really confounding for the speaker. And the acoustic modeling had to do some major growing up to be able to handle that.

• 0:45Speaker 01

  Big thanks to our partners, Leno, Fastly, and Launch Darkly. We love Leno. They keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com.

• 1:04Speaker 01

  This episode is brought to you by our friends at O'Reilly. Many of you know O'Reilly for their animal tech books and their conferences, but you may not know they have an online learning platform as well. The platform has all their books, all their videos, and all their conference talks. Plus, you can learn by doing with live, online training courses and virtual conferences, certification practice exams, and interactive sandboxes and scenarios to practice coding alongside what you're learning. They cover a ton of technology topics, machine learning, AI, programming languages, DevOps, data science, cloud, containers, security, and even soft skills like business management and presentation skills. You name it, it is all in there. If you need to keep your team or yourself up to speed on their tech skills, then check out O'Reilly's online

• 1:49Speaker 01

  learning platform. Learn more and keep your team's skills sharp at O'Reilly.com/changelog. Again, O'Reilly.com/changelog.

• 2:05Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at kingdog.com/community and follow us on Twitter. We are at Practical AI FM.

• 2:31Daniel Whitenack?

  Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International, and I'm joined as always by my co-host Chris Benson, who is a Principal Emerging Technology Strategist at Lockheed Martin. How are you doing today, Chris?

• 2:49Chris Benson?

  I am doing very well. How's it going today, Daniel?

• 2:52Daniel Whitenack?

  It's going wonderful. I don't know if you, our listeners can't see me, but you can see me. I'm in a new room. My wife and I moved this last week, which was exciting. I don't know if it's this way worldwide, but for those in the US, you'll know that the housing market in May of 2021, which is now time, is crazy. So, we sold our house and got a rent for a while, and that seemed to make sense. So, I've got a new podcast recording room. That was the big news.

• 3:16Chris Benson?

  It's funny that you mention that because I know I've mentioned to you that I'm planning to move as well.

• 3:20Daniel Whitenack?

  Yep.

• 3:20Chris Benson?

  Very local to where we're at.

• 3:21Daniel Whitenack?

  Sure.

• 3:22Chris Benson?

  But just, we have some family land that has five acres.

• 3:24Daniel Whitenack?

  Yeah.

• 3:25Chris Benson?

  But the price of wood is at an all-time high.

• 3:27Daniel Whitenack?

  It is, yeah. Building is crazy.

• 3:29Chris Benson?

  That has pushed the cost of building up way up there. So, we're trying to figure out when we're gonna do it. So, nothing to do with AI.

• 3:35Daniel Whitenack?

  Yeah, yeah.

• 3:36Chris Benson?

  But still, you know, definitely impacting the family.

• 3:38Daniel Whitenack?

  Yeah, so I've been like, working from various strange locations over the past week and opening up my computer and dealing with a bunch of data parsing issues and weirdness over the past few days. And it's been either moving a box or setting my laptop on top of a box and opening it to deal with data parsing issues. That's been like my life for the past few days, which isn't so bad. It's not so bad.

• 4:02Chris Benson?

  You do what you gotta do to make things work.

• 4:04Daniel Whitenack?

  Yeah.

• 4:05Chris Benson?

  We get there.

• 4:06Daniel Whitenack?

  Yeah, for sure. Chris, I think I mentioned to you a few weeks ago, I think it was, I was able to be on a guest on a different podcast, which is always fun, and that was the Voicebox Podcast. One of the hosts of that podcast is Jeff Adams, who is CEO of Cobalt Speech and Language, and a former member of the Alexa team at Amazon. And Jeff has graciously responded to our request to be a guest on Practical AI. So, welcome, Jeff.

• 4:34Speaker 00

  Hi, it's good to be here. And I want to state for the record, I am not selling my house.

• 4:38Daniel Whitenack?

  Okay, perfect. Yeah.

• 4:41Chris Benson?

  Touche.

• 4:41Daniel Whitenack?

  Well, you must like where you're at right now.

• 4:43Speaker 00

  I'm just holding. It's like poker in real estate, just hold them.

• 4:47Daniel Whitenack?

  You're holding.

• 4:49Speaker 00

  Yeah.

• 4:49Daniel Whitenack?

  Makes sense. Yeah, I'm taking the more impulsive route. Well, before we get into all the great speech technology things that you're involved with, maybe we could just start out and talk a little bit about your background and how you got into this sort of technology and ended up doing what you're doing now. You want to give us a little bit of that background?

• 5:10Speaker 00

  Sure. I got into speech technology a quarter of a century ago, and that's the first time I've ever said that, because I just realized it, and it makes me feel really old. So, I've been doing speech recognition, speech technology for just over 25 years. I've worked for companies like Dragon, Nuance, Yap, and Amazon, you mentioned. And then, more recently, for the last six, seven years, I've been the CEO and co-founder, or sorry, just plain outright founder, of Cobalt Speech and Language Company. So, I've been at a handful of different companies and seen a lot of things over the last 25 years, and I really love speech technology.

• 5:49Daniel Whitenack?

  When you were just getting into this world of speech technology, what was it that drew you to the technology in the first place? How did that happen?

• 5:58Speaker 00

  Well, I was a math grad student at the time at the University of Oregon. And was deciding that academics was not really where I wanted to spend the rest of my life and was trying to figure out what to do. I have always been really fascinated by the interplay between mathematics, statistics and language and trying to figure out what I could do with that. And I saw, this was in, boy, I really am aging myself. This was in 1995. This was the relatively early days of the internet and the world wide web. And I went on the internet to hunt for jobs, which was not a common thing at the time. And I found someone looking to fill a position for statistical language modeling. And I said, that sounds really cool.

• 6:42Chris Benson?

  Oh, that sounds cool.

• 6:43Speaker 00

  So I threw my hat in the ring and I got the job and I moved from the Pacific coast to the Atlantic coast and took a job with this speech company Kurzweil Applied Intelligence. It was founded by Ray Kurzweil, the inventor and futurist. It was fun. It was a really cool experience and I, that's where I got my on the job training. I learned a lot about speech technology and specifically how language, specific statistical language modeling fits into the whole process of speech recognition.

• 7:13Chris Benson?

  I'm curious, you've talked about your resume. It's kind of a who's who of that industry of kind of all the, all the big names as they've come along. When you joined Kurzweil, how big was that company? Was it, had it grown yet or was it still small?

• 7:26Speaker 00

  You know, the company was, I don't know, five or 10 years old at the time. So it wasn't a brand new company, but it was maybe 30 people, something like that.

• 7:37Chris Benson?

  Yeah, quite small.

• 7:38Speaker 00

  Yeah. Ray Kurzweil himself, he had his fingers in several different companies, but he would occasionally come by and walk up and down the halls. I was very flattered that he greeted me by name without having to look at a nameplate on the door or anything like that. He knew my name. It was exciting.

• 7:53Chris Benson?

  That is cool.

• 7:54Daniel Whitenack?

  You mentioned statistical language modeling. Could you give us sort of sketch of maybe back at that time when you were trying to do some of those things? What could statistical models do? What were the main challenges that you were solving? And maybe a sort of fast forward to the capabilities or the functionality that we can do in speech-based tasks? How has that expanded? Could you give us a little bit of that sort of sketch?

• 8:20Chris Benson?

  As you do that and can tie it a little bit into some of these various companies that you've been involved with over time.

• 8:26Speaker 00

  Yeah.

• 8:26Chris Benson?

  I'd also be interested in hearing the progression as you went through your own career.

• 8:30Speaker 00

  Yeah, absolutely. By the way, just to be clear, the first 10, 15 years of my career in speech were really focused on statistical language modeling. Since then, I've broadened my scope and horizons and been more involved in speech recognition more generally, and then more recently, speech technology more generally. But let me go back and answer your question. When I first started in 1995, doing statistical language modeling, first of all, let me just define what that means for some of your listeners that may not know what that is.

• 9:00Chris Benson?

  That'd be great.

• 9:00Speaker 00

  Statistical language modeling, in a nutshell, you could think of it as predicting the next word. So imagine you're reading a book, or a newspaper article, or a magazine article, and you get to the end of the page, and you're in the middle of a sentence, and you're about to turn the page. Can you predict what that next word is going to be? And not just predicting which word it is, but of all the possible words it could be, give me a probability of it's, you know, there's a 3% chance that it is the word yogurt, and a 5% chance that it's the word horse, or whatever else, whatever, I can't imagine what sentence might lead to those high-scoring probabilities.

• 9:36Chris Benson?

  That's what I was thinking.

• 9:37Speaker 00

  But anyway, this is the problem of predicting the next word. And the state of the art at the time, in 1995, was a technique called n-grams, where you would have counted up from a large body of text, or we call it a corpus of text, you would have gathered a large corpus of text, and at the time, a large corpus of text would have been maybe 50 million words of English. And of those 50 million words, you could count, well, how many times did I see the word yogurt? And how many times did I see the word horse? And more specifically, if you know the last word on the page before you turn the page, if you know the last word was favorite, or, you know, my favorite, or something like

• 10:20Speaker 00

  that, of all the times that you saw the word my favorite in your 50 million-word corpus of speech, of all of those times, how many times was yogurt the next word? And how many times was horse the next word? You could then infer the probability of various words based on how many times you'd seen them in a corpus. The biggest problem in statistical language modeling, the fundamental problem is the zero problem. There are many things that happen. You might see my favorite yogurt in real life, but you may never have seen that in your 50 million words. And so if you're trying to infer, what's the probability of yogurt coming next? You might naively say zero. I've never seen it before, so it can't happen.

• 11:00Speaker 00

  And the problem of statistical language modeling is figuring out how do you predict things that really could happen? But you've never seen them happen before. So over the years, statistical language modeling was dominated by this approach using N-grams. If you're using sequences of three words, we would call those three grams. Or if you want to be fancy, trigrams. And over the years, as computers have gotten bigger and data has become more plentiful, we've been able to increase N. Originally, we were working with bigrams and trigrams, that is, sequences of two or three words. And now, over the years, it's that's grown to, you know, trigrams and four grams and five grams.

• 11:45Speaker 00

  We give up on the Latin and the Greek prefixes at some point.

• 11:48Daniel Whitenack?

  Yeah, that gets a bit hard at some point.

• 11:50Speaker 00

  Yeah. And the thing that has astounded me is that over 25 years, as so many other things have improved and developed, it is still really hard to beat N-gram technology for modeling statistical language models. They are still the workhorse today of most speech processing systems, although they're trained on many more than 50 million words. It might be 50 billion words or even approaching trillions of words. And we don't just look at two or three words, but it's much more common to look at four or five word sequences when you're modeling those now. Cause you have so much more data to be able to do that. Their simplicity means that they're fast to use in practice. When you're trying to look up a probability, they're very fast. They're still the workhorses.

• 12:35Speaker 00

  There are approaches to getting more accurate statistical models.

• 12:40Speaker 00

  At predicting that next word that are more accurate, they're better. But typically they're very time consuming and so they're hard to use in practice. There are approaches now with deep neural networks to model those. And anyway, so this is the story of, you know, the more things change, the more they stay the same. That a lot of things have grown more sophisticated in speech technology. But language modeling has just grown bigger, larger amounts of training data, larger sizes of how many n grams we keep around to do the modeling and so forth.

• 13:08Daniel Whitenack?

  So that's like the language modeling piece, which mostly what you've been talking about is sort of combinations of words or tokens.

• 13:16Speaker 00

  Right.

• 13:17Daniel Whitenack?

  At what point did you start exploring the audio side of things? So audio in and not knowing maybe what words are in that audio. When did you start exploring that? And how eventually does that connect to this world of language modeling?

• 13:31Speaker 00

  Yeah, so my first job was working for a company doing speech recognition. And we were doing speech recognition mostly for people dictating documents or emails on their laptop or on their computer. The standard approach then and still now to a large extent takes the statistical language modeling of kind of predicting what words might come next. So that you know what words to kind of listen for. And then the acoustic models that are able to listen to some audio and recognize the sounds of the language in that audio. Recognize the difference between a K sound and an A sound and an I sound and whatever. The different phonemes that you might get. And the whole process of speech recognition was to merge those together

• 14:16Speaker 00

  to get sequences of words that A, sounded like what you're hearing and B, are reasonable sequences of words. So you put those two together. Now, in my first dozen years of working in this field, I ran the language modeling group at the company where I was, eventually at Dragon and Nuance Systems. I ran the language modeling group and we always had a friendly rivalry with the group next door. That was trying to improve the acoustic models of recognizing, you know, what sounds those are in the audio. And we would have each new release of our software, like, for example, Dragon, naturally speaking, each new release, the language model team and the acoustic model team would be fighting to see who could contribute the most to

• 15:00Speaker 00

  the accuracy improvement for the next release. And it was a pretty fair fight. Sometimes we won, sometimes they won. When I left Nuance and went to join this small company, Yap, in Charlotte, North Carolina, where our focus was on transcribing voicemails. When I went to Yap, I took a step up in a sense, and suddenly I was in charge of all of the speech recognition. So I had to put aside my biases, you know, rooting for the language model, and I had to also care about the acoustic model. And I was able to find some really talented people who knew about both and could push forward our research on that. And we got to be very accurate. I think we were the most accurate in the industry at the time at transcribing voicemails. And that was a lot of fun. But

• 15:45Speaker 00

  then you asked me to sort of step through the history of my career and talk about how the language model versus the acoustics kind of come together. So they came together in my career at that point. And then that continued when I was at Amazon, leading the Alexa team. We were doing the same. We had a lot of, a lot of effort goes into language modeling to figure out what sequences of words are people likely to say to a smart speaker. And on the acoustic side was, I think, more challenging for Alexa, because Alexa was the first breakthrough product that could handle speech at a distance, so that it had to accommodate all of the reverberations and echoes in a room. When I talk to one of my smart speakers now, my voice is bouncing off of any number of walls, windows, computers,

• 16:30Speaker 00

  whatever, before it gets to that smart speaker. And, and that can be really confounding for the speaker. And the acoustic modeling had to do some major growing up to be able to handle that. But we still had acoustic models and language models as sort of these separate entities that were both contributing to the recognition. And when I left Amazon and founded Cobalt, I did another step up in terms of the scope that I was looking at. I was not worried just about speech recognition anymore, but now we're worried about speech synthesis, speaker verification, to identify who's speaking, and any other aspects of speech processing. But those core components. They've grown up, they've become more sophisticated, especially in the case of acoustic

• 17:12Speaker 00

  models, and they've become bigger in the case of language models. I still recognize them from 25 years ago. They're still the basic components that come together in more or less the same way.

• 17:34Speaker 01

  This episode is brought to you by Snowplow Analytics. Snowplow is the behavioral data management platform for data teams. Maximize the value of your behavioral data using Snowplow Insights, a managed data platform that's built on leading open source tech leveraged by tens of thousands of users. Capture and process high-quality behavioral data from all your platforms and your products, and deliver that data to your cloud destination of choice. When marketing needs to make data-informed decisions, when product needs next-level understanding, and when analytics needs rich and accurate data, Snowplow is the solution for data teams who want to manage the collection, processing, and warehousing of data across all their platforms and products. Get started and experience Snowplow data for yourself

• 18:18Speaker 01

  at snowplowanalytics.com. Again, snowplowanalytics.com.

• 18:38Chris Benson?

  So, as we've seen these technologies really get out into the marketplace and be used in a huge number of different use cases and different industries, different businesses, how are you seeing this technology? What are some of the things that you've seen it being used for that are outside? I mean, I think probably most people are familiar with things like Alexa and the like, but I've run into use cases where I've found speech technology and things that I wasn't expecting or didn't realize that had come into that. And if you could give us a little bit of insight into some of the places that you've seen it go that maybe we're not aware of and maybe that we haven't been using on a day-to-day basis?

• 19:14Speaker 00

  Yeah, I don't know. Chris, have you been listening to my podcast? It sounds like you're making a pitch for my podcast.

• 19:20Chris Benson?

  I might be. I'm not supposed to admit that now. You know, you're not supposed to call me out on that, but, oh my gosh. Yeah, this is the great teaser, yeah. I set you up and you called me out on it.

• 19:33Speaker 00

  But in fact, I have this podcast, The Voice Box, where that's exactly what we look at is the variety of things people are doing with speech technology. And I'll give you a few examples, right? So people are using speech technology to diagnose disease. I can listen to you speak and detect, oh, I think the way you're speaking, you're depressed, or you have Alzheimer's, or I can even detect things like congestive heart failure and things that you wouldn't necessarily expect to be related to speech. So that's one. We've got speech in education where we're helping, not just coaching people, but in general, but coaching on like people learning a new language, where we can give them feedback on how well they're speaking. We've got custom

• 20:16Speaker 00

  synthetic voices that people build. I'm mostly thinking of things, projects that we've taken on at Cobalt, but this is not limited to what Cobalt is doing. People are doing really interesting things. There's a great company Vocal ID that is, records people's voices. People who know that they have to have some surgery, they'll lose their voice. They can record their voices ahead of time and then create a synthetic voice that sounds like them so that later, after they've lost their voice, they can still speak with their voice.

• 20:44Chris Benson?

  Oh, that's really cool.

• 20:45Speaker 00

  It's a fantastic company and a great technology.

• 20:48Chris Benson?

  And it's almost like maintaining identity as well.

• 20:50Speaker 00

  Exactly.

• 20:50Chris Benson?

  Separate, you know, from just the the physicalness of the voice itself.

• 20:54Speaker 00

  Right. We've got people who are using voice to listen in on like pilot conversations to detect when they're fatigued, to recommend that they trade off with the co-pilot or whatever, whatever you have to do to wake someone up that's fatigued. We've got instances in education and finance and medicine. And the thing that has been the most fun for me in my career has been these last six years or so, where I get to see so many different interesting applications. And I don't have to think of them all. Other people are coming to us saying, hey, I'd like to use voice for financial transactions, or I'd like to use voice for whatever it is. And we get to do that. We get to work with them and build those things out. And we've got some great technology

• 21:37Speaker 00

  that that people are using in a lot of really cool applications. It's fun to be a part of that.

• 21:42Daniel Whitenack?

  In terms of like the human device interaction in different industries, I think people started, of course, interacting a lot with computers via various peripherals, but eventually, you know, keyboard and mouse generally. And I think people have really got used to that. Of course, that's changed maybe a little bit with smartphones and other things. So how have you seen maybe industry wide and also anecdotally, maybe in people that you interact with in terms of how the value of a speech driven interaction might be different than just creating a text chatbot for your customer service? How have you seen that evolve as speech

• 22:24Daniel Whitenack?

  technology has gotten more diverse and more performant over time?

• 22:28Speaker 00

  Yeah, that's a great question. People usually rather communicate by speech. That's natural, right? Speech came long before written language.

• 22:36Daniel Whitenack?

  Yeah, we weren't born with a computer mouse in our hand.

• 22:39Speaker 00

  Right. So speech is part of us and it's natural and it's an easy way for us to communicate and to be communicated to, right? In both directions. And so it's really important. I have seen over the last five years or so, I think I've seen some people push it a little too far where they think, well, what we need to do is have all interaction be through speech. But I don't think that makes sense, right? There are times where speech just doesn't work as well. If I'm like, well, I'm talking to you right here. If my wife needed to get my attention, she'd come into the room and probably write something on a piece of paper and hold it up to me, right? In that case, you wouldn't want to use, I wouldn't want her to use speech and interrupt us.

• 23:20Speaker 00

  And there are times it's a lot easier to scan information quickly when it's written. And gestures are important. So I think the right thing to do is people have been pushing the pendulum way over towards speech, at least some people have. And I think that the pendulum is going to come back a little bit and people are going to find the right balance of speech and keyboard and mouse and monitor and gestures on a touchscreen and whatever else. So gestures with a watch, like movement of your arm and things like that. And I think it's going to come back and speech is going to take its proper role, which will be Prominent, I don't know dominant, I don't know, but it'll be, uh, speech is going to certainly

• 24:02Speaker 00

  play a central role that has only been possible in the last five or 10 years, really.

• 24:07Chris Benson?

  So it sounds like almost that there's a sort of shift of mindset, or at least a tendency now to think of interactions more in a multimodal sense than like, I'm just going to consider text, or I'm just going to consider speech, or I'm just going to consider video and camera feeds. There's more of an appetite to think about interactions more holistically. Is that something you're seeing?

• 24:29Speaker 00

  Yeah, absolutely. I do see a shift toward multimedia interactions with computer where speech plays an important role, but it's not exclusive. And one thing I should say is, you know, we're talking here about computer-human interaction predominantly, but speech technology also plays a role in human-human interaction, where the speech technology might just be listening in on a conversation when you're talking to someone helping you at the bank. And it might be helping the person at the bank, uh, recognize what you said, or maybe giving you, you know, you're asking about, you want to invest in an IRA, and it will,

• 25:07Speaker 00

  the speech system might eavesdrop on you and pop up some helpful information to the representative at the bank, so that they better know how to help you. Or it might coach them to say, you're talking too fast, slow down, or whatever. So, so it's not just computer-human interfaces, but speech technology can also be used improving human-to-human interactions.

• 25:32Chris Benson?

  You raise a great point that we are multimodal ourselves, and that we are not always directly multimodal, but that we have all these kind of ancillary interactions, as you pointed out within your bank example, of kind of hearing the next person over, having their conversation. Do you think, I mean, going forward, is that, is speech integrated into that multimodal in all of these direct and indirect ways, kind of the way forward, as, you know, whereas we have just kind of come through a time period where we tend to think in terms of single modes, and that's now finally changing, and maybe we're getting, I dare say, a little bit more human in the way that we're actually approaching it by pulling these together?

• 26:10Speaker 00

  I hope so. Yeah, that's what I hope we're doing. I hope the singularity is not of, you know, our robot overlords taking over, but rather us. Using all of this enhanced technology to help us deal better with each other and with the task that we have to do. And that that will definitely be multimodal. It can't just look at one or the other aspect of the way we communicate and and interact with each other.

• 26:36Daniel Whitenack?

  And I like the reference back to your Kurzweil roots, you know, starting out there. Full circle. I'm curious, you've mentioned a bunch of things and how I think it's fascinating how maybe some of us don't even realize all of these different areas where speech is or could be applied in both human computer device interactions and human to human interactions. I'm wondering, I know Cobalt Speech and Language has just some, some really great people, both developers and linguists and experts in this field. I wonder, as a group, what are some of the, the major challenges or maybe open problems or issues that your, your company is focused on sort of addressing in, in the technology to help push it forward?

• 27:21Daniel Whitenack?

  What, what are some of those main things on, on your radar as a company?

• 27:25Speaker 00

  So I think the biggest challenge in any application here, probably in anything having to do with AI and machine learning, but in particular, speech and language, is access to appropriate data. That's the thing that always holds us back or always is the obstacle. Someone says, I want to be able to detect when someone's speaking if they are inebriated. Okay, fine, but now we have to go get some recordings of people who are inebriated and not and whatever. And getting access to sufficient amounts of data to be able to be accurate enough about that is always the challenge. One of the corollaries to that is that it

• 28:02Speaker 00

  is very challenging to bring this technology to speakers of other languages than the big ones, you know, the English and the Chinese and the German and whatever. Someone is speaking some dialect that only has five million speakers, which sounds like a lot of people, but Five million speakers is not yet at the point to attract the attention of the big players in the field to develop speech technology for them. But it's still five million people that are missing out on this spoken language revolution in tech. And so it's hard to get for these lesser represented languages, it's hard to get

• 28:44Speaker 00

  sufficient data that's labeled appropriately to be able to build the kind of technology that we already are starting to enjoy in the, I don't know if we have a good word for it, but you know, not the first word, but let's call it the first linguistic world or something like that.

• 28:58Daniel Whitenack?

  Yeah. Major languages.

• 29:00Speaker 00

  Yeah.

• 29:06Speaker 02

  Change log plus plus is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal and help sustain our production of practical AI into the future. Simply follow the change log plus plus link in your show notes or point your favorite web browser to changelog.com slash plus plus. Once again, that's changelog.com slash plus plus.

• 29:49Daniel Whitenack?

  So Jeff, you probably know you're you kind of led us into a topic that is very much a passion of mine in the organization that I'm a part of, you know, bringing the benefits of some of this technology to local language communities all around the world, which I just think is really wonderful. And I affirm that and applaud you for your efforts in that area. One of the stories that I know that you've mentioned to me and I've read a little bit up on was this project that you've done with a group of BYU students for preservation of Cambodian stories. So I was wondering if you could tell us a little bit about how that project came up and and maybe what it's all about.

• 30:28Speaker 00

  Sure, I'm an alumnus of Brigham Young University and participating member in the Church of Jesus Christ of Latter-day Saints. And in our local congregation here in Massachusetts where I live, we have a large number of Cambodian speakers. And over the years, I've had a lot of opportunity to work with them, get to know them. I've learned a few words and phrases to be able to greet them and ask them how they're doing and so forth. I've always had this affinity for Cambodian, or Khmer, that's how you say Cambodian in Cambodian. So when I saw a story in an alumni newsletter that I got from BYU, that there was a group at BYU that was actively collecting recordings of personal histories of Cambodians,

• 31:11Speaker 00

  especially those who had been affected by the atrocities of the Khmer Rouge, that they were going out and they had collected, I think, 4,000 hours of stories, and that people had then manually transcribed like a thousand of them, struck a chord with me. I thought, these are people I care about, these are people that I've worked with. And my first thought was, oh, let me make sure that my friends here in Massachusetts, who were refugees, might have their own stories to tell. Let me make sure that they can contribute their stories to the collection. And then I thought, well, wait a minute, we have here a situation where lesser-resourced languages, where we have now thousands of hours of high-quality recordings with good transcriptions for a big chunk of them.

• 31:54Speaker 00

  I thought, those are the ingredients you need to develop a speech recognizer. And Cambodian doesn't have a good quality speech recognition system, so I thought we should do that. So I've asked some folks on my team here at Cobalt as a sort of a pro bono project to see what we could do on this. So we reached out to the group at BYU, and they said, oh, that's a great idea, because they could then use the speech recognition to help transcribe the rest of their audio going forward. And so BYU put up four or five students to do a lot of the labor of organizing the data and making sure that things are recorded properly and transcribed properly and ready to be processed in training the models.

• 32:34Speaker 00

  And we've at Cobalt had people contributing in sort of an advisory role and mentoring them. So it's good all around. Students are getting an amazing opportunity of learning about how speech technology works, and we're helping them do that. Anyway, the long story short is, we are close now. Maybe, you know, a couple of months away from having good quality Cambodian speech recognition, assuming, I'm gonna knock on wood, assuming that nothing goes wrong in the, in the meantime, because, you know, a lot of things have to come together just right to make it happen. But anyway, so Dan, I think that's the story you wanted me to tell, that we at Cobalt are working in conjunction with folks at Brigham Young University to develop Cambodian speech recognition. And now it's got me thinking,

• 33:19Speaker 00

  well, are there opportunities like that for other languages? Could we find some other partnerships where someone who wants a speech recognition for Swahili, or Aymara, or some other language, that we can go and say, here's some audio, I don't know where we're gonna get it from, and here's a partnership where we can, you know, work together to, where people from the community might work with us to develop the technology. Anyway, it's a passion of mine to figure out how to bring this technology to the whole world, and not just leave it in the hands where it's gonna make us the most money.

• 33:53Daniel Whitenack?

  Yeah, that's so wonderful. And I think maybe you can read my mind a little bit at this point, because I was just thinking towards the end of your story, hey, what learnings might you have from this initial experience that we could apply in other language communities? It sounds like there's a, in this case, there was partnership, right? There was the local language community, there was the academic institution, and there was an industry partner. As you look back on that, what do you think? Because it sounds like you've made progress, and hopefully, you know, like you say, knock on wood, things are coming out. In that process, any maybe tips or suggestions for people that are trying to maybe work on particularly

• 34:35Daniel Whitenack?

  like AI for good types of projects, they have a passion for that, and they're trying to maybe establish a partnership that would actually result in some value, and not be a sort of fun weekend hackathon project, but it would actually result in some value for the community. Any tips or thoughts or learnings that you've had over that time with that partnership that you could pass on?

• 34:59Speaker 00

  That's a good question. The folks at BYU that are working in this Cambodian oral history project, they had done all of their recordings and transcriptions before we came on the scene. And they didn't do it with the idea that they that their data was going to be used to build a speech recognizer. If they had known that going into it, they might have done a few things a little differently in the way they transcribed the audio, or the way they recorded the audio, or whatever, that might have made it a little bit easier for us. So a lot of the work that we're doing now is in sort of adapting our training scripts and models to the data that they've collected. I can't really say that I can't really fault them in any way, because that's not the purpose that they collected the

• 35:41Speaker 00

  data for, right? But if they knew that they would be doing that, it would have been nice for them to reach out ahead of time and say, hey, we're about we're about to collect all this data, and it's going to be useful for speech recognition models. Is there, you know, is there anything we should know in how we record it and how we transcribe it and so forth? So it's kind of a wish that doesn't really make sense in practical terms. But in general, if someone hears this and says, oh, I'm going to go start collecting data, or, you know, doing some work in this area for my language, I would say, why don't you reach out first and let's get involved from the beginning, rather than have to fix things after the fact?

• 36:18Chris Benson?

  You mentioned the Khmer Rouge, and just for context, for listeners who may not be familiar with it, since this dataset was in oral history, I thought, you know, that was a big enough event dominated that country's history for decades. That was a decades-long genocide. For those who aren't familiar, Cambodia is a very small country, and it ended up, I think, decimating about 25% of the country, which was like 1.5 to 2 million people died. I find, as someone who also loves history in addition to technology, I just find the the roots of that dataset as an oral history to be pretty fascinating. And I think there's a beauty in that you are able to also extract this extra unexpected

• 37:00Chris Benson?

  value from such a dataset to be able to to produce this kind of output. So sometimes when you see these two things coming together unexpectedly like that, and as you were talking through that, and I was kind of recalling back the history that the Cambodian people suffered through, there's definitely a beauty to that as an output. So.

• 37:16Speaker 00

  Yeah.

• 37:17Daniel Whitenack?

  Yeah, and it's kind of interesting to me on that point, Chris, that like, Part of the reason why this happened was because it sounds like the language community itself, those people, had a desire to record this audio, had a desire to create this asset. It didn't start with technologists coming in and saying, "Hey, we'd like to make this technology for you. Record all of these statements that I give you in a Google sheet" or something like that, which it's perfectly great if technologists want to want to come in and say, "How can I serve a language community?" But oftentimes, if it's like something they don't even have a desire for, then you're probably not going to get very far. So it's

• 38:02Daniel Whitenack?

  cool that there can be these situations in which the the community itself has a desire, and that can additionally have this benefit of expanding the possibilities of technology for that community. That's really cool. So Jeff, that's super inspiring. And I'd maybe like to move a bit to ask you about the future a bit in terms of what you're excited about exploring in speech technology, but maybe haven't yet. It sounds like you've explored most of the nooks and crannies of all over speech technology, but what are some things that maybe excite you in the future? And you're hoping that Cobalt gets to explore them, or that you get to explore them, you haven't quite yet. Is there anything like that that comes to mind?

• 38:45Speaker 00

  The list is long, and we would be here for another half hour if I tried to go into it all. But I'm going to cheat a little bit and answer your question without answering it. That is, the thing that keeps me going and that drives me is not knowing the answer to that question. The fact that I know that a few months from now, I'm confident that a few months from now, someone will have come to me with an idea that I had not thought of before. That's intriguing, and that I'll say, "Oh, yeah, that's great. Let's do that." And while there are things right now that I wish we had more time and the funding to work on, the thing that's really exciting is the things that I don't even know about yet that are coming in the in a few months, or years, or whatever.

• 39:27Daniel Whitenack?

  Yeah, and I guess maybe that gets to all of the information and complexity of speech and language.

• 39:33Speaker 00

  Yeah.

• 39:33Daniel Whitenack?

  Like, I don't know that I would have expected, like, oh, you could maybe detect someone having heart issues or something from their speech. That's pretty, I mean, maybe doctors, that's obvious to them, but it's not obvious to me as a, as just sort of an everyday person. So, you know, that's really a result of, hey, speech and language is really complicated. And despite our best efforts, there's still a lot of complication to explore. What do you think is driving that?

• 40:00Speaker 00

  That's part of it, that there's a lot to still explore. There's a lot we still don't understand about how to process it. It's a very complex process. A friend of mine said that speech and language is the most complicated process developed by the most advanced species. It's like the pinnacle of what we can try to handle and process and then take care of. But it's not just that speech and language is, is complex, hard to understand and hard to get right. It's also that it pervades everything so that the applications, it's hard to anticipate where those next applications are going to come from. There's so much explicit and implicit about how we think about speech and what we do with speech.

• 40:42Daniel Whitenack?

  And just because I know that our listeners, definitely based on what we've talked about, they're going to want to know about all of these unexpected ways that speech technology is kind of coming about. How can people find your podcast, The Voice Box? Where can they go to, to find it?

• 41:00Speaker 00

  It's on all of the major podcast platforms, so you can go find it there. I don't know if there's a better way to tell people where to find a podcast. Maybe you know that.

• 41:09Daniel Whitenack?

  Yeah.

• 41:10Speaker 00

  Maybe you know better than I do.

• 41:11Daniel Whitenack?

  We'll definitely link it in our show notes, for sure. And I hope people find it and go check it out. And also check out some of those links that we put to what Cobalt's doing and some of the things like this project with the Cambodian audio. We'll put all of those things in our show notes, so definitely check those out. But thank you so much, Jeff, for joining us on the podcast. It was a really good time and a pleasure to talk to you, as it always is.

• 41:34Speaker 00

  Daniel, Chris, thank you so much for your time and for having me.

• 41:40Speaker 02

  Thank you for listening to Practical AI. We appreciate your time and your attention. If you enjoyed this episode, help us out by spreading the word. Think of a friend, think of a colleague, somebody who would benefit from listening to it, and send them a link. We'd really appreciate it. Practical AI is hosted by Chris Benson and Daniel Whitenack. It's produced by Jared Santo with music by Brake Master Cylinder. Thanks again to our sponsors, Fastly, Linode, and Launch Darkly. That's our show. We hope you enjoyed it, and we'll talk to you again next week.