Speech recognition to say it just right

Catherine Breslin of Cobalt joins Daniel and Chris to do a deep dive on speech recognition. She also discusses how the technology is integrated into virtual assistants (like Alexa) and is used in other non-assistant contexts (like transcription and captioning). Along the way, she teaches us how to assemble a lexicon, acoustic model, and language model to bring speech recognition to life. Sponsors: Linode https://linode.com/changelog – Our cloud of choice and the home of Changelog.com. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2019 OR changelog2020. To learn more and get started head to linode.com/changelog https://linode.com/changelog. AI Classroom https://datadan.io – An immersive, 3 day virtual training in AI with Practical AI co-host Daniel Whitenack Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We move fast and fix things because of Rollbar. Resolve errors in minutes. Deploy with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Featuring: Catherine Breslin – Website https://www.catherinebreslin.co.uk, GitHub https://github.com/catherinebreslin, LinkedIn https://www.linkedin.com/in/catherine-breslin-0592423a, X https://x.com/catherinebukChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Catherine Breslin - Google Scholar https://scholar.google.co.ukCobalt Speech and Language https://www.cobaltspeech.comBusiness Insider: The 100 coolest people in UK tech - Catherine Breslin https://www.businessinsider.com/tech-100-2018-the-coolest-people-in-the-uk-tech-industry-2018-9? IR=T#42-the-woman-teaching-alexa-dr-catherine-breslin-59The Female Lead - Catherine Breslin https://www.thefemalelead.com/profile/catherine-breslinSilicon Republic: 15 architects building the data-driven future https://www.siliconrepublic.com/enterprise/sci-tech-100-2018-architects-futureStylist: Meet the woman who teaches Amazon Alexa how to understand humans https://www.stylist.co.uk/visible-women/catherine-breslin-amazon-alexa-machine-learning-scientist-women-in-tech/188131Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  And if you've ever used an old IVR system where you phoned up and talked to it, you very much get the feeling that you're being walked through a flowchart by voice. And at some point, you're being asked for some piece of information and you have to fill it in.

• 0:12Daniel Whitenack?

  Yeah, I did that just yesterday.

• 0:15Speaker 00

  They're still very popular. We're still working hard to deploy lots of advances. But this idea of modeling a conversation as a flowchart is quite popular. And now advances in language understanding have meant that we can be a little bit more flexible about what we're asking for and when.

• 0:35Speaker 02

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com/changelog.

• 0:53Speaker 02

  Do not underestimate the power of the independent open cloud for developers. Yes, I'm talking about Linode. Linode is our cloud of choice and it's the home of changelog.com. What we love most about Linode is their independence and their commitment to open cloud. Open cloud means being unencumbered by outside investment and maximizing value for the community, not shareholders. And that's exactly what Linode represents. No vendor lock-in, open at every layer. If you want to learn more, head to linode.comopen. Again, linode.comopen.

• 1:38Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at PracticalAIFM. Okay, here's Daniel and Chris.

• 2:03Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co-host Chris Benson, who is a Principal AI Strategist at Lockheed Martin. How are you doing, Chris?

• 2:19Chris Benson?

  I am doing very well, Daniel. How's it going today?

• 2:22Daniel Whitenack?

  It's going very well, you know, a lot of people working at home.

• 2:25Chris Benson?

  Yeah.

• 2:26Daniel Whitenack?

  This is not new to me and really to you either, I don't think.

• 2:30Chris Benson?

  No.

• 2:30Daniel Whitenack?

  Although travel is down, but it's been a productive few days for me. So I have no complaints. What about you?

• 2:40Chris Benson?

  Other than the travel, for March and April, I was going to be traveling quite a lot. And obviously not traveling at all. Working from home is normal for me, but it's, we've entered a little bit of a weird thing. And our, our school system locally has closed down indefinitely for coronavirus. So as we speak, my wife is in another part of the house, homeschooling my daughter. And that's a new thing for us.

• 3:03Daniel Whitenack?

  Yeah, yeah. And I saw, of course, there's been a lot of posts related to like AI and how that's intersected with this whole coronavirus thing. I know there was a good blog post that I saw the other day that kind of talked about that a little bit. I'll link that in the, in the show notes for people if they're curious. But that's not quite the topic that we're, we're here to talk about today. Back in January, Chris, you and I were at the Project Voice Conference down in Chattanooga, Tennessee, here in the US. And at that conference, I was able to meet Katherine Breslin, who is a solutions architect at Cobalt. And we have Katherine with us today to talk about all sorts of things related to speech technology.

• 3:48Daniel Whitenack?

  Welcome, Katherine.

• 3:50Speaker 00

  Hi, Daniel and Chris. Thanks so much for inviting me on your show. It's great to be here to chat with you.

• 3:55Daniel Whitenack?

  It's great to have you here. So before we jump into speech technology and voice assistance and speech to text and all sorts of things, could you just give us a little bit of your, your background and how you got into speech technology and then eventually ended up doing what you're doing now?

• 4:14Speaker 00

  Sure. So I'm sure it doesn't take much to work out that from my accent, I'm not from the same part of the world as you. So I'm over here based in Cambridge, UK. I've been living over here for, I've been in Cambridge for about fifteen years. And I moved here after I was at university and I was studying engineering at university. And at the time, I really had no idea what it was I wanted to do as a career. But I was really intrigued by the idea, which I learned about in my final year of undergraduate studies, of getting computers to seem smart, to do things which people can do really easily. And we looked at sort of vision and medical imaging and sort of

• 4:57Speaker 00

  vision-related technologies when I was an undergraduate. And then I thought about, what about language and speech? How do, how do people do this? And how can we make computers do those? And that led me down the path, which is how I ended up where I am now. And so I ended up sort of studying speech and language technology and understanding how we could make computers do some of these things and studying PhD, studying a masters and eventually ending up working various different places along the way. So my first career was effectively in, in research. I was a PhD student and I did research for Toshiba in their Cambridge lab here.

• 5:42Speaker 00

  And ended up doing a postdoc position as well, taking a postdoc position at the university all to do research into speech and language technology.

• 5:52Speaker 00

  And then around about sort of two thousand and ten, two thousand and eleven, the wider industry was taking off and people were actually building this technology into products and services. And I ended up leaving the research world and moving to work on products. So I worked on, I moved to Amazon. And when I joined Amazon, I learnt about Amazon Alexa, which was just about to be launched when I joined Amazon.

• 6:17Daniel Whitenack?

  A good timing.

• 6:18Speaker 00

  It was really good timing to work there. So I, I didn't originally start working on Alexa. I started working on some other products when I joined Amazon. But over time, things sort of coalesced under the Alexa umbrella and I worked on there for a few years before moving on to do what I do now, which is helping other businesses who want to build this technology.

• 6:39Chris Benson?

  Awesome. And I know that Cobalt, so tell us a little bit about Cobalt. I know that there's people there that have a long history with with speech, particularly, I think Jeff Adams is there at Cobalt, right? And he also was at Amazon. Is that, is that right?

• 6:54Speaker 00

  That's right. There's Jeff Adams is the CEO of Cobalt. He founded Cobalt about five and a half years ago now to help businesses who want to build speech and language technology, but didn't have the team to do it or the expertise in-house. And Cobalt's really grown over the last few years to take on people who have experience working in this technology. So, people from Amazon, people from other companies who've been able to build this technology in their careers and bring them all together to be effectively the the speech team for for other companies.

• 7:29Chris Benson?

  So I guess, starting off, you're an expert in this area, and Daniel is too. By the way, I am of the three of us, probably the one least knowledgeable in this area. And so I would love it if you would give me kind of an overview of what mainstream speech technologies look like, how they're being used. And, you know, we've talked about Alexa, and obviously, there's the competition to Alexa out there. Is it more than just virtual assistants? Is, you know, what does the landscape look like today?

• 7:57Speaker 00

  Yes. Let maybe we can start with virtual assistants. And I think they're a great way to think about it, because they contain all the different technology bits underneath them. So a virtual assistant is a pipeline of different technology that all works together to understand what you've asked and to do it. But the the underlying technology can be used in different ways. And we can talk about some of those later as well. But we can start by thinking about a virtual assistant and what happens when you ask a virtual assistant to do something. So if you say, hey, computer, play me some music, and then it starts playing you some music. There is a number of things that have to have happened for that to come true.

• 8:39Speaker 00

  So the first thing is that the computer has to firstly wake up and start to start listening to you when it hears, hey, computer, whatever it is that you've decided is going to be the the wake word. And that is often a very small, low-powered speech recognition system, which is sitting on a device or on your phone, that's listening very specifically for particular words. Then it's got to run speech recognition on what you've asked it to do. So speech recognition goes from audio to text. It's transcribing what it is that you've asked for. So it says, hopefully, play me some music. Although speech recognition systems are not perfect and they make some mistakes. So we hope that most of the time, though, it gets your request accurate enough when it transcribes it.

• 9:23Speaker 00

  But that's not enough for the computer to know what to do. The computer has to sort of bucket that into one of many things that it can do. So you could have asked for playing some music, or you could have asked for buying some music, and it has to distinguish those two things. You could have also asked for the weather forecast or asked for the answer to a factual question, which is slightly easier to think about. So there are some things that you ask about which are close together and some things are further apart. And the computer has to distinguish those with some sort of language understanding technology. And if you're asking about anything complicated, it not only has to sort of bucket what you've asked, but also what particular entities you might be asking about.

• 10:06Speaker 00

  So you could say, play me some music by Sting. And there it has to know that Sting is the name of the artist that you're actually after, that you're interested in hearing music from. So this language understanding technology is going to pick out what you want to do and the sorts of things that you want to do that with. So the artists you want to listen to, the city you want the weather forecast in, the album you want to hear, the thing you want to buy or add to your shopping basket. All of those things we have to pick out.

• 10:37Speaker 00

  And then there's some computer system which is going to take that request and go and execute it and actually figure out what music to play back. And if you play music back, you might just hear the music start to play, but you might also hear an announcement about the music it's going to play. Or if you ask for the weather forecast, you might hear it tell you the weather forecast in words. And that technology, that text to speech technology is the last part of the pipeline. And that's sort of like the opposite of speech recognition. In this case, you're going from text and converting it into speech that can be understood. So you've got these things together in this pipeline. You've got speech recognition, language understanding and text to speech, which will combine together

• 11:21Speaker 00

  to give you a virtual assistant which is going to act on what you tell it to do.

• 11:26Chris Benson?

  So I got a follow-up question for you and I'm wondering, and this is partly from the fact that I'm kind of thinking of it as a software developer in terms of kind of how I'm thinking about asking the question. And I'm asking as the person in this conversation who knows the least about the topic. So there is a series of tasks that you just talked about, but unlike if you're following, you know, like a webpage form where you have a very explicit set of things that are going to happen in a particular order and stuff. In this case, the types of questioning is you have follow-ups on what kind of music, what artist, that kind of stuff. How does the system kind of understand a collection of state, a collection of related

• 12:11Chris Benson?

  conversation? How does it conceptualize like when that's done? How does it, you know, given the fact that it is a loose, not tightly controlled system that we're talking about?

• 12:20Speaker 00

  So yeah, this is an interesting technical as well as a design challenge, I think. Because if you were to say to your virtual assistant, play me some music, you might design it such that it would, if in that case, it had no information about the music you wanted to play, but it would just pick something randomly that it thought you might like. And then it wouldn't ask for any follow-up about what it was that you actually wanted to listen to. But another person might design their virtual assistant to actually follow up on you and say specifically, what music do you want to hear? So there's um different design choices in how you build these things as well. And something like music, you can have a choice that just plays random music, so you don't have to specify any further.

• 13:05Speaker 00

  Whereas if you're asking for the weather, you probably actually want to know where in the world you're asking for the weather for. Otherwise, it's no good to anybody to get a random weather forecast. So these design choices have to come into it in the very early stages and thinking how much effort you want to put on the user. The second thing you have to bear in mind is that these systems are not that great at conversation yet. And if you try and initiate long conversations with people, they can get confusing and they can frustrate your users too. So another design goal is to sort of keep the conversations that they're having as short as possible.

• 13:40Speaker 00

  And so with this in mind, you can think about which of these. I talked about taking intents or taking bucketing what people were asking for and the the things that they were asking about. So when we bucket what are you users requesting into different categories, we call those intents. So you might have an intent which is a play music intent. You might have an intent which is a turn on the lights intent or a give me a weather forecast intent. And this is just just a general shape of the request that the user's asking. And then within that intent, you will have a set of what gets called in various ways, either a concept or an entity or a slot. The terms are somewhat used interchangeably.

• 14:24Speaker 00

  And you have some sort of entity that you're asking about. So in the case of music, it might be an artist's name or an album name. In the case of a weather forecast, it might be the city. And those entities, we can decide that the computer has to ask for them. Or maybe they can have some default behaviour or maybe some random behaviour. So the designer has to choose which of these entities are desired or necessary. And in the same way you do for other programming defaults, I think.

• 14:54Daniel Whitenack?

  So I'm curious, I mean, you mentioned the idea, so two ideas, I guess. One around the fact that these types of assistance are composed of these various components that all work together. And then second, that, you know, part of a current limitation of these systems is long conversations. We'll get into sort of the speech to text and maybe speech synthesis parts here in a second. But I was wondering if I'm right in assuming that that limitation in terms of the length of conversations, or maybe how dynamic they can be, is driven by limitations in the sort of natural language understanding that you mentioned, the, I guess, the middle bits.

• 15:36Daniel Whitenack?

  So after you've detected the speech and you've converted that into text, maybe, then you have to decide what to do with it. Is it those limitations that limit how dynamic things can be at this point?

• 15:46Speaker 00

  I think there's a separate step, which I sort of glossed over a little bit, which is managing the state of a conversation. And in the past, we've designed conversations to be sort of like flow charts. And if you've ever used an old IVR system, where you phoned up and talked to it, you very much get the feeling that you're being walked through a flow chart by voice. And at some point, you're being asked for some piece of information and you have to fill it in.

• 16:11Daniel Whitenack?

  Yeah, I did that just yesterday.

• 16:12Speaker 00

  Yeah. That's still very popular. We're still sort of working hard to deploy lots of advances. But this idea of modeling a conversation as a flow chart is quite popular. And now, advances in language understanding have meant that we can be a little bit more flexible about what we're asking for and when. So we might be able to have a user say in one single utterance everything they're asking for, whereas before, we might have broken it up and asked them specifically over a number of conversation turns. And this means that we have to introduce a technology which is tracking all the information that the user's given us, a sort of dialogue state. We have to move between dialogue states as the user gives us different bits of information,

• 16:57Speaker 00

  or as they ask us to do different things. So there's really a couple of points that are limiting that what we can do. One is the language understanding technology at the moment. It can recognize broad categories of things, but a lot of what users might ask for doesn't necessarily fall in something which can be easily modeled as intents and slots. If you're just having small talk with a computer, some sort of chit chat, that's a different kind of conversation and that's not naturally modeled with intents and slots. So we can rule out having some of those sorts of conversations with this this way of building technology.

• 17:38Speaker 00

  And the other thing is if you have a conversation turns which which happen over time and the user starts to ask for more complicated things, or they start to refer back to things that they've talked about earlier. Sort of tracking those over a conversation is also quite hard. If you ask your virtual assistant, you say, what's this? Then it has to know what this is. It has to be able to try and figure out what you're talking about, which can be relatively straightforward. If there's a song playing, you might assume that it's the song that's playing. But if you've talked about something previously in the conversation, it could be that. So there's a lot of complicated language-related

• 18:18Speaker 00

  technology to do that tracking over time, which doesn't work well at scale yet.

• 18:38Daniel Whitenack?

  Hi there. This is Daniel Whitenack, one of the co-hosts of Practical AI. And when I'm not working on Practical AI, I'm developing my own AI applications or I'm training teams at other companies. I've been doing this for over 10 years now and I've trained more than 1,000 people. Now I'd like to invite you to my new live online training event called AI Classroom. In AI Classroom, I'm going to teach the practical skills I've learned over the years using the latest open source AI technology. You will learn both AI theory along with practical hands-on implementations in both PyTorch and TensorFlow.

• 19:16Daniel Whitenack?

  After attending AI Classroom, you'll be able to understand the latest models, implement your own models in code, train computer vision and NLP models, create model inference servers, and experiment with state-of-the-art methods like reinforcement learning. AI Classroom is taking place this May. It'll be taking place live and completely online in a high-quality virtual classroom, so no travel is required. There will be two cohorts with convenient time zones for Eastern and Western hemispheres, so don't miss out. Tickets and more information is available at datadan.io. That's datadan.io. And early bird pricing lasts until April 3rd. See you online in AI Classroom.

• 20:13Chris Benson?

  So Catherine, I'm kind of curious, as we kind of launched right into assistants upfront. I'm also kind of wondering if we could lay out kind of landscape on how is speech technology being used outside of assistants strictly and in the larger landscape? What is there in the speech community out there beyond these assistants that we've been talking about?

• 20:37Speaker 00

  Sure. So I think one of the obvious things you can think about is just speech recognition technology on its own. So taking long streams of audio and transcribing it. And there you can think of all sorts of applications which don't fall within virtual assistants. And maybe one of the obvious ones is automated subtitling. So especially as we're moving more online and we're having more video content created, being able to automatically subtitle that can be really helpful. Not just for accessibility for people who are watching it in real time perhaps, but also to search it later on. So if you want to... Come back and find particular places in a video where things are talked about. Having an automatically generated transcript can help with that.

• 21:22Speaker 00

  So that's that's one place speech recognition is used. It can be used in other places, though there are some industries, for example, where it becomes very important to monitor conversations between people for for legal reasons. So if you're giving financial advice to somebody, you want a record of that financial advice. But manually transcribing all of these conversations gets very tedious and time consuming. So there's great applications there to help relieve the workflow and make it easier for people to manage.

• 21:53Daniel Whitenack?

  So as we're kind of getting a bit more into speech recognition, which I think, like you said, is one of the things that pops in the people's mind as something that has utility both in assistance and in various other places. I know you're an expert in this technology and I also know from our previous conversations, I was learning from you about how speech recognition itself can be composed into various pieces, almost like like a pipeline as well. I was wondering if you could cover what those steps in a typical speech recognition system are and what their function is?

• 22:29Speaker 00

  Sure. So as I said earlier, the the beginning, the job of a speech recognition system is to take audio and to give you an estimate or guess of what words were spoken, so to to transcribe that audio. And there's three different parts that we typically break this down into. So the first part that we have is what we call a lexicon. So if you think about words in a language, they are composed of different sounds. So the word cat, for example, is composed of the sound k-a-t. And the word bat is composed of b-a-t. So we can take this phonetic representation of different words and create a lexicon,

• 23:13Speaker 00

  which tells you how each word is pronounced. And in a language like English, we might find that there are about 50 different sounds that we can combine together to to make all the different words. So the lexicon maps words to their pronunciations. We have then a model in the system called the acoustic model. And the acoustic model, as you might guess, models the acoustics of sounds and speech. And this is the bit of the model which is going to tell you for each little bit of sound that someone is talking, which phoneme is likely to be spoken, which of those 50 sounds is likely to be happening at that time.

• 23:56Speaker 00

  And this acoustic model is going to be built on lots of audio, that we know what's happening in the audio. And we can build a machine learning model which is going to be able to tell us, given a huge database of sounds, where we know what sounds are being spoken, can we predict for new sounds, for new audio, what sounds are being spoken? And then we have finally a language model. And the purpose of the language model in a speech recognition system is to predict sequences of words. So from a language model, we might say, if the input to the language model is "Hi, my name is", then it's very likely that the next word is going to be "Catherine", but the next word is very unlikely to be "Goodbye",

• 24:39Speaker 00

  and so the language model is sort of predicting which sequences of words are more likely than others. So if we put these together, we have an acoustic model which tells you from some audio which sounds are likely to be spoken at that time. The lexicon tells you how those sounds combine into words. And then the language model tells you how those words combine into sequences of words. And so we use these three models sort of in a combined decoding procedure, decoding algorithm, to run, work together to tell you for a new piece of audio, what words are likely to be spoken in that audio.

• 25:14Chris Benson?

  So, got a question, as you've been talking about these acoustic models and language models, and I guess, almost a basic question, when you talk about these, what kind of models are they? Are they statistical models? Are they, are they deep learning models? Are they, are there other technologies that are being used to blend in? Could you kind of talk about what those are fundamentally?

• 25:36Speaker 00

  Yeah, so, one thing I should mention is that there is, what I'm talking about in terms of breaking down into acoustic and language model lexicon is... A technology which is very common right now, but there are in the research community, there's a lot more effort to come up with new neural network approaches to build one speech recognition system, which isn't decomposed into these three parts. But I'll carry on talking about the three parts because that's how a lot of our commercial speech recognition systems are built right now. So the lexicon, which is basically a phonetic pronunciation of lots of words, often we have really good, high-quality, handcrafted lexicons. So they're written by phoneticians and you can imagine for a language like English, we

• 26:19Speaker 00

  have good lexicons which have been developed over many years. Now, obviously, language is always changing and new words come up and new ways of using language come up. And so we still have to be able to predict pronunciations of new words, but we have very good lexicons to tell us pronunciations of the vast majority of English words already. And so predicting new words, the pronunciation of new words is a smaller task.

• 26:47Speaker 00

  The acoustic model and the language model then end up being statistical machine learned models. And the language model is trained on text and the acoustic model is trained on audio. So two different models trained on two different types of data. So the acoustic model, for instance, we've had great success in recent years. If you think back, like 10, 15 years ago, speech recognition systems didn't really work all that well. And they were using often for the acoustic models Gaussian mixture models. And one of the things that's really happened in recent years, in the past 10 years or so, is the switch from Gaussian mixture models to neural network acoustic models.

• 27:32Speaker 00

  And that's had a really big impact on the performance of speech recognition systems. It's made them significantly more accurate. Over the past few years, and that's come in tandem with other advances in computing power and storage and memory and computation and all of those other things. But the actual switch from GMM models, Gassia mixture models to neural networks and the acoustic modeling side has really made speech recognition systems much more accurate.

• 27:56Daniel Whitenack?

  So I'm curious, as I was thinking through what you're talking about with these various bits, obviously, like you said, there's a need for this sort of expert information upfront, and that's available for languages like English and probably less available for other languages. But I was also thinking about the side of things, which is accents. And in terms of like, the lexicon and determining how each word is pronounced, and then also for the acoustic model, I guess that's where the biggest impact is. But it seems like that filters all the way through, because even for the language model, if I understood right, you're, you're kind of taking a sequence of sounds or phonemes and then translating

• 28:36Daniel Whitenack?

  that in a sense to actual words. So the sounds would even affect that bit, I guess, as well. So how does accent come in into this?

• 28:45Speaker 00

  Say, yeah, you're right. Accent sort of touches every bit of the system. If you have, the lexicon itself is a place where you might have multiple pronunciations of a word, but there would be really standard pronunciations of a word. So you might have the and the as pronunciations for the, or either, either.

• 29:05Daniel Whitenack?

  Like alternative?

• 29:06Speaker 00

  Yeah, alternative pronunciations. But you wouldn't necessarily like handcraft in the lexicon every single accented pronunciation or something, because that would just mean you had loads and loads of entries in the lexicon. And it would get difficult for the the computer to tell them apart over time. It would make it much more difficult for the speech recognition system to work. So we might have, if we knew we were working with particular accents, we would have different lexicons. And you see that, for example, we might have a different lexicon for the US and the UK, because there are substantially different dialects of English. Then we have the acoustic model. And the acoustic model is modeling, as I said, the sounds of the language. And this is a neural network model right now.

• 29:50Speaker 00

  And if you can train this acoustic model on different variations in speech, so different accents, but also not just accents, but different noise conditions that are happening in the background. So different microphones that people are using, different distances, they're talking from the microphone. So there's a difference between if you're talking close up to a microphone versus if you're talking across a room to a microphone. So all of these variations go into the acoustic model and the model there along with some accent.

• 30:20Speaker 00

  And finally, the language model, you find that accents don't just change the sounds that people say when they say things, but they might change the phrasing that people use and the order that they say words in. And that's where the language model will pick some of that up and model it.

• 30:36Daniel Whitenack?

  So I'm also curious for the acoustic model, you mentioned that the kind of upgrade in recent years, it's really boosted performance in terms of the use of neural networks. And for something like text or for images, we've talked on the show about how both of these types of input data are encoded into neural network models. So for example, like with text, you have maybe you do some type of encoding or you have a vocabulary that you're able to convert essentially strings or words into into numbers. Similarly, for images, there's ways of encoding images. But I

• 31:16Daniel Whitenack?

  was wondering, we haven't really talked a lot about audio data as an input to neural network models. So could you describe, you know, what's unique about using audio as an input and how encoding that sort of data is maybe different?

• 31:32Speaker 00

  Sure. So if you take a stream of audio, it has some differences from text, it has some differences from images, you might think, but there's a lot of stuff in common as well. So if you take a stream of audio, what we tend to do is we split it into small segments. So each segment of audio might be about 25 milliseconds long. So we have a long stream of audio which gets split into discrete segments. And each of these segments is going to be one input frame, it's going to be one feature, one vector, one feature vector in our input to our machine learning model. So the next step then is to take these 25 millisecond segments of audio and convert them into a vector or a presentation. And so we'll typically take this small segment

• 32:16Speaker 00

  of audio and we'll do a Fourier transform on it. To give you the frequency distribution in that segment of audio. And then we will have a filter bank, which is a particular kind of set of filters, triangular filters, which are spaced in particular that they're centered on specific frequencies which follow a scale to sort of mimic the human ear a little bit. So the filters might be further apart, spaced further apart in the higher frequency band, because in the lower frequency bands, that's where it's more sensitive. So we pass the frequency spectrum through this filter bank, just to get a set of

• 33:02Speaker 00

  filter bank coefficients, which we can use as features for our neural networks.

• 33:16Speaker 04

  In case you missed it, Manning hooked us up with three free ebook copies of Build a Career in Data Science. We had one of the authors, Emily Robinson, on episode number 81 to discuss. If you want to get your eyeballs on this excellent resource, all you have to do is leave a comment on the episode page. Tell us about your career in data science and how the book might help you. Head to changelog.com slash practical AI slash 81 and click the discuss link on the playbar. You have until April 12th to enter. Once again, that's changelog.com slash practical AI slash 81.

• 34:08Chris Benson?

  So Catherine, I'm kind of curious. I've been learning a lot from you here today. Thank you very much. What is the state of speech recognition right now? Kind of for higher resource languages, what does the accuracy look like at this point? What are the current challenges that you guys are dealing with going forward? And what kind of improvements are you expecting or striving toward to meet those current challenges?

• 34:33Speaker 00

  Yes, I think you're right to think about high resource languages versus low resource languages because we've been doing speech recognition research in English for for many years now. So obviously, there is a lot more data available, a lot more benchmarks, a lot more knowledge that's been spread around. And so there are still some challenges when it comes to building good speech recognition systems. And they have different dimensions of difficulty to deal with. So noise is one dimension. So if you're speaking in a nice quiet room, that's very different speech recognition performance to if you're speaking in a car or if you're speaking in a noisy environment, like you have some machinery behind you, things like that.

• 35:15Speaker 00

  So so noise is still one thing which makes speech recognition hard. There's other challenges as well, such as the style of speech. So when someone is speaking, if they're standing up, addressing people, or if they are reading text, they speak in very different ways to if you're just having a back and forth conversation in a meeting room with some friends. People are much less formal when they're having lots of heated discussions and their voice and the way they say things are slightly different. So that's another sort of dimension of difficulty there in that if you're trying to recognize people reading a passage of text for a large group of people, that's easier than trying to transcribe people in a meeting.

• 36:00Speaker 00

  So where they're passionately discussing lots of different ideas. Things like accent is also something which can be difficult to deal with. If you have heavily accented English, that can make it harder to recognize. And also, if you have to recognize specific domains, so specific types of language, that can be more difficult. So if you have a speech recognition model which is trained on general conversational English, but you try and use that to recognize somebody giving lectures on chemistry, you see a degradation in performance because the two scenarios are not well matched up. And so I think in high resource languages, we're really good at sort of building very good general purpose speech recognition systems.

• 36:45Speaker 00

  But when it comes to building specific speech recognition systems to work in specific noise conditions or for specific types of tasks like the chemistry lectures or for specific accents, specific types of condition, that's where I think it gets much more difficult. Even in high resource languages, we don't always have lots of data in those scenarios. And so that's one of the areas that we work on a lot is to try and build speech recognition systems customized to different domains, to different scenarios, to different purposes.

• 37:17Daniel Whitenack?

  So you brought up something that kind of, I guess, made me look a little bit broader at the problem. And I was thinking mostly of, oh, I'm going to speak into a microphone and then, you know, the computer's going to figure out the corresponding text. But in scenarios where you have multiple speakers or noise, like you mentioned, there's a lot going on there.

• 37:37Speaker 00

  Yep.

• 37:38Daniel Whitenack?

  Do people deal with that sometimes as kind of another layer of this three-step process? Like maybe speaker segmentation or like picking out speakers? Is that, can that be like bolted on to the other type of system?

• 37:54Speaker 00

  It can be, yes. So you can identify speakers in different ways. If you have a room which speakers are sitting around and you have a microphone in the middle that is fixed, you can use a microphone array, for example. And a microphone array has got more than one microphone in the middle of the table. Often, some of these might have, say, seven microphones in a circle. And there you can actually do something, which I think is quite smart, which is figure out the direction that a voice is coming from. Because when people speak, their voice takes a little time to travel. And if you have two microphones in the middle of the table, their voice will get to the microphone closest to them just fractionally before it gets to the one further away. And so using that, you can figure out where in the room someone is.

• 38:40Speaker 00

  So this is a great way to separate speakers in a room if you have a microphone array to help you. You might not always have a microphone array. If you've got some sort of online conferencing where everyone's busy talking on their own computer and they are talking into a central system, you might actually have access to all of those separate microphones, which is super helpful. But if you don't, you can use the characteristics of people's voice to be able to tell them apart as well. So there's different ways you can do it and you can figure out who is speaking when. And that's a task we call diarization.

• 39:13Daniel Whitenack?

  So much new jargon. It's always good to parse through that.

• 39:17Speaker 00

  Yeah, you forget that each field has its own terminology.

• 39:20Daniel Whitenack?

  Yeah, exactly.

• 39:21Chris Benson?

  So I got a question for you. I know that there are end-to-end approaches out there for speech recognition, end-to-end approaches specifically without multiple models involved. I'd like to know what the state of that, where are people with that? I think Google has had some speech-to-speech models they've talked about in the past. And I'm sure there are others as well. Can you tell us what that looks like at this point?

• 39:44Speaker 00

  Yeah, sure. So the idea of an end-to-end model is that you have a single neural network model. It's usually a neural network because it's the most powerful model we have. A single neural network model where you put audio in and you get words out. You know, need for a separate language model or a separate acoustic model or even a handcrafted lexicon. Neural network is going to model all of that internally, which I think is a really neat idea because it just streamlines the whole process, makes it much easier to comprehend.

• 40:14Speaker 00

  And you're right that Google are now one of the ones who've been pushing the boundaries in this area. And it's a very big topic of research right now. And I know that if you go to the speech recognition conferences, a lot of the papers are looking into end-to-end methods for speech recognition right now. One of the downsides of this approach, I think, is that you typically need quite a lot more data to be able to build them. So if you are... Building a speech recognition model and then you quickly want to convert it to a new scenario and you need a lot of audio data to do that, that can get a little bit expensive. Whereas if you've separated your models out into an acoustic model and a language model, you can maybe take advantage of the fact that you need text data to train your language model,

• 40:59Speaker 00

  you don't need audio and so you can do better that way. And you can separate out whether you're adapting to the language that someone's using or to the acoustics of the situation and make the data collection a little easier. And to my knowledge, although this is rapidly changing, and I can't claim to know the latest results, but these models tend to be sort of either on a par or just slightly worse than our current, what we call hybrid systems, the acoustic model, language model, lexicon systems.

• 41:30Daniel Whitenack?

  So I'm curious, kind of piggybacking off of that question. It seems like, you know, maybe in the so early 2010s, like there was this ImageNet moment for AI with with computer vision. Now it seems like we're in a phase where like everything is about natural language processing. It's kind of having a moment in terms of text and these large scale language models like BIRD and GPT-2 and all of these things. And I was curious, like, if you think there's going to be a similar sort of acceleration of speech and AI at some point? And if so, maybe what's holding that back now? I was kind of trying to process this in my own mind and thinking, oh, well, for text,

• 42:14Daniel Whitenack?

  like, we've got the whole internet of texts that we can crawl and pull and maybe there's not as much speech data, but there is, there's a lot of audio data out there. So I don't know, is progress, do you think, could be accelerated by the availability of more data? Or is it the methodologies or what do you think?

• 42:34Speaker 00

  I think that the speech community have had an awful lot of shared. Tasks. One of the, you said about ImageNet, and ImageNet is a large shared dataset of images. And there have been, over the past years, plenty of large shared speech recognition tasks that different research groups have contributed to and worked on. So I think that in the way that text and images have relied on building up big data sets, I think that already exists in the, in the speech recognition community to some extent. Although maybe the, the size of the data sets is not as large as we need. Maybe the annotations are not quite there, but certainly having data sets available

• 43:15Speaker 00

  that people can work on and benchmark against each other has been in place for a while. And I think that's actually driven some of the improvements, like I said, from GMMs to neural networks over the past few years in that having these shared tasks has meant that people can easily build and test and compare different systems. Transcribing audio is a little more time consuming. One of the reasons that I think is so popular, has been so successful, is that it can just take large amounts of unlabeled data to build from. Whereas transcribing all the speech recognition data, you might need to build up a large dataset, is, is time consuming. And there are efforts to do that, to build up larger data sets.

• 43:59Speaker 00

  And I know that one of the breakthroughs of ImageNet was having just large amounts of annotated images available for people to use.

• 44:06Daniel Whitenack?

  So like when we're talking about a speech recognition model, whether it be the sort of hybrid approach that you're talking about, or the end-to-end approach, how much audio is, is needed to achieve something that's fairly useful, at least on, on general conversational data, maybe not domain-specific data?

• 44:25Speaker 00

  Yeah. We have speech recognition models, acoustic models in various languages. And one of the successful ways to build in a new language is to use transfer learning. So to take an English acoustic model, take off the last layer, put on a new last layer that represents, say, German, and train the last layers using a smaller amount of German data. So we're in a really good position, having good English acoustic models, so we can use transfer learning to new languages. To do transfer learning to a new language, obviously, the more data you have, the better. But sort of of the order of 100 hours up to 1,000 hours of audio can get you really good starting points for your model that you want to build.

• 45:03Chris Benson?

  So would automated annotation in some way, kind of going back to the, you know, that like, what does it take to get there? As I've been listening to you and Daniel talk about that, would have being able to get that large volume of data annotated and transcribed in an automated way so that it wasn't such a burden? Do you think that would contribute significantly in that direction or not?

• 45:24Speaker 00

  Yeah, that's one of the other things that people actually do in practice as well is speech recognition, especially if you have a commercial system that's up and running. You can have a throughput of some thousands of hours of audio, but maybe you only have capacity to transcribe 100 hours of that. So using that small amount of human transcribed data is going to improve your performance. But then you can do sort of unsupervised or semi-supervised learning using that automatically transcribed data as well. And it's not going to give you the same order of magnitude of gains as having the transcribed data. But if you have more automatically transcribed data, and then you use that to update your acoustic models, that can also give you some some good gains.

• 46:09Daniel Whitenack?

  Awesome. Well, there's so much here to dig into further. I know I want to after the conversation, but I was wondering if we could just kind of close out by talking a little bit about, you know, what you're excited about in terms of the future of speech technology? What what are you excited about implementing or developing yourself? Or what are you following? What gets you excited in in this topic?

• 46:33Speaker 00

  What am I excited about? I am excited about the fact that we have worked out how to build this technology in English and how to scale it up. And now that there's a huge opportunity to take it out to new languages and to build things that can be helpful to people all over the world, particularly if you think about virtual assistants are putting it all together into some sort of voice interface. I think that's a really nice, neat way to interact with computers that can make technology accessible. So there are huge parts of the world where people don't necessarily read and write in the same way that we do. And therefore, Having this technology available to them is a great way to make things more level. And I think it's a great way to make technology accessible to people who can't always just use a computer.

• 47:17Speaker 00

  So I know I have elderly relatives who can't use a mouse or a keyboard very easily. And for them, voice technology has been really helpful to help them be able to do things that they wouldn't otherwise have been able to do. And there is different work being done in making voice technology work for people who have maybe medical conditions which affect how they speak. And so sort of helping these people be able to live more independent lives. So I think there's a huge amount of need and desire to have this technology working for a broader range of people. And I think that's what we'll see in the next few years is sort of widening access to this technology.

• 47:56Daniel Whitenack?

  Yeah, that's super encouraging and glad you mentioned that. I'm excited to to follow your work in the coming years and really appreciate you helping us learn about these technologies today and taking time to share with us. Thank you so much.

• 48:09Speaker 00

  My pleasure. It's been great to chat with you.

• 48:15Speaker 04

  Thank you for listening to this episode of Practical AI. And a big thanks to Catherine Breslin for coming on the show. You can find everything Catherine is up to in the links in the show notes. This episode was hosted by Chris Benson and Daniel Whitenack. It was produced by Jared Santo. Hey, that's me. Our music is brought to you by the mysterious Breakmaster Cylinder. We have awesome sponsors supporting the show. You know, Fastly, Linode, and Robar have our back. If you and your organization would benefit from speaking directly into the ears of all the AI practitioners out there, consider sponsoring the show. Podcast advertising is very effective, especially when you're talking to the exact people you want to. Plus, you get that warm, fuzzy feeling of supporting something you love. Head to changelog.com/sponsor to learn more. We'd love to work with you.

• 48:59Speaker 04

  That's all for now. We'll talk to you next week.