Next-gen voice assistants

Nikola Mrkšić, CEO & Co-Founder of PolyAI, takes Daniel and Chris on a deep dive into conversational AI, describing the underlying technologies, and teaching them about the next generation of voice assistants that will be capable of handling true human-level conversations. It’s an episode you’ll be talking about for a long time! Sponsors: O'Reilly Media https://www.oreilly.com/changelog – Learn by doing — Python, data, AI, machine learning, Kubernetes, Docker, and more. Just open your browser and dive in. Learn more and keep your teams’ skills sharp at oreilly.com/changelog https://www.oreilly.com/changelogRudderStack https://rudderstack.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog – Smart customer data pipeline made for developers. RudderStack is the smart customer data pipeline. Connect your whole customer data stack. Warehouse-first, open source Segment alternative. The Brave Browser https://brave.com/cha913 – Browse the web up to 8x faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built-in Basic Attention Token https://basicattentiontoken.org. Download Brave https://brave.com/cha913 for free and give tipping a try right here on changelog.com https://changelog.com. Featuring: Nikola Mrkšić – GitHub https://github.com/nmrksic, X https://x.com/nikola_mrksicChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: -PolyAI https://www.polyai.com -The Science Behind PolyAI https://www.polyai.com/technology -How Customer Service Voice Assistants Handle Different Languages and Accents https://www.polyai.com/voice-assistants-languages-and-accents -PolyAI Maintains Lead Over IBM Watson Assistant and Other Popular Language Understanding Models https://www.polyai.com/polyai-maintains-lead-over-ibm-watson-assistant-and-other-popular-open-source-language-models -PolyAI’s ConveRT Model Outperforms BERT and GPT-Based Models in Salesforce Research Evaluation https://www.polyai.com/polyais-convert-model-outperforms-bert-and-gpt-based-models -Few-Shot Slot Labeling with ConVEx: The Most Accurate Value Extractor on The Market https://www.polyai.com/few-shot-slot-labeling-with-convex-the-most-accurate-value-extractor-on-the-market Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Chris Benson?

  I think a lot of people that aren't working in the field would assume that we're going to take a few years and kind of solve all this NLP and N related stuff.

• 0:08Speaker 01

  Yann LeCun famously said that that he's going to solve NLP in two years.

• 0:13Chris Benson?

  How many years ago was that?

• 0:15Speaker 01

  I think like six, seven. He's made great progress.

• 0:18Chris Benson?

  Yeah, yeah, no doubt. No, I heard a similar thing. I think it was Eric Schmidt in a tweet. He was like, speech is a solved problem.

• 0:25Speaker 01

  I think Eric Schmidt showed a lot more wisdom than Yann LeCun did there, despite probably less intimate technology understanding.

• 0:33Chris Benson?

  Okay, but I gotta ask this now. I'm curious because you said that and I know that it's not a static thing. It's done. I know that you're going to continue to make great progress. You've been doing that and you've been telling us about it through this. Where is this going and how does larger vision for the problem evolve over time to where you as a young man, a millennial, says this Gen Xer who's significantly older? How is this evolving over your lifetime to where you are remaining impassioned about solving this problem in the long term? What does that look like?

• 1:06Speaker 02

  Big thanks to our partners, Leno, Fastly and Launch Darkly. We love Leno. They keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com. This episode is brought to you by our friends at O'Reilly. Many even know O'Reilly for their animal textbooks and their conferences, but you may not know they have an online learning platform as well. The platform has all their books, all their videos and all their conference talks. Plus you can learn by doing with live online training courses and virtual conferences, certification, practice exams and interactive sandboxes and scenarios to practice coding alongside what you're learning.

• 1:48Speaker 02

  They cover a ton of technology topics, machine learning, AI, programming languages, DevOps, data science, cloud, containers, security and even soft skills like business management and presentation skills. You name it, it is all in there. If you need to keep your team or yourself up to speed on their tech skills, then check out O'Reilly's online learning platform. Learn more and keep your team's skills sharp at O'Reilly.com/changelog. Again, O'Reilly.com/changelog.

• 2:29Speaker 03

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM.

• 2:55Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with S I L International, and I'm joined, as always, by my co-host, Chris Benson, who is a Principal Emerging Technology Strategist at Lockheed Martin. How are you doing, Chris?

• 3:11Chris Benson?

  I am doing just fine. How's it going, Daniel?

• 3:13Daniel Whitenack?

  It's going great. No complaints. Took a few days off last week and a short week this week, so I'm rushing to get things done this week, I guess, is how it is.

• 3:24Chris Benson?

  It's supposed to recharge the batteries and make you feel all refreshed, but you just end up having to do all the same work and get it done faster. I get it.

• 3:31Daniel Whitenack?

  Yeah, I have known people in my life that do a really good job of, like, front-loading some of that before they leave.

• 3:38Chris Benson?

  Yeah.

• 3:38Daniel Whitenack?

  For some reason, I've never been able to figure that out.

• 3:41Chris Benson?

  No, neither have I. I think that's like a superpower that people have. It's a vacation superpower.

• 3:46Daniel Whitenack?

  Yeah, yeah.

• 3:47Chris Benson?

  It's not something I've ever acquired, sadly.

• 3:49Daniel Whitenack?

  Yeah, maybe someday.

• 3:50Chris Benson?

  Yeah, I have to go into vacations highly stressed out because of everything that I was trying to get done. So I guess that makes the vacation all the more important.

• 3:58Daniel Whitenack?

  Maybe so, maybe so. Well, one of those things that I'm definitely working on right now and trying to get out the door is a couple of changes to some of our internal speech dialogue-related technology, which is part of the reason why I'm really excited today about the topic, because we've got a great guest, the CEO and co-founder of Polly AI, Nikola Mirksic. Welcome.

• 4:23Speaker 01

  Thank you for having me. It's great to be here.

• 4:25Daniel Whitenack?

  Yeah. So before we get into all sorts of speech and voice and dialogue-related things, maybe you could just share a little bit of dialogue with us about your background and how you got to do what you're doing now.

• 4:38Speaker 01

  Yeah, for sure. So, look, I'm the CEO and co-founder of PolyAI. I did a PhD with a guy called Steve Young back at Cambridge with my two co-founders, Sean Wen and Eddie Su. And, you know, we worked on building dialogue in an academic context for a long time, since 2006, when Steve founded the group. And, you know, Steve started working on this stuff back when speech error rates were about 20%, which means that kind of like one in five words that you'd say would be kind of like misrecognized. And typically, it would be the one word that you needed to get right to understand what the meaning of the sentence was.

• 5:13Daniel Whitenack?

  Of course.

• 5:15Speaker 01

  So, you know, kind of like building a formalism on top that would kind of like let you model the uncertainty, predict all the right things, know when you've got something wrong, disambiguate, ask a question, implicitly confirm something. It's an art, right? And I'm sure we'll talk about it all of it, but the squat or the deadlift of, you know, just kind of like doing machine learning and NLP, right? Because it involves natural language understanding, dialogue management, response generation, interacting with the external world, its knowledge bases, specific different tasks, all the way into the natural language generation, figuring out what to say in human language again, and then producing it again in an audio format, which is, you know, as human sounding as possible. So it's a real

• 6:00Speaker 01

  compound movement and you have to get it all right. So it's a really difficult task. It's why I got into it. It's why I stayed passionate about it. And, you know, my whole team is really a group of people who worked on it long before this new hype of conversational AI came about. And we worked on it for a while. Our previous company also spit out from University of Cambridge, Vocal Q, was acquired by Apple in 2015 to make Siri more conversational, to give it a bit more of an ability to have. A back and forth conversation and that kind of like in academic terms, the multi-turn task-oriented dialogue is something we've stayed passionate about. And at PolyAI, we're building voice assistants for customer service, helping brands create something that's a superhuman customer experience, right?

• 6:42Speaker 01

  For anything that's short to moderate or even high complexity, putting in automated agents that sound at least as good as your best agents. They have answers to all the right questions. They're always current, up to date, and they're able to provide a superb, a superb level of customer service. And if they're not, they hand off to their human colleagues as kind of like supervisors or tutellations.

• 7:05Daniel Whitenack?

  So you, you mentioned one thing, like a phrase, a multi-turn, uh, voice-enabled dialogue, I think, I think is what it was. So maybe you could just kind of set that in context for some of the, like, what are the categories of sort of dialogue and voice, uh, enabled dialogue that are out there? Like, probably most people are familiar with Alexa. Is that multi-turn, voice-enabled dialogue? What are the sort of categories of things out there that people are doing, um, in terms of interacting via speech?

• 7:39Speaker 01

  Yeah, I mean, there are many ways you can kind of layer, layer out the taxonomy here. I think, you know, when you think about Alexa, Google Assistant, Siri, uh, and you ask if they're multi-turn, if you can really have a dialogue, a conversation, as opposed to just have many questions answered. The truth is primarily they're single-turn question answering or kind of like simple task execution systems. But then again, you know, they're working really hard on making them multi-turn. Now, one reason why, uh, it's really hard to build a general multi-turn voice assistant for, you know, consumers of all shapes and sizes is that they have very different requirements. They're trying to do different things. So like, it's actually a task of enormous complexity.

• 8:21Speaker 01

  When it comes to the things that we do, they're a bit less complex in scope because we build things to help you change your ticket for an upcoming flight. Or maybe you're making a reservation for a restaurant or you're trying to debug your router, which stopped working and you're having like connectivity issues. Or, you know, you're calling your bank and updating your address. So these are all things we do. And the one thing that's important about that task-oriented bit of the nomenclature is that it lets you evaluate. And when you can evaluate, that means you're doing good science and you can improve. Now evaluating something that does as many things as Siri or Alexa, it's hard, right? Because building them is hard, evaluating them is hard.

• 9:05Speaker 01

  Knowing what you should be expecting and where product market fit for them is, it's one hell of a task.

• 9:11Daniel Whitenack?

  Yeah. So when you're saying a turn, um, just to kind of get into some of this jargon, a turn would be like you say something to your smart speaker and get a response. Is that sort of what you mean? And then in your multi-turn, like if you're trying to debug your router or you're changing your flight ticket or something, that is likely going to take more interactions than a like uttering something and getting a response. Is that correct?

• 9:37Speaker 01

  For sure, for sure. So yeah, I mean, the whole reason we talk about turns is that for the most part, the dialogue systems today, from kind of like the voice assistants of the large tech giants to automated customer service to chatbots are built on what's known the turn-taking paradigm, right? So the assumption, and it's a strong one, and it's not something that necessarily holds in human speech is that you're going to wait for me to finish before you start speaking. And the assumption is that you've also absorbed all the information that I tried to relay over to you before you started speaking. Then we're taking turns speaking and yeah, a multi-turn conversation is kind of like anything that takes a lot more than one turn to achieve a task, right?

• 10:17Chris Benson?

  I also have something. I actually want to take you back for a minute because you said something interesting. I'm curious about your perspective compared to someone like myself who doesn't have your expertise doing this. When you were talking about, you know, Alexa and Google Assistant being hard, and you use the word hard associated with that several times. I really couldn't help wonder, as someone who has been doing this as long as you have, you know, starting with those Steve Young days and moving forward to the present. When you say it's hard, I'm kind of curious how, what you're thinking, what's, you know, you're, you're compressing it all into a single word, but how are you thinking about that? As you were saying that, I kept wondering that.

• 10:53Speaker 01

  Okay, okay, no, that's a really good question. I say it's hard because it's enticing, it's fun, it's a big problem, I expect to spend, you know, the rest of my life solving it and to be maybe, maybe a small cog in the wheel of how that ends up being solved. It's a hard problem on a pure kind of like academic level because it's that compound movement of different NLP tasks that all need to work really well and they need to communicate with each other, which is something that we've not really yet cracked, right? So say the thing at the center of a dialogue system, language understanding, is not a task that's fully well-defined just yet, right? So say, think of speech recognition. In most languages, you say something, there's exactly one way, one way of writing it down. But natural language understanding, well, what does it mean? How do you choose to interpret it?

• 11:35Speaker 01

  What are the kind of like things you're choosing to take away from even like an order or something as simple as that, right? So there, a lot is left to the interpretation and that means that it stops really being, you know, a science or even a field of engineering where you have a clear metric to beat. Because really, what we've shown over the past 10 years, especially with machine learning, is that you give something a clear evaluation and, you know, the sheer force and the intellectual power of the people working on it will crush it, right? If you say, think about question answering and the Squad dataset, right? I remember the leaderboard on the Stanford website, I believe. And, you know, first I think like the scores were like pretty low.

• 12:19Speaker 01

  And then all of a sudden, like three, six months in, we got to the point where like performance was unbelievable. I, I couldn't believe that it's that good. And it's because when you define a clear scope for a problem, we'll build a machinery to solve it, right? Now, when it comes to building these voice assistants, we don't know what machinery to build. The truth is, we've built a lot, right? Alexa is manned. I like to compare these assistants to aircraft carriers, right? And you think about Alexa, 14,000 people, right? Building that thing. And if you think about the ROI, they're not building it because they're making a ton of money on it. They're investing in the future because, oh, I mean, Amazon always has a math around it, right? And, you know, I hope they're right.

• 13:01Speaker 01

  In any case, it's really good for us because they've actually indirectly funded a big growth in the area and they've allowed us to, you know, in turn, build a lot of stuff. Ourselves, but we were doing it before they got interested, right? And the problem itself is, it's hard, yeah. I mean, it's a good word to use because you have to solve a lot of these different problems. You have to solve things that are not just in the domain of machine learning, but also like human computer interaction, like the voice user experience is something that academics tend to overlook. They don't appreciate that sometimes just the tone or something like that is much more likely to prolong your conversation and imbue the caller with enough patience and goodwill to go through a conversation, right?

• 13:45Speaker 01

  Equally, people who are very good at user experience don't tend to be mavericks at machine learning and then kind of bringing it all together to build something. It takes a lot of different personas, people, it's, even that is hard. So, yeah, I guess that's what I mean.

• 13:59Daniel Whitenack?

  You're talking about the human computer interaction and maybe you could speak a little bit to like, I think, the way in which people interact with, whether that be like a text chat bot or a voice assistant or something, is different than how they might interact with another human. What are some of those differences in terms of ways that people interact with those systems versus like their friend and meeting them at the coffee shop?

• 14:25Speaker 01

  Oh, for sure. I mean, I think that we could spend a lot of time just talking about the differences between voice and chat, right? And how people interact there. I mean, people will dispense with pleasantries and they'll tend to use shorter sentences when they operate with technology. They'll swear more, a big chunk of input going into all the large tech companies assistants are.

• 14:48Daniel Whitenack?

  I'm sure you have a huge database now of incredibly explicit and abusive language that people have said to their bot.

• 14:59Speaker 01

  We've maybe got a few colorful examples. I can tell you for a fact that we have a lot less, you know, percentage wise than the large tech companies where the number of turns coming into these assistance with profanities can reach double digit percentages. Part of it is just human nature, like what do you do when you can like use a new technology that understands you? Well, you swear at it because why not, right? Let's see what it does. And, you know, I mean, hopefully you've tried it at some point or hopefully, I don't know why hopefully, but, um, you know, things like Siri will have really good kind of like, you know, backup mechanisms to be sassy or to tell you off when you, when you've cursed at them or even when they think you might have, which can happen because speech recognition is not perfect.

• 15:42Speaker 01

  But yeah, I mean, the other thing that's always interesting with technology is, you know, what do people build first when a new kind of framework comes to mind? And I used this in one of our first kind of like investor pitches, but at the time, the top four applications with Alexa were things that allowed the system, I think, to read recipes out loud, which is kind of cool. And then the remaining three were meowing, farting and barking, right? Yeah. And finally enough, uh, the top, uh, revenue grossing app, when the iPhone was released was something playing one of those kinds of sounds. You can probably guess which one. And, uh, I guess it's a pattern of how technology evolves, but people tend to do these kind

• 16:27Speaker 01

  of like simple things that are just hacks where they have a bit of fun and then they go and they build like life change things.

• 16:47Speaker 02

  This episode is brought to you by our friends at RudderStack and we're calling all data engineers to check out RudderStack Cloud and start building smart customer data pipelines. RudderStack is warehouse first, no more silos. RudderStack, build your customer data lake on your data warehouse, not theirs, enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API first. Rudderstack can be easily integrated into your existing development processes. And because they're open source, you can see all their code. So you don't have to worry about vendor lock-in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. Rudderstack is free up to 500,000 events and pricing scales transparently from there.

• 17:31Speaker 02

  Learn more and get started at rudderstack.com. Again, rudderstack.com. That's R U D D E R S T A C K dot com.

• 17:49Daniel Whitenack?

  So Nicole, you're talking a little bit about different applications of voice technology, but also the way people interact differently with even chat versus voice. From your perspective, as someone who's really, you know, hands-on working with customers in this space, what makes for a good voice use case? From my perspective, people maybe don't have a great grasp on yet in terms of like, yeah, we all think voice technology is maybe going to be like a huge thing. And we can see really cool applications of it and maybe even really useful applications, but it might be hard for people to visualize, you know, what is a good voice application? And what are the benefits of that as compared to, you know,

• 18:33Daniel Whitenack?

  creating a text-based search or creating like other things? Like, when should I be thinking maybe voice?

• 18:39Chris Benson?

  As you answer that, can you kind of differentiate between voice and chat? Just for people who aren't intimately familiar with the use cases?

• 18:46Speaker 01

  Yeah, for sure, for sure. So, you know, I think that there is a bigger question of like, where you want to use voice as an interface to technology? And then there's just the more narrow question of where you want to use voice or text when dealing with customer service, right? And the only other interface you really have, other than language, are good graphical user interfaces, right? So let's say, you know, like smartphone apps and, you know, kind of like the web, right? I mean, obviously. The language-based ones are better if you're on the move, or if you're just simply you're not in front of a computer, um, or if you want to do something really quickly. Now, how the whole like AR/VR space will evolve, it's hard to predict, but we know it's coming. And there, the role of voice in particular

• 19:31Speaker 01

  is gonna be, you know, kind of like much, much larger than what you see when you interact on the web or with mobile. Mobile, in fact, is the worst one because, you know, you're kind of holding your phone. It's a bit awkward. You're typically surrounded by people. It's awkward to speak into your phone. And really, you know, the place where voice on a phone has really been successful is kind of like hands-free while driving. And there, Siri gets a tremendous amount of usage. When you think about the web, I think that's where chat is a natural interface for customer service compared to speaking often. Because, you know, you might be at work. You might be, you know, speaking to your bank or dealing with something. And you don't want your colleagues to know that you're actually doing that at work. So chat is pretty useful. Or maybe it's early in the morning

• 20:16Speaker 01

  and you don't want to wake up other members of your household. But in reality, you know, sixty to seventy percent of all customer service interactions happen over the phone, right? And they happen with voice because, you know, in this day and age where you could easily transcribe all this and have all sorts of channels, we're doing this podcast, right? Well, while recording it, I see you guys and you see me, but the end product is just voice because people can consume it anywhere. It gives them a pretty good feel for kind of like what kind of people we are, how we talk, our style. A lot of emotion goes through that voice. And it's also a really high bandwidth channel, right? I can probably type a bit faster than I speak, although it depends, I tend to speak quite fast. But really, I have a lot more fun when I speak

• 21:01Speaker 01

  and, you know, it lets me kind of express myself a lot more fully. And, you know, if you think about just the need to capture that channel when it comes to customer service. When COVID hit, you know, everyone thought big, you know, crises like these tend to accelerate technology adoption, right? And for, you know, close to a decade now, companies have invested in digital transformation in order to push people to digital channels. Mostly chat, either web chat where there are humans on the other end, or kind of like chat with an automated system. And the hope there was it's cheaper. Some people, especially those who are heavily invested in these projects, would tell you that You know, it's the channel of the future.

• 21:45Speaker 01

  Younger people prefer doing it. Well, look, I mean, I'm a millennial. I've got a PhD in computer science. I grew up playing computer games and, you know, like not seeing the sun. And guess what? When I need customer service, I like to call, right? And like, I have a bit of anxiety calling in, right? Like, like most millennials do. I still prefer to call because it gets the job done. The alternative is you're typing and then someone responds in four minutes because they're actually speaking to 10 people at the same time. And it's not really a better experience. And with COVID, you know, people thought, well, hey, now's the time for chatbots to take over. They'll go from their 10, 15% of the market share, heavily augmented by the fact that you're being forced onto that channel. And the hope was now it's going to go to like a much higher percentage.

• 22:28Speaker 01

  Truth is, COVID hit, call center volumes went up. Now, the staff went down because of social distancing or mandatory lockdowns. But really, people kept calling and it's dispelled that myth. Now, in our case, it's been really great for PolyA because we build voice-based systems for customer service. And it's been a big boon, especially in getting into those industries that previously might have hesitated to build this kind of futuristic technology. But it's not going away, right? And as time passes, you've got a smart speaker in every part of your house. At some point, you'll have some kind of wearable. They'll capture your voice really, really well. It's going to be really convenient to just say like, oh, hey, turn on my thermometer and order pizza. I mean, we can talk about these scenarios.

• 23:12Chris Benson?

  I'm curious, though, as we're about to dive into that next, do me a favor and set some context for me. For those of us, like both of you guys are experts in natural language processing. I'm one of those interested people, but I'm not an expert like you. And we've talked about having this multi-turn dialogue and these interactions. As you're going and solving this for people out there and providing these capabilities that we're all getting excited about, could you talk a little bit about what it is that you have to be thinking about in that pipeline, you know, as you're doing multi-turn? What are the things that are part of that consideration for those of us who are not as intimately familiar with that?

• 23:51Speaker 01

  Mm hmm. Okay. So if you think about kind of like that that cycle of building a dialogue system, especially if it's voice based, the first step in it is speech recognition, right? Kind of like transcribing what you think the user said. If you're doing it in the best way possible to maximize performance, the output of that is not a single sentence, but instead something that is a lot more complicated. A bit more complicated is an N best list, so kind of like maybe ten different hypotheses of what you might have said. So let's say I want to get Serbian food, right? And I said it fast, so is it Serbian, Siberian, Syrian? You're not sure, right? Or like, I want to go for three people, right? Like, did I say three or did I say free? Well, of course, what?

• 24:33Speaker 01

  If we go into just like that technology, the language model there would basically say, hey, it's more likely that they said three people. But then again, free people is also something that you tend to see quite often in text, so it's not impossible. So a good system will tell you, I think it's three people, but it might be free, right? And equally Serbian, Syrian, a few other hypotheses, right? So then the next thing that comes is natural language understanding. Taking what the user said and parsing it and saying like, in some ontology that I have previously defined, that I need to interact maybe with the external world. So let's take booking as an example. If I say, hey, I want to come in with me and my fiance, right? That actually means two people, right? And it's not NLP as in like parsing who the entities are,

• 25:17Speaker 01

  because while it could be useful in a composite task of counting up how many people there are in the request, really what you need to know is that I've initiated a booking request and how many people I've asked for, and it's two people. And that's actually a really, really hard thing to do, because like actually parsing those words and saying, uh huh, two people, that's complicated, right? Then the next thing is like, let's turn it into, in in a very traditional dialogue system literature, this would be called a dialogue act, where I've kind of like informed that number of people is two. And you know, that would then go off into a dialogue manager that would say like, okay, well, do I have something for two people? And then what you'd know is, well, like, sorry, when? And then the system would have to go and say like, well, like, you know, request the time for the booking, right?

• 26:02Speaker 01

  Now, request time, if you respond like that, you sound a bit like. Not even like Jarvis, you'd sound like the Terminator, or, well, I don't know, you sound like a really bad voice assistant, right?

• 26:10Daniel Whitenack?

  Requested time.

• 26:12Speaker 01

  Yeah, for sure. So you know, you need to turn that into like, hey, what time would you like to come in, right? And then you have to sit, so that's kind of like natural language generation, another big subfield of NLP. And then finally, if you want to produce it in audio, you'd have to use a text to speech engine, right? And that would convert it into audio, you would play it back. And then the big thing, kind of like in understanding everything and having a good conversation would be kind of like the bigger task of dialogue management, looking at the whole previous set of things that were said and using it to augment the prediction in every subsequent turn. So let's say I said, you know, you might have thought that I said three people, but if I repeat three people in the next turn, then that kind of like alternative hypothesis is probably true and the system should figure it out like,

• 26:56Speaker 01

  hey, why is he repeating it? Doesn't sound right. Let's try that other one. Did you say three people? Well, so you can choose to confirm if you're uncertain. And there, there's a lot of machinery around how you handle that probability distribution, uncertainty, a lot of Bayesian methods that come into play. It's a pretty serious discipline.

• 27:13Daniel Whitenack?

  Yeah, so now if you take that like series of steps that you've laid out, obviously, PolyAI is working in all these areas, but I was wondering if you could maybe talk about like, where do you feel like you're having to spend most of your time? There's probably open challenges in each of those areas, but maybe where's the the biggest open challenges in terms of advancing this field along that pipeline of things?

• 27:39Speaker 01

  Yeah, when it comes to kind of like, you know, where we focus, one place where we don't spend a lot of effort is speech recognition, the speech recognition task itself. Cause that's one which is pretty well defined, commoditized. A lot of people are playing there, a lot of progress has been made, the big tech companies are pouring in millions. And that's great for us because we're just getting a better product than we then get to build on, right? So we typically use often several speech recognizers in a single deployment to get that variance out so we can extract. The best possible prediction out of all of them. So the more uncorrelated Google and Amazon are, the better our performance gets. And we love that and we thank them for all their hard work.

• 28:20Speaker 01

  Now, when it comes to the piece where we really excel, and this is where we're really, really differentiated from your 1500 chatbot providers that a lot of them claim to do voice, but their idea of doing voices, I'll put speech, a speech recognizer and a text speech engine there, it's gonna be great. This is why they don't have many appealing voice applications out there. The piece that's really then exciting is what we like to call spoken language understanding, as opposed to natural language understanding, so SOU versus NLU. And the difference there is you really have to consider the fact that there's a bunch of different speech recognition hypotheses that you can operate over to really figure out what's going on. The second bit is, you also have to look at what happened previously in that conversation

• 29:04Speaker 01

  to know, again, how to kind of like tilt the outcomes to improve the accuracy. And then finally, one thing that we do really well, and that's really important, is as the conversation progresses, you can anticipate where the conversation is going to go. If I've asked you for how many people are coming in, or if I asked you about, you know, like, for example, what our systems can do is parse my name right. And Nikola Miršić will never be a common name in English-speaking environments. It's a, it's a hard name by Serbian standards. But if you know that, like, say I told you my phone number and you're authenticating me, then if you inform the speech recognizer that Miršić is coming up, well, then they're actually quite likely to parse it correctly,

• 29:47Speaker 01

  even though it's an impossible collection of syllables in English. It's very unlikely even in Serbian, so. But if you know that it's coming, they can kind of wait out for a bit, like, "Ah, doesn't it?" Right? So that's really important. And that spoken language understanding, that is what lets us do voice really, really well.

• 30:03Chris Benson?

  Could you also, just for those who are coming along with us, as you talk about spoken language understanding, which may be a new term for some people, could you also just real quickly define, you've kind of talked about some of the qualities of that? Is there more of a formal definition, or is this more of an informal way of addressing it? I'm just kind of curious, as you bring people into the terminology.

• 30:24Speaker 01

  Yeah, it's a formal research problem and it kind of like touches on different ones, but if we attempt a formal definition here in a specific dialogue task, where we're, say, trying to accomplish something. It is this problem of taking an audio stream and turning it into actionable, parseable, kind of like slot value pairs typically, or something like that. So kind of like slots are things like maybe you say date, or a location, or number of people, right? So kind of like extracting that structured information that in the backend, your logic, so not AI, like your pure kind of like business logic knows what to do. Either it sends a booking request, it sends a query for a specific kind of information,

• 31:07Speaker 01

  or something like that. So rather than NLU, which is, you know, again relatively complex to define. Um, when it comes to dialogue, it's this again idea of extracting the same kind of information from a written sentence. Now, the thing about a written sentence is that uh, there isn't any noise injected by the speech recognizer. Whereas in SLU, there's an audio file which, you know, it's not only about a speech recognizer that may struggle to recognize particularly complicated word that may be, you know, from a pharmaceutical or, I don't know, a travel domain or something that doesn't come up frequently, or it's a problematic last name, but really maybe it's just background noise. Maybe it's the fact that the accent of the person is not something you're expecting. Your model is not very good at it.

• 31:52Speaker 01

  Or it could be that, you know, increasingly they're speaking from two rooms away and you're, you know, seven microphones in, and Alexa device are insufficient to capture what they're saying. But, you know, we're people, our expectations are growing and we we expect that these things will, will work for us. And that's what makes the problem fun as well because it's kind of like, um, you know, shifting goalposts, right? Just when we got it to work, when you normally speak on the phone, speakerphone. Once it works on the speakerphone, there's a baby crying in the background. And then you're driving and there's a baby crying in the background and someone's talking over you, right? So, and then you, you might want to switch language, right? So it's fun. Like, it's a hard problem, as I said.

• 32:40Speaker 03

  We deserve a better internet and the Brave team has the recipe for bringing it to us. Start with Google Chrome, keep the extensions, the dev tools and the rendering engine that make Chrome great. Rip out the Google bits, we don't need them. Mix in ad and tracker blocking by default, quick access to the Tor network for true private browsing, and an opt-in reward system so you can get paid to view privacy respecting ads. Then turn around and use those rewards to support your favorite web creators like us. Download Brave today using the link in the show notes and give Tipping a try on changelog.com.

• 33:22Daniel Whitenack?

  As you were talking through some of the things about spoken language understanding, one of the things that you mentioned was things related to like specialized jargon maybe, or particular accents. I guess my question is like, let's say you're onboarding a new client. They're in a specialized domain and they're trying to create this, you know, new voice assistant. At this point, how difficult is it to kind of onboard a person into that? How much, you know, data do they have to provide? And like, how, how much are you able to sort of transfer things from your other use cases and common data that you have? Maybe both in terms of like restrictions between clients?

• 34:06Daniel Whitenack?

  Cause I'm sure you can't always share data that you've gathered from certain clients and use it to create things for other clients. How does that process work at this point? And how much pre-trained models can you use and that sort of thing?

• 34:18Speaker 01

  For sure, for sure. Well, I mean, I'm sure most of the listeners of this podcast know about the importance of pre-training for deep learning applications, right? I mean, it's typically like you figure out how to pre-train well and then like really good things in that sub-domain machine learning follow, right? So when it comes to natural language understanding, I can tell you, and I could, you know, talk about it for hours, about, you know, collecting data sets of a thousand, 2,000, you know, training examples. And we did this back at Cambridge. My co-founder Sean had this really good data set and- Revolutionary paper, he had one of the first papers on kind of like training an end to end dialogue system and that involved that kind of whole compound movement. It's really well cited paper, really good piece of work.

• 34:58Speaker 01

  But you know, for that paper, he collected a data set of six hundred training examples. And then for another paper of mine, I needed a bit more data. So he collected a bit more and then I'd go through and annotate, he'd go through and check the annotations. I do it again. It takes about a collective, kind of like one week of work and leaves you with permanent, you know, kind of like mental health problems.

• 35:21Speaker 01

  I don't mean to, I mean to cry jokes about mental health, but it is, it is a daunting task. It's no fun at all. So when we started PoliA, we were like, this has to stop, right? We're never gonna build amazing things if we're dependent on doing that. Cause bear in mind, like, we're pretty highly qualified for creating this kind of data. So if it takes two people in the kind of like last stage of their PhD, after years of doing this stuff to create that data set, like, that's not scalable. So what we then started doing was, uh, pre-training representation models for dialogues that would kind of like look at billions of conversations, things like Reddit, Quora, Twitter,

• 36:01Speaker 01

  and learn good representations for a dialogue so that if I give you a set of kind of like, you know, turns, like you spoke, I spoke, you spoke, I spoke. And then I say like, hey, model, use the representation of the dialogue so far and the representation of a potential follow-up to determine whether it's a good follow-up to that conversation. So we would pre-train in that way. And if you do it like that, then you get a lot of training data out of things like Reddit, Quora, Twitter. And Reddit in particular is an incredible resource because people talk about all sorts of things on Reddit, right? And there are people from all over the world and in different languages as well. But also just like, let's think about English in all possible different dialects, you'll see

• 36:46Speaker 01

  anything phrased and rephrased there in a good way. And, you know, you can train this thing for a long time. And this encoder convert that we had, we had built. It's something that's, you know, in the family of models like BERT or GPT, where really, you know, it's pre-trained on a lot of data. But unlike those models, it's not really a language model. It's an encoding model for dialogues, right? And it's purpose-built and purpose-pretrained for conversational AI. So then, when you use this model to do things like intent detection, value extraction, all of these tasks that form that big compound movement of dialogue, it's a model that's really powerful, right?

• 37:28Speaker 01

  It takes much less data to get to a high level of performance than something being trained from scratch. And, uh, you know, there's been a lot of benchmarking of this model. Salesforce recently came out with a study that confirmed that this is the most accurate, or rather, the best thing to pre-train with to get the most accurate models with a limited amount of data, or any amount of data, really. So this is really important, right? And then what we're able to do and what we bring into all of our deployments is this model, right? Then, in all these deployments, wherever there's not sensitive information, it's just more conversations that are used to subsequently tune that model. But we don't need to use specific, you know, nitty-gritty details of,

• 38:12Speaker 01

  you know, uh, I don't know, how you collect British postcodes or how you spell Serbian last names. Like, that stuff is a bit more proprietary. We have a lot of different technology that's used to counter those specific sub-problems. And the truth is, we often have to solve some of these challenges that are a bit separate, often very engineering-heavy. But when it comes to like that data barrier that people think about, like that first step of building a dialogue system, which you need like a lot of data to train, like, we need a lot less data because we've already spent years pre-training this thing.

• 38:44Daniel Whitenack?

  Yeah, this brings me to probably my favorite subject. Before the interview, I was reading your post about the convert model. And you mentioned that it's pre-trained speech encoder in multiple languages. So, and I know that multiple languages is something that's emphasized on your website. And what is your thought process behind that? And maybe, let's say that we're specifically talking about this, uh, language model. How have you gone about setting up that language model such that it enables you to solve problems in multiple languages?

• 39:16Speaker 01

  This is a personal academic passion of mine. So quite a few years ago, now one of my best friends, Ivan Bulic, who is a great multilingual NLP researcher. He's Croatian, I'm Serbian. We met in Beijing over a few beers and we were like, well, how did we end up working together later? He worked on like multilingual NLP, I worked on dialogues. We were like, hey, can we like do something multilingual in this context? And then we got then really interested in this was the time where kind of like word level, word vectors were all the rage, things like the word to vec model, glove and all those things. You know, Mikolov, Pennington, all of those guys. And like this was the first wave of like massively mindlessly data-driven NLP, right? And like, I stand for that, I love that, right?

• 40:01Speaker 01

  But I also love languages and its nuances, right? So the question is like, well, okay, you train something in English, how do you port it to another language? Typically, older school NLP had like this pipeline of things running, you know, lemmatizers, stematizers, you know, parsing the sentence structure. And like that's different in different languages, like the, you know, subject, verb, object, you know, word works differently in different languages. The morphology murders you in different languages, right? Like if you go from word gender is a thing that exists in some, but not in others. In one language, word order matters, in others, you can do whatever the hell you want, right? And I mean, it's sort of fun, right? But when you think about like then creating a dialogue system that works across all these languages, it's daunting, right?

• 40:45Speaker 01

  I mean, you can't just go and translate because a lot of stuff is lost in translation and you know, like, just the multi-sense words in one language could translate into something catastrophically different.

• 40:55Daniel Whitenack?

  Rhetorical questions?

• 40:57Speaker 01

  Yeah, that's the tip of the iceberg, right? I mean, even mundane things like the word bill might mean something, you know, like an account or a bill could be the same word in one language, but they're not in others, right? And that's just very confusing, like they trigger different actions that both exist. Now, what we started doing then was training word vector spaces, which would embed complete vocabularies of different languages into the same high-dimensional mathematical representation, right? So words like, I don't know, shion, hund, uh, dog, suka, hound, whatever, hund, they were all like in this bubble. In one place, right? Now, of course, there are problems with this because they're multi-sense words.

• 41:40Speaker 01

  But you know, the multi-sense ones tend to float away in a bit of a different direction. And then you have machine learning models trained to operate over those mathematical objects instead of operating over a unitary representation of the word dog in Serbian or in English, right? And if you do that, then the beauty of task-oriented dialogue, when it's a specific task, is that you don't need to understand the nuances or the rhetorical questions. You need to understand that someone asked for a table near the disabled toilet, right? And at that point, the fact that you're just parsing a limited number of intents means that you're actually able to do it across different languages at once. And that's like a big thing that we really, really care about.

• 42:23Speaker 01

  It's again, a place where pre-training comes to our rescue and where we're then able to do these things very, very well. The other thing, again, a big thank you to all the cloud providers for the millions they've poured into speech recognition research across different languages, because that's not a thing that we have the budget to do. And they do it pretty well. And you know, that's a piece we don't touch, but it's provided by those companies and everything else we do at House.

• 42:52Daniel Whitenack?

  Yeah, and so then, like, if you have this multi-dimensional space which embeds vocabulary from multiple languages, is the hope then that like, when you add, so let's say that you support X number of languages, but then your next client wants to have a dialogue in a next language, maybe that language is related to one of those you already support, but it's different. Is the hope then to sort of retrain that and add it in or transfer learn from that existing model, which is faster than training from scratch in a whole new language? Or how do you approach that situation?

• 43:28Speaker 01

  It's a really good question. You could do either. The transfer one seems cheaper and easier, but really in our architecture, the best thing to have is a unified approach that works across all the languages we want to support at once. So they're really retraining everything makes the most sense. Now bear in mind, we have a single model that is kind of like the nuclear reactor of our system. So that model needs to be retrained. And once it's retrained, we're done. Like that language is in there forever, right? So on that front, yeah, it's a lot more heavy lifting than like fine-tuning a hundred small models, but it provides this unified thing that will In the long run, save us. This is similar to kind of like what PageRank did to search,

• 44:10Speaker 01

  right? They kind of created an algorithm that just indexes an incredibly large matrix, right? And factorizes it. But once it does that, well, here you go. Search forever. And Internet changes, I refactorize the matrix, and then, you know, I'm done. Whereas, you know, previously, we had like, you know, in older search engines, you'd have to go to a specific kind of like industry, and then you'd search there. And there would be a small kind of keyword-based model that would flag the right results. And I had like special statements on what's more relevant. Whereas now you have a unified approach, and it works a lot better. And we're trying to do the same for language.

• 44:43Chris Benson?

  So, back at the beginning of our conversation, you said something that I've been holding it on to because I knew it wasn't yet the point where I could ask, you know? You said, as we were getting into the topic and you were introducing it, that you expected to spend the rest of your life working in this area. That made me, as a non-expert in your area, really wonder. I think a lot of people that aren't working in the field would assume that we're going to take a few years and kind of solve all this NLP and related stuff.

• 45:14Speaker 01

  Yann LeCun famously said that he's going to solve NLP in two years.

• 45:18Chris Benson?

  How many years ago was that?

• 45:20Speaker 01

  I think like six, seven at this point. He's made great progress.

• 45:23Chris Benson?

  Yeah, yeah, no doubt. No, I heard a similar thing. I think it was Eric Schmidt in a tweet. He was like, "Speech is a solved problem."

• 45:30Speaker 01

  I think Eric Schmidt showed a lot more wisdom than Yann LeCun did there, despite probably less intimate technology understanding.

• 45:39Chris Benson?

  Okay, but I gotta ask this now. I'm curious because you said that, and I know that it's not a static thing. It's done. I know that you're going to continue to make great progress. You've been doing that, and you've been telling us about it through this. So I'm fascinated about where is this going? And how does the larger vision for the problem evolve over time? Toward you as a young man, a millennial, says this Gen Xer who's significantly older. How is this evolving over your lifetime to where you are remaining impassioned about solving this problem in the long term? What does that look like? I'm really curious about that. I've been waiting the whole Whole way through to ask you that.

• 46:16Speaker 01

  Wow, that is a tough question. I think that, you know, there are just like a lot of like tactical things to solve, right? So like say customer service, it's a challenge that Polly as a company is focused on. And I think we'll be working on it for a long time because we're really far from having a voice assistant that you speak with without kind of like getting that, you know, the danger frustration at the start when you realize that, oh, God, it's automated, right? We want to make that like a non-problem, right? We want people to call in automated, fine, the same way that, you know, like you, you, you log into a website, you've never seen the format before, but you figure it out, right? You know, I think there's a lot of work to be put into those things becoming really, really good. It's not a small challenge, right?

• 46:55Speaker 01

  It's going to be an adoption curve that is also partially just about shifting consumer behavior, right? And um, voice assistants have done a lot to help there. They've shifted this into the realm of possible, likely. And, you know, the new generations, you know, kind of like people younger than any of us are growing up with these things. And that, I think, is really powerful because, you know, we might be the last generation which was really, really fluent in, you know, kind of like laptops, opening up a terminal, uh, you know, the heavy intricacies of the web. Like, why would you do that if it's a lot more accessible with a more natural interface, right? And, you know, I think that people who are heavily relying on the web might seem in 20 years' time to those, well, to younger people,

• 47:40Speaker 01

  like, you know, those guys that are still using the terminal or, you know, speaking about assembly, right? I know that's the best and the most fortunate analogy, but to kind of like get into the meat of your question, like, let's say that in 10, 20 years, 10 years, we've got voice assistants that are everywhere, right? They're endemic, they're just how you interact with businesses, you know, they've enabled us as humanity to move away from doing all of those mundane tasks. What's next? Well, I think like just a general interface with technology. The deeper answer to your question doesn't come without understanding what happens in AR and VR, what happens, you know, with things like Neuralink eventually where like that's an absolute necessity, right?

• 48:23Speaker 01

  Like, I really want that because that's really how we then kind of like transcend humanity. This technology then becomes a big interface of that, right? Of just like communicating and of understanding or absorbing all that information. And then, you know, you could go and fall for, you know, like singularity, consciousness in AI. How are you going to communicate with all that? I don't know where that's going to go. I'm not super bullish on that, but I know that this problem is going to get deeper and deeper. It's a bit like the Darkness Conference, right? Where, you know, people sat down for, I think, a summer and they're like, we're going to crack this, uh, you know, AI thing. It's like, yeah, right. You didn't even scratch the surface, right? So when we get to the point where we have technology, where voice is completely natural,

• 49:07Speaker 01

  I think it's really hard to imagine what the world will look like at that point. It's gonna be great. It's gonna be really, really interesting, right? We're not that close. There's a lot of work. I'll be past your age by the time that happens.

• 49:18Daniel Whitenack?

  You got a ways to go then.

• 49:21Speaker 01

  We'll see.

• 49:22Daniel Whitenack?

  I'm definitely glad to hear that perspective and also just get to hear from your great work with PolyAI. Yeah. It's been a pleasure and I know I'll probably annoy you with all sorts of speech-related questions. Um, uh, as, as time goes on, I'll look forward to seeing what, uh, what PolyAI does. But, uh, yeah, thank you so much for joining us. It's been a pleasure.

• 49:42Speaker 01

  And thank you for having me. Have a lot of fun.

• 49:48Speaker 03

  Thank you for listening to Practical AI. We appreciate your time and your attention. Follow the show on Apple Podcasts, Spotify, or your favorite podcast app. Your neural networks will thank you. We are also on the web at practicalai.fm. There you'll find recommended episodes, listener favorites, and a free sign up to join the community. Practical AI is hosted by Chris Benson and Daniel Whitenack. It's produced by Jared Santo with music by Breakmaster Cylinder. Thanks again to our sponsors, Fastly, Linode, and Launch Darkly. That's our show. We hope you enjoyed it and we'll talk to you again next week.