Real-time conversational insights from phone call data

Daniel and Chris hang out with Mike McCourt from Invoca https://www.invoca.com to learn about the natural language processing model architectures underlying Signal AI https://www.invoca.com/product/signal-ai. Mike shares how they process conversational data, the challenges they have to overcome, and the types of insights that can be harvested. Sponsors: Linode https://linode.com/changelog – Our cloud of choice and the home of Changelog.com. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2019 OR changelog2020. To learn more and get started head to linode.com/changelog https://linode.com/changelog. Brain Science https://changelog.com/brainscience – For the curious! Brain Science is our new podcast exploring the inner-workings of the human brain to understand behavior change, habit formation, mental health, and being human. It’s Brain Science applied — not just how does the brain work, but how do we apply what we know about the brain to transform our lives. Featuring: Mike McCourt – GitHub https://github.com/mkmcc, LinkedIn https://www.linkedin.com/in/michael-mccourt-0a35a4156Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Mike McCourt | Invoca Employee Spotlight https://blog.invoca.com/employee-spotlight-mike-mccourt-astrophysicist-turned-data-scientistInvoca https://www.invoca.comSignal AI https://www.invoca.com/product/signal-aiZipf’s law https://en.wikipedia.org/wiki/Zipf%27s_lawMeet the Data Scientists Behind Invoca’s Conversational Analytics Algorithms https://blog.invoca.com/meet-the-data-scientists-behind-invocas-conversational-analytics-algorithms -Invoca raises $56 million to apply AI and analytics to voice calls https://venturebeat.com/2019/10/17/invoca-raises-56-million-to-apply-ai-and-analytics-to-voice-calls -Invoca’s Signal Discovery uses machine learning to conduct conversational analysis on inbound calls https://www.clickz.com/invocas-signal-discovery-uses-machine-learning-to-conduct-conversational-analysis-on-inbound-calls/256016 -Invoca - What Will You Discover? https://www.invoca.com/product/signal-discoveryUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com/changelog.

• 0:17Speaker 01

  Linode is our cloud server of choice. Grab the Nano plan for just $5 a month, just five bucks. That gets you a gig of RAM, a blazing fast 25 gig SSD, and one terabyte of transfer. Let's be honest, you can go a long ways on that five bucks. When you do need to scale up, their prices are predictable, so you can put your calculator down, you won't need it. We've been running change.com on Linode for years, and we've always impressed by their award-winning support team. Check them out at linode.com/changelog. Once again, that's linode.com/changelog.

• 0:58Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slack with us around various topics of the show at change.com/community. And follow us on Twitter. We are at Practical AI FM. Okay, here's Daniel and Chris.

• 1:24Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with S I L International. And I'm joined as always by my co-host, Chris Benson, who is a Principal AI Strategist at Lockheed Martin. How's it going, Chris?

• 1:40Chris Benson?

  Doing great, Daniel. How's it going today?

• 1:42Daniel Whitenack?

  It's it's going great. It was a beautiful weekend after finishing a submission, a paper submission to a conference. So, you know, all, all was good. I got to relax and and be outside a little bit and feel the satisfaction of having that done. So what about you?

• 2:00Chris Benson?

  Sounds good. Just family stuff over the weekend. I had the satisfaction of both my daughter, who's in second grade, is into basketball, and we raised the goal all the way up to the 10 foot mark. And so she's, she's learning that.

• 2:12Daniel Whitenack?

  The big time.

• 2:13Chris Benson?

  And we transitioned her up to a larger bicycle as well. So both big daddy moments for me. So not, not AI related, but it was a good weekend in that way.

• 2:20Daniel Whitenack?

  That is exciting though. Yeah, yeah. Cool. Well, I'm excited about the show today. We have a topic that is very interesting related to conversational data and extracting insights. We're joined by Mike McCourt, who is a data scientist at Envoca to talk a little bit about that subject. So welcome, Mike.

• 2:40Speaker 04

  Hi, thank you for having me.

• 2:42Daniel Whitenack?

  Before we jump into that, would you just give us a little bit of a background to help us understand how you got into AI and data science stuff? And then we'll take it from there.

• 2:52Speaker 04

  Absolutely. Yeah. So my arrival into the AI world was totally fortuitous. And I'd not planned on it at all, really. My original goal was to be a college professor and I really wanted to study astrophysics and to do research in that field and to teach that being how structure formed in the universe and how we came to be here. And so, you know, I worked for probably 10 years dutifully towards that end. I majored in physics and theoretical physics in college and got a PhD in astrophysics. And then I moved all the way across the country for a postdoc position, just like a research position in science.

• 3:36Speaker 04

  And then moved all the way across the country again for another one. And, you know, as I was dutifully working towards this goal of being a professor, I was getting less and less sure that that was the life I wanted. And at the same time, I'd seen this new field of AI just like take off and blossom.

• 3:56Daniel Whitenack?

  And what year was this around?

• 3:58Speaker 04

  Well, so I was in college, 2004 to 2008.

• 4:01Daniel Whitenack?

  Okay.

• 4:02Speaker 04

  And basically never heard of AI in that timeframe. I mean, I knew of some people in the computer science department who were doing research in AI. And then like midway through grad school, 2010, 2011, I started to hear more and more about it. It seemed interesting. And like by the end of grad school, you know, to an outsider, AI was like hot. And by the end of my first postdoc, AI was really hot. You know, I saw how fast the field was moving and new inventions were coming. And it seemed like it was advancing by leaps and bounds. And eventually I got jealous and I wanted to make the switch. So.

• 4:38Daniel Whitenack?

  Yeah, it's interesting. It seems like you're even the years. So I was also in college 2004 to 2008, and then went to grad school in physics. So we've got a lot of similarities there, except I guess you stuck, you had more perseverance than I did. I didn't make it through the postdoc. I jumped ship before then into industry.

• 4:58Speaker 04

  Yeah.

• 4:58Daniel Whitenack?

  But it is interesting. I guess there's really a lot of astrophysics and also high-energy physics people getting into this space. At least that's my understanding now. I'm kind of more removed from it than you maybe. As the non-physicist in the group, I have to say I've noticed that it seems like the AI is breaking out with physicists these days. It's a good thing, by the way.

• 5:20Speaker 04

  Yeah, totally.

• 5:21Daniel Whitenack?

  Yeah.

• 5:22Speaker 04

  Yeah. So you jumped earlier than I did. I think you could see part of it than I could. But yeah, I feel like almost all of my colleagues now from grad school are in data science or AI.

• 5:33Daniel Whitenack?

  Yeah, well, it's definitely very competitive in terms of like tenure-track physics positions right now. But the demand for AI talent and data science talent is so huge that, you know, I think that's driving a bit of that.

• 5:49Chris Benson?

  Before we dive in, you know, and talk about what we were planning, I'm just kind of curious with both of you here. Any thoughts on is there a kind of a natural progression, do you think, these days for physicists to move into AI? Kind of a natural synergy between the two?

• 6:03Daniel Whitenack?

  I'll let Mike go there, maybe. He was in physics longer than me, maybe.

• 6:07Speaker 04

  I mean, the jump more recently, I think. Yeah, I think for me, I felt like physics prepared me extremely well for the jump into data science. You know, I think in physics, what you're trained to do is to take some observation about the world, some fact that you'd like to understand and explain, and then you come up with a model that you believe could explain it. And then you have to grind through the math and turn that model into something that can make predictions. And in the process, you have to collect some data, you have to inform your model, you have to fit four parameters in the model, and you have to understand your uncertainty in all those. And then at the end of the day, you have to make a prediction.

• 6:49Speaker 04

  And then then you have to convince someone else to go run an experiment and test your prediction. And if your predictions count out well, then then you believe that you've learned something about the universe and you can publish that. I think that process in AI is pretty similar.

• 7:04Daniel Whitenack?

  Yeah, it's a great parallel there that you that you outline.

• 7:07Speaker 04

  Yeah, so the the focus is different, you know, in physics, we don't really care about the predictions on their own. The predictions are there to test our theories and the the theory and the model is really the product that we want to give to the world. And working in AI, you know, the the model actually is typically something that we. Well, sometimes we hide it, or or at least it's not really the focus. The internal workings of the model isn't what we care the most about. It's the predictions that we care the most about. But I think that that process is broadly similar.

• 7:37Daniel Whitenack?

  Yeah, definitely, I I agree, and I think you expressed it much better than I could. So it seems like you you've also developed the important data science characteristic of being able to to explain things and and frame things well. So I don't know if that's something you learned in physics or afterwards, but um yeah, appreciate that. So tell us a little bit more about how you got involved with Invoca and in your path to sort of what you're doing right now.

• 8:05Speaker 04

  I got very lucky landing at Invoca. So like I said, I was kind of, I was very, I had a laser-like focus on my physics career trajectory. I'd never really considered anything else until I kind of got towards the end and realized AI seemed more interesting to me. So I was not really aware of what was going on in industry. And I made the jump a little bit blind. But you know, while I was making major life decisions, I decided I wanted to work in AI. And I decided that if possible, I wanted to live in Santa Barbara. I'd always loved the city here and I didn't know if there would be AI positions in Santa Barbara. So I just started searching and I found this company called Invoca.

• 8:50Speaker 04

  And, you know, I, as a scientist, Invoca does call analytics for marketing. I didn't know that much about phone calls and I knew almost nothing about marketing. Uh, but it looked like an awesome company, especially what was really interesting to me was the data set that Invoca has because we study phone calls here. That's an unusual data set. Most, most of the research at least in AI now is not focused on phone calls. So it's kind of a unique problem and a, and a big data set. And it looked like a fantastic company to work for. And so even though I came in knowing almost nothing about marketing, I was able to convince them that my skills as an astrophysicist would be useful to them. And landed a fantastic job at a really great company here.

• 9:36Chris Benson?

  So you kind of led right into where I was about to ask you, and that is, I had looked at Invoca's website and it talks about Invoca being this AI-powered call tracking and analytics platform. And so you've already kind of talked a little bit about this interesting data set. And, and so I guess for those of us who are not in that particular field, could you talk a little bit about what call analytics mean? And also I noticed on the webpage, it talks about campaign attribution and it refers to some actionable data. And if you could kind of talk about how all those fit together and what they mean in this context, I'd really appreciate it.

• 10:10Speaker 04

  Absolutely. So keep in mind, I'm, I'm not a marketer, so I'll do my best with explaining the value proposition of what we do. You know, I think now marketers, when a transaction happens online, marketers are used to getting all of this attribution information. So if you buy something online, they know who you are, they know what ads you've seen, uh, they know what locations you've been to, and they can use this information to really analyze their marketing campaigns and optimize them. And try to make sure that they're spending money on ads that are driving business to the company and hopefully that are relevant and interesting to consumers.

• 10:50Chris Benson?

  Yeah, so it's not just like broadcast marketing. The trend at least now is very targeted marketing, right?

• 10:57Speaker 04

  Yeah, it's changed a lot in recent decades, but as soon as you pick up the phone, they lose a lot of that information about you. If you don't finally make your purchase online, but you pick up the phone and call them and make your purchase there, then a lot of that attribution is lost. And that's important because a lot of high value transactions still happen over the phone. You know, if you're buying a new sweater, you'll do that online. But if you're getting an air conditioner installed in your house, you typically want to pick up the phone and talk to someone before you make that decision.

• 11:26Chris Benson?

  That's a great point in terms of how real life works, you know, when you're doing that.

• 11:30Speaker 04

  Yeah. And so, you know, if you're shopping mortgages or home improvement things, or if you're booking vacation on the cruise line, you know, big, high value transactions, they still tend to happen over the phone. And Invoca works to close that gap so that marketers can analyze the transactions that happen over the phone and link those to their marketing campaigns.

• 11:53Chris Benson?

  And I'm just curious, I did spend a number of years as a technology person in marketing organizations. And how do you actually make the link? Or do you, in this case, if they're starting online and, you know, traditionally, marketing has focused so much on analytics with online activity, and then they've totally moved over into this new medium, and you start your analytics there, which I guess we're about to learn about. And but how do you actually make that connection between the two?

• 12:19Speaker 04

  Yeah, that's an excellent question. And I have only a thin understanding of it. There's a whole, this is kind of the core of Invoca's technology, and I just work on the AI part. But essentially, it works when you sign up with Invoca, you get a number of phone numbers. And then when ads are shown, a number gets swapped into each individual ad. And so that when someone calls the number, that number gets forwarded to your business. But the number they called can be used to see what ad they called in from.

• 12:52Daniel Whitenack?

  Yeah, I think together we're getting there. I know in a previous employer, actually, the first data science position I had after leaving physics was with a telecom company. And I think most people don't realize this now, but it is super easy for you to, you know, spin up, you know, even 1,000, 100,000, you know, numbers, unique numbers, and then destroy them right away after you've used them these days via API and things like this. So you can be very smart with your phone campaigns in terms of using unique numbers for even just a single purpose, like doing this link, which is pretty interesting.

• 13:30Speaker 04

  Yeah, exactly. But then it starts to get complicated because you want to make sure it's a phone number that no one's used for a while. So there are a ton of people working on that here. I find it interesting.

• 13:40Daniel Whitenack?

  Yeah, it is interesting. And I think, you know, also, it's fairly transparent to the user, which is interesting. And in a similar way to, most people don't realize that they're being, you know, tracked by Facebook when they're not actually on Facebook or Instagram, right? There's people that have this like embedded thing in their website, or like most people should, that are doing any type of marketing, that links their activity on that website, assuming they're logged in to Facebook or Instagram, to their account. And I think a lot of that world is, is kind of transparent to most people. At least it was to me before I kind of got into some of these related things.

• 14:21Chris Benson?

  That point you made a moment ago about having so many numbers available, I don't think I have ever thought of using phone numbers in such a disposable way. It makes perfect sense. And I do do programmatic stuff with phone numbers. You know, there are, there are companies out there, obviously, where you can, you know, get phone numbers for your business and program them into your, your stack and stuff. But I don't think I had thought about it in the, in the context that you guys are discussing. So it's an interesting, interesting thought there.

• 14:47Speaker 04

  Yeah, no, I've had the same phone number since high school. So this

• 14:51Daniel Whitenack?

  Same here.

• 14:52Speaker 04

  This process really, really surprised me. But yeah, it's amazing to see it work.

• 14:57Daniel Whitenack?

  Yeah, and I'm assuming, I mean, I don't know, it was interesting for me as a physics person coming into this world that like you were talking about some of these problems around, oh, you don't want to reuse the number too much, you know, what's the optimum amount to do that? How many numbers do I need to spin up? What's the volume of that? And the geography and targeting, like, these are very complicated problems that aren't like, you know, the fundamental pure science problems of like nature, but they are really interesting problems, at least for me. I found them interesting as a physicist, just in terms of their complexity and, you know, also their connection to

• 15:39Daniel Whitenack?

  real life and how you could make a real life impact for people. I don't know if it was similar for you.

• 15:44Speaker 04

  Oh, absolutely. Yeah, I think the, one of the really fun things that I learned as a physicist is that, you know, you can use math to make a model of something and then optimize it and then apply that to the real world and see it work. It's kind of amazing to me that I can listen to a problem, write something up on the chalkboard, grind through the math, and then end up with something useful.

• 16:08Daniel Whitenack?

  So Mike, I think we should probably transition to the more AI related things that you're doing at Invoca, since this is practical AI and we definitely want to hear about those things. And one of the things I know that was highlighted as we were talking before the show was this product called Signal AI and some of the things you're doing around that. Could you maybe just give us a bit of the motivation behind that product and, you know, at Invoca and how it came about?

• 16:36Speaker 04

  Yeah, absolutely. As I mentioned, we do, well, the kind of core of Invoca is called tracking, so you can attribute a phone call to an ad. That would allow you to see which ads were driving the most calls. But really, you know, as a marketer, what you want to understand is what's driving revenue. And that requires understanding what actually happened on each of the calls. So you don't want to just optimize for a marketing campaign to drive more calls. You want to optimize for one that drives more calls of people who actually go on and buy something.

• 17:07Chris Benson?

  Yeah, you don't want vanity metrics similar to like, you wouldn't want to optimize an ad for just like clicks of the like button on Facebook or something like that.

• 17:16Speaker 04

  Yeah, exactly. You know, at the end of the day, you need revenue. And so Signal AI is our answer to that. It's a product that will actually analyze what takes place on the call. And tell you whether it was a purchase, or whether it was an appointment, or a cancellation, or whatever it is that you happen to want to track, to attribute to your campaigns. So Signal AI is a supervised model. And what that means is that the model is trained by labeled data that's provided by the user. So if you, you know, let's say that you're at Dennis's office. And what really matters to you is appointments. You're looking for new patients, new people to come in. And you want to increase the number of appointments you're getting per month.

• 18:01Speaker 04

  You would provide us with, say, 150 calls where an appointment was made, and 150 calls where an appointment was not made. And you can train the Signal AI model that would now listen to all of your incoming calls as they're routed through Invoca. And tag them as true or false for appointment. And then you could feed this information back into your, your marketing system to try to optimize. Not only which ad campaigns are driving the most calls, but which are, which ad campaigns are actually bringing people into your office for appointments.

• 18:37Chris Benson?

  So are you, are you primarily classifying the call in from the AI perspective? Or are you doing more than just classification with the AI stuff? Can you kind of talk about, you know, how that fits indirectly?

• 18:50Speaker 04

  Yeah, so Signali, exactly. It's classification. You know, was it an appointment? Was it a rescheduling, cancellation?

• 18:57Chris Benson?

  I'm just kind of curious, are there any other aspects of like particularly around NLP, natural language processing that you guys are focused on, like aside from classification?

• 19:07Speaker 04

  Well, I think for Signali, a lot of the processing that we do, you know, at the end of the day, what we want is classification. Classification is actually difficult on phone calls for a number of reasons that I find interesting. One is that phone calls range really widely in length. And so if you listen to phone calls, you need to businesses, a lot of them are, you know, three minutes long or even less than a minute in some cases, and some of them are, can be two hours long. And so a classification algorithm has to be able to handle this wide range in lengths of phone calls. And that can be tricky because if the classifier is looking for a particular pattern in a phone call,

• 19:52Speaker 04

  you know, if the phone call is really, really long, the odds that that pattern is going to appear randomly increases. So it's tough to be able to handle that that range of lengths in phone calls. And then another thing that that I find really interesting about phone calls is that, you know, English is such a rich language that no two people are going to express themselves the exact same way. Even for something as simple as scheduling, sticking with the example of a dentist's office, you know, scheduling an appointment for a cleaning at a dentist's office, that's a relatively simple thing to do. But if you listen to a hundred phone calls where people are all doing the same thing, they have the same goal, no two of them are going to use the exact same sequence of words.

• 20:37Speaker 04

  You know, there's enough freedom in English that we can all express ourselves a little bit differently. A classifier algorithm that's working on phone calls has to be really sensitive at pulling these patterns out of speech.

• 20:50Speaker 04

  But at the same time, on phone calls, there are certain things that are exactly the same from one call to the next. You know, if there's a hold message on the call, if you call or I call, we're going to hear the exact same sequence of words in the hold message. And so if a classifier algorithm is too sensitive at pulling these patterns out, you know, it'll get tripped up on these recordings that are present in phone calls.

• 21:15Daniel Whitenack?

  Yeah, so I have so many questions. This is super interesting. I guess the first is, the other thing that I was thinking about when you were talking about, you know, challenges specific to phone calls, you were talking about the sort of structure of them, how there are maybe repeating things, but a lot of times there are varied lengths and there's a lot of variance in terms of language. You know, there's also the element of accent and all of that, I'm assuming. So one of the things I was wondering was, are you primarily in the system analyzing the raw audio? Or are you converting that to text and then doing sort of text-based methods to do classification? And along with that, I'm assuming that like,

• 21:59Daniel Whitenack?

  phone calls, for example, the the audio is not going to be as good of quality as, you know, our wonderful quality practical AI recordings or other audio recordings, maybe. So like, are there also issues with that? Those are some of the things going through my mind.

• 22:15Speaker 04

  Yeah, these are excellent questions. So by the time it reaches me and on the data science team here, the phone calls already been transcribed. And so I'm fundamentally working with text data, but they're transcripts of phone calls. And so I don't personally work on analyzing, you know, the waveform of the audio and turning that into into text. But, you know, we do have, you see significant differences with dialect and accent. The way that people will explain things varies quite a lot across the country. And, you know, again, doing classification, like, you know, if I'm canceling, if, let's say we have a model for appointment set.

• 22:59Speaker 04

  And it's trained on phone calls from people in California. You know, me as a Californian cancelling an appointment, that might seem more like the truths for appointment set because I'm from the same region than someone from, you know, Maine, who's making an appointment. And so we really do have to be careful about how we handle dialect and regional differences. And then, yeah, as you mentioned, the, the audio quality varies widely. You know, somebody may be making a call from home and the audio quality is fantastic. And you can hear everything and understand every word that's said. Or they may be calling on a cell phone from, you know, driving on the freeway and the audio quality is terrible.

• 23:41Daniel Whitenack?

  What sort of real variance and impact does that make in the actual, since you're kind of the consumer of the text and it's already been transcribed? Do you see like an incredible amount of noise in the text because of those issues? Or is it fairly clean and then there's kind of outliers here and there?

• 24:00Speaker 04

  You know, I think there's kind of a continuum from transcripts that are nearly perfect to ones that are almost all noise. So we get the whole range and we treat it almost as a, like a quasi linguistic phenomenon. When a word is mistranscribed due to noise in the audio, uh, that's kind of like an artificial synonym. You know, so words like higher, tire, fire, higher, those might all be synonyms on, you know, for the purposes of, uh, phone call transcripts, even though they have different meanings in the dictionary.

• 24:35Chris Benson?

  So I'm curious, as you go through this process, and I've been listening to you and Daniel, and, and Daniel's also a real NLP expert, and so I'm trying to learn from both of y'all. Are you using multiple models in your workflows as you're coming through this? Or is there, is there one master model? You know, how, which approach are you taking? And how do you account for variability within the different speakers? Given the fact, I know Daniel mentioned accent a few minutes ago. So as you run into all these variations that you can have, I guess from an accent standpoint, that would be out from, because you're only dealing with text, right? But at some point, are you thinking about doing? Doing the raw audio and maybe you do things like sentiment analysis or anything. And I guess I'm, I'm running all over the place here, but I'm just trying to understand.

• 25:17Daniel Whitenack?

  Yeah. So are you using multiple models in your workflow for like different types of calls? Like calls of this accent type or of this dialect. It has one model and then calls of a different accent or dialect have a different model. Also, you mentioned like segmenting things. So like the whole music or message and that sort of thing. So do you have different models that analyze the different parts of the call? Or are you kind of just saying we have one big model that has enough complexity to handle all those things?

• 25:48Speaker 04

  Yeah, that's a great question and it's a nice segue into a new product that we're working on at Invoca.

• 25:56Daniel Whitenack?

  Oh, it's perfect then. Well done, Chris.

• 25:58Speaker 04

  Yeah. So I just to quickly answer your question about one model or multiple models. Another thing that's a little bit tricky about working on phone calls in our industry is that, you know, we have customers that might work in the healthcare industry or in the financial services industry or banking industries where they're really, really concerned about privacy. And so we have a policy with Signal AI where there's really one model per customer. And so each customer's model is only trained on their own data. So, you know, we might do, since we've transcribed millions of phone calls, you know, we might do much better if we could say, take a whole lot of that data

• 26:40Speaker 04

  and pre-train models that then get, you know, refined using each customer's data. But we've, we've decided not to do that. And so each customer's model is trained only on their own data. And so sometimes there's not that much data. We might have to make a model with only a few hundred phone calls that has to handle all these different regional variations. And so the, the way that we try to make that work in practice is partly by recognizing and stripping out hold messages as well as we can. You know, often a hold message essentially plays advertisements and that can sound just like a purchase. And that, that actually can be really confusing to our models. So we try to strip that stuff out as well as we can.

• 27:24Speaker 04

  And then once the recorded stuff is removed. Then we try to make our models really, really parsimonious. And so they only include words and phrases into their predictions. If there's solid statistical evidence that they really matter. I don't know if that answered your question.

• 27:43Daniel Whitenack?

  No, it did. And I'm, I'm kind of curious a little bit more on that. Like, given the sort of privacy things that you're dealing with and all of that stuff and your limitation to have this model per customer. Like, in your experience, how much data is, is a hundred calls enough to create something useful or how much is needed? And then I'm assuming, are you kind of updating this over time? Is there some sort of human in the loop element from the client who's able to kind of help fine tune things over time?

• 28:14Speaker 04

  Yeah, absolutely. So a customer can upload some data and train their own model. Another tricky thing about our industry is, or at least in Boca's models, is we have to, we decided to make them totally self-serve. So our models are not actually trained by data scientists. They're trained by the customer. And so we really have to, have to make them as bulletproof as we can. But a customer can upload some data and get their model up and running. And when their model makes predictions, you know, every call that comes through in Boca gets marked now as true or false. Let's say for appointments. And then the customer has the opportunity to listen to some calls and correct them. You know, if we mark a call as true, customer can listen to that and say, no, that wasn't

• 28:58Speaker 04

  an appointment and they give it a little thumbs down. And then, you know, through the thumbs up and thumbs down gestures, you know, we retrain the model and it should get better over time. There's also opportunities to collect more data and upload it. You know, let's say as, as time moves on, your business grows, the conversation drifts or changes over time, you can upload new data and update the model that way. So in practice, I think, you know, a hundred calls, it's, it's awfully hard to train a model on a hundred calls. That's also often what people are willing to give us at first. It's a hard sell to say, you can have this great model if you listen to a thousand calls and label them.

• 29:40Daniel Whitenack?

  Yeah, who wants to do that?

• 29:42Speaker 04

  Exactly. So even though that would make my job a lot easier. Ultimately, we serve the customers and we, we have to make do with what they're, the amount of work that they're willing to do.

• 30:00Speaker 00

  Have you heard of our newest show called Brain Science? Yes, Brain Science. It's a different kind of show, I know. And it's probably one of the ones that reaches the furthest out from our typical listener audience, but this podcast is what we call for the curious. And what's cool about this show is we're exploring the inner workings of the human brain to understand things like behavior change, habit formation, mental health, and pretty much what it means to be human. If you've ever thought about why you do what you do, or why others do what they do, then this show is for you. Head to changelog.com/brainscience to listen, subscribe, and learn more about this awesome show. Here's a preview of a recent episode called "One Small Act of Kindness",

• 30:44Speaker 00

  talking about empathy and mirror neurons. So it sounds like pliability and flexibility is a pretty crucial role too in relationships, because if you're not flexible, bendable, pliable, whatever, however you want to phrase that, if you're rigid, right, that's only gonna, that's only gonna, it's gonna be difficult for you to flex, to enable change, or to, what you've said before, recalculate.

• 31:09Speaker 05

  Yeah.

• 31:09Speaker 00

  You know, accept new data, make, you know, analyze that data, make a new plan, and iterate towards a new action.

• 31:17Speaker 05

  Yeah, and so one of the other things involved with this flexibility would be what researchers have discovered as mirror neurons. And so mirror neurons are these neurons within the brain that help us sort of get access to another person's emotional experience. And so there's an action component in it that it was first discovered, actually, with monkeys and the sort of mimicry that occurred by watching somebody else do an action. Well, in the same way, I can sort of watch somebody else walk through something in terms of an emotional experience, and if I'm holding space for them in my mind, like my body

• 32:00Speaker 05

  physiologically, these mirror neurons come, come to play.

• 32:05Speaker 00

  Is that why people cry when they watch movies or certain movies? Because their mirror neurons are firing because they're watching somebody go through a situation, and they're empathizing with them and can't help but encapsulate themselves into their scenario and feel what they're feeling? Is that why?

• 32:24Speaker 05

  Yes.

• 32:24Speaker 00

  Okay, so is that why anybody cries at anything when it's like, say, a movie related? Because they're, that's what's happening?

• 32:30Speaker 05

  Yeah, think about it, sort of like this emotional contagion, right? So...

• 32:35Speaker 00

  That's interesting to put it that way. We've said mirror neurons several times, but this emotional contagion, I believe, is actually a better subtitle for mirror neurons.

• 32:43Speaker 05

  Mm-hmm. Yeah, and so, some of this emotional contagion or mirror neurons, like, the research has been rooted in aspects of pain, because if I can recognize sort of the suffering of another...

• 32:56Speaker 00

  All right, to keep listening, head to changelog.com/brainscience/9. That will take you to the episode titled "One Small Act of Kindness." Marielle and I dig into this thing called empathy as a construct. We ask questions like, what key brain structures are involved? How can we better understand empathy to be able to better navigate ourselves and our relationships with others, both at home and in the workplace? It's a deep subject, a very fun subject. Again, changelog.com/brainscience/9, or search for Brain Science on your favorite podcast app and subscribe. We'd love to have you as a listener.

• 33:47Chris Benson?

  So, Mike, a moment ago, you were talking about the fact that you have this customer self-service part of your process, and I found that interesting. It's an interesting choice that you made there, and not something we hear all the time. And I was really wondering if you would kind of take a moment and tell us a little bit about how you've made that work for you, what that customer experience is like for them to do it, and the fact that they can update. And, you know, what does it mean that they can update their model from your perspective?

• 34:16Speaker 04

  When a customer signs up with us, they go through a process of determining what outcomes on calls are important for their business, whether it's appointments and cancellations or purchases and new customers. And then, they curate examples of training data that they then upload. And once that happens, our models train on that data kind of behind the scenes. We typically have a few different varieties that we try out and train.

• 34:44Speaker 04

  And then we measure how well each of these models is performing.

• 34:48Chris Benson?

  So as they upload the data itself, are they able to initiate training? Or is that essentially just kind of flagging that there's new data there? And you or somebody else on the team would then address the data and prep the data and get it ready? I guess, how automated is it between the point where where they're offering the data? And you're actually training on the data? I was just curious, you know, in terms of, it seems like there's a whole bunch of tasks that would have to happen in there.

• 35:17Speaker 04

  Yeah, from my perspective on the data science team, it's totally automated. So we don't initiate anything or okay anything. So the the customer uploads the data and the models train automatically. We do model selection automatically, choose the best performing one and then offer that to the customer with some summary statistics. You know, this is how accurate the model is. If you're whether you're sensitive to false positives or false negatives. You know, these are the relevant criteria for you. And then, once the customer sees this summary of the model, they can decide whether to go live with it or to wait and get more data to try to improve it.

• 35:57Daniel Whitenack?

  So I'm curious, you know, coming from, I guess, a more less speech perspective and more of a text perspective. I know that sometimes with like text documents from like chatbots or assistants and that sort of thing. You can do kind of unsupervised topic modeling and determine like what, you know, computationally, what are the topics that the model is kind of finding on its own? You know, and there's a few different ways to do that. But I was wondering if you've explored those sorts of ways to ease the burden on the customer to do the labeling and that sort of thing? Or maybe suggest to them like, oh, it seems like there's different things happening here than over here in your data.

• 36:40Daniel Whitenack?

  Have you considered these things and being kind of proactive in that way?

• 36:44Speaker 04

  Yeah, it's like you read my mind. So this signal AI process where a customer determines what's useful to them, labels data, uploads it and gets the model. That process works well for a lot of people, but you know, we always want to understand what's difficult or what are the pain points for customers? And two of the main questions that people came up with was, one, can I avoid having to label data myself?

• 37:12Speaker 04

  And two, you know, the Signal AI, it only tells me what I ask it to look for. You know, I want AI to tell me what I don't know about. I want AI to tell me what I should be looking for. And so we've recently come up with an unsupervised model, exactly like a topic model, as you mentioned, that will chew through calls automatically without any human labels. And it'll just ingest calls from a customer and kind of passively analyze them and look for recurring themes that take place on the calls. And then present those as kind of a summary to the customer. Like, hey, we analyzed 20,000 calls and this is what people talk about when they call your business.

• 37:55Daniel Whitenack?

  So first of all, you're right in that basically every time I go into a project, I don't want to label data and I'm saying, do I really have to label data? Most of the time in my projects, the answer is always yes. But the one thing I found as I've kind of tried some of these unsupervised methods, it may be hard to kind of understand what the clusters or what the topics correspond to, even though it, you know, computationally, they're distinct. Has that been a problem as you've tried to develop these methods? And how have you kind of faced some of those challenges associated with unsupervised methods in terms of, you know, how many topics do I consider? And how do I represent this to the user and that sort of thing?

• 38:37Speaker 04

  Yeah, this really sent me down a road where I did a lot of math. So I think it's an interesting case that often you can run an unsupervised algorithm. And just to be clear, an unsupervised algorithm is one where no human has provided labels or provided any indication of what the model should be looking for. A model just chews through raw data and looks for patterns to identify. Often, you know, these unsupervised models, they can find patterns that make sense to a computer. And they're useful for prediction and they're useful for clustering and all this sort of stuff. But if a human looks at what the patterns are, they're completely unintelligible.

• 39:19Speaker 04

  They make sense to the model, they make sense to a computer, but as a human, you can't make sense of them. And so, you know, we tried to design an unsupervised algorithm that would be where every element of it would be human interpretable. And we did that by imposing a lot of restrictions on the model. And so we looked at some linguistic features that seemed important for phone calls. And we made sure that we had a mathematical model that could reproduce those features. And then we put those constraints on our model at essentially every level.

• 39:52Daniel Whitenack?

  So that's kind of like you, like you did have labeled data. And you tried to recreate your labeled data with the unsupervised method to validate that you're creating clusters or topics that actually made sense. Is that the kind of strategy?

• 40:08Speaker 04

  No, so there are no labels that are made by humans. But there are statistical properties of language.

• 40:15Daniel Whitenack?

  Oh, I see.

• 40:15Speaker 04

  You think are really important.

• 40:17Chris Benson?

  Could you guys, for just a second, you kind of address what, maybe an example of what a couple of those might be in terms of like the characteristics you're looking for there?

• 40:25Speaker 04

  Yeah, yeah. So are you guys familiar with ZipSlaw?

• 40:30Daniel Whitenack?

  So I'm guessing that you should probably, probably just give a brief definition.

• 40:35Speaker 04

  Yeah, I think this is fascinating. So ZipSlaw, it turns out that if you look at a big corpus of text and you count up the occurrences of every unique word that appears in the text, and then you sort them. So you know how often the most common word is. And that word's typically the in written text, it's not in phone calls. So you take the most common word and you see how often that appears. The second most common word will appear about half as often as the most common word. And the 10th most common word will appear about a tenth as often as the most common word. And the 100th most common word will appear about 100th as often as the most common word, and so on.

• 41:17Speaker 04

  And so language really sorts itself out into this interesting relationship. And that's called ZipSlaw. And I think that's important, well, for two reasons. One is that on the Wikipedia page for ZipSlaw, there's this amazing part where someone has analyzed all of the English language Wikipedia pages. And they've made this plot for English. And of course, it follows this relationship because that's how the relationship was discovered. But then they went and did it for Spanish and for French and German and Croatian. And I think they did it for 30 different languages. And they're, you know, Latin-based languages and Germanic languages and Asian languages. And amazingly, all of them obey this relationship.

• 42:03Speaker 04

  And so there seems to be something really fundamental about human communication that creates this behavior. Where there there's some words that are really common and they appear all the time. And then there's a long, long tail of rare words. And the rare words tend to be really specific words that have a lot of meaning. And so so language tends to sort itself out into this relationship. I don't think we fully understand what creates it, but it is clear that it's a common property across language. And so that's an example of something that we really wanted our model to be able to recreate. And it's also interesting because a lot of statistical models that are kind of implicitly assumed in some common machine learning algorithms are kind of under the hood.

• 42:48Speaker 04

  They assume that distributions are roughly Gaussian-ish, that things roughly follow the normal distribution. And this distribution of words is completely unlike that. It breaks that assumption really, really violently.

• 43:02Daniel Whitenack?

  And so with these patterns that that you know should be exhibited in language. So let's say that you have these 20,000 phone calls, like you mentioned, and you could do some sort of topic modeling to break them up into however many topics. Let's say there's five topics, I don't know, as as an example. Would you then impose that sort of test on each of those clusters of calls on the transcripts to see if they if they follow that? Or how does that sort of test fit into the sort of clustering or the breaking up of topics?

• 43:39Speaker 04

  Yeah, so the way that our model is laid out, and again, it's designed so that not only a computer can understand it, but it's designed so that a human could look into the model at any any piece of it and try to be able to make sense of it. We sort of laid it out hierarchically. So if you think at the very bottom, we have the data, all of the phone calls. And the phone calls have all these properties that make them hard to analyze. So the word choice follows Ziploc, this funny distribution, and there's all this uniqueness and dialect and all that. So the phone calls at the bottom are really messy. And then we want to move up through layers of abstraction.

• 44:22Speaker 04

  And so, you know, at the very top of the model, well, we wanna, we wanna say, this is what has to be common across all of the calls in the data set. And, you know, if they're speaking English, that's gonna be the dictionary. It's gonna be the list of words that can be spoken. And that, you know, the dictionary may be an infinitely long list because people can invent new words all the time. Uh, but it's at least a countably infinite list. So that, that's the stuff that's common across all of the calls in the data set. And then our model works by kind of starting from the dictionary at the top and moving through these layers of specialization. And so we have the, this list of words that make up the dictionary.

• 45:06Speaker 04

  And then, you know, we know, well, we're speaking language and, and language follows this zip slot where there's some really common words and some really rare words, uh, that are still important. And so we have to derive a probability for each word, uh, that follows this, this power law relationship. And so that gives us, I call it a lexicon, but it's, it's essentially the words with their associated probabilities. And then where the topic modeling comes in is we say, okay, well, the, the words, the dictionary defines the language you're speaking. That lexicon now defines kind of what an average phone call looks like. And so that's gonna vary from one customer to the next or from one industry to the next.

• 45:49Speaker 04

  You know, you're gonna use different words if you're calling a car dealership or a hospital. But then that lexicon, it's still, it's still too abstract. It's just telling you what an average call looks like. And so that lexicon is gonna go through, again, several layers of specialization. And each of those are gonna be these, these power law type distributions. But it's gonna split that lexicon into, well, these are the words that you use if you're making an appointment or if you're canceling an appointment or if you're a new customer who's enrolling for the first time. Or someone.

• 46:24Daniel Whitenack?

  Gotcha. So you have the the commonalities of the language and how words are used in the language. It sounds like your goal in developing the unsupervised method is then to take things those steps lower where you what you're really trying to develop are those probabilities of or the power laws associated with the words associated with reservation or the words associated with purchase or with making an appointment or whatever the the thing is. So you have these sets of calls that have a certain more distinct statistical relationship amongst themselves that allow you to separate them out. That that's kind of the the goal that I'm picking up on. Did I say that right?

• 47:07Speaker 04

  Yeah, no, that's exactly right. We want our topics to kind of float above the data. The topics, you know, because all calls are going to be different and everyone has their own dialect and their own individual problems. So none of the conversations are going to be the same. We want the topics to be something that kind of floats above that data set and represents the themes that are consistent throughout it. And we do this through this the sort of process of hierarchically splitting the dictionary into these probability distributions. And then to try to make sure that our models are interpretable, we don't want to fit those topics directly to the data, you know, so we take the lexicon and we split it into the topics that you talk about. And then each topic, we say, well, you know, this is a phone

• 47:51Speaker 04

  call. You've got a caller and an agent, and they have totally different roles in the conversation. And so if I'm calling to make an appointment, well, the words that I say are going to be totally different from the words that the agent says. And so we go through another exercise of the splitting. And then to handle individual dialects and regional variations across the country, we also actually take a really extreme step where all of those caller side topics are then split and every caller gets their own personal variant of each topic.

• 48:25Daniel Whitenack?

  Gotcha.

• 48:27Speaker 04

  So that's the process by which, you know, we're looking for patterns that are consistent all the way across the dataset. And so we've got the data at the bottom, you know, that's totally messy and totally adiosyncratic, has all this rich, interesting information, but it's too much information. And then at the top, you know, we've got the dictionary, which is kind of uninteresting. We can all agree on it, but there's not that much information there. And then we go through these successive layers of specialization to get these two ends to meet. And then at the middle, you know, that's where we have the topics and that's what we present to the customer. And so by doing that and by tuning the math pretty carefully to match this fifth law, I think we made something that where we end up with with really interpretable results.

• 49:09Chris Benson?

  So I guess as we wind up here, what are you excited about in terms of the future of unsupervised language methods and AI and conversational data or other related topics? What does it look like for you going forward?

• 49:23Speaker 04

  Well, something that I'm actually really interested in now, you guys mentioned earlier, the possibility of having some labeled data and using that to inform the topic model. That's actually something we're working on prototyping right now, and I'm really excited about. And that's a case where we could have a semi-supervised model where, you know, let's say going back to the Signal AI problem, if someone says, "Appointment set is important for me, but I'm only going to give you 100 calls," I may have 100,000 calls that are unlabelled. And so designing a model that's semi-supervised, that can accept some labeled

• 50:03Speaker 04

  calls and some unlabeled calls, use the labels to inform the topics, so that I get not just the words and phrases, but all of the conversation topics that are relevant to this, and using that to make a more powerful predictive model. That's the direction that we're headed at the moment.

• 50:20Daniel Whitenack?

  Cool. That sounds exciting and something that I could talk with you about for for many hours. But unfortunately, our time has come to an end. Hopefully, we can stay connected and keep the conversation going. But I really appreciate you joining us to talk about this stuff and chat about how to develop insights from from speech and conversational data. It was really interesting and appreciate you joining us, Mike.

• 50:44Speaker 04

  Well, thanks so much. I really enjoyed it.

• 50:48Speaker 01

  Thanks for listening to this episode of Practical AI. Do you know someone who's trying to break into the AI game? Send them our way. We appreciate every recommendation because word of mouth is how people hear about podcasts. Shoot them a quick email or a Slack message, put out a tweet, whatever, or get crazy. Get up from your desk, walk across the room and tell them in real life, "Hey, my sorry good conversation." Practical AI is hosted by Daniel Whiteneck and Chris Benson. It's produced by me, Jared Santo, with music by the oh so mysterious Breakmaster Cylinder, and we're brought to you by awesome sponsors. Support them, they support the show. We've got Fastly on bandwidth, Linode on hosting, and Rollbar on bugs. If you haven't yet, hit up our master feed, your neural networks will thank you. It's all Changelog Podcasts in one easy subscription. Get it for the price of a free hot dog. Thanks again for listening, we'll talk to you next week.