NLP to help pregnant mothers in Kenya

In Kenya, 33% of maternal deaths are caused by delays in seeking care, and 55% of maternal deaths are caused by delays in action or inadequate care by providers. Jacaranda Health is employing NLP and dialogue system techniques to help mothers experience childbirth safely and with respect and to help newborns get a safe start in life. Jay and Sathy from Jacaranda join us in this episode to discuss how they are using AI to prioritize incoming SMS messages from mothers and help them get the care they need. Sponsors: RudderStack https://rudderstack.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog – Smart customer data pipeline made for developers. RudderStack is the smart customer data pipeline. Connect your whole customer data stack. Warehouse-first, open source Segment alternative. SignalWire https://signalwire.com – Build what’s next in communications with video, voice, and messaging APIs powered by elastic cloud infrastructure. Try it today at signalwire.com https://signalwirecom and use code AI for $25 in developer credit. Changelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Jay Patel – LinkedIn https://www.linkedin.com/in/jay-patel-127a491Sathy Rajasekharan – LinkedIn https://www.linkedin.com/in/sathyrajanDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Jacaranda Health video https://youtu.be/CgV9QVF2VYwJacaranda Health article https://kenya.ai/how-jacaranda-health-is-getting-help-for-mothers-in-kenya-faster-using-natural-language-processing/Jacaranda Health website https://www.jacarandahealth.org/How AI helps mothers in Kenya get the care they need, faster (article) https://towardsdatascience.com/how-ai-helps-mothers-in-kenya-get-the-care-they-need-faster-eb4f05b34732Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  You know, the goal is always to get that mom care as quickly as possible. We are looking at what other data, what other information will help us increase a risk rating for a mother, will help us understand which facility she should be referred to. Half of the organization works with providers and so we have a lot of data and insights from that layer of the health system. And so can we use these layers of data to really route those moms more effectively to care in a timely manner? And just scratching the surface of it is to look at conversational history and look to see if there are triggers, trigger intents that could be predictive of a future danger sign. And it actually looks like we may have enough data to develop a model just from the initial work that's been done.

• 0:47Speaker 01

  Big thanks to our partners, Leno, Fastly and Launch Darkly. We love Leno, they keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com. This episode is brought to you by our friends at RudderStack. And we're calling all data engineers to check out RudderStack Cloud and start building smart customer data pipelines. RudderStack is warehouse-first, no more silos. RudderStack builds your customer data lake on your data warehouse, not theirs, enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API-first. RudderStack can be easily integrated into your existing development processes.

• 1:31Speaker 01

  And because they're open source, you can see all their code, so you don't have to worry about vendor lock-in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. RudderStack is free up to 500,000 events and pricing scales transparently from there. Learn more and get started at rudderstack.com. Again, rudderstack.com. That's R U D D E R S T A C K dot com.

• 2:05Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM.

• 2:31Daniel Whitenack?

  Welcome to another Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International, and I'm very excited today to be joined from Kenya by Jay Patel, who is Technology and Analytics Manager at Jacaranda Health, and Saty Rajasekaran, who is Executive Director in Africa of Jacaranda Health. Welcome.

• 2:59Speaker 00

  Thank you. Thanks for having us, Daniel.

• 3:01Daniel Whitenack?

  Yeah, it's wonderful to talk to you both. We've already had a lot of great conversation even before we started recording, so I'm really excited about this. Maybe, Saty, could you give us a little bit of an introduction to Jacaranda and some of the things you're doing and how it came about?

• 3:17Speaker 00

  Yeah, absolutely. So Jacaranda is a nonprofit organization that works in Kenya, primarily in Africa. And the challenge we're trying to address is one of the fact that mothers and babies die during childbirth in this part of the world, probably six or seven times more frequently than happens in more developed countries in North America and Europe. And we've recognized that it's not really a question of not having enough hospitals or providers or services, although, you know, there's certainly challenges

• 3:56Speaker 00

  there, but the quality of care, so the kinds of care that are being provided, really needs to improve. And this has been shown, you know, in the literature and by many, many other groups as well as ours. And so Jacaranda works with governments, with the government hospitals in country to try and improve the quality of the care that's being delivered in hospitals. And we do that by using low-cost, scalable solutions that can be deployed within government hospitals to increase the number of moms who are seeking care at the right time and the right place, and to improve the care they're receiving from providers when they actually get to a hospital. And that's where our digital health tools come in.

• 4:41Speaker 00

  They are one of those low-cost solutions that we're delivering through government hospitals.

• 4:45Daniel Whitenack?

  Yeah, that's awesome. I really, really appreciate your work in this area. Jay, maybe you could sort of give us some context. So that's a wonderful story and setup and context. But, Sathy, you mentioned digital tools. Maybe, Jay, you could let us know, hey, where does AI and NLP and these sorts of things fit in? So why are we talking about this on the Practical AI Podcast?

• 5:09Speaker 04

  Sure. So what happens when a mother enrolls in our service is that she will first go to one of these, so far, 700 public health facilities that we're partnering with. She will enroll in the service and then we will start sending her messages about her health, about the health of her baby, about her pregnancy. And messages include everything from nutrition all the way up to danger signs. And she can then ask us any questions that she has at no charge. This is primarily run all off of SMS, and again, it's free to the mother.

• 5:43Daniel Whitenack?

  Yeah, maybe you could, because I think a lot of times, at least in many people's context, they might be thinking about chat as like a little window pop-up on their customer service site or something. But it sounds like you're focused primarily on SMS. Is that right?

• 6:00Speaker 04

  That's correct. We checked with our users and more than half of them are still using feature phones. So even though mobile phone penetration in Kenya is in the high 90s, a lot of users don't have smartphones and are still using, you know, like the old Nokia feature phones.

• 6:16Daniel Whitenack?

  Sure. And how, so I guess you're having these conversations. When did you start thinking about what machine learning or NLP sort of techniques could benefit you? How did that come about originally?

• 6:32Speaker 00

  I was gonna sort of tell the story from the perspective of how Jay came to be part of the team.

• 6:39Daniel Whitenack?

  That'd be awesome, yeah.

• 6:40Speaker 00

  Yeah. You know, we launched a service now four years ago with 200 moms. You know, we were super excited that we had 200 moms using the platform. It was an accident that we actually opened up two-way communication for free. We didn't originally intend for that. We just thought we'll send moms a bunch of messages and that'll improve their knowledge about pregnancy. And then I distinctly remember one of our program team members coming in and saying, "Hey, these moms are asking questions. I've been answering them, but the questions are increasing." So it turned out we had learned of this latent demand for just a whole bunch of questions needing to be answered.

• 7:21Speaker 00

  I mean, you know, back to what Jay was talking about, many of these moms have feature phones. Many don't use data on a regular basis because it's still relatively expensive here. And so Googling something isn't really an option. The complexity of language when you Google something, I mean, we all know is really hard. So the minute they realized someone was sending them messages about their pregnancy, they started sending in questions. The question volume started increasing as we started to enroll more and more moms. You know, we were getting to hundreds of moms per month, to 1,000 moms per month. We're at almost at 100,000 per month now. And we pretty early realized, early on realized that we need a way to triage the questions coming in.

• 8:03Speaker 00

  So some of the moms were asking about, you know, "What can I eat for during pregnancy? Is it okay to eat avocados?", for example, which is a surprisingly common question that we get asked. But out of, you know, every 30 questions we get asked, one or two of them will be really serious. A mom may say, "I'm bleeding. What should I do?" And we recognized that if we sort of did a first-in, first-out approach to answering questions, we'd miss that mom or be late to answering that mom who needed to know about bleeding. So our journey to think about machine learning actually came in thinking about how do we effectively categorize these messages?

• 8:46Speaker 00

  And I had, this is when we had a tiny team, I had been messing about with dialogue flow to see if we could put some of the conversations in. And quickly, you know, in addition to realizing that we were onboarding more and more users, recognizing that we actually needed someone with a lot more development expertise to come to the team, which is how Jay comes into the picture and kind of changes the way we do do business with these incoming questions. Happy to now hand over to Jay to talk about how he actually solved this challenge.

• 9:20Daniel Whitenack?

  Sure, yeah. And just before we kick it over there, I guess, so you mentioned dialogue flow, Sathy, which is an offering from Google to help build chat conversations and that sort of thing. When you started looking to solve this problem, was it clear that, I guess, machine learning and AI even could provide a solution? Or was that still something that was relatively unclear? You just knew that, you know, technology needed to be brought to the table?

• 9:47Speaker 00

  Yeah, we knew tech needed to be brought to the table. I was actually, you know, in the ecosystem here in Nairobi, there was a number of people working on chatbots. And so the original thought was, okay, maybe there's also a chatbot opportunity here where we don't even need people at the other end answering questions and we can create conversational histories and, you know, intent classification, et cetera, et cetera. And I think it just took a few weeks to realize that with the complexity of language and information coming through, we'd actually needed a different solution. So this automated chat idea was quickly discarded. Also, from a user perspective, moms didn't seem to like getting these kind of

• 10:32Speaker 00

  cookie cutter responses back from whoever was on the other end of the texting service.

• 10:38Daniel Whitenack?

  Yeah, that's definitely something I've run into as well in working in chat and dialogue. So, yeah, that's interesting. So Jay, how did then, when you started approaching this problem and getting involved, what was the process like in terms of figuring out what was the right tech solution?

• 10:57Speaker 04

  We were initially connected by two gentlemen by the name of Matt Capers, works in Square in Silicon Valley. He volunteered to help us figure this kind of stuff out. And we started out by testing a various solutions. So we had the data sets that Saty had put together from the previous two years. We used that. This was a list of in questions that moms had asked us. And so we took a few thousand of them and had our team label the questions according to what the mom was asking. So just a one word label. And it could have been nutrition. It could have been, you know, something else.

• 11:38Speaker 04

  And we fed this list of intents into the three most popular, like off the shelf, commercial models of it. In fact, four NLP models that were available at the time. Dialogue flow was one, then, you know, the usual suspects between Google, Amazon and Microsoft. And after some testing, it it became apparent that Google's NLP for this particular use case was most useful. And so then we took a larger data set. I think it was something like 13,000 questions. And the way we figured is is to run each of these questions through a translator.

• 12:22Speaker 04

  And then the output of the translator, along with the intent that we had classified for each question would be used to train the model. After that, it then took a little bit of extra figuring out, you know, how granular do we want the intents to be? So we could have four or five very broad intents. Or we could have many dozens of, or 50, you know, or 60, very, very fine grained intents. And it turned out that the best mix was kind of like the middle, the goldilocks middle. So we ended up with a with an intent list was, which was about 33 in length.

• 13:03Daniel Whitenack?

  And I mean, that sounds like a pretty labor-intensive process getting through that data labeling. What was that process like in terms of actually getting into this into this task of data labeling? And what challenges along the way did you face there?

• 13:22Speaker 04

  Yeah, it was quite labor-intensive. It was very manual. So we just threw up, you know, all 13,000 questions on a spreadsheet and then assign a few members of the team to go in and read each question and actually assign a label from a list that we had predefined. And then, you know, they'd have the option to add labels as they went along. That took several weeks and it took, you know, a lot of our team members' time when they could have been doing other things. But we couldn't-

• 13:49Speaker 00

  Yeah, everyone was super-niced and volunteered to do it, for sure.

• 13:55Daniel Whitenack?

  Yeah, you need some grace from people when you're asking for that sort of task. But I'm sure hopefully now they see some of the value coming through from that.

• 14:03Speaker 00

  Yeah, for sure. One other challenge, you know, I, this is still something that we work on is the sort of specificity of labeling something, you know, there's a lot of subjectivity involved, right? Like if a mom texts in and says, "My baby hasn't stopped crying," someone may label it as, you know, crying baby, and someone else may label it as general baby concern. And so we learned the hard way that we needed to do a lot of training around what is it that we mean for each of these labels. So I think that was the tougher part, certainly for this initial dataset. But we did, you know, Jay will talk about this, we did a much bigger round of training a year later and had to put in a lot more rigor to the process after that.

• 14:56Speaker 02

  Signal Wire is real-time video tech to help you create interactive video experiences previously not possible. It gives you access to broadcast-quality, ultra-low-latency video that's proven and trusted by Amazon, Ring Doorbell, Zoom, and others. See why the future of video communication is being built on Signal Wire. They have easy-to-deploy APIs, SDKs for the most popular programming languages, and expert support from the OGs of software-defined telecom tech. Try it today at signalwire.com and use code AI for $25 in developer credit. Just visit signalwire.com, that's signalwire.com and use code AI to receive that 25 bucks.

• 15:41Speaker 02

  Once again, that's signalwire.com code AI.

• 16:03Daniel Whitenack?

  So, Saty, you just mentioned how there was kind of this initial round of labeling and defining the different classes that you're working with. And then as you went along in that process, you realized you needed more data and to kind of switch up the labeling. Jay, what did that process look like in terms of training maybe a bigger model with more data? How did you go about that scaling and what was needed to facilitate that?

• 16:31Speaker 04

  So when we had run the first round of labeling, we didn't even, we hadn't even collected that many questions. Then across the following year, as the questions, you know, as the service grew and the questions continued to come in, we just basically stored them somewhere. At the end, we had over 100,000 questions, but we decided to limit the labeling exercise to 100,000 questions. Along with the questions, we also, the help desk also had now much more experience on what kinds of questions that they get. And so instead of, you know, me or Saty pulling together a list of intents, we asked the help desk to help, to help us

• 17:12Speaker 04

  figure out what now, now that we're evolving to a second version, what's the best list of intents and do we expand this? Are there, for example, If you have an intent for general baby concerns, how can you break that down and how do we make it more specific? What are the questions that come in that are being caught as general baby that we can identify better or help triage better? And this time we had to outsource and we used a service that or a company that actually helped us go through and label each of these hundred thousand questions. But as Saty mentioned, you know, we had to train the team and then the training was had to be quite rigorous.

• 17:55Speaker 04

  But even after the training, we had to go through several rounds of cleaning and just making sure that, you know, what we might identify as one particular intent was identified by the team as another intent because they're not really exposed to the work we do on a daily basis.

• 18:13Daniel Whitenack?

  Yeah, that brings up a really interesting question, which, you know, is striking me is that there's probably a lot of, I imagine there's a lot of health expertise within your team and those that you're working with. But it sounds like you're not primarily a machine learning startup or, you know, something like that. So what was it like culture wise? And, you know, when you're explaining what you're trying to do to your team, to, you know, your board of directors, to those you're serving, how did that sort of culture change happen? And what strategies did you employ to sort of help? Help you bring along people with the solutions that you were trying to build? Maybe that's a question for Saty.

• 18:58Speaker 00

  Yeah, I think one thing that Jay and the team do really well is try and frame for the team, how is this going to make your life easier, right? If we're able to label questions, you get to prioritize them. Then you can tag the high priority ones first. But actually, what was even more exciting was what if we could automate responses to a class of questions that we are, you know, 95% certain of in terms of intent classification?

• 19:29Speaker 00

  And that reduces the volume for the helpdesk. So by sort of sharing, what does this mean for you at the front lines essentially? I think Jay and the rest of the team really brought everyone into this kind of, this is a shared mission mindset versus, oh, it's the tech team doing something, you know, AI-esque again. So what's really cool is everyone talks intense and classification now on the helpdesk team. They all know what they're looking at. It's really fun to watch that journey. It has been fun to watch it over the last couple of years to see people kind of ignore what me and Jay were doing in the background to now being, you know, front and center and understanding how it works.

• 20:11Daniel Whitenack?

  And that's really cool to hear. I think that many of us in different organizations listening to this are probably, you know, wishing and hoping we can bring along our teams in that same way and build that excitement. I'm assuming part of that is, you know, you talk to them about those pain points, but then also they actually saw value out of what you were producing. How quickly did that bit happen? How soon was it in terms of the time when you first started showing them maybe prioritization of questions or classification of questions? And when they were actually seeing, seeing value out of that? How was that rollout period? How did that happen?

• 20:51Speaker 00

  I mean, I have an opinion. I'd love to hear Jay's opinion. I actually think we were more excited about the classification and this sort of, you know, precision and recall that we were getting after the first training round. But then after a while, we realized the helpdesk team was like, yeah, it doesn't really work that well, which is what really pushed us to try and improve the classification. And I say us, I mean, Jay and the team really worked on, you know, capturing statements like when someone says yes or thank you, how do you filter that out? You know, the sort of real life annoying things that happen when you run something at a relative scale. We're doing 2,500 questions now. I think questions per day.

• 21:34Speaker 00

  I think where we really started to see. It making a difference is once those practical things were ironed out, then I think the team started to see, huh, this actually works. And that's what enabled us to get their buy-in to do that bigger round of training later. I don't know, Jay, what do you think? Does that track? Or is that just my assumption from high above?

• 21:56Speaker 04

  No, that makes sense. And there was a lot of hiccups getting it to work. There's a lot of patience from the team, especially the help desk. Um, maybe they don't have full context on why it's working, but they had enough trust in us to let us be about experimenting. And as you mentioned, once we get, once we figure out how to filter the stuff that the help desk doesn't need to answer, that's when they really bought into, okay, now this is how it's having a practical impact on my day.

• 22:26Daniel Whitenack?

  Makes sense. So I'm always interested, if you listen to the podcast, in the very practicalities of how this all works, which you've brought up, Jay. And one curiosity I'm having is around the integration of all of this. I think one of the areas that machine learning and AI practitioners often get blocked in is, you know, they can train a model, but sort of integrating it in a workflow or in existing systems is often very difficult and error prone. It sounds like you're dealing with SMS here, and I know that there's APIs like Twilio and other things that you can use to interact via SMS. But then, you know, you've got your model sitting somewhere. Um, I don't, I don't know where that's, you know, stored and sitting.

• 23:09Daniel Whitenack?

  And then somehow you've got to integrate those things together along with actual humans in the loop. So what does that integration piece look like for Jack Aranda?

• 23:20Speaker 04

  So on one, one end, we have a messaging platform that's called Rapid Pro. That's plugged into the telcos and handles the traffic for the SMS. On the other end is the ticketing platform or the ticketing software where the helpdesk is responding to these messages. And the triage or the NLP model sits in the middle. And what happens is just by reading and writing the APIs for these three platforms. We grab the messages as they come in, run them through the translator, run them through then the NLP model. And all the information that comes out of both of these gets posted to the ticketing platform.

• 24:00Speaker 04

  Any responses then go straight out the other way, but bypassing the middle bit and hit rapid approaches for delivery to the mother. But basically, all this was integrated by way of the APIs for the three platforms.

• 24:14Speaker 00

  You know, Daniel, I was just thinking that as reflecting as we're talking about this, we used to be really cagey about talking about, oh, do we use, you know, Google's platform or IBM, whatever. And the sort of, the real experience has been that it's what Jay affectionately calls the glue that holds all this together is really where all the hard work and iteration and innovation came in. The development of a ticketing software that works with this workflow and can incorporate the AI in a useful way. That's actually been the sort of innovation here over and above the use of machine learning or NLP in what we do.

• 24:54Speaker 00

  And so it's kind of fun to hear our evolution of cage-ness being like, now we just, we tell everyone what we do. But we recognize that there's so much stacked in here that is unique to what Jhakaranda does that we don't have to be super cagey about it anymore.

• 25:09Daniel Whitenack?

  Yeah, that's a very interesting observation, Sathy. And I think it's true that, you know, it's one thing that you can go to like GitHub or somewhere and you can just see all the implementations of all of these models that people are releasing and, you know, state of the art models. But the process of getting that to work for you and your context and integrated with your systems and your employees and that glue, like you mentioned. To me, that's, that's almost always where projects.

• 25:39Daniel Whitenack?

  Get blocked or held up. And so sharing sort of what you're doing in the sense of the AI or NLP side actually is in some ways, you know, it's significant. It's definitely a driving factor, but it's only a small factor of a much larger system that needs to be put into place. So yeah, I definitely, I definitely appreciate that comment and context for our listeners as well. I know. So one of the things that was mentioned very briefly, I forget who mentioned it was translation. So I want to talk about that a little bit more, but maybe before we talk about that, could one of you maybe just describe a little bit for those that aren't familiar with Kenya or African languages? What is the sort of linguistic diversity look like where you're at?

• 26:25Speaker 00

  So in Kenya, you know, we have two major languages in the country. So that's English and Swahili. But of course you have local languages in the various regions, the various counties that we have. And then you have dialects of those languages as well. Because of the strong public education system that the countries had, you're actually, we're actually quite lucky that we can send text based information and receive text based information, primarily in English and Swahili, because you know, everyone is comfortable texting in that. The one challenge, and this is a pretty big challenge, is when you start to see mixture

• 27:06Speaker 00

  of the languages and more informal Swahili, which is known as Xheng. So you get a real mishmash of languages, more hip words coming in. And that tends to break things a little bit, although I believe now we've gotten around it. Yonah Jay, we want to talk about some of those language related issues.

• 27:28Speaker 04

  Yeah, so most of the messages we get, some 60 or 70% are in Swahili, but not pure Swahili. It's always a mixture of Swahili and English. And even the English messages have Swahili words in them. But the Xheng or the slang, as Saty mentioned, that in itself is a mixture of English and Swahili and other languages, and it evolves quite quickly. And so it's not something that is maybe as stable as an official language.

• 27:59Speaker 04

  And when we're running all of this through the translation, what comes out at the other end is quite often very garbled and often doesn't resemble what the original question was. However, the NLP model seems to be able to parse it for context and apply the correct intent most of the time. And we have gotten the accuracy to about eighty-seven odd percent for like general questions and for danger sign questions. It's in the mid to low nineties and we'll continue to try and improve that.

• 28:50Speaker 02

  ChangeLog++ is the best way for you to directly support Practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of Practical AI into the future. Simply follow the ChangeLog++ link in your show notes or point your favorite web browser to changelog.com/plusplus. Once again, that's changelog.com/plusplus.

• 29:36Daniel Whitenack?

  So Jay, you're mentioning some of the results and the sort of current state of what you're doing. You mentioned there's kind of these different categories of questions and you're tracking your metrics in those different areas like danger sign questions and others. Could you just describe maybe a little bit the makeup of your dataset in terms of? How many of these questions that are coming in are sort of danger sign questions that you need to triage with very high priority? And what sorts of questions are those? And what does that percentage look like with, in terms of the rest of the questions and general information questions?

• 30:12Speaker 04

  So about 30% of the questions that come in could potentially be a danger sign. And danger signs include questions like bleeding, or I have swelling in my, in my feet, in my legs, I have a headache. And of those 30%, I think overall of, out of the questions, might be 3 or 5% which are actual danger signs. And what we're trying to do is throw a wider net so that even if we capture a lot of questions which are not strictly a danger sign, we want to make sure that we do capture those which are. And the help desk can then, you know, filter for urgent and high priority, figure out

• 30:56Speaker 04

  what needs to be answered now, and then what can wait an hour or two. And then questions about, you know, nutrition and whether it's okay to eat avocados during pregnancy, that kind of thing can, can wait maybe half a day or a day.

• 31:08Speaker 00

  Yeah, and maybe just to add, what, what happens is, you know, the agents who are texting back will escalate messages that they feel a nurse needs to review, and then the nurse may pick up the phone and call the moms. And so that's where that 3 to 5%, you know, confirmed danger sign, mom needs to be referred to the hospital metric comes from. But it's definitely an area that we're really actively looking at in terms of analytics, you know, data science, and even a little bit of predictive modeling now, to be able to, okay, in the haystack of danger signs, find the needle of absolutely needs to be referred right now, without having to make the phone call first.

• 31:49Daniel Whitenack?

  Yeah, so you mentioned potentially in the future, kind of providing some automated responses. How do you view that workflow coming in and for what types of questions? And, you know, sort of over the progression of the system, how do you see automation being used best versus the way that it's interacting with humans in the loop?

• 32:15Speaker 04

  We are responding automatically to a subset of the questions we get already. And how it started is that when the AI detects that this is a danger sign question, we would send the mother a menu saying, hey, it sounds like you're asking about danger signs. Please select the specific issue you're having from the menu below. And that would list, you know, the three or five danger signs. What we realized is that, or what we noted is that we got a response rate of around 3% to that menu. And it should have been obvious in retrospect that moms and probably everyone else hates interacting with menus. I hate interacting with menus.

• 32:57Speaker 04

  And so we iterated to version two where, you know, if the AI detects it's a headache question, we'll send only the headache response. If it's a swelling question, we'll send only the swelling response. And then we follow up with the mom and ask also automatically, did this information answer your question? And so we went from a response rate of 3 odd percent to about 45%. Now, if the mom says yes, that's great. And we close the ticket so the help desk doesn't have to worry about it. But if the mom says no, it didn't, or if she doesn't respond, then that particular question gets red flagged for the help desk to look at as a priority.

• 33:38Daniel Whitenack?

  So you've got agents sort of in the loop. You're always responding, which I think is really wonderful, taking that perspective on it. How does that feedback maybe from your agents or the new questions that come in? How does that feedback into your data set in terms of model updates? And when you update your data set and how you update your data set? How are you handling that loop?

• 34:03Speaker 04

  Great question. So the second round of labeling, the 100,000, it was expensive. It was painful and we didn't want to have to go through that again. So in the help desk ticketing platform, we've built an option for the agents to correct where they note that the AI had flagged the intent incorrectly.

• 34:26Speaker 04

  So every time they see that, they'll just from the dropdown menu, select the correct intent. And now, you know, once we collect enough data, uh, we can just feed that back into the model and update it without having to manually label, uh, you know, hundreds of thousands of questions every, every now and then.

• 34:45Daniel Whitenack?

  Yeah, that's wonderful. I know a lot of my questions are on the, the data side of things, but as both of you have emphasized that, you know, that's a real key part of any of these types of solutions. And I know in the health space in particular, you know, data is, is difficult in certain ways to deal with because, you know, we're dealing with people's personal health information, information about maybe personally identifying information, very sensitive data. It sounds like that one of the strategies you're taking is, is definitely people sort of opting into this service and making sure you have some information from them. How has it been on that side of things in terms of,

• 35:31Daniel Whitenack?

  you know, keeping your data secure, making sure that things are kept confidential while at the same time, being able to sort of combine this data in a meaningful way to create useful models?

• 35:43Speaker 04

  The first step, and Saty can add some context, but the first step is to collect as little as possible. So we don't know our users' names. We don't ask for things like age or other demographic information. Obviously, we need a phone number and we'd like to know where, which health facility they signed up in and how many months pregnant they are so that we can tailor the message campaigns according to the stage of pregnancy or whether they've delivered. Um, and then on the backend, just making sure that everything is stored according to industry best practices on, you know, the, one of the major cloud providers and using their security tools.

• 36:26Speaker 04

  I think that rather than me trying to manage a server here, just using the resources that are already available helped us better able to secure the data. Saty?

• 36:37Speaker 00

  Yeah, I would just add that it's such an evolving conversation because I feel like knowledge and literacy around machine learning, what it requires from an infrastructure perspective.

• 36:51Speaker 00

  What that data is being used for, I mean, it's hard enough for the general public as a Nokia feature phone and is in the middle of a village. You know, how do you consent appropriately and provide a terms of service appropriately? So that she's fully aware of, you know, how this information is being used, you know, and we're using it to improve the service that they get. The other sort of area where I think there's a lot of work to be done is in helping, you know, our government partners understand how that data is used. You know, what are these systems? What are these processes?

• 37:33Speaker 00

  A question we get asked frequently is where is the data stored? You know, is it in Kenya or somewhere else? And I think that kind of, that's indicative of actually missing the more challenging question, which is how is this data being used by, you know, a machine learning platform? How are you using it to respond to women? What's your threshold for risk, et cetera? And I don't think we're there yet in terms of conversations here. But there's a lot of groups doing some great work around building capacity to improve those conversations. And I think that's true, not just for Kenya, that's true around the world with the conversations we're having with data and privacy.

• 38:13Daniel Whitenack?

  Yeah, to some degree, this is definitely new for everyone. And also, you know, like you're talking about helping people understand the ways in which their data might be used. I know a lot of companies have developed different principles and other things around that. So it's really interesting to hear from your perspective how you're approaching that, I think. In terms of looking, you know, maybe forward a bit, I'd be curious to hear maybe first on the technical side, but then also just on the user side. What do you feel like are the challenges and opportunities moving forward that you haven't addressed yet that you'd like to dive into? Um, maybe on the technical side first, Jay, what, what do you, what are the main challenges

• 38:57Daniel Whitenack?

  you're facing um in terms of scaling this or improving your models or extending this system?

• 39:05Speaker 04

  One of the technical challenges is getting the accuracy up higher than what it is, much higher than what it is. So the off-the-shelf or commercially available models that we can plug into without actually being machine learning engineers ourselves. They do it well enough. They don't do it quite as well as we'd like. And so one of the next steps is to partner with some machine learning experts. And try and figure out how do you go from processing or just pattern matching on these words to maybe building in some sort of understanding and context into what the questions are about and then respond appropriately.

• 39:43Speaker 04

  In terms of scaling, I think we've gotten our costs down. It's running pretty cost effectively. In those terms and in terms of the number of questions that we can process. We seem to have gotten a handle on that. We just need to increase the accuracy. And maybe Saty can also mention what's next in terms of some of the predictive analytics stuff that we're looking at.

• 40:10Speaker 00

  You know, the goal is always to get that mom care as quickly as possible. And so now we're looking at what other data, what other information will help us increase a risk rating for a mother, will help us understand which facility she should be referred to. Something we didn't even talk about is half of the organization works with providers and improving the skills of providers. And so we have a lot of data and insights from that kind of layer of the health system. And so can we use these layers of data to really route those moms more effectively to care in a timely manner?

• 40:48Speaker 00

  So that's work that's going on now. And just scratching the surface of it is to look at conversational history and look to see if there are triggers, trigger intents, you know, that could be predictive of a future danger sign. And it actually looks like we may have enough data to develop a model just from the initial work that's been done. And then Daniel, you were asking a question about the users and I think the ultimate goal is to support moms with what they need in terms of information, whether that's, you know, they want to know where should I get my baby vaccinated? What kind of diet should the baby be on? You know, like, how do you transition from foods? And these are questions the moms are asking us, right? They want more and more information.

• 41:32Speaker 00

  Because in the context that we live and work in, that information, that kind of support is hard to come by. So the lower cost, the more digital, the more close to the mom we can get on her phone or, you know, maybe phone plus, um, the better. So we're looking at things like voice. We're looking at a home record on, you know, in terms of their own medical information that they wish to choose on their uh, free to access for them so that it can help them provide a case history to a provider easier. So there's a lot of kind of future thinking around this, but the principle is always what does that mom need to get care quicker? So lots of work going on in the background right now.

• 42:14Daniel Whitenack?

  That's wonderful. Well, I really appreciate what you all and Jacaranda are doing. I think it's, it's wonderful and also a great illustration of how AI and NLP can be utilized by an organization in a very practical way that really benefits the users. So I appreciate you sharing this story and joining me on the podcast. It's been wonderful to talk to you both. So thank you very much.

• 42:41Speaker 00

  Thanks a lot. It's been great chatting.

• 42:42Speaker 02

  Thanks. Yeah, it's been fun.

• 42:47Speaker 02

  Thank you for listening to Practical AI. We have a bundle of awesome podcasts for you at changelog.com, including our brand new show, Ship It with Care Hard in the Zoo, a podcast about getting your best ideas into the world and seeing what happens. It's about the code, the ops, the infra and the people that make it happen. Yes, we focus on the people because everything else is an implementation detail. Subscribe now at changelog.com/shippit or simply search for Ship It in your favorite podcast app, you'll find it. Of course, the Galaxy Brain move is to subscribe to our master feed. It's all changelog podcasts, including Practical AI and Ship It in one place. Search changelog master feed or head to changelog.com/master and subscribe today.

• 43:31Speaker 02

  Practical AI is hosted by Daniel Whitenack and Chris Benson with music by Breakmaster Cylinder. We're brought to you by Fastly, OnStarfleet and Linote. That's all for now. We'll talk to you again next week.