Data science for intuitive user experiences

Nhung Ho joins Daniel and Chris to discuss how data science creates insights into financial operations and economic conditions. They delve into topics ranging from predictive forecasting to aid small businesses, to learning about the economic fallout from the COVID-19 Pandemic. Sponsors: O'Reilly Media https://www.oreilly.com/changelog – Learn by doing — Python, data, AI, machine learning, Kubernetes, Docker, and more. Just open your browser and dive in. Learn more and keep your teams’ skills sharp at oreilly.com/changelog https://www.oreilly.com/changelogRudderStack https://rudderstack.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog – Smart customer data pipeline made for developers. RudderStack is the smart customer data pipeline. Connect your whole customer data stack. Warehouse-first, open source Segment alternative. Changelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Nhung Ho – Website http://www.astro.yale.edu/nth4, LinkedIn https://www.linkedin.com/in/nhungho, X https://x.com/nhungkiaChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Intuit Website https://www.intuit.comIntuit Blog | Nhung Ho, Director of Data Science https://www.intuit.com/blog/author/nhung-ho-director-of-data-scienceIntuit Blog | How AI Can Help Build Resiliency for Small Businesses in a Global Economic Crisis https://www.intuit.com/blog/technology/how-ai-can-help-build-resiliency-for-small-businesses-in-a-global-economic-crisisUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I like to point back to two thousand eight where you think it's like a once in a generation event. And then like ten years later, you have another once in a generation event. These events, like you can't write them off as one time anomalies. You actually have to figure out how to fold that in. Uh, and be able to essentially like automatically regress against them, learn from them, and then figure out kind of how to apply it forward. Like you said, we're at the tail end of the pandemic right now. Hopefully, fingers crossed, right?

• 0:29Speaker 01

  Yep.

• 0:29Speaker 00

  But what happens if, you know, another large worldwide event happens ten years from now? We can't just clip that chunk of data, remove it and act as if it never existed. You need to build that into the methodology of your models.

• 0:44Speaker 01

  Big thanks to our partners, Leno, Fastly and Launch Darkly. We love Leno. They keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com.

• 1:03Speaker 01

  This episode is brought to you by our friends at O'Reilly. Many even know O'Reilly for their animal tech books and their conferences, but you may not know they have an online learning platform as well. The platform has all their books, all their videos and all their conference talks. Plus you can learn by doing with live, online training courses and virtual conferences, certification, practice exams and interactive sandboxes and scenarios to practice coding alongside what you're learning. They cover a ton of technology topics, machine learning, AI, programming languages, DevOps, data science, cloud, containers, security, and even soft skills like business management and presentation skills. You name it, it is all in there.

• 1:44Speaker 01

  If you need to keep your team or yourself up to speed on their tech skills, then check out O'Reilly's online learning platform. Learn more and keep your team's skills sharp at O'Reilly.com/changelog. Again, O'Reilly.com/changelog.

• 2:05Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM.

• 2:31Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International, and I'm joined as always by my co-host, Chris Benson, who is a Principal Emerging Technology Strategist at Lockheed Martin. How are you doing, Chris?

• 2:48Chris Benson?

  I am doing very well. How's it going today, Daniel?

• 2:50Daniel Whitenack?

  Oh, no complaints. Lots of work to do, but all interesting things and fun things. You know my world, lots of natural...

• 2:58Chris Benson?

  About the same, just lots of work and trying to enjoy spring weather. So I have been, since I'm living in the South, I live in Georgia, I have a standing desk. I think I've mentioned this to you in the past.

• 3:10Daniel Whitenack?

  I didn't know you had a standing desk.

• 3:12Chris Benson?

  I do. Like, I have a regular desk inside, which I need to replace, but I have a standing desk in my sunroom.

• 3:19Daniel Whitenack?

  Oh, okay.

• 3:20Chris Benson?

  And so it's an open-air sunroom.

• 3:22Daniel Whitenack?

  For those of you that aren't from the American South, a sunroom is basically just like a room. It's like a patio on the side of your house that has like actual glass walls instead of being open.

• 3:34Chris Benson?

  I just realized that sunroom is jargon.

• 3:37Daniel Whitenack?

  Maybe so. I don't know.

• 3:38Chris Benson?

  You know, I don't think of sunroom as jargon, but you know, clearly...

• 3:41Daniel Whitenack?

  Obviously, in certain parts of the country, sunrooms aren't as maybe appealing as other parts of the country.

• 3:47Chris Benson?

  I'm showing my bias right there. See, I didn't understand that. So, okay, we're good. We're good.

• 3:52Daniel Whitenack?

  Yeah.

• 3:52Chris Benson?

  And just doing work and enjoying the weather. Nice way to combine the two. I highly encourage anyone out there, if you have an outdoor area with a roof over it, put a standing desk there. You'll love it.

• 4:02Daniel Whitenack?

  Put a desk out there?

• 4:04Chris Benson?

  That's right. It's great.

• 4:05Daniel Whitenack?

  Yeah, it's a good idea. Yeah. Today's topic, I'm really excited about and the guest, we were talking before the show. This is actually a topic that's really close to my heart because as our listeners will know, my wife owns a small business and has tried to navigate, I think with a lot of success, that business through these COVID times. And of course, that involves a lot of complicated stuff, especially her business is a manufacturing business. So there were all sorts of supply chain things and cash flow things and planning things that were just really difficult. And I think daily having very interesting financial discussions with her team to try

• 4:48Daniel Whitenack?

  to make it through that time from PPP loans to all sorts of things.

• 4:53Chris Benson?

  She needs good tools, doesn't she? That's what I'm getting from that.

• 4:56Daniel Whitenack?

  She needs good tools and good ways to make sense of that financial data and plan accordingly. And I'm really excited on that topic because today we have Intuit's Director of Data Science, Nung Ho with us. Welcome, Nung.

• 5:09Speaker 00

  Hey, guys. Thank you for having me.

• 5:11Daniel Whitenack?

  Yeah. Before we get into all of that great stuff about customer experience and working with financial data and cash flow planning and all of those cool things, could you just give us a little bit of a sense of your background and how you ended up as Director of Data Science at Intuit?

• 5:28Speaker 00

  Yeah, definitely. You know, it's interesting. Seven years ago, when I joined Intuit, I came from, we would then call a non-traditional background. So if you look at my background, I actually have a PhD in astrophysics. And before I started into it, I was doing a lot of data reduction, image processing, signal processing, and studying dark matter dominated dwarf galaxies. Right? A whole complete other world away from like what I do now in applied AI. But what was interesting was that at that time, right, the field was still fairly nascent. And a lot of us who came from non-traditional backgrounds kind of came uh over into data science and AI as a field.

• 6:09Speaker 00

  Because we realized, like, holy heck, we actually were doing, using the same techniques, we just had completely different names for these techniques in our respective field.

• 6:18Daniel Whitenack?

  That's so true, right? I remember, so I came from physics too, and I was like, had this experience where it was like, I was looking at all these data science methods. And I was like, oh, like, when you're saying you're doing this, you're just using ordinarily squared. Like, how many times have I done that in my life? Like, a million times? And like, you're just using a different word for it. That's kind of weird, but okay, I'll do that.

• 6:41Speaker 00

  Yeah, exactly.

• 6:42Chris Benson?

  What is it with physicists? I'm, I'm telling you, I think that there are more people in data science and AI from physics than any other field that transfers over. You guys are ruling the world here in that way.

• 6:53Daniel Whitenack?

  Yeah, well, I think also physicists have this, whether it's um, and maybe Nong, you have a perspective on this, but I feel like in physics, for whatever reason, it breeds the sense of like, I'm not qualified to operate in this area, but I do feel like I have the expertise to work in this area. So I'm gonna go ahead and jump into it. I don't know, Nong, if you get that same sense?

• 7:15Speaker 00

  Oh, yeah, for sure. I mean, there, there was, um, you know, kind of like figuring out the language of data science and how to translate the skills that you have and the things that you know into this like new industry and this new accepted skill. But just knowing that, hey, you know, I write a lot of code, I know a lot of math, I do a lot of matrix math i- in my daily life, uh, previously. That seems like these are all the things that we do in data science, so let's just go for it. And uh, exactly as you said, Chris, right? Like, uh, now if you look at kind of the, the composition of the backgrounds of folks who are coming into data science, it's all over the place, um, a- and the umbrella has really expanded. But when I started, I, I actually had.

• 7:57Speaker 00

  I had to go through a program to help me repackage all the things I knew, like, oh, ordinarily squared. Oh, actually, like, it's regularization and regression. So just being able to speak that language to convince folks that, hey, we had the skills. Right? Because um, uh, in the past, you know, I would write these like massive programs to take raw data from telescopes and then actually convert them into uh, meaningful information and then write papers. And a lot of like the day-to-day work is exactly the same. So that's kind of a bit, you know, my journey towards data science. And I specifically uh, wanted to come to Intuit because when I was in grad school, I used Mint a lot.

• 8:39Speaker 00

  Right? Like, every dollar counts and you have to kind of be, you know, pretty obsessed with it. But I had a really poor user experience with Mint. And I'm like, you know, I'm pretty sure I can do better than what exists right now. Uh, and um, and so when I, when I came for my interviews and they said, hey, you know, you could totally work on it, you can come in and fix this. Like, yes, please, sign me up. And it was just, you know, like, this openness to, you know, allowing us to innovate. And I, I found it kind of refreshing that I told my boss, hey, I actually hate one of your products. Can I come and make it better? He's like, yes.

• 9:17Daniel Whitenack?

  That's great. That's great to have that environment where you can sort of be comfortable sharing those things and brainstorming solutions. Of course, I, I mean, there's always boundaries within a workplace, but it's really cool to hear that you found that environment.

• 9:30Speaker 00

  Yeah, and, and you know, it's, it's been great ever since. So as I mentioned, I've, I've been at Intuit for seven years. And one of the things that, you know, I specialize in and my team specialize in is um, we specialize on the applied AI side. So building a lot of uh, production models, features and systems that go directly to our end customer. Um, cause that's something that I'm personally super passionate about. I really like to, you know, take a look at some of the core problems that our customers have right now and figure out, well, how do I bring my skill set in helping solve that for you and make it easier for you? Um, because I, I, you know, previously was a customer and I think, uh, similar, uh, to you,

• 10:12Speaker 00

  Daniel, is that I, I have 10 siblings and half of them own small businesses. So, you know, it really is like a space that really speaks to me. And, you know, it's really fun to be able to actually build things that you can see your family using. Uh, I can really see that benefit.

• 10:29Chris Benson?

  I just want you to know you have my sympathies. I, I thought I have four siblings, you know, there's five of us and I thought we were, were alive. So yeah, lots of respect there.

• 10:38Daniel Whitenack?

  Do your siblings own businesses, Chris?

• 10:40Chris Benson?

  Uh, actually, yes, yes. Uh, several of them do.

• 10:43Daniel Whitenack?

  You can relate in that sense, I guess. Yeah.

• 10:45Chris Benson?

  Yeah, yeah, I can. But she's done way better than I have. I have a bigger family and they're all doing that. Keep going. I'm sorry. I didn't mean to interrupt.

• 10:52Daniel Whitenack?

  Yeah.

• 10:52Speaker 00

  No, I think that most people are like, how do you deal with 10 siblings? It's, um, you know, at any one time you, uh, you have a high probability of liking at least one of them, right?

• 11:02Chris Benson?

  There you go. That's a good attitude.

• 11:05Daniel Whitenack?

  And I think also like the small business owners, they, they have to, like we were talking about in physics, like this sort of comfort level with like operating in a space that you're not necessarily qualified to operate in and just like relying on the skills that you have in your past and picking up what you need going forward. I feel like small business ownership is, is very similar. It's like, oh, I've, you know, I've never cleared a container from China through customs with, you know, my raw materials in it. It's like, there's some information on the internet about that, but I'm going to figure it out. And maybe some of that, I, you're going to talk about the cash flow of things and other

• 11:45Daniel Whitenack?

  things, I think, um, on the financial side. But a lot of that too, it's like, maybe in, you know, I've learned a lot of terms from, from my wife, which I'm not a financial person, but she learned, you know, how to do like profit and loss statement and all of those things in her school, which is great. But there's all of these other like things and like, for example, forecasting and, you know, forecasting sales and raw material need and all that. That's like, it is a very data science problem. It has to be data driven. And, you know, if you get it wrong, then your business goes under. So it's very high stakes too. But, uh, maybe before we jump into some of like the individual tools that you've worked on, Nung, one of the things that you mentioned,

• 12:31Daniel Whitenack?

  In terms of the sort of general space that you're working in within Intuit is customer experience and AI for customer experience. So when you're thinking about customer experience at Intuit, what what does that mean? Maybe for data science people out there that are trying to understand, what does it mean to do data science for customer experience? How do you look at that?

• 12:52Speaker 00

  Yeah, that's a great question. The way that, you know, we as a company approach is we have a system called Design for Delight and Customer Driven Innovation. And the approach is to really get into the minds of our customers and understand what are the most important problems that they have. Because you can only build great solutions if you really understand what the problem is, and not just like what it is that you want to build, but why it's actually important. And so we use this, the system called Design for Delight, where we do a lot of customer interviews, you work with a customer experience researcher, and you know, you just kind of see how your customers are using your products. Where are they getting stuck? Right? What is confusing for them? Where are the tasks that, you know, they're

• 13:35Speaker 00

  spending a lot of time on that they wish they were not doing? And then there's also the other piece, which is what did they wish that you actually had that would make their lives a lot easier? And these are things that, you know, like, so there are some traditional methods where you can infer it by looking how a customer uses the product. But honestly, it's just faster to talk to a wide variety of customers and have them tell you directly. So, starting from that base where you really understand what the problem is, that then we take it back and we say, okay, what are some of the hypotheses that we have in creating the solution? And the solution could be done by data science, it could be done via an experience design, it could be done via some very simple rules, right? But what is the

• 14:19Speaker 00

  solution that you want to build to solve that problem? And then you go into the mode of decomposing it into, okay, like, there's the experience piece, there's the machine learning piece, there's the engineering piece. Then we go off with these hypotheses, you know, build a couple of quick MVPs, Minimum Viable Products, for our listeners who are new. Once we have these MVPs, the thing that I learned, which actually took me a little bit to appreciate, is that, you know, when you build an MVP for a customer and you show it to them, you're like, tell me how you would use this. Would this be beneficial to you? In the past, we would put in mock data and then, you know, the customer would say, yeah,

• 15:01Speaker 00

  you know, if this was my data, I would react this way. I would see this. And it was actually not useful at all because you're asking them to imagine what they would do. And so we pivoted and moving forward from about three or four years ago, when we found this insight, we actually just loaded the customer's data in. Uh, we ran it through our models. We made the predictions, uh, whatever. And then we showed it to the customer. And then immediately, like the feedback that we got back was so much richer. It actually helped us like narrow in on the right solution. And, and it was just like, you know, there's like aha moment for me that I, I guess it's obvious now, but was not obvious back then.

• 15:42Chris Benson?

  I love the, uh, as you've gone through, I've had like three or four questions and you keep answering them before I ever get it. No, no, no, that's good. That's not a bad thing. I'm really curious. I was gonna ask you about kind of like, how do you know when you get to the quality of those inputs? So that it can translate to data and you were kind of saying, well, now we've, we're just going and taking the data. What is it about when you said, I took the data, can you dive into that for a moment in terms of saying, how do you know that you've just ended up with the right data set to innovate in such a way? That your customers are going to be delighted at the end of the process? Uh, how do you recognize that you've achieved that moment?

• 16:18Speaker 00

  Yeah, it's a little bit of an art, but I would say that, you know, as a scientist, one of the things that we do is we always have a couple of hypotheses and a couple of approaches on how to solve the problem. And so at any one time, we would have multiple solutions. And we would beforehand identify what the success criteria is first that we want. Then we show it to the customer and we compare them. So this is like a super lean, very baby early A B test. Before we go full into A B testing. What I find that gets in the way of this is a lot of times folks build models, but they don't think about, well, what is the success criteria for me? What am I trying to get the user to do?

• 16:59Speaker 00

  Like if, if I talk to, you know, our, our marketing teams, it's like, we want to get a user to convert. But that's like maybe like 50 clicks before you even get there. But what is the thing that I'm going to like actually physically affect and ask the customer to do? That's my target. And so like having that very robust target and measuring it across your different hypotheses and experiments is one way to do it, but it can be a little bit challenging to come up with that.

• 17:27Chris Benson?

  I can imagine. I've I know that I have participated in similar processes before and speaking from experience, not as much as you have, it was really hard to do. And so I'm very impressed with that.

• 17:39Daniel Whitenack?

  Yeah, I'm curious. One perspective on this is that like you look at the customers and this customer experience survey, and then you identify where they're getting stuck, like you said. What if it's like scenarios where maybe it's something that you need to develop that's outside, like the customer doesn't know that they need this yet, or it's outside of their scope of maybe, you know, what they even have imagined, like, oh, I didn't imagine that I could have this. But how do you come up with a way to validate that those are sort of good ideas, the ones that might not be driven by the specific customer pain that you've observed in an interview,

• 18:22Daniel Whitenack?

  but maybe sort of new, new data product development is a good way to put it.

• 18:26Speaker 00

  Yeah, that one, it's very interesting because, you know, for me, I tend to be fairly customer back. And so I go back to the roots of like, well, why is this thing a pain and not focusing so much on the what? I think a lot of times folks stop at the what, like, what is it that the customer want? It's really more about the why piece, because, you know, otherwise you can't innovate, right? You can't wait for someone to say, I really want this thing. They might not know that they actually want that thing. And so getting down to the why and then working with the teams on coming up with, okay, well, what are some ways that we could actually solve this in experiences that are new? In my team, we do a mix of replacing really, you know, rote

• 19:11Speaker 00

  existing things and backing it with machine learning. And in other instances, we're building tools that just, you know, the customers didn't even think they wanted at all. One example of that is like in the cash flow space overall, like when you think about cash flow, for me, naively, I initially said money in, money out, delta between money and money out. That's what I care about. And then you talk to a small business and they're like, well, actually, you know, I have like invoices, I have bills, I have, you know, loans that I need to pay, I have assets. And I care about all of those things and I need to know where they all are. But when we were talking about our, to our small businesses, we said, well, okay,

• 19:55Speaker 00

  like you wanna know your invoices, you know, what are outstanding invoices so that you can get paid? They said, yeah. We pushed and we said, well, what if we can tell you not only when an invoice would get paid, but like the date range within which it would get paid? Would that be something, you know, you would want? And they were like, you can do that? I didn't know that was possible at all, right? For them, they were really thinking about, just send me a reminder if it's gonna be due soon. But we expanded then, we said, we could actually build this new capability that allows you to go much further and have more control. And so I think that that's like kind of one example of starting with a why, which is like, they need to understand their cash flow.

• 20:45Speaker 01

  This episode is brought to you by our friends at RudderStack. And we're calling all data engineers to check out RudderStack Cloud and start building smart customer data pipelines. RudderStack is warehouse first, no more silos. RudderStack builds your customer data lake on your data warehouse, not theirs, enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API first. RudderStack can be easily integrated into your existing development processes. And because they're open source, you can see all their code, so you don't have to worry about vendor lock-in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. RudderStack is free up to 500,000 events and pricing scales transparently from there.

• 21:29Speaker 01

  Learn more and get started at rudderstack.com. Again, rudderstack.com. That's R U D D E R S T A C K dot com.

• 21:51Chris Benson?

  So I'd really like to dive back into that even more because I'm very intrigued by you, you know, knowing the data that you have and figuring out, how do you start that process? Because you know, we talk to people all the time that are doing all sorts of cool things with data, but I'm really fascinated by your creative process that's kind of encompassing that. And so when you're thinking about, I'm going to innovate for the customer in a way they have no idea that I can, how do you approach that? How do you know what you're going to do with the data that you have? Or know that you need to go get that data in order to achieve that? And using data science as a mechanism to get there? How do you put it together? Before you do the do, how do you conceive of it?

• 22:34Speaker 00

  Before I do the do? I like that.

• 22:36Chris Benson?

  Yeah. Sorry.

• 22:39Speaker 00

  That's something that I think a lot of us in the field are still working through because every new problem is a little bit different. Some are much more straightforward, right? You know exactly which data source you're going to get. You know exactly where to start and you even have an idea of what the initial features are that you want to start with. For others, you have to get a little bit creative, right? And trawl through the data a little bit. One of the ways that I've tried to fast track this is that, you know, as data scientists, we don't work in silos. A lot of times we work with our analytics teams, we work with our product partners and we work with our design teams to jam together and say, okay, these are some, you know, pie in the sky ideas that we want to work on that could really solve a big customer problem.

• 23:24Speaker 00

  Once we narrow in on that, we actually source it from the broader team because they have kind of like an initial starting point. And you know, for me, I would rather not do a random walkthrough parameter space to figure out where to start. It's easier to just crowdsource it. From the team.

• 23:40Chris Benson?

  Yeah.

• 23:41Speaker 00

  And so, um, sometimes it's actually from our engineering partners who say, hmm, you know, I'm aware of this data that we're capturing right now. Do you think that that would be useful for you? We go, aha, okay, actually, never thought of that. But let's go and try it out.

• 23:54Chris Benson?

  So it's almost kind of taken the creativity of the entire organization, the entire larger team, and then figuring out where to allocate resources against that creativity based on what what's rising to the top?

• 24:05Speaker 00

  That's right. That's right. Um, and so when we start at the problem first, and then we figure out whether or not we have the data. Cause I have never worked on a project where all the data we ever needed was available. Sometimes you actually need to go and capture that data. You need to go and buy that data. Right? Having everybody in there together kind of opens up the door and more avenues for you to kind of see what you're not seeing right now.

• 24:28Daniel Whitenack?

  I'd really like to dive into this idea. There's an article that we'll, um, link in our show notes about some of the things that you've been doing during the pandemic time. And I'd love to dive a little bit into those. One of the intriguing things that I kind of noted was that pandemic hit. And then of course, as any business, including Intuit, it's like, there's this kind of natural um, inclination to be hesitant to do a lot of different things because, you know, we have no idea what's going to happen next. We're really unsure about the future, but it seems like, you know, your team and Intuit really took the vision that like, hey, now is a time to innovate.

• 25:11Daniel Whitenack?

  And now is a time to try something different. It seems like that really paid off well, but I I'm wondering if you could go through that mindset and how you kind of came to be motivated that this sort of cash flow tool and things were things that you should be focusing on during this time during the pandemic.

• 25:28Speaker 00

  Yeah, then, you know, a year ago, like the the world kind of went upside down, right?

• 25:35Chris Benson?

  Yeah, I remember that. Yeah.

• 25:38Speaker 00

  And I think we all kind of had plans for ourselves and businesses had plans for what growth was going to look like. You know, for us, we had a roadmap of what we were going to build for our customers, assuming that the growth was going to continue. And then kind of overnight, you know, when the shutdown happened, like you can look at it in our data for some of our small businesses, like revenue dropped ninety nine percent. And like you, you don't see step functions like that in the data ever. So this was just a global worldwide massive event that everybody was feeling simultaneously. And through every single new locale or state where a lockdown occurred, this drop happened.

• 26:18Speaker 00

  Okay. And we said, this, like, we have to do something for our small businesses because this not only is such a scary time for them, but it's also a like huge time of uncertainty for everyone across the world.

• 26:34Chris Benson?

  And your data is sort of a history book for a monumental event in world history when you think about it.

• 26:39Speaker 00

  Yeah.

• 26:39Chris Benson?

  You know, seeing that financial data do that, that is a powerful story right there that you're describing, seeing that step function.

• 26:46Speaker 00

  Yeah, it absolutely was fairly shocking, right, to be able to see it so immediately. And it was so fast as well. And so, you know, we really came together and we said, what can we do to help our small business customers? And, you know, like, we kind of sourced across the entire company, a whole host of different solutions. So if you can recall, you know, early on in the pandemic, there were a lot of GoFundMes for small businesses. We hosted quite a few, we partner with GoFundMe to enable our small businesses to raise money. But but that's a little bit of a temporary solve, right, for that moment, because, you know, we didn't know how long the pandemic was going to last. And there's the PPP loan

• 27:28Speaker 00

  program, right? When that was announced, we realized, actually, a lot of our small businesses and a lot of customers in general, including my my sisters and family who I helped through the loan process. Eligibility was really confusing. Like, you needed to know so much more about your business than most people were equipped to be able to do at all. And so we rolled out another program using like our, you know, knowledge engine to build like the Ada Assist, which helps small businesses like quickly say, am I eligible for these loans? What are the loans that I'm eligible for and get that? And then, you know, for my team specifically, you know, because we're so close to the data, we looked in there and we said,

• 28:09Speaker 00

  wow, like cash flow right now, the historical data that we have about your cash flow is wrong. Like those of us in this field, like the historical data is like where you build your models off of. And if you have like this, this drastic shift and this drastic change, right, like you need to go in and make huge modifications to your model to accommodate for that. And not just that, like the level of granularity of details that you provide to customers, um, the information that they want to know is different. Some businesses did super well and some are, you know, really, really suffering.

• 28:44Chris Benson?

  When you're looking at that data and you've just, you're going through the middle of this uh enormous event, one which we're still in the tail end of, presumably, um, hopefully the tail end. And you are, are you're looking at how so many organizations are trying to cope and you have that real time data and the history developing on that. And you're, you're figuring out how your own organization is going to address that, serve its customers and deal with the same set of circumstances. Did that give you, or the company, I'll let you answer it any way you want, any sense of insight uh or revelation in terms of

• 29:24Chris Benson?

  how to approach the business of finance in a different way than you ever had before? Something that might actually be sustained past just the pandemic. So truly a lesson learned, not just a coping mechanism for the moment is, is there anything that you and your colleagues talked about, about like, now that we know this, nothing will ever be the same in how we look at this and, and we're making changes?

• 29:47Speaker 00

  Yeah, so uh I'd say that there's two parts. Um, I can speak for, uh, you know, my team and how we build models. But I'll start with how we as a company realize like, and really reinforce one of our beliefs from before the pandemic and made it much stronger. Um, is that we have our product called QuickBooks Capital and so we provide small business loans to our customers. And I would say the, the magic or secret sauce there is that we have a lot more data than a standard lender would, right? For our customers, we have all of their accounting data. And from that, we're able to actually offer loans to customers who traditionally would not be able to get loans.

• 30:28Speaker 00

  What we saw during the pandemic was that there is a large population who were underserved in general, who would not have been able to get loans, who we were able to offer PPP loans for. Uh, because we knew their data very well, we knew their risk profile, we could actually underwrite. So I think that really reaffirmed our decision that, you know, we need to continue to invest in figuring out ways to continue to serve the underbanked and underserved population. That's at the company level. And there's a lot that you can do in the data to figure out whether or not, you know, you're capturing the population or you're actually doing the right thing there. But, uh, you know, for, for, on, on the other side, the, the more technical side, uh, with our teams is that we've seen just,

• 31:12Speaker 00

  you know, with these large worldwide events, right? Like, um, I like to point back to two thousand eight where you think it's like a once in a generation event. And then like ten years later, you have another once in a generation event. And so, you know, th- these events, like you can't write them off as one time anomalies. You actually have to figure out how to fold that in, uh, and be able to essentially like automatically regress against them, learn from them, and then figure out kind of how to apply a forward. Right? Like you've said, we're at the tail end of the pandemic right now. Uh, hopefully, fingers crossed.

• 31:48Chris Benson?

  Hopefully.

• 31:48Speaker 00

  Right?

• 31:49Chris Benson?

  Yep.

• 31:49Speaker 00

  But, um, what happens if, you know, another large worldwide event happens ten years from now? We can't just clip that chunk of data, remove it and act as if it never existed. Right? And so you need to build that into, um, the methodology of your models.

• 32:04Chris Benson?

  Real quick, and, and Daniel, I, I saw you about to say something. I just wanted to say something before you left that point. In the different industry that I'm on in defense, I noticed it because it's directly to your point there. The intelligence community, which is the, the industry I'm in, has just released its annual assessment of world events. And one of the big things that goes directly to your point is the fact that they're talking about with the pandemic over, there will still be fallout politically and economically, uh, in various jobs. And such for years to come. So the outcome of today will lead to other events that are unforeseen, some of which may be significant. I just wanted to note that that's a great point and it really spans across industries and different contexts. And Daniel, I apologize for cutting you. I saw you just about to speak.

• 32:49Daniel Whitenack?

  Oh, I forgive you, Chris. I'll always forgive you. So you were talking about like this idea that models that you build need to be robust against these sort of once in a generation events that you can sort of no longer expect that this is a once in a generation, but you have to be ready for them. Again, we'll link to this blog post in our show notes, but I'm sort of looking at the cash flow planner and maybe you can talk a little bit about the goals there. But one of the things I see is like developing a personalized forecast. And when I think about forecasting, like you were talking about, it's like, oh, well, how much good historical data do you have to develop that

• 33:32Daniel Whitenack?

  forecast? So how is it even possible to develop a sort of robust forecasting system that's robust against these sort of, you know, really crazy events that we're dealing with?

• 33:46Speaker 00

  Well, I can tell you it's dang hard. That's the short answer. One of the things that we got creative with in the forecasting example is, well, let's take sales as one type of input forecast. Right? Traditionally, you take your sales in the past and then you can project that forward standard time series forecasting. What we were seeing during the pandemic is that there are actually a lot of customers who have extremely sparse data because what used to be daily sales are now turning into maybe once a week, you're getting a sale, but you didn't have. And that was the totality of your sale. And so incorporating that in and being able to deal with sparsity of

• 34:28Speaker 00

  data is one way that we folded that in. There's also actually learnings to be had. This is where you can actually transfer the models that you build from for one business to another. There are certain businesses who essentially all of their sales and all of their events happen once during the year. Like, well, my favorite example is a Halloween store. Like, there's a very short lifetime for a Halloween store where they have nothing, they have everything, and then they have nothing. Again, for a while. And so like for those, we actually can take the models that we've built for them, learn for them and actually apply them for companies who are going through somewhat similar events right now. So that's where it's been advantageous to have that

• 35:10Speaker 00

  broad view across the industry and being able to, you know, transfer across.

• 35:26Speaker 02

  Change log plus plus is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and help sustain our production of practical AI into the future. Simply follow the change log plus plus link in your show notes or point your favorite web browser to change log.com slash plus plus. Once again, that's change log.com slash plus plus.

• 36:12Daniel Whitenack?

  So now I'm kind of interested to, I guess, get a little bit more into the weeds of this response to the these changing times in your data and your forecasting. And we don't have time to get into all of the specifics, but if you take and you look at, you know, some of the trends out there in AI right now, in terms of, well, we've got all sorts of things. We've got, of course, the trend to use, you know, neural network based models. We've got sort of semi supervised things. We've got reinforcement learning and plus this sort of, you know, quote, more traditional type machine learning models or statistical models. As you're thinking and you're looking about this scenario where like looking to the forward, we don't know.

• 36:57Daniel Whitenack?

  What sort of events are gonna come up? And we need to be able to, you know, adapt our models as our data increasingly changes over and over. So if you sort of match that landscape of AI techniques up with, you know, this problem that you're looking at, which of those sort of become most relevant? And have you experimented with a variety of those? Or what have you found most useful in your context?

• 37:22Speaker 00

  Yeah, that's a great question. We've actually experimented with quite a bit. It wasn't until very recently that deep learning kind of beat your traditional time series methodology in terms of performance. In the space that, you know, myself and my team work in, one of the considerations that we have to make is model performance and efficacy compared to, in our case, explainability and being able to actually help our customers understand why we're making these decisions. And that's specifically because we're dealing with financial data, and it can have very large implications if you don't really trust the output of the recommendations we provide for you.

• 38:06Speaker 00

  And so, you know, in most cases where it's not as customer-facing and the decision is not directly to your dollar, right, we use, you know, more black box techniques, as I call them. But for this case specifically, for time series forecasting, we tend to stick with models where we can explain why these decisions are made. Simply because like for our customers, like you have to be able to build trust. They won't use it unless they know what goes into it. I'm sure, you know, your wife may have comments and feedback about this, Daniel. But when you're dealing with money, right? You have to be fairly sensitive and really understand what it is, the tolerance level from the customer.

• 38:49Speaker 00

  In terms of like how far you can go and how black boxy you can go.

• 38:52Chris Benson?

  How might that evolve over time? And I'm asking a speculative question and I recognize that. But there's a lot of concern about anything that's sensitive. I'm in a sensitive industry. You're in a sensitive industry dealing with people's money and their finances. And as these technologies evolve, both on the capability side and the the need for explainability, A for the purpose of satisfying regulatory concerns, for the purposes of gaining trust so that you can continue to innovate the way that you and your team have been doing. How do you envision, you know, when you're thinking about it? How do you envision explainability evolving and and the level of importance? Do you think we're just going to reach a level and it kind of plateaus?

• 39:34Chris Benson?

  Or how does that weave its way through the industry over the years ahead in the way that you're thinking?

• 39:40Speaker 00

  Yeah, I mean, I think like we are early now, especially in our space in fintech, is just getting customers comfortable with having an AI system work on their behalf and making decisions on their behalf on a daily basis. But if you look at, you know, some of the other areas where, you know, AI has kind of had a bit of a head start, people get comfortable real fast, right? And it's about

• 40:06Chris Benson?

  This is a good point.

• 40:07Speaker 00

  It's about building trust. And once they trust in the decision making power of the system, what I've seen is that they tend to need to know less about how it works because there's that like large baseline in time that they've had to build trust in the system.

• 40:22Chris Benson?

  Yeah, it's a great insight.

• 40:23Speaker 00

  Like I expect, you know, in the near future that we'll be able to use much more sophisticated techniques because we've built up that trust. But, you know, trust takes time and it's very easy to violate that trust as well. So you have to be fairly careful and make sure like you bring your customers along. But, you know, sometimes also you just need to Push it through because, you know, folks don't like change, right? So that there's that balance.

• 40:49Chris Benson?

  As a quick follow up, you know, when you just talking about folks liking change, you know, you've made these innovations that you've talked about through the COVID period because your customers really needed you to do that and stuff. And I guess, you know, there's that old saying about don't let a good crisis go to waste and all. And I realized that the sentiment on that is probably not appropriate for the COVID time. And I'm trying to acknowledge that. But at the same time, it's during times of change that great things can really come about and happen. As you're thinking about that and knowing that that is necessary and the explainability can be achieved to some degree, gain that trust and do that. How might that is you, you have amazing tools going forward in the years ahead that are being developed.

• 41:32Chris Benson?

  Deep learning goes farther, reinforcement, other branches that we don't yet know about that could come about at some point from current research. How might you approach some of those in terms of, you know, how do you make that judgment on now's the time to push? Because they really need it and I can really help them and I know that they'll trust it quickly versus being more conservative and saying, you know, I'm afraid to. How do you assess that?

• 41:55Speaker 00

  Yeah, I would say just A B test it, right? A small A B test.

• 41:59Chris Benson?

  Oh, good point.

• 42:00Speaker 00

  Costs very little, right? And there are certain portions of the population who are okay with it, right? So doing a random test with a small population on a very bold idea that pushes it far ahead, you might get actually really, really great results. And then you can kind of step your way up into opening up more is how I would, you know, think about approaching it because like I can tell you like this population we serve for some of our products with accountants, they're not necessarily, you know, the most open to trying out new technologies at all. But, you know, if you go through just with an A B test and test without them really knowing it, right? Then you really get that like a true reaction.

• 42:41Speaker 00

  And then you can actually start like bringing the people along as you have that data and that evidence.

• 42:46Daniel Whitenack?

  Gotcha. Chris knows I'm always the one like asking super selfish, practical questions because I always struggle to do practical things myself. And I want to make sure, I know there's people, listeners out there that are all the time thinking about practical data science things. And I noticed in reading about some of your work, you're talking about having, you know, individual models for different transaction streams and combining, you know, aggregating those together. And I'm just thinking as like in a practical sense, you've got, you know, models per transaction stream per user and you know, how many users does this QuickBooks have?

• 43:27Daniel Whitenack?

  My question, I guess, is like, when you're dealing with this many models and this many predictions, do you have any sort of practical advice for in terms of model management? And like model debugging and that sort of things, you know, I think a lot of people can sometimes get something into production, but then managing it after that, especially when you're managing it for multiple customers, becomes really difficult. So any, any practical advice on that front?

• 43:58Speaker 00

  Yeah. Oh, man. I, the, the operational component of, you know, the work that we do cannot be understated. Right? Because it's not just about building the model. It's actually taking to production, dealing with the ETL, grabbing the data, serving the customers. So you need to really make sure that you have fairly robust monitoring in so that you can be proactively alerted if something goes wrong, because the worst case scenario would be a customer contacting you and saying, hey, you totally messed up my cash flow. Now I can't pay a vendor this month, right? That that would be disastrous. And so, uh, building and, um, monitoring.

• 44:41Speaker 00

  Uh but you you can't, you know, feasibly monitor for every event for everything. And so doing a sampling and monitoring along the way. I will also just put in a plug, which is that when you're building, um for us, we have, you know, three million customers in the U S alone for QuickBooks and then multiple transaction stream grabbing in all that data. We quickly get into the realm of, you know, like over ten million uh models uh being built and served. Start in batch mode, right? There's this like obsession with building models in real time for everything. Like every single team I talk to, they're like, when am I gonna get the model in real time? Well, let's start with batch, right? The complexity for a batch model is just much easier to handle.

• 45:22Speaker 00

  You validate your hypothesis, you validate the use case, and then you can say, you know, is the ROI enough? If we move it to real time, that's one area. There's also the thing that we've made an investment in that has really helped is uh investing in a robust machine learning platform that allows us to do distributed training and distributed scoring. So marrying, you know, the best of um uh essentially the map reduce framework, right? Uh and doing that distributed computing with our models. And the cash flow is not the only use case where we have individual models for users. For a transaction categorization, um which is where any transaction comes in, we figure out which accounting books it goes into.

• 46:05Speaker 00

  Uh we have individual models serving in real time for customers in that case as well. But, you know, you also have to figure out like when is the scenario where you actually wanna get to that level of complexity. So for us, we always start with one base vanilla model for everyone uh to start with. And then see how far we get, and then we figure out, okay, how do we blow this up in terms of scale and complexity? And how much headroom is there in order to solve the problem well?

• 46:31Chris Benson?

  Gotcha. You said something as you were going through there that grabbed my attention as well as you were going, and that was you talked about platform. And you can answer this one any way that you want, because not every company wants to talk about the details of their infrastructure and stuff like that. And I totally get that. But how do you approach platform? Because that is one of those practical questions for practical AI that everybody has to contend with. And we often ask guests how their organization is tackling that. So I'd love to hear how Intuit is approaching a data science and AI platform and what that means to you, because it means different things to different people.

• 47:08Speaker 00

  Yeah, so I've been at Intuit for long enough where I've lived in the world before we invested in a platform and the world after we invested in a platform. And I can tell you that, like, you know, for us, a platform should enable us to be able to ship machine learning models that are high quality, that have scale built in faster. So those are essentially kind of like the requirements of what a great platform would be able to allow you to do, essentially. So our platform solves for being able to deploy models quickly, train models in a consistent way, and then monitor the performance of those models when they go out the door.

• 47:49Speaker 00

  And you can find this, you know, we have a close collaboration with AWS. And underneath, we use SageMaker as one of the core components of our platforms. Different companies have a build versus buy. In our case, we partner specifically for this. But, you know, for me, like, as a data scientist, like, what I want out of a platform is to make the job of actually the maintenance, the monitoring, the scaling easy, so I can focus on the algorithms. That's, you know, what we've optimized for.

• 48:22Chris Benson?

  Awesome. So at the beginning of our conversation, you mentioned how

• 48:28Daniel Whitenack?

  Your engagement with Intuit really started from a felt need that you had and a problem that you saw and you brought that problem to Intuit. As we kind of close out here, I'm wondering, as you look to the future, what are those things in your mind like, oh, I really like, I see this problem and I really think AI or data science has a great solution for it, I just haven't, haven't done it yet? Or maybe it's things out there that you just like, you're really excited to, to try, but you haven't tried yet. When you're looking to the future, what kind of keeps you up at night and runs through your mind?

• 49:03Speaker 00

  Yeah, man, I am constantly chasing, you know, for me, one of the, the holy grail problems and like, I would be happy to retire if we could solve this, which is actually, how do we, you know, go from the state right now where if you have to actually run your books, right, every month, if you're a small business. Right now, a lot of small businesses are doing it manually. They're entering things by hand. They're like figuring out the complexity of accounting, like, as we talked about early on, like, a lot of small businesses don't go into the business of small business thinking about books, right? They think about, wow, I really want to do this thing.

• 49:43Speaker 00

  They don't, it's not, no one gets joy from doing accounting, to say, at least, right? But it's a really difficult problem, right? To go from a raw transaction data to a state where we can actually tell you, this is what your books are like, this is your cash flow, this, this is how much taxes are owed, and this is how you should run your payroll. So for me, like, I'm always chasing after, how could we actually fully automate the entire accounting process for our customers so that they can actually spend time doing the things they like? And we can have the machine learning solve this problem. And it's it's really difficult because it's so multifaceted.

• 50:24Speaker 00

  And um, and that's where, you know, in the future, we're really gonna be, you know, innovating quite a bit, um, both on the algorithmic side. Um, but also on uh, potentially making fairly major changes to our product. To get, you know, folks comfortable with this. So I'm super jazzed about that. I know not every kid grows up saying, you know, I want to be an accountant. And I I want to build machine learning for accounting. It's such like an old problem, but it's for me, it's a super exciting problem.

• 50:53Chris Benson?

  You know, it's fascinating to hear you say that because as as you say that, I'm thinking back to what you were talking about customer experience. And and and holding that as high as you do, there is a self-awareness in you in you and your organization. When you recognize that while your product is doing great things for someone's business, that's not what they want to be thinking about. That's what they want you to handle. So you're basically saying the highest thing is for them not to have to worry about us at all. We're just there doing the magic, but they never have to deal with us. And that is almost like perfect customer experience when you think about it because you're optimizing their own experience. So they can do the thing they love. I love hearing that answer.

• 51:32Daniel Whitenack?

  I've learned a lot in this and I think that really appreciate the the insights. And thank you so much, Nung, for joining us. As I mentioned, I'll point our listeners to some links in their the description. So make sure you check those things out and uh, check out what uh, Intuit is doing in this space. But really appreciate you taking time. It's great to talk to you, Nung.

• 51:52Speaker 00

  Yeah, thank you for having me, guys.

• 51:58Speaker 02

  Thank you for listening to Practical AI. We appreciate your time and your attention. Follow the show on Apple Podcasts, Spotify, or your favorite podcast app. Your neural networks will thank you. We are also on the web at practicalai.fm. There you'll find recommended episodes, listener favorites, and a free sign up to join the community. Practical AI is hosted by Chris Benson and Daniel Whitenack. It's produced by Jared Santo with music by Breakmaster Cylinder. Thanks again to our sponsors, Fastly, Linode, and Launch Darkly. That's our show. We hope you enjoyed it and we'll talk to you again next week.

• 52:56Speaker 05

  Game on!