When data leakage turns into a flood of trouble

Rajiv Shah teaches Daniel and Chris about data leakage, and its major impact upon machine learning models. It’s the kind of topic that we don’t often think about, but which can ruin our results. Raj discusses how to use activation maps and image embedding to find leakage, so that leaking information in our test set does not find its way into our training set. Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean’s developer cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, team accounts, worldwide availability with a 99.99% uptime SLA, and 24/7/365 world-class support to back that up. Get your $100 credit at do.co/changelog https://do.co/changelog. Changelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Rajiv Shah – Website https://www.rajivshah.com, GitHub https://github.com/rajshah4, LinkedIn https://www.linkedin.com/in/rajistics, X https://x.com/rajisticsChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Rajiv Shah | University of Illinois at Chicago https://comm.uic.edu/profiles/rajiv-shahRajiv Shah | DataRobot Blog https://www.datarobot.com/blog/author/rajiv-shahDataRobot https://www.datarobot.comUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Cool things on Twitter, like cool, like little visualization blog posts to kind of play around with. I think there's, there's a ton of good stuff out there that I kind of like playing around with. Transformers and NLP is an easy one. In visualizations, you see things like Deck GL that you can just make the coolest kind of visualization pieces around. There's a lot of cool stuff there, but I will tell you, part of it is, is I always tell people that are getting into data science. Don't get suckered in by that. Most of the data science that you could use, that's useful in an enterprise, hasn't come out in the last three years. It's been around for much longer.

• 0:30Chris Benson?

  That's a really good point.

• 0:31Speaker 00

  Don't get into reading the latest archive posts and trying to take those account because going back to those classic problems, those classic either Kaggle competitions or other projects, and learning those techniques is going to get you much further along than following the latest pieces.

• 0:46Chris Benson?

  Don't start with GPT-3 as your very first thing.

• 0:52Speaker 02

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at ChangeLog because of Rollbar. Check them out at rollbar.com and we're hosted on Linode cloud servers. Head to linode.com/changelog.

• 1:09Speaker 03

  Whether you're working on a personal project or managing enterprise infrastructure, you deserve simple, affordable and accessible cloud computing solutions. So you can take your project to the next level. Simplify your life with Linode's Linux VMs to develop, deploy and scale your applications faster and easier. Get started on Linode today with $100 in free credit for our listeners. You can find all the details at linode.com/changelog. Or if you're not at your desk, just text CHANGELOG to 474747 and get instant access to that $100. Linode has 11 global data centers and provides 24/7/365 human support with no tiers or handoffs, regardless of your plan size. In addition to shared and dedicated compute instances,

• 1:54Speaker 03

  you can use that $100 credit on S3 compatible object storage, manage Kubernetes and more. Visit linode.com/changelog and click on the Create Free Account button to get started, or just text CHANGELOG to 474747. Get started today on Linode.

• 2:19Speaker 03

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We're at Practical AI FM.

• 2:45Chris Benson?

  Welcome to another episode of the Practical AI Podcast. I'm Chris Benson, a Principal Emerging Technology Strategist at Lockheed Martin. And with me as always is Daniel Whitenack, who's a data scientist with S I L International. Hey, how's it going today, Daniel?

• 2:58Daniel Whitenack?

  It's, uh, it's going really good. It's a beautiful day here in Indiana. And, uh, this weekend, I know in the last couple of times we've, we've started out the podcast, I've had a few projects, like, you know, tech projects, like the network at my wife's business and those things.

• 3:16Chris Benson?

  You always got something going on.

• 3:17Daniel Whitenack?

  I always got some sort of tech, but this weekend was like, I was outside most of the weekend. So took a hike and then, uh, had a bonfire at my wife's family's place and, uh, birthday party. And so we were outside most of the, most of the weekend, which was kind of nice, nice break from being in front of a screen.

• 3:37Chris Benson?

  I'm so jealous because as we record this, we just came out of the weekend where Hurricane Delta came in off the Gulf. And so we have had torrential rains all weekend.

• 3:45Daniel Whitenack?

  Right.

• 3:46Chris Benson?

  Inside, not fun, that kind of stuff. Yeah. Not, not good. So I'm looking forward to sunny weather coming up. So.

• 3:52Daniel Whitenack?

  Yeah. I also tried out this, um, there's an app called iNaturalist. Um, and you like, take pictures of like birds or plants or other things. And there's like a whole like, uh, crowdsourced community on there that will help you like identify what you're looking at and that sort of thing. My wife and I, both in the spring and are through the summer and fall, we like to go mushroom foraging. And, um, I've tried like all sorts of different apps that try to identify, it seems like something that should be like totally doable. But there's definitely a user risk in that as well. So talking about like a classification model and the risk associated with it, that's a pretty high one.

• 4:32Chris Benson?

  Yeah, I was, I was thinking that as you were saying that I was thinking if I was doing that, I would either end up dead or I would end up with an inappropriate level of, of something I shouldn't have in my body. Definitely. So.

• 4:44Daniel Whitenack?

  Yeah. So it's cool to find this i naturals thing. It's like a citizen science sort of deal. And yeah, if you're interested in like, you know, plants and animals and, you know, fungi and that sort of thing, it was pretty cool to find that. So.

• 5:00Chris Benson?

  Joking aside, I actually do have that app. It's wonderful. I've been using it for a while. So I like it a whole lot. Yeah. For me, I'm just like, I'm hoping for good weather here. I'm excited. I got to tell you something. Tomorrow, by the time this comes out, it'll have just been in the past, but I believe a recording will be available. I'm giving the keynote at an IEEE conference tomorrow.

• 5:19Daniel Whitenack?

  Oh, cool. Congrats.

• 5:21Chris Benson?

  Yeah. It's the Digital Avionics Systems Conference and I will be doing artificial intelligence and autonomy state of the union, which is kind of ties where the entire science of autonomy and the sciences of artificial intelligence come together and where they intersect and where they don't.

• 5:37Daniel Whitenack?

  Cool. Ken, well, you know, do you know if the, that will be posted online?

• 5:41Chris Benson?

  I think it will. I'm going to check. And if so, I will certainly include it in the, in the show notes.

• 5:45Daniel Whitenack?

  Cool. Yeah. Excited to listen.

• 5:47Chris Benson?

  So we have a very exciting guest today, based on reading some of the stuff online on Twitter that we've looked at. Today we have Rajiv Shah, who is a data scientist at Data Robot. He's an AI researcher and he's also a professor at the University of Illinois at Chicago. Welcome to the show, Raj.

• 6:06Speaker 00

  Thank you very much. Thank you for having me.

• 6:08Chris Benson?

  So wondering if, before we dive in to our topic today, which is going to be all about leaking information from your training and data sets, if you could tell us a bit about your background and kind of how you got to, to this point.

• 6:24Speaker 00

  Like a lot of early folks kind of into data science, I probably have a long kind of meandering journey, but partly I started off as an engineering undergraduate, didn't want to do engineering, went off, studied law, ended up with a PhD in communications where I ended up kind of studying just that intersection between technology and people, which is really interesting and solving a lot of interesting research questions. And I used kind of various methods, both qualitative and quantitative methods, to kind of solve these problems. And, you know, this was at the time the internet was booming, so there was a lot of interesting questions around that. I did that for a while, kind of moved out of academia. I was a sysadmin for a number of years, so worked in IT. And then this data science thing came around, and I was like,

• 7:09Speaker 00

  hey, wait a minute, right? Like, I have a little bit of a background in math, and my social science background had taught me like how to ask interesting research questions and kind of probe them. And then, the nice thing is Andrew Ng made these wonderful courses, and I was able to kind of scale myself up on some of the newer algorithms and approaches that I hadn't learned. And so, that was kind of my introduction into data science, and landed my first data science job at State Farm, and worked there for a couple of years, went over to Caterpillar, worked there as a data scientist, and now I'm over at Data Robot.

• 7:40Chris Benson?

  So just out of curiosity, was that his original machine learning course by any chance?

• 7:46Speaker 00

  Well, yes, it was his original machine learning course, yes.

• 7:49Chris Benson?

  I did the same one. It was kind of the first big, major, deep learning course that he put out there, and I think it started a lot of us down this path in a professional context.

• 7:59Speaker 00

  Yeah, I can't say I finished the course, but yeah.

• 8:03Chris Benson?

  Understood. It was a hard course. I remember, it kicked my rear.

• 8:06Speaker 00

  Yeah, no, he has a great knack for explaining things, right?

• 8:09Chris Benson?

  He does.

• 8:10Speaker 00

  His side gig was starting Coursera, so you gotta like that.

• 8:13Daniel Whitenack?

  Do you find that you're sort of, like with that background, thinking about people and technology, which is, it seems super relevant to AI specifically in terms of like, you know, I'm thinking, there's a lot of interesting thought around, you know, augmenting humans with AI, but also systems like, you know, if we're talking about like active learning or something, which I think is extremely powerful, where, you know, you are sort of having a human in the loop with these systems. Has that made you think? Maybe more about those or brought a fresh perspective on those as opposed to just, you know, kind of thinking about the tech side of it.

• 8:54Speaker 00

  Yeah, so it really helps me like remember how kind of socially grounded everything is. I think one of the valuable things of where you do a PhD and you collect your own data, you end up with kind of learning some skepticism towards kind of quantitative data because you realize like how it's collected, like what was the questions asked, like how does that formed on the survey? Right? You know, how do they handle the missing all those different types of things that we often just accept when we're running algorithms? And so kind of having that understanding is really strong. The other pieces that's really kind of come around now in data sciences. I think there's a movement in data science away from just focusing on algorithms, but thinking about the entire value chain going from your raw data

• 9:37Speaker 00

  to how you finally use that model in a production setting. As well as the larger questions like how does that interact with the rest of the organization? How does that interact with the rest of society where now, right? I mean, we see now at conferences where they're being asked to say, you know, yes, you can tell us about how you solve the problem, but we also want to know what are the larger social, political, economic impacts of your research? And wanting researchers to be cognizant of that.

• 10:04Daniel Whitenack?

  Yeah, and of course, that dips definitely into, you know, you mentioned a little bit of a background in laws as well. And of course, that dips into like, you know, ethics and of course governance and all of those things which we're seeing a definite surge in. And I'm guessing that you have thoughts there with that that background. But it's really interesting to hear your perspective coming from that background. And have you found very many sort of other people in data science coming from communications or from law? You know, I hear a lot coming from, you know, maybe engineering or science or, you know, math or physics or whatever it is.

• 10:49Daniel Whitenack?

  Are people surprised when they hear about your background?

• 10:52Speaker 00

  A little bit, but I think there's a few people now kind of coming in this way, right? I mean, you had Patrick Hall on kind of a couple of weeks ago, right? Talking about, you know, what are the implications that data scientists or there are people that are owning the data science products need to think about within liabilities within an organization? So. For me, it's kind of natural as data science expands where now the decisions that we're making are really impacting organizations in society that hey, there's a lot more people that are going to be involved and kind of giving scrutiny to what's going on.

• 11:23Daniel Whitenack?

  Yeah. Well, I'm curious, maybe kind of from that perspective, how did this topic, you know, Chris mentioned that we're going to be talking about this topic a little bit today about data leakage. How did this topic come to the forefront of your mind? How did it become something that you started thinking about?

• 11:41Speaker 00

  Yeah. I mean, so target leakage is a very common problem in data science. I think most data scientists who've been out there practicing for years, they know it, they're aware of it. They know that, and just to kind of level set for everybody, what target leakage is, is when you use information from the future when you're making a prediction. It's a very common problem that can occur. And so a simple way that it might occur is you're building an HR model. You want to predict what the salary is going to be for kind of your new computer, your CS hires. And you build a model, you take on all their background, and it spits out a number and says, this is what my, the annual salary should be

• 12:23Speaker 00

  for this particular person. But now, what happens if, when you're predicting annual salary, one of the variables you used was the monthly salary, because remember, you start off with historical data from the past. And especially in an enterprise setting, it's easy. You have lots of features and variables in the data sets, you grab them, and you might have something that's entirely related to your target when you're building the model. So it's kind of leaking that future information into your model. The great thing is, is when you're building this, the model looks flawless, right? Often, when you have target leakage, the performance of your model looks really good. And it's only when you actually put it into production and you realize, wait a minute, I don't have that monthly number, right? Like it's always missing now and your model all of a sudden

• 13:08Speaker 00

  doesn't look nearly as good as when you were training it. And so this is kind of the fundamental problem and it happens in many particular ways, where essentially kind of your model cheats a little bit and kind of gives you performance that's really misleading.

• 13:23Daniel Whitenack?

  Is this something that happened to you personally as you were starting to get into data science where you had some of these experiences, maybe where you had to answer for a production model that was having issues?

• 13:37Speaker 00

  So it didn't happen to me for all the way at the level of a production, but I see it all the time. I mean, my kind of intuitive guess is like 75% of all data science models, at one point have some target leakage. Now typically, right, as you're kind of building the models, hopefully you're debugging it, trying to understand how well the model's working, right? Using your explainability and interpretability tools. And hopefully with that and some domain expertise, right? This is where as a data scientist, you really have to understand what the data elements are in your model. If you don't, right, it's very easy to fall into target leakage. But with those, usually, right, most of the time, 99 out of 100 times,

• 14:20Speaker 00

  those errors are caught hopefully before you finally get to production.

• 14:24Chris Benson?

  So what was it? Because as we're all working in this space and stuff, you've really put some thought into this and you've really noticed it. You've probably more than most have. What really oriented you on this particular issue? Because it's fascinating, but I'm not sure it's something many of us pay attention to very well upfront. And so what kind of got you focused on this in particular along this line of thought?

• 14:50Speaker 00

  So I really like data science and one of the things I like to do is often is when I see other interesting projects that are out there, I like to rerun them and just try to understand like, how did they solve this problem? What did they use? For me, I have a hands-on learning style and it's just much better to kind of be in the data and with the code rather than just reading a page of formulas. I remember years ago, Chicago was one of the leaders of open data. And they put forth a model for predicting kind of restaurant inspections. The idea was, we want to figure out and sign our inspectors to go to restaurants that likely were going to fail inspections versus ones that aren't, right? Inspectors are rare resources, we want to just allocate them efficiently.

• 15:34Speaker 00

  And I remember when I went through that model, they had a couple pieces of target leakage. One is they used weather. This is, weather is, if you're building a model that you're predicting way out in the future, often you don't actually have the actual weather, what's going to happen a month from now, for example, right? So that's a form of kind of target leakage that can happen. And they also had a more subtle thing where they were using the inspector ID information in their model. And the reason this was target leakage is because in the past, some inspectors often kind of were very easy inspectors, some were really harsh on find everybody. But when they actually were going to deploy this model, they wouldn't know what inspector was likely to go to the restaurant. And so going through this process, I realized like, wait a minute, they made a mistake.

• 16:18Speaker 00

  And I talked to the folks in Chicago and the actual model they implemented was a bit different than what they had shared publicly. But it kind of got me attuned to, you know, these types of problems happen if we kind of don't carefully scrutinize the data. And then just along the way, I see it routinely when I work with other customers, where they bring in their models, we walk through them, I'm like, wait a minute, like, this variable's really good, should this be in the model? And then somebody thinks about it and they're like, oh no, you know, you're right. Like, that shouldn't be in the model, that's leakage. You see it in Kaggle competitions, right? If you look at the history of Kaggle competitions, there's many Kaggle competitions where along the way, they discovered target leakage and they're like, oh, we have to redo the competition.

• 16:59Speaker 00

  Or the competitors all use the target information and make these unrealistically good models as part of the competition. And then the one that really highlighted it to me was, there was an article in Nature studying aftershocks, so kind of earthquakes and you have aftershocks. And the great thing was the researchers put the data out there, they put their code out there, so I was able to kind of play around with it. And once I started looking at it, there's kind of some funny things that kind of made me a little concerned. One thing is, is whenever I see a paper and I don't see a baseline model, that always concerns me, right? This is kind of for me, a data science best practice is, before we kind of jump into using fancy deep learning methods, like, what does a simple model do? Right?

• 17:44Speaker 00

  We do this if we're working, whether we're doing a time series problem, or kind of if an enterprise has a rule-based system, right? Let's find out, like, if we just do the bare minimum, like, what the starting point is. And then we can see, hey, now I'm gonna apply this fancy data science methods and see how much of an increase I can get. Does that sound fair? It does. And so that was one issue that I kind of had with the paper. The other issue when I started looking at it was how they had organized and partitioned their data. So within data science, typically, there's an assumption often that the rows of our data are not related to each other. And so what happened in this case was, some of those aftershock earthquakes actually happened near the same time, near the same place.

• 18:29Speaker 00

  And so I made the assumption, well, if you treat them like they are related to each other, we can still solve the same problem, we can still use data science, but what we use is a different partitioning method. We organize the data differently when we're training the models. Typically, kind of the default you'll see, pretty much every blog post out there goes with random partitioning. Right? All your rows of your data are the same, it doesn't matter where we put them. But the reality is, is often in a lot of problems, there's some relationships in your data. Maybe, for example, you have data and it's broken up by state. So you have a bunch of observations for Texas, and you have a bunch of observations for Florida. And what happens is, is there's some regional variations that, if you know something about,

• 19:14Speaker 00

  a little bit about Texas, you can probably make a pretty good assumption what the next person from Texas is gonna do. And so what happens then is, when we randomly distribute our, this data, and we put some of Texas data in our test, kind of where we wanna see if our data will generalize to, the model can very easily cheat. It can say, hey, this is what I know about Texas, versus all the other characteristics that we really want the model to learn from.

• 19:41Speaker 00

  And there's an easy remedy. It's something called group partitioning, where we just make sure all the Texas information stays together in one partition when we train. All the Florida's information stays together in one. It's a very simple thing to do. It takes a little bit of extra data science work to think about it and to kind of code it up. But then you can run it and see, hey, is there a big effect? Is there an effect of this state? You know, that is actually kind of leaking information as I'm solving it. Okay, I said a lot. I kind of want to make sure you guys are with me on this so far.

• 20:12Daniel Whitenack?

  Definitely. It was good. Thank you.

• 20:14Speaker 00

  You know, and that's what brought me back in this aftershocks one is that's what I noticed is that there was some type of kind of leakage going on. And for me, the, the big issue isn't that there's leakage. I mean, like I said, it happens all the time. Most data science teams have it. And I hopefully, I mean, most data science teams have confronted leakage. I think most of us who have experienced data science, it's not a big deal. It happens all the time. Right? It's nothing to be embarrassed about. You fix it, you move on and you've learned, you write you just another kind of data science battle scar that you have and you've learned from that. But my concern in this one case was, you know, this got to an article that was published. When I brought up the issue, it wasn't like we were on the same data science team and like, oh, shoot, let me fix that.

• 20:57Speaker 00

  Let me rerun the results and see, is it going to make a difference or not? But there was quite a bit of pushback. And so for me, this brings up kind of the larger concern about as we welcome more people into using our data science tools and techniques, which I'm all for, right? I work for an auto ML company, right? The goal is to kind of make data science available to a much larger set of people. But how at the same time can we balance and make sure that folks kind of learn some of these fundamental concepts and best practices for how to do data science and solve these problems?

• 21:29Daniel Whitenack?

  So I'm curious with this particular problem, it seems at least in some cases, like one of those things like. You may not be able to deduce, or it may not be obvious to you upfront when you're first creating a model and trying to solve a problem, where you might be having data leakage. It's sort of like you see some weird behavior afterwards, or you see like degradations in your production model, or you see some maybe suspicious evaluation results, and that might trigger you to kind of dig in deeper. And we can dig into those here in a second, like kind of detecting where you're leaking. But are there ways upfront as you're getting into a problem

• 22:13Daniel Whitenack?

  that you could kind of set yourself up for success, you know, before you have to kind of retrospectively debug where you have leakage?

• 22:22Speaker 00

  No, absolutely. And I think part of it is just that good problem framing upfront. Data scientists, I think, are getting much better about this, but thinking about what is the problem? What are you trying to solve? How are you going to implement this at the end of the day? Right? What is the data looking like? What are the production systems that are coming in? Right? So a common problem that often happens is data gets updated in databases all the time. And if you're not using kind of snapshots, your model could fail to that. You could fail because of that. So let me kind of explain that. So imagine you're doing a model on claims, and you're using the text of the claims. Well, if the text of the claims is constantly being updated during the lifecycle of the claims from when it was taken to when the final process was,

• 23:07Speaker 00

  well, that information is changing over time. So hopefully, you have that conversation when you're setting up the model and the project and being like, hey, is this data that I'm using at training time, are the fields going to be the same as when I actually go and set up my production pipelines at the time of prediction? So I think a lot of this can be addressed by just thinking through and having the domain experts and knowing what the data looks like at the time of prediction. Does that help?

• 23:34Daniel Whitenack?

  Yeah, it does. I'm wondering, like, how do you connect this also? Like, is this connected to, like, you were talking about partitioning your data? And, you know, there's a lot of talk when you first learn data science about training and tests and holdout sets and like cross-validation. And you talked about this sort of group partitioning. Is there one, or is it sort of a case-by-case basis in terms of, like, every project you do, you should consider what unique partitioning is? A lot of data scientists sort of just use one-size-fits-all, basically. Like, cross-validation is easy because I can just write this, like, one-liner in my code

• 24:18Daniel Whitenack?

  and then, like, I'm set for not overfitting, right? So, do you have any thoughts on that?

• 24:25Chris Benson?

  And you might want to define a couple of those things along the way, by the way, for listeners who may not be familiar with everything.

• 24:31Speaker 00

  Yeah, no, so that's great. And it kind of gets to, right, are there some good best practices I could use for avoiding target leakage, right? And one of them is, what is my default partitioning scheme, right? When I get the data, right, we talked about the issues with the random. How can I do it? Well, I think, kind of, the general best practice for data science is to use a technique called nested cross-validation, where what you do is, when you partition your data, you're making sure that you kind of break this up into, I don't know if I can sufficiently define nested cross, into kind of different folds. But what we want to do is make sure there's different data that we're using for both the validation, as well as the hyperparameter tuning of the models. That's another subtle thing that can come into play.

• 25:17Speaker 00

  So, this is where, kind of, I, for folks that aren't new, I just have them set, right, nested cross-validation is the default. It's a pain in the butt to code up, but it's kind of gets you in the best place. But, kind of like we said with the aftershocks, that's not 100% bulletproof. I mean, this is where you still want to have folks that understand a little bit about data science and have been burned, and are just very skeptical. I think one of the takeaways is always to be skeptical of your model and its performance, and think about what else could it be that's going wrong? Because... Otherwise, it's going to happen when you go to move it to production.

• 25:52Chris Benson?

  Absolutely. And as you were talking about that, I was also thinking, like, on the aside, you know, at least you have a lot of the control as you're busy doing the, you know, planning out the model and figuring out your training data and you can find those things and correct them. When you're dealing with the production environment or maybe planning at this stage for the production environment, how can you apply this line of thinking to that effectively, especially if maybe you don't have control of that, you know, exclusively? You may be working with, you know, infrastructure people, database people, all sorts of different ties. Even if you've recognized it for training, how do you get it out into the real world and accommodate these same issues?

• 26:33Speaker 00

  I've seen far too many models that data scientists have made that haven't accounted for all the production issues that end up just sitting on laptops that never actually get put into production. So even if it's a pain to deal with those IT folks on that and figuring out those pipelines, what data they have, getting snapshots of real production data, you got to do it. It's the only way to actually get your model in there. I realize, right? Lots of data scientists would rather just sit on their laptop, be handed some data and just kind of plug away at it. But if your goal is to actually bring value to the organization, to get models implemented into production, like, you got to have those conversations and figure that stuff out. And it's much better to do it earlier before you have, you know,

• 27:18Speaker 00

  you or your team spending time kind of looking at these models than doing it. So.

• 27:23Daniel Whitenack?

  So let's say that I've tried at least to, you know, use some reasonable partitioning scheme. I've tried to do the best that I could, but I put my model into production and I detect that there's something suspicious going on. How would you go about sort of retroactively looking into this issue of leakage? Like, it could, it could be other things, right? There could sort of be like model drift or something where like, you know, the state of your data was one way and then like, you know, something fundamentally changed about, you know, the data or the problem or the time period or whatever, and you've got some

• 28:04Daniel Whitenack?

  sort of drift of the performance. But it could also be some, you know, like this, like leakage issue or something. So how do you kind of go about that when you have to do this more on the debugging side?

• 28:15Speaker 00

  So one thing is, I really like that you're talking about monitoring your models and thinking about the data drift and concept drift, because I think that's an important element, right? Those things could happen and it could be entirely separate from target leakage. That could be affecting your model. So, yes, who knows for kind of thinking and doing that. And then, yes, then we have to start kind of taking this apart and deciphering and seeing, has the data changed from the time I trained the model to when it's in production? Is that's what's causing kind of this lack of performance? Or going back and looking at those features? So I have a longer talk that I have on target leakage that I did with a colleague of mine, Yuri, who's spoken at Open Data Science Conference. And he has a kind of different categories of target leakage.

• 29:00Speaker 00

  So like, we've talked about data partitioning as one of them. We've talked a little bit about kind of that initial set of data is another source of leakage. Thinking of that example of using that correlated feature or overwriting information is often another piece there. Another example of when that can happen is if you have, for example, image data and it has some type of labels inside it, like using the explainability tools, interpretability tools, hopefully then you can figure that out and go back and then fix that data for that. But two other places that it often happens is feature engineering, that this is one, it's a very subtle way, but when we kind of engineer and build those features that we're talking about decoding and deciphering.

• 29:45Speaker 00

  Often, people will make the mistake of doing it and creating this feature engineering on their entire data set, right? People will do EDA on the entire data set and that allows you to learn the ins and outs of your entire training data sets. But that also subtly leaks information now because you don't have that holdout data set that's really kind of partitioned off from you where you haven't seen it from. And so that's a real subtle one that you won't be able to catch until you really think about how they actually did the feature engineering and they're like, oh, shoot.

• 30:15Daniel Whitenack?

  Yeah, so, yeah, I think that's a really important point that you bring up and something that really is not stressed at all. So like, I think if I'm getting what you're saying, like, let's say that you have a data set, you have a feature and you want to know, like, oh, is this normally distributed? Do I have to apply some sort of power rule to this to make, you know, fit the assumptions of a model? Or like, what, so you could do those things and look at nice histograms or plots on your entire data set, right? So like, you get your whole, like, customer transaction data. You do that, but then you kind of partition off and you train your model on only part of that data.

• 30:56Daniel Whitenack?

  But the, the way in which you've created those models or the transformations you've done on those features is informed by data that you're not using in training.

• 31:07Speaker 00

  Exactly, yeah.

• 31:09Daniel Whitenack?

  So I guess, I mean, one thing you can do, of course, is, you know, when you're preparing, you can adjust your exploratory data analysis methodologies. But yeah, I could see how that one would be a really hard one to catch because it also would require, I guess, like, really good documentation around how you did that feature engineering initially, which I know maybe we're, at least myself, not always the best at documenting things the way I should. How does that fit into this and the teams you've worked on in terms of maybe not so much the data side of things, but the process side of things? What are some of your best practices around how you document your, your process on projects?

• 31:54Speaker 00

  Yeah, no, I mean, this is a very tough one to get because, so I think one of the, the initial Twitter thread that we were kind of talking about a month or two ago that led to this podcast was a thread by kind of Jacob Schreiber that looked at a machine learning package in genomics that it was doing this exact same thing, was using all the training data to get the insights. And then later, yes, they used cross validation, but they'd already kind of been kind of corrupted because they'd already learned everything from the training data. So solving this problem i- is very tricky here because it relies on your data scientist being very aware, aware of these issues and not doing it. Because I think, right, decoding it from somebody's code.

• 32:36Speaker 00

  Is a bit tricky to kind of go back and see how they engineered their code. Did they do the proper splits? If we start thinking about things like time series where you start having lagged features, it gets really tricky to be able to diagnose and look at somebody's.

• 32:51Chris Benson?

  It's a hard one.

• 32:53Speaker 00

  Yeah, I don't have a very good answer for you on the best process for it because.

• 32:57Chris Benson?

  Well, I have another way of going about it, which I kind of put you on the spot for a second and you can make it up because I literally was wondering if I'm trying to tie everything that we've covered together because there's a bit of complexity to it and I'm trying to make it easy to understand for my own selfish purposes so that as we get past the episode, I can start applying. Can you kind of think through a use case and it can be fictional or it can draw from something that you've done, but kind of take us from the beginning to the end a little bit in your thinking and say, I'm doing this now and then I'm doing that. It can be anything. There's no wrong answer or you know, it's whatever you want, but but kind of sequentially take us through so that as we come out of that myself and maybe other listeners that are interested can kind of go, okay, I get that.

• 33:42Chris Benson?

  I can, I can literally turn to my work after this podcast and go apply that.

• 33:46Speaker 00

  So I'm not sure I could do it quickly in 30 seconds. I will say we have built out a webinar that goes into detail on target leakage as well as there's an online course available that. So for me, this target leakage is really a fundamental concept that anybody new to machine learning really needs to kind of go through and learn all these different types of areas. So I've spent some time kind of actually taking all these different pieces together and building a little course with example problems for people to kind of go through. And that way it kind of triggers. So that way hopefully they can learn from seeing some other examples of where target leakage might occur and they don't actually have to go through the pain themselves of experiencing it.

• 34:25Chris Benson?

  Awesome. And we'll make sure and link that in our, in our show notes as well. I think that's great and something I want to go through as well.

• 34:32Speaker 00

  So yeah, I mean, there's a couple of rules of thumb, right? About just partitioning that data early, using all the interpretability and explainability tools to understand what are those factors in your model? There's having the domain knowledge, so making sure you're understanding what you're trying to predict, what are all the predictors? You know, is there anything that's kind of leaking in? Understanding your IT system, making sure that, for example, records aren't being updated, using good press practices for, you know, creating features, for training your models. We talked a little bit about nested cross-validation. So another common mistake data scientists will do is on their hyperparameter tuning.

• 35:12Speaker 00

  So algorithms, for example, have, some of them literally have tens, maybe hundreds, of different knobs and levers, hyperparameters, that we can kind of turn and modify when we're building out our models. A lot of data scientists, not all, a lot of them like to spend a ton of time, way, if you ask me, way too much time on hyperparameter tuning. But a common thing that can happen is, what you're doing is, you're testing, you're moving the knob, you're moving the switch one position, you test the model, you move it another position, you test it, and you literally do this hundreds, thousands of times. And if you're doing all of this and using the exact same dataset to test it on, what

• 35:54Speaker 00

  can happen is you can actually do what's called overfitting to your model. You can essentially kind of learn that validation dataset that you're using to test it on by testing it a thousand times. And that is another form of kind of leakage that can happen, where then you've kind of built a model that's much more optimistic, that you have much, that has memorized its testing data, and so you think it's working quite well, but actually it isn't. You just kind of got the, figured out the one lucky one that understands this testing data.

• 36:26Chris Benson?

  It's memorized rather than generalized.

• 36:28Speaker 00

  Exactly.

• 36:46Speaker 03

  ChangeLog++ is the best way for you to directly support Practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of Practical AI into the future. Simply follow the ChangeLog++ link in your show notes, or point your favorite web browser to changelog.com/plusplus. Once again, that's changelog.com/plusplus.

• 37:37Daniel Whitenack?

  So Raj, it's been great to talk a lot about the sort of practicalities of this issue. I know there's things even, you know, we've talked about that are for sure, you know, things that I want to help better integrate into our team structure as well. I was curious now, you know, that you've been with Data Robot for a while, you've been in academia, you also have a sort of unique perspective, I think, on machine learning and AI and are concerned, at the same time, very concerned with these sort of practical side of things and productizing things. What are some of the things that you're excited about exploring, you know, over the next

• 38:19Daniel Whitenack?

  year or so in terms of maybe it's new verticals that you think that AI or machine learning is really going to make an impact in, or maybe it's, you know, methodologies or even just like practical things that you're implementing that you think are going to make a big difference for you? What are some of those things, like what are you thinking about as you fall asleep in the evening if you're thinking about machine learning and AI type things?

• 38:45Speaker 00

  There's two pieces for me for this question. One is just for like the techno side of my data science of just like feeding me in terms of like the cool things on Twitter, like cool like little visualization blog posts to kind of play around with. I think there's, right, there's a ton of good stuff out there that I kind of like playing around with, right? So transformers and NLP is an easy one. In visualizations, you see things like deck GL that you can just make the coolest kind of visualization pieces around. So I think there's a lot of cool stuff there as part of it. But I will tell you, part of it is, is I always tell people that are kind of getting into data science, don't get suckered in by that, right? Most of the data science that you could use that's useful in an enterprise hasn't come

• 39:29Speaker 00

  out in the last three years, right? It's been around for much longer.

• 39:32Daniel Whitenack?

  That's a really good point.

• 39:33Speaker 00

  Don't get into reading, you know, the latest archive posts and trying to take those account, right? Because... Going back to those kind of classic problems, those classic either Kaggle competitions or other projects and learning those techniques is going to get you much further along than kind of following the latest pieces.

• 39:51Chris Benson?

  Don't start with GPT-3 as your very first thing.

• 39:56Daniel Whitenack?

  I kind of have a maybe a follow up to that, which is like, you know, you're talking about like those Kaggle data sets or other things. And you've emphasized previously in the conversation, like one of the like really important pieces of this whole puzzle in terms of creating value is like understanding infrastructure, understanding like production systems. What do you recommend to like, because sometimes I talk to new people getting into data science about this very thing, right? Like, is it more useful for me to learn this sort of state of the art models? Or what should I learn? What do you tell people that are sort of just getting into the field in terms of how

• 40:37Daniel Whitenack?

  heavy they need to, you know, jump into things like databases, like, you know, DevOps and CICD, like, you know, those things that are kind of common software engineering world things? How do you kind of prep people that you talk to?

• 40:56Speaker 00

  It's tough, right? Because the expectation now for data scientists is that they have to know it all, right? That you, especially if you're going out on the job market, if you don't know all the pieces, you really feel like I'm not measuring up. But then when you go actually study and work with data scientists in an enterprise, most of them are very kind of in a very narrow piece there and they're complaining that they have like more challenges and like to do different things. When people are first getting into the field, I encourage them to know a little bit about everything. So, you know, spend a weekend just playing around with Spark so you kind of have some idea of what Spark is. Like, don't, don't spend two months on Spark, but just a weekend so you have some kind

• 41:37Speaker 00

  of ability to kind of know what the tool is and where it fits in to all of that. And kind of go down the road like that for a lot of these kind of technologies, databases, is just have a little bit of understanding. So if you're at a conference, you could have a conversation and know, you know, what are the things you'd want to go to versus not. But I think you can't go deep in all of them, right? There's just not enough time to go deep into all of them. And what I always encourage people to do is a much more project focused way of learning data science, where you solve some type of project. Hopefully it's a passion project, something you care about, but and you solve that end to end. And along the way, you're going to learn a lot of those subtle skills of right?

• 42:18Speaker 00

  How to, you know, set up a webpage or how to kind of fire up a web server in Python, whatever. But you'll pick up those skills along the way to kind of solve the problem. And then you'll really own those skills because if you solve a problem with it, you really kind of free that domain. You totally understand how to solve the problem and you can go deep with somebody on that.

• 42:38Daniel Whitenack?

  Yeah, cause if it's a like, even if it's a side project or a passion project, what you're saying, which I definitely agree, I think like if you're gonna get into this, like choose something that you're gonna want to work on because it makes it just so much easier to, to put in that, that time. But yeah, it's not gonna be like, you know, let's say that I'm creating like a special webcam or a camera, you know, detect my pets versus like, you know, raccoons or, you know, something that, you know, is like bothering my trash or like.

• 43:12Speaker 00

  You have a problem with raccoons, Daniel?

• 43:14Daniel Whitenack?

  Yeah, well, actually right now we, we have a mouse hanging around, but I don't know if I could catch that on the camera.

• 43:21Speaker 00

  Sorry, didn't mean to mess you up.

• 43:22Daniel Whitenack?

  Yeah, no, if, if you do that, like the, you know, the solution to that isn't gonna be a Jupiter notebook, right? You may want to like play around with your video and whatever in a notebook. There's, I'm not saying anything actually bad about Jupiter notebooks because I use them every day. But at the end of the day, like how you're gonna solve that, it may be specific. Maybe you want something like that you can look at on your phone, like a, a phone app, or maybe it's a webpage or a dashboard or something that you want to look at with that. And so it's, it forces you to think about some of these intricacies of like integrating a model into other things, um, which is something, I, I don't know if you agree Raj,

• 44:05Daniel Whitenack?

  but it's something I see missing in a lot of, in a lot of training is like. A lot of buildup around, even sometimes around model management, which some people are getting a little bit more into that, but not as much into sort of integration and that sort of world.

• 44:23Chris Benson?

  Yeah. And just to insert before you answer there, I mean, I think that's a surprise for a lot of data scientists is that, to your point, Daniel, is that at the end of the day, a model that's going into production is a software component that has to be deployed.

• 44:37Speaker 00

  So. No, it is a good point because, and it's been often missed in data science, where the conversation has been around algorithms and not necessarily what's going to take to kind of work with IT to get your model being used. And you're right, I mean, IT often has specific requirements that you want. And, you know, going back to your earlier thing about the Jupiter notebook, I'd say, go ahead and try running your Jupiter notebook and, you know, using your production model off that. And that way you'll learn what can go wrong.

• 45:06Chris Benson?

  Why it doesn't.

• 45:07Speaker 00

  Right? Sometimes the best way to learn is to kind of fall down a little bit.

• 45:11Chris Benson?

  True.

• 45:12Speaker 00

  But no, absolutely. Often I think about data science as kind of the webmasters of, right, 2020, right? When I remember there was, when the internet was coming along, there was kind of the webmasters that they did everything, right? They designed the webpage, they run the networking switch, they kind of came to your house, solved your modem issues. And data scientists often kind of are put into that same category of having to fix every problem. And like the piece there where we're talking about productionizing, this is where, right, many places now have said, hey, you know, right? If you're a data scientist, you're going to explore data, you'll be on this part of the data science team. If you're the software engineer type that is good at coding, that knows how to take that and put it into production,

• 45:54Speaker 00

  you'll be, let's say, an ML engineer or data engineer and do that. So, you know, as kind of data science matures and we get a little bit more of these specializations and hopefully boundaries around this to help kind of do that. Because you're right, we need those skills at the end of the day to kind of get your model working.

• 46:11Daniel Whitenack?

  For sure. I think it's a great message to end out with here on, because you know, as the name suggests, Chris and I are both concerned with sort of the practicalities of this and bringing a practical side to AI. So I really appreciate your perspective there and you know, perspective from really spending time building, you know, data products and productionizing models. It's really useful. We're going to link to those things that you mentioned during the show, the webinar and all of that. And so I encourage our listeners to definitely check that out. Also, check out, you know, like we mentioned, Raj and I have talked a bit on Twitter, so you can connect to him and I and Chris and the show

• 46:56Daniel Whitenack?

  there. We've got our community on Slack. If you go to changelog.com/community, you can talk about, you know, join our community and talk more about the topics there. And LinkedIn, we just had someone from LinkedIn on our show last week, and that's another place where you can find us. So make sure and connect with us. And I really appreciate you coming on, Raj. It was a great discussion and appreciate the very practical ideas that you've given me as well.

• 47:28Speaker 03

  If you enjoy Practical AI, we would enjoy a five-star review on Apple Podcasts, a blog post in response to something said on the show, and/or a recommendation to a friend or colleague. Those word-of-mouth recommendations really do make a difference. Practical AI is hosted by Chris Benson and Daniel Whitenack. It is produced by Jared Santo, with music by the mysterious Breakmaster Cylinder. Thanks again to our partners who support this show's existence. Shout out to Fastly, Linode, and Robar. That's all we have for you today. We'll talk to you again next week.