Testing ML systems

Production ML systems include more than just the model. In these complicated systems, how do you ensure quality over time, especially when you are constantly updating your infrastructure, data and models? Tania Allard joins us to discuss the ins and outs of testing ML systems. Among other things, she presents a simple formula that helps you score your progress towards a robust system and identify problem areas. Featuring: Tania Allard – Website https://www.bitsandchips.me, GitHub https://github.com/trallard, X https://x.com/ixekChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: “What’s your ML score” talk https://youtu.be/obLHa21U0qo“Jupyter Notebooks: Friends or Foes?” talk https://youtu.be/vyAG0bPJ6EsJoel Grus’s episode: “AI code that facilitates good science” https://changelog.com/practicalai/51Papermill https://papermill.readthedocs.io/en/latest/nbdev https://www.fast.ai/2019/12/02/nbdev/nbval https://github.com/computationalmodelling/nbvalBooks “DevOps For Dummies” by Emily Freeman https://www.amazon.com/dp/1119552222Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at change law because of roll bar. Check them out at rollbar.com and we're hosted on Linode cloud servers. Head to linode.com slash change log.

• 0:17Speaker 01

  Linode is our cloud server choice. Grab the Nano plan for just $5 a month, just five bucks. That gets you a gig of RAM, a blazing fast 25 gig SSD and one terabyte of transfer. Let's be honest, you can go a long ways on that five bucks. When you do need to scale up, their prices are predictable. So you can put your calculator down, you won't need it. We've been running change dot com on Linode for years and we've always impressed by their award winning support team. Check them out at linode.com slash change log. Once again, that's linode.com slash change log.

• 0:58Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show at change dot com slash community. Follow us on Twitter. We're at Practical AI FM and now into the show.

• 1:24Daniel Whitenack?

  All right. Welcome to another episode of Practical AI. This is Daniel White neck. I'm a data scientist with SIL International and I'm joined as always by my co-host Chris Benson, who is a principal AI strategist at Lockheed Martin. How's it going, Chris?

• 1:41Chris Benson?

  Hey, it's going great, Daniel. How are you?

• 1:43Daniel Whitenack?

  Doing, doing pretty good. The winter sickness is still going through our household, so still dealing with that, but, uh, but otherwise doing pretty good. I think I've avoided most of it at this point, or at least got over the worst of it. So that's good. But how are things on your end?

• 1:59Chris Benson?

  They're going well. Um, you know, I wanted to note that thing from last week about us, because I had posted on some social media and people were really surprised. So I thought I'd share it with the listeners. So if you were A long time listener of the podcast. We've been doing this for about a year and a half. And Daniel and I actually have never met in person until last week. We were at Project Voice in Chattanooga.

• 2:21Daniel Whitenack?

  The wonders of the internet.

• 2:22Chris Benson?

  Yeah, no kidding. And so I had commented online how incredibly cool it is that you can develop such a great friendship and collaboration. And yet have never met each other in person. So anyway, that's passed now. We've now met in person. It was like meeting family, I think for us. And so

• 2:38Daniel Whitenack?

  Definitely.

• 2:38Chris Benson?

  Yeah. And so anyway, just a very cool thing. I decided to relate that. So yeah, doing great.

• 2:43Daniel Whitenack?

  Yeah, awesome. We're starting out. So this is one of our first episodes for the new year, a couple before this, but definitely want to start out this year promoting practical uses of AI and practicality in developing AI and machine learning systems. And I think we've got a great guest today to emphasize a lot of those things. So we're joined by Tania Allard, who's a developer advocate with Microsoft, a Google machine learning GDE and a Python Software Foundation fellow. Thanks for joining us, Tanya.

• 3:17Speaker 04

  Hi, it's a pleasure joining you guys.

• 3:19Daniel Whitenack?

  Awesome. Well, maybe before we jump into some of the things you've been talking about and working on recently, if you could just give us a little bit of your background and how you got into machine learning and ended up at Microsoft, that would be awesome.

• 3:35Speaker 04

  Yeah, well, I started doing machine learning software during my PhD. I was using machine learning applied to material science and it was basically to try and identify some materials that could be candidates for a teacher placement. So with a lot of optimization, a lot of material assessing here and there. And over the course of my PhD, I realized that I really enjoyed the computational side of things much more. That the experimental and research bit. So after that, I transitioned into research software engineering, which was basically research engineering in Research institutions in the UK.

• 4:15Speaker 04

  I then migrated into working for HelloSoda, that is a company that has machine learning as a service. And I was doing data engineering there, machine learning engineering, research engineering. So it was pretty much everything as you normally do in a small company. And then that brought me into Microsoft because I was already doing a lot of community work, community engagement, doing GD work with the GD community in Google, all open source work that I do. And they just seem like perfect fit now for me. That's how I got into Microsoft.

• 4:52Daniel Whitenack?

  That's awesome. Yeah, and you mentioned a few things like you held different positions like data engineering and machine learning engineering and data science and computational research. Sometimes these sort of titles get a little bit fuzzy. I'm wondering from your perspective, I guess, one, how you would define what you did differently as a data engineer, maybe versus some of the more data science-y things? And then also, like, if that more focused engineering experience kind of influenced how you do machine learning?

• 5:23Speaker 04

  Yeah, right. I can say I've been working across the machine learning pipeline in all the different roles. And as you mentioned, a lot of these roles are very, very fuzzy or have a lot of things in common. So when people talk about, I don't know, data scientist and data engineering roles in machine learning, researcher or machine learning engineer, rather, they try to use these Venn diagrams. And I found that this is not very descriptive. But rather is, for example, if you're working in the data science-y thing side of the pipeline, you're focusing much more on the statistics, on developing novel algorithms or models that would help your business or your company to get more software data or you get good insights.

• 6:08Speaker 04

  But then you will probably have or need some software engineering skills as well to take that into a production format with the rest of your dev environment or your dev team.

• 6:21Speaker 04

  Whereas when you're working on the data engineering side of things, you're focusing much more on all the processes that are extraction, transform, load. But sometimes you still have to know how that data is going to be integrated into your model so that the data is actually usable by the rest of the team. And then the machine learning engineer role is basically the one that binds it all together. It makes sure that everything is robust, accessible, that can be taken into production, that folks that are using the data that is being transformed or being transformed by the data engineer is actually pulled into a reproducible manner that you can always track where the data is coming to and from at all times.

• 7:02Speaker 04

  So that everything is tightly integrated within your data science infrastructure.

• 7:08Chris Benson?

  So, you know, I was just looking, and I know Daniel had seen it already, at your talks that you gave called "What is your ML score that you went through at All Things Open and AnacondaCon?" And in that talk, you were focused very much on QA and testing of machine learning systems specifically. And I wanted to start off by just asking if, as we've kind of talked about the roles there, if you could also talk a little bit about what is a machine learning system? You know, it's kind of a little bit of an ambiguous term. And could you kind of define what that means? And what types of things are included in a machine learning system or not?

• 7:46Speaker 04

  Yeah, well, it depends a lot on what data problems your company is working on or sorry, or the kind of machine learning projects that you're normally working on. But normally, the machine learning system is going to be comprised of wherever you're getting your data, which can be a canonical database, then how the data is going to be pulled into your machine learning model or your prediction or classification model, whatever it is you're trying to do with that data. And then how you take that model into something useful by your customer or your teammates. And this can mean, are they going to access it through an API? Is this going to be a standalone web app? Is it going to be on a mobile device?

• 8:31Speaker 04

  And they're gonna be accessing this model. So all of this apparently movable parts that conform to your data warehouse or your database, the infrastructure where you're running your prediction, where they're training your model, uh, how you're collating your database, all of that forms your machine learning uh system. So it's a lot of uh data bits, a lot of infrastructure. And it could be like things on the cloud, for example, as well, like using the public cloud infrastructure.

• 9:00Daniel Whitenack?

  And uh I know like when you're talking about testing or validating the ML system, I was just thinking like if I was to ask, you know, a software engineer and ask them about testing, probably one of the things that comes to their mind is like unit testing or integration testing. Whereas if I ask a data scientist or maybe a machine learning engineer what they think about testing, probably the first thing that comes to their mind is testing of like a specific machine learning model, like the performance of that model in terms of accuracy or whatever the metric might be. And now we have this other kind of category of like machine learning system, which it sounds like is is more broad than a machine learning model.

• 9:42Daniel Whitenack?

  How is the testing or validation of an ML system or a machine learning or AI system different from the testing of a specific machine learning model?

• 9:54Speaker 04

  Right, so I think you said ML, well, the machine learning system comprises all of your pipeline. It has to be a bit more holistic, a bit more integral and cover all of the parts of that. So if we go back to traditional software engineering, your your testing that your your piece of software is returning the results that you're expecting it to do. It's because you already know what those results or those behaviors are is it's relatively straightforward to design your test cases, create your unit tests, your regression and your smoke test cases.

• 10:29Daniel Whitenack?

  And and they should always return the same thing, basically.

• 10:32Speaker 04

  Exactly, so it should be deterministic in that sense. When you're testing a machine learning system or a machine learning model, in in many cases, you don't even know what that end result is. Or because you have your your data, you have your labels, if you're doing, for example, a classification problem. But you need to make sure that your system is doing what it's meant to be doing and that it's repeatable, that you can repeat all of that. So in that sense, you have to ensure that your testing, your data, your features, ensure that the data conforms to that distribution that you're expecting and the behavior that you're gonna see. And also the cost that adding more features to your predictive models is adding is

• 11:15Speaker 04

  that's a mi- a major component, especially when you're doing things on the cloud. Sometimes when you add one feature, you marginally increase that accuracy that you were talking about. But then your compute time or the use of resources that you're using doubles or triples. So you also have to take that into consideration to to balance whether that very marginal increase in your accuracy i- is actually worth all that extra compute cost that you're incurring on. Also, when you're doing your your model development, you have to look at things like your metrics, whether the impact of of your hyper parameters is also causing impact on on your computer resources,

• 11:58Speaker 04

  testing for implicit bias, testing for your staleness of your model, because you might then need to retrain your model after a certain period of time if you're acquiring new data or there are significant changes to the API. And then again, you need to test your infrastructure. You need to make sure that you're able to deploy them, to deploy your models, your infrastructure, probably using techniques like continuous integration and continuous slavery. Because that's essential, especially if you release a new version of your model, although just tested it before getting into production, it turns out that there is a bug or there's something that needs a rollback. Being able to know

• 12:40Speaker 04

  how long that rollback or that release is gonna take you is crucial. So in that sense, you have to have integration tests against your entire pipeline from data acquisition to data transformation, prediction, and then result serving, whatever that is for your system.

• 12:57Chris Benson?

  So one of the things I wanted to ask, you know, I know that Daniel and I come from a software development background, kind of before we were much into deep learning. And so it's kind of the idea of testing and why you test and the importance of testing is kind of second nature. But for somebody coming in to this, you know, into deep learning and trying to do these things, it may not be. So I'd like to ask you quite simply, why is testing or validation of machine learning systems so important? And what would be the downside of not doing that?

• 13:28Speaker 04

  I think one of the main advantages of you being able to test your machine learning model is explainability. As we're going into more complex frameworks or in more complex deep learning algorithms, it starts becoming increasingly difficult to explain how to reach to a certain prediction and why, especially when we're releasing machine learning out into the world and is affecting other people. I think it's crucial for us to know that it's actually predicting what we want it to predict, that it's being transparent and clear, and that we can always trace all the predictions that we're doing.

• 14:08Speaker 04

  Also, test for implicit bias is crucial, especially when we have data sets that are biased toward a certain feature or towards a certain portion of our population. Having tests in places throughout all of our pipeline ensures that we can mitigate those biases early on as well. So I think those are some of the most important reasons for us to be testing our algorithms.

• 14:32Daniel Whitenack?

  Yeah, and I mean, in certain cases, I know you're in Europe right now, and there's certain regulations that have come down the pipeline there. And of course, you're influencing the rest of the world. So I guess it may be partly your own, you know, you're trying to create a development environment that is responsible and you're able to repeat things and actually make incremental progress on things. But also, you might be under certain regulations, right? That you actually have to be able to, you know, give someone an explanation to some degree of what you did with their data. Is that right?

• 15:08Speaker 04

  That's correct. I think it was now over a year that all the things GDPR regulations took place. So one of the most significant things that this brings is that if a customer or someone from whom you're withholding any sort of data comes and tells you... A I want to see what you're doing with my data, or I want to to have access to all the data of myself that you're storing. Uh you should be able to comply within a standard period of time. So if you don't have mechanisms in place for you to trace this data, or trace what you were doing, or even to delete, because now your customers should be able to ask you to delete all the data that you have in place for them.

• 15:51Speaker 04

  You should comply. That that's now our regulation. And I think also having a better understanding, as I said before, of where your data sources are and what you're doing with the data and how you're moving it from one place to another is very, very important for for reproducibility and assurance of our systems.

• 16:11Daniel Whitenack?

  So I know people that might be coming into AI and machine learning, maybe from a science background. Or even, um, you know, there's a lot of different backgrounds, even things like economics or or finance or that sort of thing. Some of these things around infrastructure, C I C D, monitoring, might be sort of intimidating to them. Um, I was wondering if you had any, you know, thoughts as far as who a data scientist needs to work with to make sure that all the right testing is in place for a machine learning system? Because it does um impact, like you said, there's implications for infrastructure, for scaling up, for changes to data.

• 16:51Daniel Whitenack?

  Who does a data scientist need to be talking with to make sure that all the right testing and quality assurance pieces are in place?

• 16:58Speaker 04

  I think a good workflow would be to have uh the data science team working very, very closely with the machine learning engineering team, if there is one, or otherwise the software engineering team. If you have them sitting together or working very closely, it's easier for both teams or the three teams to better understand what the requirements are, how people are bringing things from research and development environments into production. Because something that I've noticed in some companies or in some teams is that you have the R and D or the machine learners sitting in a corner doing all of their things, developing their models, and then

• 17:38Speaker 04

  they have to throw things over a wall and and hope that the software engineer will take that into production. In most cases, the software engineer doesn't have an idea on how the model works, the canonical database or the canonical data sources and the transformations that need to take place for that data to be usable. And that's when you sometimes see that folks spend weeks or months working on a model, but then they spend another couple of months or weeks sitting on that model just waiting for it to be taken into production. Having from day one a collaborative approach where folks define what resources are you going to need, how this algorithm is going to reach out to their customers and what

• 18:23Speaker 04

  sort of data is going to be accessing to is crucial for both teams to be able to take this from R and D into production in a seamless way. That doesn't mean that you as a data scientist need to do everything or need to be super good at CI and CD and testing and know Kubernetes and all of these complex things. But if you have these teams together, it's easier for both to understand the world of the other world.

• 18:51Chris Benson?

  Yep. So, you know, one of the things I'm really getting from you here is that when you're actually working on getting a model into your overall production lifecycle and you're integrating with the existing software development and deployment life cycles that you are moving into, it's really part of a larger effort, which kind of fits into what a lot of organizations are already doing as they kind of add this in. One of the things I'd like to know is, given that kind of larger team that we've been talking about, how should different roles within that team think about their responsibility for testing? In other words, if you are an infrastructure engineer, what should you be testing? If you're a data scientist, what should you be testing? What should the machine learning engineer be testing?

• 19:35Chris Benson?

  How should testing be divided out among those different roles?

• 19:39Speaker 04

  I think, as you mentioned it, finding these roles and assigning responsibilities is crucial. You as a data scientist, you would mainly be in charge of assessing your data and your features. So again, this goes back to when you have your dataset, make sure that the distribution of each feature match your expectations. This is a very basic sanity check, but sometimes, because we do it so, so often, we don't think about documenting it or going in depth into that. Also, making sure that the relationship between your features and your targets and the correlations make sense. And that sometimes it needs to go beyond creating just a correlation heat map as well. As I mentioned before, testing the concept of your features.

• 20:24Speaker 04

  Also, something that is very, very important and aligns very well with this GDPR. Also ensuring that your system maintains privacy across entire pipelines. Sometimes we are very, very concerned about privacy of our raw data, because that's our most valuable asset. But you need to ensure that your system or every transformation or every data manipulation that you're doing complies with that privacy as well. Also, make sure that you are aware of how much time it's taking you to develop a new feature or a new production model, because that's also going to help a lot with time churn, which also will prevent you from going into half baked features

• 21:07Speaker 04

  or having very, very tight data jungles as well. If you are, for example, the machine learning engineer, you're going to be testing for model development practices and monitoring those models. So again, making sure that everything is checked into our repository, that there is version control, making sure that there is a peer review process, that it not only has to be the data, the senior data scientist, but everyone in the team has to be responsible for that. Making sure again that you're checking your impact metrics, checking the impact of your tunable hyperparameters. And also check against.

• 21:49Speaker 04

  Uh simpler models, sometimes because we are so into deep learning and deep learning is the most popular framework. Or the most popular approach at the moment, or and we have to, we want to use very sophisticated models. Sometimes it's also good to go back to the basics and compare against a much more simple or a simpler model. Just to have a baseline and and ensure that we're actually going down the right path. And that it makes sense, the additional cost that we're going into compute. And again, test for your implicit bias. And finally, the the fault that is in charge of our infrastructure, sometimes it's a DevOps person.

• 22:28Speaker 04

  Check for the reproducibility of training, making sure that the model specification is up to date and and correct. And we have properly versioned all of our data, all of our hyperparameters, all of our models, the training and everything. And integrate the full, test for the integration of the full machine learning pipeline. Make making sure that again, everything is reproducible, whether your infrastructure. Sometimes you will have infrastructure that is um your development, your staging and your production. Making sure that across those three environments, you can get reproducible results.

• 23:08Speaker 04

  Sometimes you'll have changes in infrastructure that will imply changes in your predictions. That gives you indication that your infrastructure is not reproducible. So you'll have to to make sure that that's not the case as well. And again, as I said before, test that you can do releases and rollbacks in in a reproducible, reliable and robust manner. Because if it's only one person, for example, in charge of deploying things into production. And what's going to happen when that person is on holidays, for example, you're not going to be calling them at three o'clock in the morning. For them to revert back to a previous version. So make sure that there is a robust way to do that without having a bottleneck.

• 23:51Speaker 04

  There is also um very, very important and ensuring that this rollback can be done safely. You know, in a controlled manner, so that anyone can do that. Ideally, you would have an automated pipeline that would take care of that. So testing all of these little bits will be assigned within the specific role responsibilities. And it makes everyone's lives easier as well.

• 24:14Daniel Whitenack?

  Yeah, and I'm so glad you went into the details of those different areas. It's really helpful and there's definitely a lot there to work on. There's, there's a lot probably for every team, whether a small organization or a large organization that they can always be improving on. I know a lot of the things that you talked about, I know I have work to do on my end. But one of the things I liked about the talk that you gave on this was that you developed a sort of very practical, I guess I would call it a rubric for kind of scoring yourself in these different areas, which is what you meant by giving yourself an ML score and then kind of helping you focus your effort on where you're lacking and where you can improve on.

• 24:56Daniel Whitenack?

  I was wondering if you could go into a little bit of the details of that scoring system and that rubric to help me develop my own ML score and make sure I'm putting in effort where I can make the most difference.

• 25:08Speaker 04

  Sure. Throughout this conversation, I've placed a specific emphasis in data science testing, machine learning, engineering and infrastructure testing. So within these three areas, you have the different steps that I've already mentioned, the things that you should be testing at a minimum. So for example, if you have no testing in place for any of these steps, that would give you zero points because you have no testing. If you're testing, let's say from the data science perspective and you're manually testing the distributions or checking the distributions of your data and then ensuring that the training of your model is reproducible, but all of this is done manually, you can assign yourself probably one point.

• 25:49Speaker 04

  If you or your company has reached a more mature level where all of these tests are done in an automated fashion, probably through your favorite CI provider, it can be GitHub Actions, Travis, Azure pipelines or GitLab CI, then that would give you two points. And as you go creating your tests, you can add those points and then compare against the three stages, whether it's data science, infrastructure and machine learning engineer. And more than likely is that you're going to be very good in one of these areas and not so good. You'll have a lower scores in one or two areas.

• 26:31Speaker 04

  And there where you have the lowest score is the one that you should be paying more immediate attention. Start trying to level up that score.

• 26:40Daniel Whitenack?

  Yeah, that's super helpful. I was just trying to think through while you were talking what my score would be.

• 26:45Chris Benson?

  Me too.

• 26:46Daniel Whitenack?

  I don't know, Chris, if you were doing the same. Yeah, for me, so I guess there is the manual and the automated tests, and then there's the three sections. And a data, data science, machine learning development and infrastructure. I think probably, at least in my organization, on the stuff I work directly in, I've got a fairly good amount of manual tests going, but definitely not everything's automated. And probably, we have been more focused on the data science and machine learning development side than on the infrastructure side, just because, you know, our, our organization being a nonprofit isn't already operating a ton of infrastructure and have some of those things in place, although they do for a variety of projects.

• 27:28Daniel Whitenack?

  So I'm guessing I've got like a one going on and maybe a two in a couple of the first sections. And then maybe where things are needed is more on the, on the infrastructure side. What about you, Chris?

• 27:41Chris Benson?

  I think it's more or less the same. As you were saying that, I think the things that, for me personally, and I'm part of a larger team, but the things personally, the closer we are to kind of what I grew up in, in the software development side, I think we're actually, and this is where I'm a little bit different from you, probably more automated and actually attending to tests on the infrastructure side. And as you move over toward the data science, it's probably, you know, pretty decent on manual tests, not, probably not a lot of automated tests. And so I think it varies a lot for us, whether it is a team effort. And I think we probably get a little bit higher score because different people are attending to different parts of it versus when me or, or somebody else is doing something alone.

• 28:22Chris Benson?

  And I'd say our scores probably fell off. So I think it's definitely probably a testament to throwing people from different perspectives on the team probably yields us a higher score since we're combining that. But boy, I'll tell you, after talking through this and then hearing Tanya talk about her scoring, I'm starting to realize all the places that I need a little bit of work.

• 28:40Daniel Whitenack?

  I think that's cool though, because it's like. You know, if you were to just present all this, like what Tanya presented in general, like all the pieces of the machine learning system, it could be overwhelming and a bit crippling. But if you're able to kind of zero in on where you need to put the most effort, I think it's really helpful in terms of, you know, starting somewhere and at least getting some more testing off the ground. I'm curious, Tanya, as you've kind of gone around and presented this to various groups, what's been the feedback that you've got in terms of where people, is there a consistent place that you think data science teams or like machine learning teams are maybe not putting a lot of effort where they need to? Has there been any sort of trends in that sense?

• 29:25Speaker 04

  I think something that has come up a lot is infrastructure, like in the sense of they, they have a dev person that normally takes care of the infrastructure and, but everything is very flat in the sense of they always have the same test, they always run the same processes without adapting to the specific cases or specific situations for machine learning. Because sometimes we would have a bit of different behaviors or we would need something a bit different if you are serving, for example, a model that is going to be accessed by a lot of people over a web app, for example, or a more simple ecommerce app that folks are going to be accessing to buy products or the such. So I think the

• 30:10Speaker 04

  understanding and the testing of machine learning infrastructure is very often overseen by a lot of teams.

• 30:22Speaker 00

  Have you heard of our newest show called Brain Science? Yes, Brain Science. It's a different kind of show, I know, and it's probably one of the ones that reaches the furthest out from our typical listener audience. But this podcast is what we call for the curious. And what's cool about this show is we're exploring the inner workings of the human brain to understand things like behavior change, habit formation, mental health, and pretty much what it means to be human. If you've ever thought about why you do what you do. Or why others do what they do, then this show is for you. At the changelog.com/brain science to listen, subscribe and learn more about this awesome show. Here's a preview of a recent episode called "One Small Act of Kindness," talking about

• 31:07Speaker 00

  empathy and mirror neurons. So it sounds like pliability and flexibility is a pretty crucial role too in relationships because if you're not flexible, bendable, pliable, whatever, however you want to phrase that. If you're rigid, right? That's only gonna, that's only gonna, it's gonna be difficult for you to flex.

• 31:26Speaker 05

  Right.

• 31:27Speaker 00

  To enable change or to, what you've said before, recalculate.

• 31:31Speaker 05

  Yeah.

• 31:31Speaker 00

  You know, accept new data, make, you know, analyze that data, make a new plan and iterate towards a new action.

• 31:39Speaker 05

  Yeah. And so one of the other things involved with this flexibility would be what researchers have discovered as mirror neurons. And so mirror neurons are these neurons within the brain that help us sort of get access to another person's emotional experience. And so there's an action component in it that it was first discovered actually with monkeys and the sort of mimicry that occurred by watching somebody else do an action. Well, in the same way, I can sort of watch somebody else walk through something in terms of an emotional experience. And if I'm holding space for them in my mind, like my body physiologically, these mirror

• 32:24Speaker 05

  neurons come, come to play.

• 32:26Speaker 00

  Is that why people cry when they watch movies or certain movies because their mirror neurons are firing because they're watching somebody go through a situation and they're empathizing with them and can't help but encapsulate themselves into their scenario and feel what they're feeling? Is that why?

• 32:46Speaker 05

  Yes.

• 32:46Speaker 00

  Okay. So is that why anybody cries at anything when it's like, say, a movie related because they're, that's what's happening?

• 32:52Speaker 05

  Yeah, think about it, sort of like this emotional contagion, right? So.

• 32:56Speaker 00

  That's interesting to put it that way. We've said mirror neurons several times, but this emotional contagion, I believe, is actually a better subtitle for mirror neurons.

• 33:06Speaker 05

  Yeah, and so some of this emotional contagion or mirror neurons, like the research has been rooted in aspects of pain, because if I can recognize sort of the suffering of another.

• 33:18Speaker 00

  All right, to keep listening, head to changelog.com/brainscience/9. That will take you to the episode titled "One Small Act of Kindness." Marielle and I dig into this thing called empathy as a construct. We ask questions like what key brain structures are involved? How can we better understand empathy to be able to better navigate ourselves and our relationships with others, both at home and in the workplace? It's a deep subject, a very fun subject. Again, changelog.com/brainscience/9, or search for "Brain Science" on your favorite podcast app and subscribe. We'd love to have you as a listener.

• 34:10Chris Benson?

  So on a slight change of topic here, Tanya, we certainly, I think, would be remiss if we ended our discussion about testing and machine learning systems and integrity without mentioning notebooks. I understand you gave a talk that was called Jupiter Notebooks: Friends or Foes recently, and I was wondering, what was your conclusion in that talk, especially, given the emphasis in this episode on integrity and reproducibility? Could you share some of your thoughts there?

• 34:39Speaker 04

  Yeah, I've given that talk a couple of times, and it's been very well received, because Jupyter Notebooks are a tool very, very commonly used by data scientists. And I'm going to say, I love Jupyter Notebooks, but I always try to use them within reasons. And even with the teams that I work, I try to have these standards on how work them, have processes again in place. There are a lot of very, very good things of the Notebooks, but there are also a lot of hidden things and caveats. So the more aware you are of this, the better use you can make of this tool. And then again, it comes to having these processes and these workflows in place. For example, something that is relatively easy

• 35:24Speaker 04

  to do is If you have someone to help you or you spend some time is, for example, when you're working with notebooks and then you check them into version control, having, for example, a GitHub that will make sure that all the outputs are cleared out, that you are conforming to certain standards, that your paths are not referencing to local paths before those are going to be checked into version control. And then, again, having testing of your notebooks, making sure that your environment is reproducible, that makes a very, very dramatic change in how folks are using it. Because I know some software engineering folks, they absolutely

• 36:05Speaker 04

  hate it. They absolutely hate notebooks, because it also allows for a lot of bad practices in the more traditional sense of software engineering. But I think, again, if you adhere to this, you know, style guides, having, enforcing workflows that will allow for this quality assurance, this goes a long way. And also, being smart about what you're using the notebooks for and when it's good or more advisable to move from notebooks into a more traditional development practice, as having your scripts and your tests and importing your modules. And being able to discern between these two use cases, or these two different

• 36:49Speaker 04

  approaches, rather, is very, very valuable.

• 36:52Daniel Whitenack?

  So I'm curious, so I guess first, I have a couple of follow-up questions, because there's so much here. And of course, Jupyter notebooks are everywhere, so it really does influence a lot of people's workflow. I guess the first thing, so you mentioned a couple of the checks that you might do when you're checking your notebook into version control, but you also mentioned maybe some caveats where notebooks kind of can break down. I was wondering if you would go into that a little bit more. Maybe, I know, for example, one area that I've seen is where, and this is something that Joel Grus mentions, we had him on another episode, and he gave a talk as well, more controversially titled, I guess, I don't like notebooks. But he was saying, you know, there's a lot of hidden

• 37:35Daniel Whitenack?

  state in notebooks where, you know, you might run things out of order, and it's hard for another person to jump in and actually recreate that state that you might have actually been a little bit, you might have had some misunderstanding of how you got to a certain place in the notebook. So wondering if you had similar experience?

• 37:55Speaker 04

  Yeah, right. I I totally agree with him in that sense. There is a lot of hidden state. Because it gives you so, Jupiter notebook gives you so much flexibility when it comes to executing a cell and getting output straight away and then going back and forth. But I think especially for people that are getting started or have never had proper software engineering practices, this is not obvious. So sometimes they will work notebooks, jump from one cell to the another, get to our result, whatever that result is, that seems satisfactory for them and they're like, okay, I'm done. And just check or share their notebooks in the state that it is.

• 38:38Speaker 04

  But then if you go again a step further and you have these practices of, okay, I finished this notebook, so I'm then I'm gonna clear everything, restart my kernel and run all of the cells to make sure again that my results are reproducible, then you're adding this extra quality checks. There are tools, for example, like NVVal that I I love and I've worked with a lot that allows you to do this regression test. And they're very, very useful because you've already have the the state of your notebook saved, so it runs again in background to your cells and checks whether what you're getting is the same one you get,

• 39:18Speaker 04

  what you got before. And this is very, very useful for for regression test, for validation of of somebody else's work. But then again, something that is very, very obscure is the dependencies that you're using. Unless you are actively sharing your environment through pinned versions in a con environment, or you're using Pip for poetry or Docker or something of the such, it's very, very obscure. And a friend of mine has done a lot of study around how slight changes in packages version can actually change the result that you're getting in a workflow or in a study. And this is, I think, all of the the hidden state and all of the weird practices. And then

• 40:03Speaker 04

  especially when folks only learn to use, for example, Python through Jupiter notebook, through Jupyter notebooks, it becomes very problematic. Because then they're like, okay, so I normally import a library like, let's say pandas. If I develop something in a Jupiter notebook, how do I import this notebook into my notebook? So. People started misusing the notebooks, if that makes sense?

• 40:28Chris Benson?

  It does, it does. I think when, in my own thought process, I think I come from, I have this bias where I'm coming from the software development background, as I had noted before. And so I do think Jupyter Notebooks are wonderful for kind of experimenting and trying and doing your, experimenting with feature selection, stuff like that. But I also know, speaking for myself, at the first point where my model starts to stabilize a bit, and I'm doing less experimentation and variance and, you know, from a minute to minute kind of thing, that's where I'm always looking for that first moment where I say, "Okay, it's time to get it out of the notebook at this point and really start wrapping it with software development best practices," you know, as we've discussed.

• 41:10Chris Benson?

  I'm kind of curious, what is that point in your own workflow, speaking for yourself? And also, what would you definitely, as part of that, not want to see happening in a Jupiter notebook? You know, at what point would you be saying, "I've stayed in a Jupiter notebook too long," in your personal workflow?

• 41:28Speaker 04

  Yeah, I think I very much agree with what Jujie said. Jupiter notebooks are excellent for prototyping, to do very fast things, and get the initial part of the R and D process off the ground. They're amazing. Another use case that I found that is really good is parameterizing Jupiter notebooks using tools like Papermill. But then, once my model starts stabilizing and I need to start, you know, making more consistent predictions, or validations, or proper training, I try to go into a more traditional software engineer practice. There is this tool by Fast.ai called MBDev,

• 42:09Speaker 04

  where they try to bring all of this literate programming into Jupiter notebooks, so you can have your code and your tests, and then develop your library from there. I think it's good to start bringing the software engineering practices into the workflow of people, but then again, once I start finding myself that I am reusing and calling a lot of functions or methods that I've declared into a Jupyter notebook and I have to reuse it and probably for the same workflow or other workflows, that's also an indication that, okay, this doesn't working, this has to become a standalone module or standalone package that I can use and share

• 42:53Speaker 04

  and reuse and maintain separately rather than, you know, having bits and pieces in multiple code bases or multiple Jupyter notebooks and having to keep that updated if needed.

• 43:07Daniel Whitenack?

  Awesome. Yeah, I would definitely encourage everyone, all the listeners, to go and listen to your talks, both on the ML Scoring talk, but also this Jupiter talk. This is really useful and very practical thing. So I would definitely encourage people to go there. Also, thank you for mentioning fast.ai. We actually mentioned them, you know, a good bit on the podcast because they contribute so much. So yeah, I think it's a testament to what they're doing that they're mentioned so much. I was wondering kind of to close us out here. So we talked a lot about, you know, maybe some things that certain people, certain listeners, especially who maybe are newer to the software engineering side of things, might be a little bit intimidated by, whether

• 43:51Daniel Whitenack?

  that's like, you know, Python project structure and tests and automation and all of this stuff. I was wondering, because you're also, you know, in the role as a Python Software Foundation Fellow, do you have any good recommendations for maybe people that are wanting to level up their ability to code good Python that has a lot of integrity and integrate tests and, you know, Python deployments and all those things? Do you have any good suggestions for people in terms of learning resources and ways for them to level up in that sense?

• 44:29Speaker 04

  Yeah, well, there are a ton of resources out there. That is the problem, that it's so easy again to

• 44:35Daniel Whitenack?

  There's so much.

• 44:36Speaker 04

  fall into this rabbit hole, right? There is, if, for example, if someone wants to get into more DevOps kind of thing, Emily Friedman, that is one of my colleagues in my team, she just released a book called DevOps for Dummies. It's not focused on machine learning. It's a very, very good resource to get you into that DevOps-y mindset, not understanding how you would integrate continuous integration and delivery into your projects. And then you can interpolate some of those things into your own data science stuff. Something else that I recommend is, I've talked a lot about collaborations across teams and team members. Sometimes just sitting down with the software engineer and have conversations

• 45:21Speaker 04

  or pure programming sessions where you both sit and start writing tests or just discussing about continuous iteration, continuous testing, continuous deployment, testing of your programs and your models goes a long way. Because again, you're learning from the other one and getting things off the ground.

• 45:43Daniel Whitenack?

  Awesome. Yeah, those are great suggestions. I know for me personally, I was a little bit nervous to sit down and pair program or be next to some software engineers when I was first getting started out of grad school. But that's probably one of the ways that I learn the most, the fastest. And so I can definitely recommend that. And hopefully you've got some good engineers on your team that are receptive to that. Well, Tanya, it's been super instructive and really great to talk with you today. We'll have links to the various talks and the other things that we discussed in our show notes. And really, really appreciate you taking time out of your busy schedule to talk through some of these things with us.

• 46:24Daniel Whitenack?

  And I hope we can meet at a conference or somewhere in the near future. Thank you so much.

• 46:30Speaker 04

  Thank you for having me. It's been a pleasure.

• 46:35Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for change log is provided by Fastly. Learn more at fastly.com. And we catch our errors before users do hear change law because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode Cloud Servers. Head to linode.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address. Get our weekly email, keeping you up to date with the news

• 47:20Speaker 00

  and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.