Model inspection and interpretation at Seldon

Interpreting complicated models is a hot topic. How can we trust and manage AI models that we can’t explain? In this episode, Janis Klaise, a data scientist with Seldon, joins us to talk about model interpretation and Seldon’s new open source project called Alibi https://github.com/SeldonIO/alibi. Janis also gives some of his thoughts on production ML/AI and how Seldon addresses related problems. Sponsors: DigitalOcean https://do.co/changelog – Check out DigitalOcean’s dedicated vCPU Droplets with dedicated vCPU threads. Get started for free with a $50 credit. Learn more at do.co/changelog https://do.co/changelog. DataEngPodcast https://www.dataengineeringpodcast.com?utm_source=changelog&utm_medium=podcast&utm_campaign=practicalai – A podcast about data engineering and modern data infrastructure. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Janis Klaise – GitHub https://github.com/jklaise, LinkedIn https://www.linkedin.com/in/jklaise, X https://x.com/jklaiseChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Seldon https://www.seldon.io/Seldon Core https://github.com/SeldonIO/seldon-coreAlibi https://github.com/SeldonIO/alibiBooks “The Foundation Series” by Isaac Asimov https://www.amazon.com/dp/B01EFDEMS8“Interpretable Machine Learning” by Christoph Molnar https://christophm.github.io/interpretable-ml-book/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at ChangeLaw because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com/changelog.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, C I C D, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front-end web servers that need to be full duty CPU all day, every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co/changelog. Again, do.co/changelog.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slay with us around various topics of the show at changelog.com/community. Follow us on Twitter, we're at Practical AI FM. And now into the show.

• 1:47Daniel Whitenack?

  Welcome to Practical AI. I am Daniel Whiteneck, data scientist with S I L International. And I'm joined by my co-host, Chris Benson, who is a chief strategist for AI and high-performance computing at Lockheed Martin. Hey, Chris, how you doing?

• 2:03Speaker 00

  I'm doing fine. How's it going, Daniel?

• 2:04Daniel Whitenack?

  It's going good. No complaints. It's been kind of gloomy weather here for a while in Indiana, but you know, such is this time of year, I guess.

• 2:14Chris Benson?

  Sounds good. I'm about to head out of town for Live Works in Boston. We're going to give a talk and it'll be over by the time this episode comes out, but right now I'm looking forward to heading off and doing that and meeting a bunch of people up there.

• 2:27Daniel Whitenack?

  Awesome. Well, I'm super excited about our guests and our topic today. As our listeners know, we have a particular passion for the practical side of AI, hence the name of the podcast. We realize that, you know, a lot of times the blockers for AI projects are not necessarily the sophisticated modeling, but the whole productionizing things and operationalizing things. And one of the companies that's really leading the way in this area, at least I feel like, is a company called Seldon. And today we have Yannis Kleis, who's a data scientist at Seldon, joining us. Welcome, Yannis.

• 3:06Speaker 03

  Hi, I'm very pleased to be on the show.

• 3:08Daniel Whitenack?

  Awesome. Well, we're really happy to have you. Maybe we could start things off by just hearing a little bit about your background and how you eventually got into AI and data science things.

• 3:22Speaker 03

  Yeah, sure thing. So my background is actually in mathematical modeling. So I did my PhD at the University of Warwick here in the UK. And there's a doctoral training center for complexity science, which is basically applied maths to real world systems. And with a particular interest on sort of systems that are made up of simple rules, but that can result in interesting emergent behavior. So just to give you a couple of examples, so people in my cohort worked on various problems, sort of ranging from traffic modeling

• 4:04Speaker 03

  to modeling the spread of infectious disease. One of my papers was also roughly related to that kind of thing. So that's my background.

• 4:14Daniel Whitenack?

  Awesome. And did that sort of academic training, did that transfer naturally into kind of industrial data science work? I know a lot of people, including myself, maybe had some kind of awkwardness or weirdness and kind of trying to transition from academic science into data science and industry. So it sounds like your work maybe was more applied, though.

• 4:39Speaker 03

  Um, yeah, I I can I can definitely relate with what you're saying because because the incentives in academia and in the industry are are completely different. Uh, but they the great thing about my department was that it was very multidisciplinary and uh people were working on all kinds of things. So uh I mentioned things like uh epidemiology research, uh but also there was a small group of people doing machine learning research with also some industry applications. So that this was a smaller uh portion of the of the center because we didn't have as many uh staff members uh doing research in machine learning.

• 5:21Speaker 03

  But but there were some good ones and towards the end of my PhD, I I started taking more interest into machine learning and started going to uh paper clubs and um be more interested in uh student talks about about the topic. And as I was looking for a job in the industry, uh and preferably a modeling job, I I realized that this is this is the best way of going about it and I I should really uh really pick pick up the subject in my spare time if I can.

• 5:55Chris Benson?

  So uh so Yanis, uh I was wondering, uh as we were prepping for uh for getting online and talking to you about this, um is by chance uh the company named Seldon in any way related to the Harry Seldon figure in the Foundation series?

• 6:10Speaker 03

  Uh, yeah, I was partially expecting this question and that's that's exactly that's exactly true. In fact, I'll tell you, before I joined Seldon, I I had not read the Foundation series, even though I'd heard about it. Uh, and soon after I joined, I I asked this exact same question and afterwards I I I read the book, so uh so yeah, thanks for the question.

• 6:33Chris Benson?

  No worries. And just for listeners, it had been a while since I had read the series. It's a fantastic series, been around forever by Isaac Asimov. And so I was just, the idea there is that this figure, Harry Seldon, is what's called a psycho historian. And it's essentially evaluating society based on mathematics. And so they're in a galactic empire and it's kind of, he predicts that it's about to fall and it's going to be 30,000 years of chaos. And he uses this psycho history analysis to figure out a way to take certain actions to narrow the chaos down to a mere millennium. And so the series is about what he does.

• 7:18Chris Benson?

  And he creates these two foundations at each end of the galaxy, filled with scientists and engineers and stuff. But just since we made the reference, I wanted to relay that to anybody who had not read the book series.

• 7:29Daniel Whitenack?

  Yeah, so it sounds like, I mean, a kind of a central piece of that story is prediction, maybe kind of bridging that gap. I'm assuming that's kind of the reason why the Seldon name was chosen. But, you know, as you, so kind of filling in the gaps of your story, Yannis, you came to Seldon. What is Seldon's relation to what they're trying to do in machine learning and AI? And assuming it's related to that because of the name and the relation to prediction and all of that.

• 8:04Speaker 03

  Yeah, sure. So the Seldon name is actually very fitting, as you say, because it's all about predicting the future.

• 8:13Speaker 03

  But what we are doing at Seldon is we are doing machine learning deployment primarily. And this is everything that happens after your data scientists have finished their job and developed some models, trained them and achieved good enough performance so that they're ready to go out and be applied in the business. But there's a lot that needs to be done for that step to materialize. So typically a data science model.

• 8:48Speaker 03

  Well, there's this whole setup of how you deal with the data. Um, and then the modeling, modeling uh scripts, which are typically Python or maybe R scripts, or maybe some other languages. Uh, and after the training is done, then you have maybe some, some artifacts, some model weights, uh, which you can then load again, uh, and then make predictions. But that doesn't really make it easy for, uh, uh, people i- inside the business to use the model. So what needs to be typically done is, uh, productionizing the model, which would, in the simplest case, involve, uh, wrapping it with some, uh, light, uh, API, like a

• 9:30Speaker 03

  REST, REST endpoint, for example, so that it can, it can start living in the company's infrastructure and other business apps can start communicating with it by, by sending requests, uh, getting predictions back, and then those predictions being acted on.

• 9:44Chris Benson?

  Could you extend that a bit and kind of talk about what the products that Seldon has are and what projects it's engaged in and kind of give us a sense of, of what your customers or users are, uh, and why they're coming to you, you know, what, what, what is it they're trying to solve, uh, when they, when they engage in your products and projects?

• 10:03Speaker 03

  Uh, yes, of course. So, um, Seldon is an open core business, meaning that our primary product, which the company is built on, is open source. Uh, it's called Seldon Core, and it's a machine learning deployment platform that runs on top of Kubernetes. And it basically enables, uh, people to, to wrap up their trade models, uh, which can be trained using any framework. So you can be, your data scientist can work in Python or R or even Java or any, any other framework. And then

• 10:39Chris Benson?

  Even Java.

• 10:40Speaker 03

  Well, yes, even, even if you feel like it, yes. And because it's model agnostic, we don't get in the way of the data science modeling part. Uh, what we're interested in was, once that data science part is done, you can wrap up your models in, into Docker images and then deploy them using Seldon Core. And, and they, it, the models will be running on Kubernetes, uh, seamlessly and you can, uh, you can start using them in your business.

• 11:07Daniel Whitenack?

  Yeah, and I mean, so I've kind of heard of Seldon like a little bit in the past, but leading up to this conversation, I tinkered around with some more of what Seldon is doing and was super impressed, so much so that I'm encouraging one of the teams that I'm working with to use some of this for deployments because, you know, it was like really, so I had a model for like reading comprehension and, you know, getting that into production was as easy as writing a particular, you know, Python class with a predict function and then, and then using some of Seldon's tooling to basically just specify that, hey, I want a REST API

• 11:52Daniel Whitenack?

  on this port and my class is called this name and then Seldon kind of took care of the rest and that deployment piece. So I was super impressed with the whole workflow and I'm sure I've missed, you can tell me what other pieces of Seldon I missed. I'm sure I only touched on a little bit of what you offer.

• 12:12Speaker 03

  Yeah, and that's very great to hear that you're getting some value out of it already. So continuing on that, so Seldon Core is the open source deployment platform. And then on top of that, we are building an enterprise layer, which is supposed to make things that little bit easier and also more accessible to people that are not necessarily that familiar with the command line. So what we envision with the enterprise option is a centralized place to monitor all your models in deployment. So both monitoring them,

• 12:54Speaker 03

  sending off new deployments or decommissioning old ones and having a rich interface of inspecting the models, how they're doing and also have team collaboration and authentication of different levels of permission, who's allowed to put new models in production, who's allowed to decommission the models and that kind of, that kind of stuff.

• 13:20Daniel Whitenack?

  Cool. And I know that Chris and I have talked several times on this show, I think, and in our conversations outside of the show about this sort of weird friction that exists between engineering teams and data science or AI teams often in that, you know, the tooling that the AI people are using is really weird in comparison to what engineering teams are used to. Like, you know, what are these Jupiter things floating around? What do they do? And that creates a lot of friction oftentimes in terms of actually building value out of the AI stuff. So in terms of who's kind of latching on to this tooling that you're building, do you see kind of people coming from both of those

• 14:05Daniel Whitenack?

  sides? So from like a maybe a DevOps side and those trying to productionize what data science teams have passed off to them and maybe, but maybe there's people also from the AI side? Or what are you kind of seeing in terms of trends in that sense?

• 14:18Speaker 03

  Yeah, that's a very interesting question, actually. And from what we've seen both from community members reaching out on our Slack channel, but also in conversations with some of our clients, it is typically a very varied bunch. And I think it depends quite a lot on how data science, both the modeling part and the production part is organized in companies. And there tends to be a difference, especially with bigger enterprises and smaller companies in that regard. So in bigger enterprises, typically places like banks, there appear to be quite a lot of silos between teams. So data scientists, for

• 15:03Speaker 03

  example, would only be responsible for the development of models. And once they're kind of happy, then they just chuck it over the wall to the DevOps people or the engineers who actually have to put it in production. And that's not always the best way of doing things. As the data scientists then don't get any feedback on how their models are doing, and the data engineers don't get feedback about, well, how you actually, what is this thing? How do you productionize it? Whereas in small companies, it's it seems to be people are doing many, many roles at once. And this is something that when I joined at Seldon last year, I also had to pick up.

• 15:43Speaker 03

  So instead of just doing pure data science and modeling, I had to basically had to take up the engineering best practices. Because we are a part of an engineering company. But yeah, the people are varied.

• 15:58Chris Benson?

  So Yanis, how did Seldon actually get interested in doing model inspection and interpretability? What was the motivation that drove the company in that direction?

• 16:09Speaker 03

  So to answer this question best, it's maybe fruitful to discuss a bit more about the capabilities of our open source deployment platform.

• 16:18Chris Benson?

  Absolutely, sure.

• 16:20Speaker 03

  So when people think about productionizing models, usually they think of single models. So you have a model, you wrap it up in a single Docker image and then you deploy it. So if it's a TensorFlow model, maybe you use TensorFlow Serving, for example. Or if it's a Python model, maybe you write a Flask app. And then that's kind of it. It's a single model. You send requests of data through and get predictions back and then you use them in whatever you see fit. With Seldon Core, there's a lot more functionality that you can have. With Seldon Core, we have this inference graph abstraction, which is part of every deployment.

• 17:04Speaker 03

  And it can be as simple as a single model, which is probably most use cases in the business. But you can do a lot more interesting things. For example, you can have several models running in parallel. And then you might want to make predictions using all of them. And then before returning the prediction, you might want to combine all of them using some custom business logic. For example, well, it could be as easy as majority vote, or it could be just returning all predictions. So you get ensembling at inference time for free. Another way you can use it is instead of ensembling models at prediction time,

• 17:47Speaker 03

  you can route traffic to models. So you could have Say two models and have a router. It could be an A B test, for example. If you've developed one model and then put a second one model in alongside it, you you want to split traffic fifty fifty and see which one performs best. Or you could have custom business logic. Maybe you have several models and you know that one model does best on a particular at a particular time of day, for example. And another model does better at a different time of day. So that's kind of domain expertise that can be coded in into the routing. Or similarly, if it's, for example, a recommendation use case,

• 18:31Speaker 03

  maybe you have a couple of models and some of them do better on certain demographics than and some of them do better on other demographics. And then you can code that into your custom business logic router as well. So you can do quite a lot of interesting things with Seldon Core beyond single model serving.

• 18:51Daniel Whitenack?

  Yeah, so in terms of the so you just recently, you know, released and are promoting this Alibi project, which it's described as being concerned with model inspection and interpretability. So is the kind of I've heard of interpretability in terms of kind of interpreting individual models, maybe sort of black box models that you try to gain some insight into. But then what is the difference when you're meaning kind of model inspection versus model interpretability? Does inspection have to do with this kind of routing logic? And where things act like which models actually got used for a particular prediction? Or what's the difference there and the distinction?

• 19:33Speaker 03

  Yeah, so that's a very good question. So when engineers talk about monitoring live systems, then they typically think of things like, for example, requests per second and the total load on the nodes that the models are living on and that kind of stuff. But when you're approaching the question of monitoring machine learning models from a data science perspective, then there are a lot of very interesting and useful things that you might want at inference time rather than just raw predictions. So.

• 20:08Speaker 03

  And this is what I sort of mean about monitoring and interpreting machine learning models. So just to give you a few examples. So one is, we've already touched upon it and we'll go into detail is about model explanations. So you might want to go through your request logs historically and see, okay, well, why, why did this model make this particular prediction at this, on this particular instance? And, and you could, you could have a, a component in the Seldon core graph that is a model explainer. And then you can basically send that historic request through and get a model explanation back as to roughly the logic that it took at that particular time.

• 20:52Speaker 03

  Other, other use cases are, you might be interested in whether your data distribution is evolving in time. So if it's changing, your model might become stale and then you might want to flag it for retraining or, or decommissioning. So you would use components that do outline detection on the data or more generally concept drift on the data. So if your, if your data distribution is constantly changing over time to, to, to create those alerts and, and those can all be part of your deployment, of your Seldon deployment.

• 21:27Chris Benson?

  So what does the tooling look like for these sorts of things? I mean, obviously it sounds like, um, you know, I know we're about to get into Alibi and it's probably leading the way, but could you kind of describe the landscape of tooling? And, and, uh, you know, is it, is it custom logic in many cases or how, how are people dealing with things if they're not using Alibi?

• 21:46Speaker 03

  Uh, yeah, that's actually quite a broad question. And in my mind, I, I always split these sorts of questions into two parts. There's, there's really the, uh, there's the engineering part and then there's the data science part. And on the engineering side, it really is about. Okay, how do we, what sort of components do we need that will talk to a model in production? And how will that look inside, say, the Seldon deployment inference graph? And maybe sometimes these components need to be stateful as well, which complicates things. And then on the data science part, it's about, it's less about the engineering of these components, but more about the algorithms.

• 22:30Speaker 03

  So what do you actually use if you want an outlier detection running alongside your model and detecting anomalous data instances? So that's really the data science piece. And we've done some work specifically on outlier detection for Seldon Core. And we have examples on our GitHub about that. And with Alibi, we're sort of doing more, but more in the direction of model explanations.

• 22:58Daniel Whitenack?

  Yeah, so maybe just describe, as I'm thinking about like, so I'm thinking about like Seldon Core and how it gives you these deployments, which might be like one model or multiple models tied together with business logic. Let's say I'm already using Seldon to do those sorts of things. So how does Alibi fit into that? Is it like a component that's kind of like a library that you call from like within those components? Is it kind of something that runs on the side and reaches out to those things and tells you certain things? Or analyzes logs? Like what is the, how does it kind of operate within this ecosystem?

• 23:40Speaker 03

  Yeah, good question. So with Alibi specifically, so we got interested in model explanation as a company, maybe around about four months ago. And we sort of thought, well, okay, it would be good to support model explanations for deployed models in our enterprise product. And then, so on the data science team, we did a bit of research of what kind of techniques are out there. And in the academic literature, there's actually a lot, a whole lot going on. And there's a whole host of things that you could try to try and interrogate what your model is doing.

• 24:23Speaker 03

  But when it came to the open source code, it's mostly research code. So it's, it's not easy to use. So what we decided is that if we want to approach this seriously and, and basically we, we need to recreate some of those algorithms. And so Alibi was born as an open source Python package for some of the, the more famous or well-known algorithms for model explanation. So it's, it's a completely standalone library. So you don't need to be interested in model deployment or any kind of productionization to try and use it. And you can use it in a, in a Jupiter notebook.

• 25:03Speaker 03

  You can play around with some models that you've trained in the same notebook and see, see what kind of explanations your models can offer on the decisions that they made. But the, the way that it ties in together with, with Seldon Core and eventually Seldom Deploy, which is the, our enterprise layer, is that it will be the backend for producing these explanations of, of your models that are running in production.

• 25:38Speaker 04

  The Data Engineering Podcast is a weekly deep dive on modern data management with the engineers and entrepreneurs who are shaping the industry. Go behind the scenes on the tools, techniques and difficulties of data engineering so you can learn and keep up with the knowledge to make you and your business successful.

• 25:54Speaker 05

  Can you give a bit of an outline about the motivation for choosing Jupiter notebooks in particular as the core interface for your data teams?

• 26:02Speaker 06

  Yeah, and actually, when I first joined Netflix, it was a sort of toss at me. And I was definitely like, are we crazy? And the answer was like, we might be a little crazy.

• 26:11Speaker 04

  Go to dataengineeringpodcast.com to listen, subscribe and share it with your friends and colleagues.

• 26:29Chris Benson?

  So I noticed on the Alibi website, you note that Alibi provides, and this is kind of the air quotes here, "consistent API for interpretable ML methods." And so, you know, what's in that API? And is it an API that's used during training or testing? Or what other scenarios are you there for?

• 26:49Speaker 03

  Yeah, well, API is maybe a bit of a grandiose term. It literally refers to the way we've structured these various kinds of explanation algorithms within the library. So it's as simple as, so you could think of it in terms of the scikit-learn API, which we all know, which is, which typically has two steps for any given model or estimator. You have the fit step, which takes in a training set, and then you have the prediction step. So you can make predictions on new instances. And with the model explanations, it's a reasonably similar process, actually. So some explanation methods require access to the training set to be useful.

• 27:34Speaker 03

  So you would have a similar fit step, not always, but in some methods. And then really, explanation time, explanation almost exactly maps to the predict step. So you would have, rather than model predict, you would have an explainer explain and pass in a single instance that you want to be explained why the model made the prediction that it did.

• 27:58Chris Benson?

  Cool. Is this kind of model agnostic in the same way that, you know, seldom deployments are in the sense that if you want to use TensorFlow or PyTorch or whatever, then you, these things kind of work out of the box? Or is this kind of restricted to particular, like, reference implementations or that sort of thing?

• 28:17Speaker 03

  Yeah, that's a very good question to ask. So in the first iteration of Alibi, we're now in version two, but we're still mostly focusing on what we call black box explanation methods. So and when I say black box, I don't mean a complicated neural network that you've created. What I mean is that all you have is access to a predict function. And this can be very general. It can be, it can be literally something that just takes in arrays and spits out other arrays. So it can be, it can be as general as an API that's already sort of running in production. So, so we have a couple of methods in Alibi that all work on these sort of black

• 29:01Speaker 03

  box, black box models. So it's, it's, it's very portable. We do have in our roadmap more model specific methods because once, once you start to know a bit more about what your model actually is, then, then you have a lot more way and a lot more interesting methods as well that you can apply if you know the model architecture or the loss function, for example.

• 29:26Daniel Whitenack?

  So now that we've kind of talked a lot about what Alibi is in general, what the API is like, and how you integrate it into your workflow, I'd be interested because we've talked about, you know, model interpretability in general on this show before. But I'd love to dive into a few more specifics because this is, you know, really a practical project that people can use. So one of the things that you talk about, so you talk about a lot of different methods, maybe one that we could start with is you talk about anchors and anchor explanation. Could you kind of describe what that is in general and when it might be useful?

• 30:05Speaker 03

  Yeah, sure. And just before I dive into the anchor method in particular, I just want to pick apart a bit what we really want from model explanations, right?

• 30:18Daniel Whitenack?

  So Yeah, that'd be great.

• 30:19Speaker 03

  So there's the sort of a couple of different notions of model explanation. In particular, people talk About global explainability, where you typically want to know how the model performs sort of on average, say, on the whole of training set to try and draw some conclusions from it. And then there's the local explanation, which is, which has been the focus of Alibi in the first few releases, which is answering the question, given a specific instance and a model prediction, why did the model make this prediction? But as you can already tell, this question is not well defined. I mean, you can't really answer it, why the model made a prediction.

• 31:02Speaker 03

  You need to try and pick it apart a bit more. So one thing you can do is you can try and ask human interpretable questions, what you would like about this model decision to be answered. And then, given those questions, you can try and find algorithms that sort of approximate the answers to those kinds of questions. So, for example, you might ask, okay, given this instance and this prediction, what is sort of a minimal subset of features and their values, given which the model will make the same prediction regardless of everything else? So that's what we call an anchoring question. So that's where the anchor technique comes in.

• 31:46Speaker 03

  So you ask an interpretable question and then you go ahead and see, okay, how can I write an algorithm to do this?

• 31:53Daniel Whitenack?

  So in that case, with the anchor explanation is like, what's the sort of, how would you kind of phrase these anchor questions, I guess? Or what would be the thing that you would be looking to come out of Alibi that would help you kind of with that explanation or make that explanation logically?

• 32:12Speaker 03

  So the great thing about this is that these interpretable questions are asked when you want to design a new explanation method. But once the method is designed, you only need to be aware of what question that method is answering. So the anchors method would return, say, a small subset of features of the original instance and their values, which would result in the same model prediction, say, 95% of the time. So let me just give you a quick example, one example that we also have. On the on the website, so it's the sort of semi famous census income data set from

• 32:52Speaker 03

  the early nineties, composed in the US, and it's it's a binary classification problem of predicting whether a given individual will be a low income individual or a high income individual. And the threshold is whether they make less than or more fifty, more than fifty thousand dollars a year. So and and the the various features of each individual are their education level, their occupation, their relationship status, but also their gender and race, and various kinds of other features. So so it's a it's a fairly standard setup. You've got tabular data and you can train a binary classifier to to solve this problem.

• 33:37Speaker 03

  So once you have your classifier trained, you can then pick individual instances and run the anchor algorithm to try and find for each instance which which really are pertinent set of or subset of features that were important for this particular prediction. So for example, I I might pick a, say, a woman who's mid thirties, who's been separated, and maybe her profession is working in the government. And I I run the model and it says, okay, it's a it's a lower income individual. Then I run the anchor explanation and what I get out is, okay, my anchor for this explanation

• 34:22Speaker 03

  is that their marital status is separated and their work category is government work. And with those two present, the model will make the same decision ninety five percent of the time, regardless of all the other features. So that's where the name anchor comes from. It anchors this decision to to really what makes what makes the most importance for this particular individual.

• 34:46Chris Benson?

  So wondering if you could kind of tell us a little bit about maybe a different method that you've implemented that might be useful in a different situation, or or possibly in the same situation in a different way. I don't know if that made a lot of sense, but.

• 34:58Speaker 03

  Um, yeah, sure. So I I can I can talk through another method, or rather, a couple of methods that we have. In Alibi, so when we first kicked off Alibi, we sort of wanted to implement methods that didn't really have good implementations, but that were interesting and had received attention in the academic community. So Anchors is one such method. And actually, it was, I'm not sure if you know, it was designed by the same people behind Lime, which sort of kicked off the whole interpretability of machine learning area.

• 35:37Daniel Whitenack?

  Yeah, that's the, I'm trying to remember, is the one that kind of gives you linear estimations of certain relationships or something like that?

• 35:47Speaker 03

  Yeah, so what Lime does, it kind of fits a surrogate model, a linear surrogate model around the instance. It's basically trying to approximate the nonlinear decision boundary with a linear one. So and then the weights of that linear model can get interpreted as feature importances. So it was sort of, it's sort of a very familiar technique, but it has its shortcomings. And then the authors of Lime came up with Anchors, which they feel is sort of a better solution for this problem.

• 36:21Daniel Whitenack?

  Cool. So we talked about the anchor explanation. You list a few different methods. Like you were saying, there's multiple methods that are implemented in Alibi. There's, you know, the trust scores and counterfactual instances and other things. But I was wondering, maybe, maybe as we kind of move towards the end of our of our conversation here, if you could just give us a little bit of an idea of, you know, where Alibi is heading? And maybe also like what what Seldon has in mind kind of for the for the future of Seldon core and Alibi and maybe maybe other things? Like, where are you headed?

• 36:57Speaker 03

  Yeah, so for Alibi, we have a reasonably ambitious roadmap. I mean, I mentioned sort of our API mimics the Psychatlen API, but for For model explanations. So we we kind of in an ideal world, we would be the psychic learn of model explanations, bringing bringing various techniques together i in one place uh for people to to use and compare and contrast what makes sense for the use case. So other than that.

• 37:26Daniel Whitenack?

  So that would be like like psychic learn, you can, you know, if you're wanting to do classification, you can choose from any of, you know, whatever it is. I I actually don't know how many like twenty or thirty or whatever uh implemented uh methods for for classification. So you're kind of imagining Alibi would kind of have that zoo of explainability methods that you could kind of call using a a standardized API. Is that is that kind of the the thought?

• 37:53Speaker 03

  Uh, yeah, yeah, that that's kind of our our ambition with with Alibi eventually.

• 37:58Daniel Whitenack?

  Cool, cool. Yeah. What about um, you know, Seldon uh Seldon in general? Are there are there other things on the on the horizon that you can share or or maybe just things that you're excited about in the AI community or at Seldon specifically?

• 38:11Speaker 03

  Yeah. So so I'm not sure if you're aware, but this week is actually the the CogX event uh here in London, which is the sort of festival of AI where lots of companies sort of go and and uh and present their the sort of emerging technology to to facilitate all kinds of um machine learning use cases in the industry. So what we're actually uh doing there this week is we're we're demo demoing a sort of a a first uh sort of version of our enterprise uh product Seldon deploy, which which has also anchor explanations running with the live requests. So that's that's that's kind of quite exciting from a product development perspective.

• 38:56Chris Benson?

  So I guess as you are are looking at that, is that I'm just I was curiosity with that particular uh event, is that uh is that more of a conference or is it more of um kind of companies coming together to just demo their their thing? What what what style of event is it? And is this kind of a are you essentially taking the opportunity to announce new product uh features in that with with the one you just talked about?

• 39:19Speaker 03

  Yeah, so it's it's it's kind of a bit of both. There are speakers, mostly industry speakers, but it's also so companies have their own stands where they can demo their technology and that kind of stuff. So I expect there to be fewer people from the sort of pure machine learning research community, especially as it also coincides with ICML this week.

• 39:43Daniel Whitenack?

  Cool. And I mean, first off, congrats on the the announcement. That's really, that's really exciting. I can't wait to see what's next and the hype around around that for sure. But let's say people are listening to the podcast, they want to, you know, get hands on with Alibi and or Seldon Core and or Seldon Enterprise. What are the best ways for people to kind of get get up and running with Seldon or or Alibi? Where can they, where can they find resources and get help?

• 40:13Speaker 03

  Yeah, probably the best resource is a is a docs. So you can just go on docs.seldon.io and that will take you to the relevant documentation, either for Alibi or for Seldon Core, whichever takes you takes you fancy. So it has, it has a lot of information to to get you up and running.

• 40:30Daniel Whitenack?

  Cool. And do you have any, no worries if you don't, but as you've kind of explored this whole area of, you know, interpretability and explainability, do you have any recommendations maybe for people that are interested in learning more about that subject in general? As far as resources where they could learn about what has been done or what people are researching? What's, what's the best way to find those, that sort of information?

• 40:57Speaker 03

  Yeah, so one thing I can definitely recommend and something that we kind of got a lot out of when we were first developing Alibi is Christoph Mollnauer's book on interpretable machine learning. So it's, as the title says, it's about, it's a book about interpretability techniques. And it's, it's fully open source, it's available online and it's, it's really well written. So if people want to get into the topic, then then I think that's the best, best place to start by far.

• 41:24Daniel Whitenack?

  Awesome. Well, thank you so much for joining us today. I know I'm excited to dig in more to Seldon and to Alibi. Now I understand a lot more about kind of how it fits into my workflow. I encourage people, if you have questions about or thoughts about Alibi or Seldon, of course, they have their Slack channel, but we also have our Practical AI Slack channel, which you can find at changelog.com/community and our LinkedIn page as well. If you just search for Practical AI. So we'd love to hear your thoughts on machine learning, interpretability, and inspection. Really appreciate you joining us, Yannis,

• 42:08Daniel Whitenack?

  and hope to stay in contact and really looking forward to seeing what Seldon does in the future.

• 42:14Speaker 03

  Yeah, thanks very much for having me on the show. And I would just like to quickly thank my coworkers, Arno van Meurthen and Giovanni Vacanti, who both are working on the Alibi project. And without Stu, I couldn't have done the work.

• 42:29Daniel Whitenack?

  Awesome. Appreciate that. And I really hope that we can run into you at a conference sometime and catch up. Thank you so much for joining us.

• 42:38Speaker 03

  Okay, great. Thank you.

• 42:42Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for Changelog is provided by Fastly. Learn more at fastly.com. And we catch our airs before our users do here at Changelog because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode Cloud servers. Head to linode.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address,

• 43:25Speaker 00

  get our weekly email, keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.