Eliminate AI failures

We have all seen how AI models fail, sometimes in spectacular ways. Yaron Singer joins us in this episode to discuss model vulnerabilities and automatic prevention of bad outcomes. By separating concerns and creating a “firewall” around your AI models, it’s possible to secure your AI workflows and prevent model failure. Sponsors: Me, Myself, and AI https://sloanreview.mit.edu/audio-series/me-myself-and-ai/ – A podcast on artificial intelligence and business produced by MIT Sloan Management Review and Boston Consulting Group. Each episode, Sam Ransbotham and Sheervin Khodabandeh talk to AI leaders from organizations like Nasdaq, Spotify, Starbucks, and IKEA. Me, Myself, and AI is available wherever you get your podcasts. Just search Me, Myself, and AI. Changelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Fastly https://fastly.com/podcast – Compute@Edge free for 3 months — plus up to $100k a month in credit for an additional 6 months. Fastly’s Edge cloud network and modern approach to serverless computing allows you to deploy and run complex logic at the edge with unparalleled security and blazing fast computational speed. Head to fastly.com/podcast https://fastly.com/podcast to take advantage of this limited time promotion! LaunchDarkly https://launchdarkly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Fundamentally change how you deliver software. Innovate faster, deploy fearlessly, and make each release a masterpiece. Featuring: Yaron Singer – X https://x.com/yaronsingerChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Robust Intelligence https://www.robustintelligence.com/Robust Intelligence Blog https://www.robustintelligence.com/blogUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The goal of any technology is to basically support humans, right? Not replace them. Wherever it can help automate and kind of de-bias human decision-making, we want to support that and not replace that. But at the same time, we want to make sure that we're not replacing human judgment. I think that the challenge that we have in our world, I don't know that we will have that choice. Meaning that we don't have a person sitting in every kind of like critical decision junction and making that critical human judgment. It doesn't scale.

• 0:32Speaker 01

  Big thanks to our partners, Leno, Fastly, and Launch Darkly. We love Leno, they keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com.

• 0:51Speaker 01

  This episode is brought to you by Me, Myself, and AI. It's a podcast on artificial intelligence and business. It's produced by our friends at MIT Slow Management Review and Boston Consulting Group. The question is, why do only 10% of companies succeed with artificial intelligence? That's the question they aim to answer with this podcast. Here's Google Cloud's Will Granis on an unusual AI challenge.

• 1:13Speaker 02

  When I think about what AI is, I find the algorithms mathematically fascinating, but I find the use of the algorithms far more fascinating. Because from a technical perspective, we're finding correlations in extremely high dimensional nonlinear spaces. It's statistics at scale in some sense, right? We're finding these correlations between A and B. And those algorithms are really interesting and I'm still teaching those now and they're fun. But what's more interesting to me is what do those correlations mean for the people?

• 1:41Speaker 01

  All right. Me, Myself, and AI is a collaboration between MIT Slow Management Review and Boston Consulting Group. It's available wherever you get your podcasts. Just search Me, Myself, and AI.

• 2:09Speaker 03

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at change.com/community and follow us on Twitter. We are at Practical AI.

• 2:35Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International, and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

• 2:51Chris Benson?

  Doing very well. How are you today, Daniel?

• 2:52Daniel Whitenack?

  I'm doing well. I think this episode will release in the new year, but there's a lot of sort of year-end stuff happening to tie up loose ends and project plan for next year. So all good things, but yeah, trying to do all that stuff.

• 3:07Chris Benson?

  The end of your time warp here, even though everyone's hearing it after the fact.

• 3:10Daniel Whitenack?

  Right, exactly. I'm pretty excited. I know that you and I have talked many different times on the podcast about where AI models can fail or go wrong, either in terms of data or in terms of their behavior. But maybe we haven't talked as much about like ways to mitigate that problem practically and how people are approaching that. And we're really excited today because we have Yaron Singer with us, who is CEO of Robust Intelligence. And of course, they know all about how AI models fail and how to do something about it. So welcome to the show, Yaron.

• 3:54Speaker 00

  Yeah, great to be here. Thanks for having me, Daniel and Chris. This is, you know, I'm a big fan. I listen to, you know, I think I've listened to pretty much all the episodes. So it's great to be here.

• 4:03Daniel Whitenack?

  Oh, wow. Yeah, that's great. Well, it's wonderful to have you on the show. Could you maybe just start out by getting us up to speed, maybe for those that are out there that aren't really aware of the different ways in which AI models might fail and the sort of risk associated with that? Could you give us a little bit of an intro to? Maybe some of the highlights from that and either things you've seen or things in the community that kind of motivated you to think about this problem.

• 4:34Speaker 00

  Absolutely. You know, when we say that AI models fail, what do we mean, right? Like failure can come in all these sort of like different shapes and forms. You know, so maybe we, you know, we kind of like start with some, some examples that, you know, I think a lot of us in the AI community are familiar with. One example that's been like kind of a very famous one was the Microsoft chatbot example, where Microsoft kind of like developed this, this chatbot, this AI chatbot. That was really great technology and, you know, but what they did is they basically, you know, the idea was this for this chatbot to like kind of mimic actual human conversation. And the way that they did that is they actually used, you know, kind of like data from Twitter, right, kind of to have, you know, trained that bot.

• 5:17Speaker 00

  And when people figure that out, what they actually did is they actually, you know, kind of fed all sort of like kind of weird, different things to the, to the chatbot, to the point where they realized that they can actually make this chatbot racist. Right? So at the end, what you got is that kind of pretty quickly, I think, within, you know, a few, like, I think it was like 36 hours after it was kind of released, you know, you had this chatbot that was kind of like spinning out all these sort of like awful racial slurs and Microsoft ended up shutting it down. Right? So, you know, it's just kind of like one example where you have this like really elite technology, right? And once it falls into, you know, the wrong hands, right? It can obviously, you know, it can kind of be led astray and kind of be manipulated in all these kind of weird ways that we, you know, we never expected and intended. So there are all these sort of ways that, you know, that AI models fail.

• 6:02Speaker 00

  And, you know, even like kind of as of, as of recent, like we've all read about, or some of us have read about, you know, Zillow's example, where if you went on Wired and you read like in June 15th, there's like kind of this beautiful headline about how Zillow is going to like improve its, you know, its predictive pricing like on, on housing using kind of sophisticated neural networks. But then if you kind of read Wired again in November, it explained to you why Zillow failed with its AI pricing, AI-based pricing. Right? And basically, they kind of like let go, I think of like 25% of the You know, of Zillow's kind of like employees and it had huge kind of like economic implications for Zillow.

• 6:43Speaker 00

  And basically, you know, there the the failure came because of changing conditions. And specifically, it was because of the pandemic. So the models were trained using old data and then they applied them. And then, you know, on, you know, on the world, which, you know, was experiencing a pandemic. And that data was, you know, was very different. So this is what, you know, we call distributional drift. And we saw failure of the AI models there. So these are just kind of like two, I think, kind of one famous example and one of the more recent examples. But of different, you know, totally different ways of in which AI systems can, you know, can just completely fail for different reasons. Yeah, so so those are kind of like some examples and and implications are, you know, are obvious, right?

• 7:24Speaker 00

  You know, either, you know, we have like kind of a very bad reputation, very bad risk. We're putting kind of people's safety at risk. So those are kind of big problems of experiencing.

• 7:35Daniel Whitenack?

  I definitely see on the one hand, you've sort of got these like behavioral problems of models. Where like depending on what data you feed them or train them on or update them with, you kind of get this non-ideal behavior in in one way or another. I'm curious about your perspective now, sort of being in this this field and seeing how clients are using AI models maybe more and more over time. What is the sort of, from your perspective, is there an increasing risk in the way in which people are using models? Like the the tasks that they're applying to, applying models to in business use cases? Where like the chatbot one is is interesting and, you know, is bad PR for Microsoft.

• 8:20Daniel Whitenack?

  But like, are people starting to maybe apply AI models in maybe more risky business use cases? Versus sort of more toy problems or research type of settings?

• 8:33Speaker 00

  That's a great question. I think in general, regardless of like one organization or another, like we are in a world that is adopting algorithmic decision-making that is completely based on AI. And this world is adopting this sort of automated decision-making at an exponential pace. And some examples are examples that have been out there for a while that we know. But it goes for things that are as simple as AI models being used for determining, basically, insurance rates, for home insurance, for home insurance and car insurance, but also for health insurance. And when these AI models can have these sort of different failure points,

• 9:16Speaker 00

  failure modes, that has huge ramifications. Same with lending. So AI is used a lot in lending and deciding like who gets a loan, who gets a house loan, who doesn't get a house loan, who gets a car loan, who doesn't, and how much they have to pay. And we also see this in things like predictive policing, where police departments across the country are using AI models to basically decide on whether, where they're going to be putting more forces. And all these things are, I think, kind of like the intentions are good. People want to make good decisions, they want to do this in a fair way, and automate that process somehow. But at the same time, with everything that we know about AI and its discipline nature,

• 10:00Speaker 00

  the risk is enormous.

• 10:02Chris Benson?

  I got a question about that. We've talked about some of these use cases, and you kind of think of, maybe divide them up a little bit. On the Microsoft side, there's failure that is intentional in nature. There's sort of these, for lack of a better word, adversaries out there, and they're taking advantage of the weakness of a model, and seeing what they can do with the chatbot. And then you have these kind of environmental influences, such as the Zillow thing, where the world changed out from under you. It was no one person's intention to do that, but nonetheless, it had the same effect. Does the intentional and the unintentional, does that matter as you're dealing with these failure situations?

• 10:43Chris Benson?

  Does it change the way that you approach the problem? Or does that intention or lack thereof, just because it happened, is that a factor in things?

• 10:53Speaker 00

  That's a very good question. Like, does it matter whether the AI failures are due to an adversary that's trying to... To create, you know, these failure modes, or they just, or they just sort of like happen because of, you know, changing natural conditions or whatnot. So I think the answer to that is, you know, yes and no, it like kind of it matters and it does not matter. And in our approach, you know, and that's sort of like kind of the approach that we have in our company, right? In our company, the part where we think it does not matter, right, is where basically we put all these things under the category of risk. Right, where what we look at is we basically we abstract that like kind of we abstract the root cause of the risk.

• 11:32Speaker 00

  Right away from, you know, we we sort of say, well, it doesn't really matter, you know, kind of what has caused that model to fail. Right, the important thing is that, you know, that the model does not fail for what, you know, whatever, you know, whatever reason. So in that sense, we're kind of like, we take an approach where we're agnostic. Right to whether it was an adversary that fooled the model, whether it was the pandemic that changed the conditions, right? Whether it was, whether somebody really intended to, you know, misfeed, to put in like racial slurs. Or it's just something that, you know, for whatever reason, you know, was picked up from the internet. So somehow like the the root cause does not really matter, right? What's important is that we somehow are able to reduce it, right, from a technical perspective.

• 12:14Speaker 00

  Right to um to kind of like understand it and and being able to protect the models from, you know, from this. Now, yes, it does matter, right, in the sense of, you know, kind of like the algorithms. You know, they began abusing to, you know, kind of protect models from, you know, one kind of failure to like another kind of failure. Right, so, you know, the algorithms that you would use to protect the model from what we call like a distributional drift. Like kind of changing in conditions like due to the pandemic are different from the algorithms that we use. Right to protect the model from being kind of handed kind of like adversarial input.

• 12:50Daniel Whitenack?

  On that point, are you seeing like as you're engaging with clients and and companies? Are you seeing, I guess, two questions? One is, what is their perception of the main category of risk in those two cases? The adversarial or the sort of data drift and distributional change types of things? And what maybe from your perspective is the sort of reality, I guess. I guess maybe some companies have a natural, naturally a higher risk vector for adversarial parties coming against them just because of what they do or whatever. But yeah, I don't know if you have any thought on that.

• 13:26Speaker 00

  Yeah, I think that's a really interesting question. I think that it really depends on, you know, the company, but not only the company, you know, it actually, you know, like even, you know, different teams within a company can have different concerns, right? You can imagine that you have a company where they have a team and that team is responsible for fraud detection in that team, right? So the team that's, you know, dealing with fraud detection in the company, right? They are constantly dealing with adversarial input and they very much care about protecting their AI from, you know, the threat vectors of adversarial input, right? The same company can have another team and that team is dealing with forecasting, forecasting of different events, right? And, you know, forecasting different events, like they don't care about, you know, they don't think about adversarial input.

• 14:11Speaker 00

  They worry about, you know, rainy days changing, you know, if you're thinking about maybe like a company that's suing Ritesharing or things like that, right? So they care about like the underlying, you know, the continuously kind of changing conditions and how that affects the, you know, the models and the predictions. And if you want to do this well, right, then you want to be able to, you know, kind of build a system, right, that protects models from both these types of cases, right? The cases where there's like, you know, there are adversaries that are, you know, like really just trying to change and manipulate, you know, financial transactions, right? As well as the threat vector of just changing conditions, right? With no adversary in place, but just sort of like changing conditions that can change for the, you know, the predictions of the models.

• 15:08Speaker 03

  ChangeLog++ is the best way for you to directly support Practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of Practical AI into the future. Simply follow the ChangeLog++ link in your show notes, or point your favorite web browser to changelog.com/plusplus. Once again, that's changelog.com/plusplus.

• 16:05Daniel Whitenack?

  So as I was looking through some kind of updated best practices out there, actually, I think what triggered this was, you know, Chris and I spoke recently on a show about the OpenAI API because it's now sort of generally available. And I was reading through a bunch of their documentation in terms of like how they can, what they're thinking about in terms of risk and like best practices in terms of using the API to maintain safety and security and privacy and all of these things. One of the things that they highlighted in there was the sort of automation and human in the loop element of this, where like if you're planning on creating an application, for example, with the OpenAI API,

• 16:51Daniel Whitenack?

  it's considered much higher risk if it's to automate something, right? And there's not going to be any sort of human in the loop to review things. As someone who kind of works to kind of manage and mitigate the risk in these types of scenarios, how do you view, I guess, as the industry gets more sophisticated at handling the risk, are we going to be able to automate things more? Or maybe is it that we deploy our models and we have monitoring infrastructure that helps us know when things are going wrong, but it's still automated? I'm wondering how you kind of view this shifting over time and view this need for humans in the loop in terms of the output of models?

• 17:32Speaker 00

  I think this is really important and really interesting, right? And I think that what you're saying from OpenAI is just echoes what we're seeing from leading companies and platform in general. When you look at the state of the art, it's all going towards automation, right? So if you're using Databricks today, you can already, you're essentially kind of in Databricks notebooks, you can have the option of retraining your models automatically, right? And in general, that's kind of where the world is going. And I think that like within, I want to be careful with my predictions here, but like we're a few years away from basically having most of the retraining tasks being done automatically, meaning without a human in the loop, right? And that's generally where I think AI is going, right?

• 18:15Speaker 00

  If you kind of want to think about where I was going to be like kind of, you know, three years from now, look at where AI was, you know, five or seven years ago. If you think about kind of like where we were seven years ago, this is like, it predates scikit-learn, predates TensorFlow, predates PyTorch, right? Where if you wanted to like, you know, if you heard about AI and you thought, oh, maybe that could have an advantage for my business or my organization, you would basically need to hire a PhD from some top school to like code up and see an SBM. That's kind of what they would do. The thought of that today seems like kind of very, we're kind of laughing about that, but that's really how it used to be, just like in what, in 2014, right, 2015? There's been a lot of automation since then.

• 18:58Speaker 00

  And I think that where we're going is we're going towards more and more and more of this automation. Especially when, you know, if you meet a company now and that company has just a handful of models, you know, talk to that company again a year from now, they'll probably have like hundreds of models, right? And if you met a company now that has hundreds of models, you know, a year from now, they'll probably have thousands of models. So in order to do that and scale, it's all automation. So that's, that's kind of like our, our perspective and philosophy in the company, that the world of AI is going towards automation. And as the world of AI is going towards automation, we're trying to really think and understand, how do we ensure an organization that is using AI, especially kind of in an automated

• 19:40Speaker 00

  way, is really making sure that is not taking on any risk, that it really eliminates all the risks that AI has, especially with assuming that it does automated retraining of models and all that. I think automation definitely presents an additional, you know, dimension of risk. And it's really important to like, understand that that risk that we're taking on, right? And be thoughtful about it.

• 20:01Chris Benson?

  I got a follow-up question to that. When, as Daniel was asking that last one, it popped to mind and you've started addressing it already. And I wanted to explore it a little bit. And that is, if you're thinking about this journey from kind of where we are now, and having human in the loop on lots of critical tasks and lots of industries, especially for retraining and things. But if we agree that we're generally moving toward full automation, there's risks associated with both of those scenarios. And there's also risks associated with that kind of in-between where it's human, automation, collaboration, often said like, in the industry I'm in, man, done, man teaming, that kind of concept. All of those two poles, plus the journey in the middle, all have a discrete set of

• 20:44Chris Benson?

  risks associated with them. How do you see that? How do you look at those risk sets? And how do you decide what matters? You know, how do you evaluate that? Because I think that's a really tough question that I end up talking with folks that are grappling with is, is well, there's problems with having humans. We're not perfect. We screw up. And we sometimes talk about it in the sense of having a human bring safety, but not always. And there are other times, you know, that you can go argue the other way. What's your perspective on that?

• 21:10Speaker 00

  Yeah, there's an inevitable trade off, right? That's where you're saying. I'm of the opinion that the goal of any technology is to basically, you know, support humans, right? Not replace them. On the one hand, we wanna, we wanna support human decision making. Wherever it can help automate and kind of de-bias human decision making, we wanna support that and not replace that. But at the same time, we want to make sure that we're not replacing human judgment. I think that the challenge that we have in our world, I don't know that we will have that choice. Meaning that we don't have a person sitting in every kind of like critical decision junction and making that critical human judgment. It doesn't scale. And that's the biggest challenge that we have.

• 21:51Speaker 00

  So with that in mind, I think that like, what we need to do is we need to make sure that it's never going to be perfect, but we need to, we need to know that we're making every possible effort to make sure that it is as safe and risk free as much as possible.

• 22:06Chris Benson?

  That's a very thoughtful answer. Thank you.

• 22:08Daniel Whitenack?

  Yeah, this whole time I've sort of been thinking and through these sort of wider, you know, broader issues and thinking through my own use cases. And as Chris knows, eventually I always get to the point where I'm like, okay, well, I understand the, the point here. Like, what can we do practically to address these things and mitigate them? I'm wondering if you could like, maybe walk us through like your journey to robust intelligence in terms of how you kind of came to, I guess, understand what you wanted to focus on in terms of, in terms of what you wanted to build and, and offer to the community? Because there are so many problems and so much to address.

• 22:52Daniel Whitenack?

  And, you know, obviously you have to focus on, on something to start with. So how did, how did you get to that point? And, and how did things kind of get started?

• 23:00Speaker 00

  So my journey into this has been through, like a lot of AI practitioners, I think it started in academia. So I was a PhD student at Berkeley and then I, I worked at Google. And then I spent seven or eight years at Harvard. I've been, I'm a professor of computer science and applied math at Harvard. And basically what, what I've been working on Harvard is exactly this topic, the vulnerability, the sensitivity and the failure modes of machine learning models. Right. So a little bit before my time at Google and then while I was at Google, I've worked on machine learning models and basically algorithmic decision that is based on machine learning models. And when you do that and you start to kind of like do the the theory behind it and you try to prove theorems, you start realizing

• 23:42Speaker 00

  that we have very little foundations for kind of like algorithmic decision making given machine learning input. And there's a good reason for why we have like very little theoretical understanding is because it's pretty terrible, to be honest. And that's kind of what I started studying and specifically like kind of I've been looking at, you know, really the the sort of the whether we can make good algorithmic decisions given input for machine learning models. And the answer is, you know, mathematically, no, sort of like, and this has been my my focus at Harvard.

• 24:10Daniel Whitenack?

  And so when you say terrible, you sort of mean terrible in the sense that you can't get to a point where you can prove in many cases that algorithmic decision making is a good idea.

• 24:21Speaker 00

  Yes, exactly. You actually prove it mathematically. You know, we have like kind of mathematical definitions of what it means to learn and what it means to like kind of make good decisions. And you can, we have like kind of very, very rigorous kind of models and statements. And then when you kind of use these rigorous, and these are the kind of like the rigorous, you know, models and statements that kind of make machine learning work. When you kind of try to apply a little bit more complex decision making on top of these results of machine learning models, it all starts to break. So we've been kind of like proving these terms about like how much data you would need in order to to make what we call good decisions. And it turns out that, you know, the data is like exponential in the dimension of the input, which is really bad. And kind of like we studied the sensitivities of, you know, of models and to like kind of very, very small errors and very, very kind of like small failures.

• 25:06Speaker 00

  And again, like kind of like infinitesimally small errors in, you know, in models can lead to errors that are arbitrarily like kind of bad. It's quite horrible and quite bad. And I've actually spent some time, you know, in my academic career, trying to kind of convince people of this, right? And giving talks about an inconvenient truth about algorithms in the era of machine learning. If you look at that title up, you'll see like a bunch of lectures coming up and, you know, and and seminars across.

• 25:28Daniel Whitenack?

  I'm sure it produces some awkward conversations at conferences.

• 25:32Speaker 00

  Absolutely, absolutely it does, right? Especially in a time where kind of like machine learning is on the rise and on the boom and you know, and your department is like, you know, this gung ho about like hiring more people in machine learning and now they have like this professor who's like, you know, kind of proves all these weird theorems about why it's not working. I'm sitting in the same department as Les Valiant, who created the foundations of machine learning, right? Who's been, by the way, like the most receptive person to like, you know, this sort of criticism and then whatnot, and he's been so supportive. So it's definitely like a lot of, some papers took like, you know, three, four years to get published, but they got published and then they got kind of like very good recognition. But we started from, you know, the theories about the impossibilities and then we moved

• 26:13Speaker 00

  to algorithms and, you know, my group developed a lot of, basically focused on algorithms for, you know, noise robust algorithms for these types of problems and kind of like focusing on what is that we can do. As we've kind of proved all these impossibility theorems and then, you know, kind of like focus on what is it that algorithms would be able to do, like a very big idea that came up was like decoupling. So what we realized that one needs to do is basically decouple the part about model building from model security or model safety. Meaning that if you try to like build, just train a model that is going to be robust to, you know, let's say adversarial input, we talked about that. I think kind of the best result known, and I think that result is now from like two or three years ago.

• 26:54Speaker 00

  It turns out that if you want to like make your model, just by retraining it, to make it more robust to like adversarial input, I think in image classification, you're going to take the accuracy of the model from 98% to 37%, right? In order to get any sort of reasonable robustness, which is a trade off that is just unacceptable, right? If you're going to come up to like a company and tell them, hey, you know how your model like has 98% accuracy? Well, in order to make it robust to like the 0.001% of input that is bad, I'm going to take that model to accuracy to like 37%.

• 27:29Chris Benson?

  Would there be an analogy there to kind of in the software world, having a separation of concerns?

• 27:34Speaker 00

  Absolutely.

• 27:35Chris Benson?

  Yeah, okay. For any listeners that are not software people, you want to address the problem you're trying to address, but at the same time, the ancillary things like security that are very important need to be addressed, but you address them separately to maximize both.

• 27:47Speaker 00

  Absolutely. Whenever I talk to people for the first time about this, I tell them, you know, look, there are two considerations. One is the mathematical consideration. And mathematically, if you wanted to make your model robust, quote unquote, by retraining it to like adversarial input, it means that you're going to like take accuracy from something like 98% to 37%. It's kind of like, it's just a mathematical fact. Now there's another aspect of it, which is the product aspect of it, right? Right? Or the kind of like the engineering aspect of it. And the engineering aspect of it is, it's exactly what you're talking about, Chris, right? Where if you're an engineer that's building a system, you probably, we should not be the one who's also responsible for like protecting that system. You know, we can just all imagine the nightmare if every time that we wrote software, we'd also have to write the antivirus and the firewall, right, for that software.

• 28:33Speaker 00

  So in software, like kind of, we've seen such tremendous success exactly because of this decoupling, right? And kind of like layering of different systems, right? And components that know how to work together, right? And almost like this agnostic way. And I think that what we're trying to do in the company and for Bus Intelligence is mimic that kind of decoupling. So specifically what we do is we build an AI firewall. And an AI firewall is a piece of software that wraps around an AI model to protect it from making mistakes. So it kind of like, it's a one line of code that you add so that that one line of code can basically stands between the data and the model. And it can, once the data comes in, it monitors, tests, and can even correct it.

• 29:17Speaker 00

  So basically, the data point does not fool the model, does not cause the model to make this sort of mistake, right? Or a bad prediction. And that's just sort of like this decoupling process where we're not trying to build a better model. All we're trying to do is we're trying to be, basically be able to catch bad data. So it's reducing it to like a much, much simpler task.

• 29:39Daniel Whitenack?

  I definitely resonated with your example around the firewall and antivirus. I know in my own building software career, anytime I felt like I start poking holes in a firewall to open up ports and configure things, I start feeling extremely uncomfortable because I have no idea what I'm doing. So that definitely resonates with me. We talked about that firewall component, this AI firewall component wrapping around the model between the data and the model. In that firewall, are you looking at out-of-distribution data

• 30:18Daniel Whitenack?

  that's coming in? Or are there other ways that some data is more particularly risky than others? So I guess my question is, how do you know as data comes in if it's risky data or if it's not risky data?

• 30:33Speaker 00

  Yes, that's a good question. So what we do is, basically, we test the models. We have a process of stress testing the models, and we do that either implicitly, if you just install the AI firewall, if you just put the line of code, we basically do that in the background, we do that implicitly. Or sometimes, when there are companies where we start from stress testing, and then we graduate to AI firewall. And stress testing, basically, what that means for us is, we run a series of tests on the model. Some of the tests can be, how does the model respond to distributional drift? How does the model respond to unseen categoricals? How does the model... just all these different scenarios and different inputs? And then we're measuring the response of the models to all these bad things that could happen.

• 31:18Speaker 00

  And as we're measuring them, we're getting a sense of, we're basically training our own AI firewall. So by understanding how different input can affect the model in different ways, when new input comes in, we know whether that input is going to lead to some sort of prediction error, some sort of prediction change. So let me give you a silly example. Suppose that you have some sort of AI model, and maybe that AI model is trying to predict whether somebody's gonna earn... About a hundred thousand dollars next year. So whenever we take that input, like we check all the different features and when we sort of see like how the changes in different features can affect the prediction of the model, right?

• 31:59Speaker 00

  So maybe like, you know, for example, we look at like, how does age, right, affect the prediction of the model? Now suppose that like kind of, um, you know, data comes in and somebody has accidentally replaced age with the year of birth. Right? So age is like, you know, maybe from like, I don't know, thirty six, age was changed into like nineteen eighty five or nineteen eighty six or whatever, you know? So that's basically some sort of like human error or like kind of error in, you know, kind of like in feeding in the data. And by that point, the AI file was trained that it knows that, you know, this feature, change in this feature can really affect the model prediction. And certainly kind of like it, it understands what the right distribution of age is. It understands that age needs to be something between, you know, probably.

• 32:44Speaker 00

  It's seen data from like ten to like ninety or something like that, right? So it understands whenever it sees like a big number like nineteen eighty five, right? There's something wrong here. And then what it can do is basically it can alert and it can prevent that mistake from happening by kind of even replacing, you know, nineteen eighty five by the mode of the distribution or something like that. Right? To kind of put out output like a better, better prediction.

• 33:06Chris Benson?

  I'm curious. It's an interesting approach that you have here. If you are already kind of building your models and you're deploying them out into production, you know, whatever industry you're in. And you have your ML Ops pipeline in place. And on the software side, kind of your DevOps or DevSecOps as it's kind of evolving into a place. How do you integrate this together? How do you take what you're talking about and put into or integrate into your existing pipeline? So that you gain the benefit of what you're, what you're describing and yet not kind of breaking your approach, so to speak, in terms of, of how you're already doing that. How does all that work together?

• 33:45Speaker 00

  Great. So our approach is that the best integration is no integration. And that's why we have, one way to integrate the product is like, kind of, we call this like a light integration. Is where, basically, we're not integrating with the model at all. All we do is we just take prediction logs, right? So if you have like a model that's running and you store prediction logs somewhere, kind of like, meaning that you have like input and the output of the model, and that's stored in some sort of CSV file. Then, basically, our product just, basically, just sits there and runs a CICD process and just continuously reads, you know, that CSV file whenever you dump in, you know, a new kind of like log file. And it just reads it, and that's it. And it just continuously tests it. And we call this continuous testing.

• 34:29Speaker 00

  So we continuously test, you know, the model, right, without ever being in the critical path, with basically zero integration. So it literally is like a, in production, it's like a two-hour integration with Kubernetes. Because, again, it doesn't stand in the critical path of anything like that. When we're doing something like AI Firewall, this is where we're integrating on the, again, it's like a single line of code that we're integrating on the, on the actual model server. And that involves some, you know, some libraries and things like that. But, again, it uses that same principle where ultimately it throws data in the form of like prediction logs in the background so that it doesn't stand in the critical path of, you know, anything real in the, in the system. So that's something that's, that's really important. And a lot of it, we do, we do for customers on-premise.

• 35:14Speaker 00

  Because having your data leave the organization is a huge pain. It's sensitive, you know, there's like a lot of kind of compliance involved and things like that. So on-premise is, is actually something that is very important.

• 35:26Daniel Whitenack?

  I want to follow up a little bit on what you were talking about, sort of this continuous testing, I think, I think is what you called it, which is, is a really cool idea that, that I like. And probably, I don't know if that's less scary to AI people than a word like monitoring or some, you know, something, but I like that idea of continuous testing. I'm wondering, you kind of gave the simple example of like, maybe this distribution of age in the model or something like that. And I'm thinking, there are likely these, maybe these categories, sensitive categories like, you know, personal details of

• 36:03Daniel Whitenack?

  Age, or you know, race, or whatever it is that you're giving examples of like the police scenarios and that sort of thing where these might be categories in which variation in that category should necessarily produce invariance in the in the output, at least if if you're monitoring those well. What's your perspective on that in terms of how to approach those sensitive categories? And is is that something you can kind of program to both in terms of variation and maybe things that should be invariant with change?

• 36:39Speaker 00

  So basically, like in our paradigm, one of the kind of the basic building blocks is the building block of tests. Basically, what you know, you discover whatever it is that you test for, right? If you're not going to test for bias, you're not going to discover bias. But if this is something that you care about, and I think that a lot of practitioners and organizations should very much care about bias, right? Then you test for it, and whenever whenever it exists, you find it. And that's actually a suite of tests that we have embedded in the product that is actually very important, right? So what we do is we automatically test the model. We just automatically go through like all the different categories. And we test like whether there's whether there's bias in prediction, whether there's

• 37:22Speaker 00

  bias in UC, whether there's bias in false positives, false negatives, like all these different things across different categories, right? And that's when people discover like all these biases that they had in the model that they never knew about. Some of it is protected categories, and some of it is just, you know, just other categories that they, you know, that they didn't know, that they found out that their model is just like, they're not training their models on the right data set, or they should do kind of different sampling of the data, right? In order to make sure that the model is not even like kind of performance biased, right? So I think

• 37:50Daniel Whitenack?

  That's cool.

• 37:51Speaker 00

  Testing for bias is such a critical thing that basically all AI practitioners should do.

• 37:56Chris Benson?

  This has been a fascinating exploration and definitely a topic that we have not gotten really into at any point in any previous episode. As we wind up, where do you see the future going, both with robust intelligence as your enterprise, as your company that is doing things, but also within the larger world of robust intelligence, the larger effort in the industry to drive forward and the evolution? We won't hold you to any predictions, but if you were to make some predictions on where you think it will go or where you would like to see it go, I'd love to hear that.

• 38:31Speaker 00

  Sure. So I think like, let's start by the things that are not interesting, right? What we all know is we all know that the world is kind of like, AI is eating the world, right? Like, we're going to be hard pressed to find an organization that is not adopting AI in a serious way in just a couple of years, I think, right? So, okay, that's not interesting, so let's put that aside. But I think when it comes to our little part of the world, when it comes to like AI risk, we think that within just a few years, there are two things. I think that any organization that uses AI in a way that can affect people is going to have to like go through some sort of stress testing by third party. I think that's going to be mandatory. I don't think that it's going to be up to like the director of data science in that company. That's just going

• 39:11Speaker 00

  to be regulation. That's number one. And number two, also on regulation and best practice, I think just a few years from now, in the same way, I think we're going to be hard pressed to find a company that is not protecting its models with an AI firewall. Not necessarily robust intelligence, right? I don't know if any other companies are making AI firewalls, but, but I think that like within a few years, and we're going to have this conversation again, three years from now, we'll go back and remind ourselves like, hey, do you remember that, you know, three years ago, you know, companies were actually deploying AI models without an AI firewall? How crazy was that, right? I think that's what we're going to find. Three years from now, that's where we're going, kind of like in my part of the world.

• 39:51Chris Benson?

  And you should consider that an invitation, by the way, for three years out to come back and we'll have that conversation.

• 39:56Speaker 00

  The data is going to be easy to remember because I think that we're like the last podcast of the year, right? So as we go into 2025, it'll be good to sort of like check, you know, where we were with our predictions.

• 40:07Chris Benson?

  Absolutely.

• 40:08Daniel Whitenack?

  Yeah, for sure. And hopefully sooner than that as well. It's been an absolute pleasure to talk through these things with you. And as for myself, and I'm guessing Chris as well, we appreciate your work in this space and pushing these ideas forward and being that voice that has occasional awkward conversations at AI conferences. It's much needed and appreciate your perspective. So thanks for joining us.

• 40:34Speaker 00

  Awesome, guys. Thank you.

• 40:38Speaker 03

  All right, that's Practical AI for this week. Thanks for listening. If this is your first time with us, subscribe now at practicalai.fm or simply search for Practical AI in your favorite podcast app. We're in there. And if you're a long time listener, do us a solid by recommending the show to a friend. Word of mouth is still the number one way people find new podcasts they love. Special thanks to our partners for supporting our work, Fastly, Launch Darkly, and Linode. We appreciate it. And to the mysterious Breakmaster Cylinder for cranking out new beats for us all the time. That's all for now. We'll talk to you again next week.