Towards stability and robustness

9 out of 10 AI projects don’t end up creating value in production. Why? At least partly because these projects utilize unstable models and drifting data. In this episode, Roey from BeyondMinds gives us some insights on how to filter garbage input, detect risky output, and generally develop more robust AI systems. Sponsors: PSSC Labs https://pssclabs.com/practicalai – Solutions from PSSC Labs provide a cost effective, highly secure, and performance guarantee that organizations need to reach their AI and Machine Learning Goals. Learn more and and get a FREE consultation today at pssclabs.com/practicalai https://pssclabs.com/practicalaiSnowplow Analytics https://snowplowanalytics.com/ – The behavioral data management platform powering your data journey. Capture and process high-quality behavioral data from all your platforms and products and deliver that data to your cloud destination of choice. Get started and experience Snowplow data for yourself at snowplowanalytics.com https://snowplowanalytics.comChangelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Featuring: Roey Mechrez – LinkedIn https://www.linkedin.com/in/roey-mechrez, X https://x.com/roeymeChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: NVIDIA GTC21 Talk on “Addressing Garbage In Garbage Out” https://youtu.be/5cp_JBrDZvkBeyondMinds https://beyondminds.ai/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Problems, data, requirements, business process, constraints are so specific, right? And most of the problems cannot be solved repetitively good enough. And many times I tell my team at Beyond Minds that state-of-the-art and models are a critical part of the solution, right? But they're probably 5%. And we need to do a lot of problems to build a system that can deliver in production.

• 0:32Speaker 01

  Big thanks to our partners Leno, Fastly, and Launch Darkly. We love Leno. They keep it fast and simple. Check them out at leno.com/Changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com. And get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com.

• 0:51Speaker 02

  With advancements in AI and deep learning evolving at lightning pace, it's more important now than ever to research the best options suited to your unique needs. This is particularly true when building custom systems and those systems that are GPU-heavy. Not only do the applications running on the system matter, but your AI infrastructure and budget constraints need to be front of mind as well. PSSC Labs, which is an HPC and AI custom solutions provider based in California, has been creating high-performance computing systems to meet their clients' unique enterprise computing challenges for more than 25 years. And with cloud computing costs growing at astronomical rates, plus companies increasingly losing control of

• 1:33Speaker 02

  their data security, it is no wonder that enterprises and government agencies need to continually look for ways to take back control of their data. Solutions from PSSC Labs provide a cost-effective, highly secure, and performance guaranteed that organizations need to reach their AI and machine learning goals. For more information and a free consultation, please visit pssclabs.com-practicalai. Once again, that's pssclabs.com-practicalai.

• 2:11Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM.

• 2:37Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

• 2:52Speaker 04

  Doing great, Daniel. How's it going today?

• 2:54Daniel Whitenack?

  It's going great. And you know, this is Practical AI and I'm super excited because today's episode is super practical. We have with us Roy McRez, who is CTO at Beyond Minds. I saw a talk by him at GTC about garbage in, garbage out, which was really cool. So, welcome, Roy. Really excited to chat about these things.

• 3:18Speaker 00

  Yeah. Hi, guys. Hi, Daniel. Hi, Chris. Happy to be here and talk to the audience about Practical AI elements.

• 3:25Daniel Whitenack?

  Yeah, definitely. You're the right person for this podcast, for sure. Your talk, which we'll link in our show notes, for sure, it was really good. I really enjoyed the practicalities of it, but also the emphasis on, hey, what does, you know, what are some of these challenges and blockers that prevent AI from producing value when you're trying to deploy it? So I'm wondering if maybe before we jump into the specifics of that, if you could just give us a little bit of information about how you got particularly interested in that topic and ended up, you know, talking about it in that context.

• 4:00Speaker 00

  Yeah, brilliant. So again, happy to be here and talk to you about this amazing topic. So, you know, I came from academic background. I think the last 10 years probably I'm doing computer vision for my living. Started in biomedical engineering, working on some medical imaging problems. Then I worked a little bit on consultancy, some computer vision project. Shifted into a PhD in computer vision, wrote several papers, were jumping around between CVPR and ECCV and talking about some very

• 4:35Daniel Whitenack?

  Doing the circuit.

• 4:35Speaker 00

  Yeah, exactly. Some very academic ideas and some, all over the place. I published papers on a random set of topics. And then I met my co-founder and we detect this problem where the academic research is amazing. Right? Three percent of all papers in the world are about AI. That's just mind blowing.

• 4:57Daniel Whitenack?

  Oh, really?

• 4:58Speaker 00

  That's what I read. I need to look.

• 5:00Daniel Whitenack?

  Is that like a recent statistic or maybe it's more now? I don't know. That's pretty amazing.

• 5:05Speaker 00

  Yeah, yeah. On a broad topic, right? But then in, at least in enterprise AI, right? Outside Facebook, Google and Amazon, the use of AI is very low and most severely, they're trying, but there is ninety percent failure rate. There is a colossealical problem there. Systematical failure. And we establish Beyond Minds, that's the company that I'm its founder in the last three years. And that's what I've been doing for a living in order to bridge that gap between academic research and practical AI and production, right? And real value from AI. That's my passion. That's what I'm doing as the CTO, as a co-founder.

• 5:47Speaker 00

  That's the company that I'm trying to build. And we're coming from a very technical data science, machine learning, AI background. Both me and the CEO, which is also coming from an electric engineering background. And we understand AI, we understand the concept. We build many, many different models, train lots of problems. But the reality, we're it meets the reality, there is a gap there. There are challenges. And if we want, we can dive a bit more into these challenges, but they are not organizational. They're technical problems.

• 6:24Daniel Whitenack?

  Yeah, and I'm curious, since you did that sort of circuit of the academic research and published many things, but now you're sort of helping clients productionize and operationalize their models now, because I'm assuming you probably still read some of that research and keep up with it. Now, when you look at that research, do you read papers differently, or like, look at what people are doing differently in the sense of like, understanding what might be able to create value in the enterprise versus not?

• 6:59Speaker 00

  That's a philosophical topic, like, what is the role of academia versus the role of the industry around AI? And I think it's good that academia is dealing with, you know, holistic problems, right, theoretical problems. That's its role, right? And I think, you know, in AI specifically, we are shifting towards different areas around more practical AI academic research, definitely when the number one affiliate or the tried AI is Google, and probably the second one is DeepMind, and then, you know, some other universities. But in general, I think it's good to have people that are dealing with SOTA, right,

• 7:39Speaker 00

  state of the art. And I had this discussion with another, with my supervisor during my PhD yesterday. And she told me that, you know, they're dealing with SOTA on one hand, but very down-to-earth, algorithmical way of thinking, what is needed to solve that problem, not what is state of the art. And that's a completely different mindset, right? And, you know, it could be that there are people that like different things. And I think, by the way, that's probably one of the problems, because if you are an insurance company, and you've built your center of excellence, and you went to Stanford and MIT, and brought the best, the best data scientists and PhDs from there, it's probably not likely

• 8:24Speaker 00

  that these people will know how to build a product in production that can actually deliver. They can do amazing research, and novelty, and innovation, right, but not necessarily deal with some production elements.

• 8:40Speaker 04

  Yeah, the last my day, just to be transparent on this, was like, sure, I thought my day was going to involve doing some AI, but really I just tried to convert audio files off of an RTSP stream into like a usable format into it and not get garbage out. That was basically my day. I didn't, I didn't train anything. I didn't even pre-process, you know, anything major. I just worked on like this file conversion.

• 9:08Speaker 00

  Yeah. On the other hand, by the way, you mentioned papers, right? My rule of thumb is paper a day. I read paper a day, not, you know, not deeply, maybe deeply like every week, once, two weeks. But I want to read an abstract, look at the figures, understand a concept once a day. And, you know, we all the time use open source, new ideas. And many times I tell my, I tell my team at Beyond Minds, which is more than 50 engineers today, that state of the art and models are critical part of the solution, right? But they're probably 5%. And we need to do a lot of problems to build this system that can deliver in production. I think that's, you know, an essence. That's my belief, right?

• 9:52Speaker 00

  From a model to a system. The model is maybe the heart, but there are so many different components there.

• 9:58Speaker 04

  You know, while we're talking about that, you've said several things that really, really grabbed me in the last couple of minutes. And we talked about the difference in academia and industry and stuff. And we also noted that there's that crossover. You have, you know, you have Google going into Stanford and cleaning out the department and pulling all of those experts in so they can take advantage of it. And so we're seeing a little bit of a difference in this practical, when, you know, when we talk about garbage in, garbage out, and the incredibly practical nature of this as data science, we are seeing a divide in industry where you do have academia and you have a kind of a select group of organizations that understand this and that they are able to do it and that they can take

• 10:40Speaker 04

  an idea all the way through the work stream and produce that system that's highly productive down the road. But then you're also seeing a large number of organizations, probably well Well, over 99% of them out there that are really struggling with that. And before we dive all the way into the details of your talk, which I also saw your video. So I kind of know what's coming. I'm looking forward to this. But there's this big chasm right there. How is that cross? Because the understanding of these, of the data science of it and how to actually do it end to end. And having all of the steps and skills necessary.

• 11:12Speaker 00

  It's a lot to know.

• 11:13Speaker 04

  Is a lot to know.

• 11:15Speaker 00

  Yeah.

• 11:16Speaker 04

  What are your thoughts about that? How does that get reconciled going forward?

• 11:20Speaker 00

  Well, I had a conversation with one of the tier one banks a few weeks ago. As part of my role as a CTO, I do a lot of sales meetings. I meet clients occasionally and definitely part of our go to market and vertical is around financial services. So anyway, a tier one bank in the US, they have 1,000 engineers in the center of excellence around AI, right? Okay, so that's tier one bank. But even if you go to tier two bank, let's say, you know, Fortune 1000 company, right? They will build a center of excellence with tens of engineers. They will give a good emphasis on bringing people and talent and organizational elements, strategy,

• 12:03Speaker 00

  and educating the executives, et cetera, et cetera.

• 12:07Speaker 04

  Yeah.

• 12:07Speaker 00

  But they're still struggling for some complex elements. And I think the big element is becoming a technology driven company, right? Because Google and Amazon are not on the same challenges, but that's extreme cases, but also not other technology companies. I think when it comes to technology driven companies, people that understand that, understand engineering in their essence, right? That the product is technology versus the product is bank or insurance or manufacturing or telco or mining or oil and gas or whatever, most of the world, right? Most of the economic. They are not technology companies and building a technology company, that's super complex, right? And it's not unique to AI, in my point of view, by the way.

• 12:52Speaker 00

  If you look at, I don't know, CRM or ERPs in the late 90s or beginning of 2000, right? So many people spent $5 million on trying to build CRM and then they bought Salesforce, right? And then, you know, that's a super classic biverses bill dilemma, which I think we are super not balanced in, in AI, for sure. And for good reason, by the way. That's one of my key assumptions as a company, my key observation about AI. If you want, I can tell you about that.

• 13:23Daniel Whitenack?

  Absolutely. Could you go into that in a little bit more detail?

• 13:27Speaker 00

  Yeah, for sure.

• 13:28Speaker 04

  He's got us pulling it out of our hair.

• 13:29Daniel Whitenack?

  Yeah, I know. You put out the carrot there.

• 13:33Speaker 00

  Specificity. Problems, data, requirements, business process, constraints are so specific, right? And most of the problems cannot be solved repetitively good enough, right? Even if you take something that sounds super repetitive, defect detection in manufacturing, right? We need to collect data, annotate bounding boxes, and then train Yolo, right? And we have a defect detection, right? We have an object detector. Amazing. But the reality shows that, you know, that's so different for production line. One is defects that are glass for smartphones and the other one is PCB boards. And then you have anomalies type

• 14:17Speaker 00

  of data and you have very clear defined defects and transparency and light. And what is the dynamic of the data? Is it shifting or it's very consistent? How you deal with the monitoring of that problem? Do you have, if it's a, is it bottles, shampoo bottles that you don't really care about accuracy? If it's more than 95, you're okay? Or it's PCB boards that go into Lockheed Martin engines and you are super sensitive, like zero tolerance to errors, right? You have human in the loop? You don't have human in the loop? Is that a super 10

• 14:55Speaker 00

  GPUs per production line and you have the most amazing model? Or it's a small IoT device that you need to shrink and quantize and optimize and deploy in C++ or whatever? So many different requirements and constraints and how the business is working, how many people and you need to deal with that, right? And how you deal with that? If it cannot be solved by an off-the-shelf product, I cannot go and buy something. So, okay, let's build an internal team, right? They will solve my problem. But then you come to another problems, which is how internal teams are becoming a technology company, right? How to deal with data? How to build processes? How to build repetitions?

• 15:39Speaker 00

  Scale all these elements that are usually the mindset of startups, of technology companies. And that's what I'm trying to bridge in life, right? Helping enterprise bridging AI into production, right? Can give you an example. I have a company that build something regarding sales prediction. And they have thirty different lines of sale, thirty different products. And they built an amazing model for one product. And scaling that to thirty required them to multiply the team. Because they need, they have, for example, someone that whole his life has his responsibility

• 16:19Speaker 00

  as a data engineer or a junior data scientist or whatever, to retrain the model every week because there is new data, something is changing, the hyper parameters need to be tuned. How you do that now? Multiply by thirty, scale.

• 16:49Speaker 01

  This episode is brought to you by Snowplow Analytics. Snowplow is the behavioral data management platform for data teams. Maximize the value of your behavioral data using Snowplow Insights. A managed data platform that's built on leading open source tech leveraged by tens of thousands of users. Capture and process high-quality behavioral data from all your platforms and your products and deliver that data to your cloud destination of choice. When marketing needs to make data-informed decisions, when product needs Next level understanding and when analytics needs rich and accurate data, snowplow is the solution for data teams who want to manage the collection, processing and warehousing of data across all their platforms and products.

• 17:31Speaker 01

  Get started and experience snowplow data for yourself at snowplowanalytics.com. Again, snowplowanalytics.com.

• 17:51Daniel Whitenack?

  Roy, I really appreciate your perspective on that. I think that the way that you think about specificity and scaling a data team is definitely a good perspective. I mean, we just talked on our conversation between Chris and I about GitHub Copilot. And like, oh, you know, there's people like, oh, now AI is writing our code. And, you know, people have been saying that like, software engineers are going to automate themselves away for however many decades it's been. A long time now. But, you know, there's such a need for specific solutions to specific problems in even just software engineering, right? And then if you bring AI into that, which AI is now sort of, like you said, infiltrating all of these different companies at all different sizes

• 18:35Daniel Whitenack?

  and all different ways, in very specific ways, then there's a lot to solve there. One of the things that I saw you talked about a little bit was this idea of stability. Could you give us a sense of, you know, what does stability mean in the world of, maybe in the world of software engineering, but then moving into the world of AI? How is stability something that needs to be, you know, on our minds as we productionize AI?

• 19:02Speaker 00

  Yeah, sure. So, you know, I think if you're coming from an electrical engineering background, so stability is something that is usually defined for, let's say, linear system or something like that, right? A bounding input, bounding output or something like that. But definitely in deep learning system, which are far from linear and far from being well-understand in terms of what is bounding even. What's going on with the data in terms of stability? What data is going to break my model, right? And you clearly know as a data scientist that AI is usually when it fails, it fails silently. Right? I gave in the video, if you watch it, right? So I call it the giraffe problem, right?

• 19:45Speaker 00

  The most naive classifier probably that you did.

• 19:48Speaker 04

  You gotta tell that joke from your video. You have to tell it, okay?

• 19:51Speaker 00

  I will do my best. Yeah, so so the giraffe problem, right? A giraffe enters a bar and the bartender is your classifier, basically. And he tells the giraffe, you are hundred percent a dog. I'm I'm sure of it, right? And you know, that's the reality in in deep learning, because the model as the bartender, right? He saw only cats and dogs. That's what he knows. And he will classify everything as a cat or a dog, even if it's a giraffe. And let's say as an extreme toy example, but the reality is full of this problem. Data shifts, noise, someone changed the the light bulb.

• 20:31Speaker 00

  The bias in the image is slightly shifted by two gray levels, right? The text is one tab corrupted, right?

• 20:42Speaker 04

  Yeah.

• 20:43Speaker 00

  Et cetera. And how you deal with that?

• 20:45Speaker 04

  There is no practitioner of deep learning, I think, who has done any amount of work in the field, who's going to not recognize the reality that that joke represents.

• 20:55Speaker 00

  Yeah.

• 20:55Speaker 04

  That is true. We have all experienced that.

• 20:57Daniel Whitenack?

  Yeah, I always use the example. It's like if you trained a self-driving car to drive perfectly and but you did it in Sweden and then you take that car and you plop it down in Australia. The first time you run across a kangaroo running across the road, something very interesting is going to happen. I mean, maybe something really bad or just something really interesting. But yeah, it occurs also. I mean, we're talking about a lot about images, but also for text and other things as well.

• 21:25Speaker 00

  Definitely.

• 21:26Daniel Whitenack?

  Yeah.

• 21:27Speaker 00

  And I think, you know, stability is well coupled with the ability of the model to generalize, right? Hypothetically, you have a dataset that is the only distribution of whatever, and you can generalize well, right? But that's clearly not practical. You don't have access to the entire distribution, and you cannot generalize to the entire distribution, obviously. And you definitely don't have labels. But when you shift into practice AI, I don't care about generalization. I want to be good on what I need to be good at and make sure that nothing that out of that good will come into my model, right? And that's why I call it garbage in, garbage out, right?

• 22:10Speaker 00

  I want to make sure that I have a training set. I have its distribution. And that's the data that I'm going to get. How are we going to achieve that, right? And I think that's one of the most important problems that cause this failure rate, the 90% failure rate that I talked about in the beginning. Because AI researchers, data scientists, they know how to build models, and they can achieve this 95% accuracy in the lab, for sure, with a stable, static dataset, right? The production is not stable, is not static, and it's definitely dynamic and it's going to shift.

• 22:52Speaker 00

  Data drifts, data shifts, whatever, something is going to change. And how you react to that changes? And why it's important, by the way. I think why we're progressing with value around AI, we're also shifting from, let's say, statistical applications, which I define as something where the difference between 80 or 81 or 82% accuracy is, okay, nice, better to the business. But there are mission-critical problems. Claim assessment in insurance, anti-money laundering, fraud detection, defect detection in PCB board, whatever. These are mission-critical, right? You know, the difference between 80 and 81 is...

• 23:37Speaker 00

  Is critical. It could be massive to the business. And I'm not even talking about applications where the business requirement is 99. I need 99. The model can achieve 90. What you do now? And that can be breached, right? And there is a way.

• 23:53Daniel Whitenack?

  Yeah, I think that now that AI is intersecting with all of those different areas. I mean, you don't have to go far in a business to encounter one of those problems where if you do behave massively different than you did in the lab, then you're going to make a major impact to the business for sure. I'm wondering, you mentioned this sort of idea that, you know, you want to make sure that the input that you give to your model is something that you expect in your input distribution. I'm wondering how maybe that mindset compares to... People talk about making their models robust, and you see people like maybe perturbing their image data sets

• 24:37Daniel Whitenack?

  to like flip them all around or enlarge certain things. Or maybe like there's that OpenAI example where the robot's moving the Rubik's Cube and they poke it with a stick or a stuffed giraffe. You know, giraffes are a major theme in this conversation, I guess. But one approach seems to be thinking about like making your model able to sort of withstand or be robust against perturbations. And maybe the other is concerned with making sure that your model never sees things that it shouldn't see. Is that a proper distinction? Or how would you sort of look at those two approaches?

• 25:12Speaker 00

  Yeah, I think it's a good way of putting that, right? You know, I'm sure all the audience, including you two, are following Andrew and Greg. You know, data is the important part here. And you know, if I can't control the data, I can't control the model and better data, etc. Super important elements, right? And you know, in the ideal world, I want to have access to all the data that I need in the lab. And I want the data in production to be static, right? Ideally. And unfortunately, that's not practically many, many areas, right? And we can do our best. By the way, I think that's one of the key consideration of tech companies that make them better is because data as a strategy is so, so significant within what they're

• 25:57Speaker 00

  doing from the very first mindset, right? How I going to collect the data, is it's going to be clean, how I going to train down the line amazing AI models of that and not, okay, let's look at the entire organization, what data I have, what I can do with AI now.

• 26:14Speaker 04

  So what, what kind of strategies do you have to deal with that in the sense of, so, you know, we have this capability to produce these models in some sort of controlled environment, they're going out there in the world and they're being subjected to this. What are some of the strategies that you use to address that?

• 26:30Speaker 00

  Yeah, so the two most critical elements that I think are very, very straightforward, but not easy to build, out of distribution detection. I want to make sure that the data that comes into the model is within the distribution. Now, let's say quite, let's say it's not super new research area, but it's not super, super advanced. So I think there is active research out of distribution detection and how to apply it to some kind of areas. But if you can build this amazing filter that can detect data that is out of distribution and make sure that it's not going into the model, detect the giraffes, right? With respect to the cat and dogs images, you will be better.

• 27:14Speaker 00

  Then you have the other part, which is confidence or uncertainty, right? The research call it uncertainty estimation. I like the term confidence estimation because it's more straightforward than not the negative. So if you have good confidence of prediction, right? I got an image, went to the model and then the model give me the prediction, right? Soft max classification, whatever, right? It's probably 95% sure about that in terms of probability and well, that's not, I'm putting in the air here, that's

• 27:48Speaker 04

  Air quotes there.

• 27:49Speaker 00

  Air quotes, yeah, that is not really probability of course, but it's clearly not, right? That's part of the problem and I want to estimate if the model is sure about this prediction, right? This usually term uncertainty estimation or confidence and how to scale that and how to calibrate that and the temperature scaling, whatever. These techniques enable you to say, wait, the model is suddenly not that sure that it's a dog. And I can really use that number. And going back to the mission critical elements, now I can threshold something and make sure that if the confidence is below the threshold, let's pass that to a human, right? I have a human in the loop in many applications, but I not necessarily know how to use it efficiently.

• 28:34Speaker 00

  So if I know how to use it efficiently, because I have the out of distribution detection and have confidence estimation. So I can now make sure that my model is somewhat now more stable. It has its defense mechanism. Garbage won't come in. And if semi garbage will come in, I will have a confidence score that will tell me suspicious. And that will improve my stability 100%. And it will improve my ability to combine this model with human, which is super important in many applications.

• 29:23Speaker 02

  Change log plus plus is the best way for you to directly support Practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of Practical AI into the future. Simply follow the Change Log Plus Plus link in your show notes or point your favorite web browser to changelog.com/plusplus. Once again, that's changelog.com/plusplus.

• 30:07Daniel Whitenack?

  So you talk about these sort of two filters, one on the data coming in, out of distribution filter, and one on the data coming out of the model in terms of quality estimation, or confidence scoring, or risk assessment of the output of your model, and forwarding that on to humans. In your experience working with companies, one of the things that I'm always curious about is like how you start to judge, you know, what you should show to humans versus what you shouldn't show to humans, and figuring out where that threshold is. Because if you go one way too far, then maybe you're letting garbage through, right? And that's causing actual problems

• 30:51Daniel Whitenack?

  in your systems and all of that. If you go too far the other way, then you sort of start eating away at the, maybe the good graces that you've built up in terms of making things more efficient. Well, now you're just sending all of this stuff to humans, and you know, maybe it was all right, and they're not changing much about the output. How do you sort of dial in that sort of system?

• 31:14Speaker 00

  Yeah, well, you know, I don't have an amazing rule of thumb to answer your question. But what I can say that is, it's this mindset of AI solution in an organization is not a technical problem per se. It's a business technology problem that combines some business decisions, business process, and technology understanding. And I think if we go back like three years to 2018, I think the first wave of AI challenges were characterized by understanding where to start, what AI can do, understanding of manager and executives, what about AI?

• 31:57Speaker 00

  But while we're progressing in that and we have better and better decision makers that understand AI and responsible AI and what AI can deliver. And they did some AI for managers course at whatever university or online course. And we can have this open discussion, right? And product managers that are understanding technology, but understanding also the business. And I think, you know, I think even today we have the AI product manager role, right? That exists in some companies, which is amazing. And that's enabled us to take better decisions, to develop better products, right? That are driven on AI. I think that's the main take, right?

• 32:39Speaker 00

  You need business people from stage one along the way, right? To take decision, to be involved in what technology can develop.

• 32:50Speaker 04

  So I love where you're going there. And I actually want to go there even even a little bit farther with you if you would. Because as we've come through the conversation and you've kind of delineated these strategies, you know, both on the front end and on the back end to solve both sides of that. You know, let's bring it back to this practical aspect of you're in your business and you're trying to get some stuff done. And you have a product manager who's trying to contend with this. What are some of the things that you have seen in the market to actually try to implement these strategies and thus mitigate the problems that we're talking about in real life? And I'm not saying it's holistic or a complete list, but what are some practical things that listeners who might be facing their own set of problems want to do?

• 33:33Speaker 04

  You know, whether it be data engineering on the front end of some sort or what? What have you seen?

• 33:38Speaker 00

  Well, here I have a rule of thumb. Don't do P O Cs, right? Try to avoid as much as possible from building this model in the lab, right?

• 33:48Speaker 04

  Yes.

• 33:48Speaker 00

  Try to understand what are the production challenges from day one. For example, very classical problem is around data, right? You want to make sure that the data that you are training your model is exactly identical from the data in production or even take the data from production to train on it, right? And make sure that the acquisition in production of the data or the data that goes around to your model is exactly the same, right? It's the same distribution. It's the same problems. It's the same noise. Try to understand the hardware, the requirements. Who is going to work with that? How is going to work? Is the data is shifting over time? Do I need to sample differently?

• 34:32Speaker 00

  Do I have a monitoring element? That's bring me actually to a super important factor about AI in production, which I think is interesting, which is highly related is evergreen AI, right? This AI over time, right? It's not, I build a model in product in the lab. I got ninety five percent. I put it in production. I'm happy. It's bring me value. Oop, three months pass. The data was changing. The model doesn't work anymore. I need to go back, recollect data, annotate, retrain, deploy again. Okay, it's working again, right? How you deal with this cycle efficiently? Because none of the organization is talking about one AI model, right? That's not the goal anywhere, right?

• 35:16Speaker 00

  We're talking about adoption, acceleration, mass organization change, transformation, et cetera. And I think Facebook published a few weeks ago that they, what was that? That they shift all the models in production to PyTorch. I think that was the topic. And the amount of models that they are running is just mind blowing, right? And yeah, everybody want to be, not everybody want to be Facebook, but I mean in terms of the technology wise and AI wise, everybody want to be Facebook. Wishing they could pull that off. Yeah, exactly.

• 35:51Speaker 04

  It's funny that you say that is you're talking about, you know, trying to get to where you're truly reflecting production. And in my experience, working with several organizations, I've often seen an attempt by the data science team working on it to do almost the opposite because they perceive that the ability to get to production data and to move with that is somewhat overwhelming. It's a daunting thing for them because it's a moving target. It's incredibly dirty often that. Maybe they're not software engineers. Absolutely. And so when you're saying that I'm thinner going, that what you're saying makes perfect sense. And yet I've literally seen teams trying to do the opposite of that repeatedly.

• 36:34Speaker 04

  So, I mean, that's really good advice you're making there.

• 36:37Speaker 00

  Yeah, yeah, I totally agree. And you know, you said about the software engineer as a joke. You know, that's not totally a joke, because to build an API that will sample data from production to your training environment and will deploy that efficiently, that's clearly a software problem, right? And how to do that consistently and monitoring, that's a scientist build models, right? And solve algorithmic problems, but to build a system and etcetera, you need engineers, you need software engineers, right? And I think that's at least my belief is that squads are probably the way of solving problems, right? You need engineers, you need data people, you need data scientists, and you need them to work together on the problem.

• 37:22Speaker 00

  And think about production, and think about all the requirements, and you need product managers, right? And you need, you need the business people as much as combine with mutual communication, same language, if possible, right? In order to tackle this problem. And I think, take for example, something that is all the time coming again and again. Security and governance, right? And maybe like, you know, if we're going to financial services, regulation, right? I'm sure non-data scientists will take these elements into consideration without the proper guidance and business people shouting at him, I need explainability.

• 38:01Speaker 00

  It doesn't make sense without explainability, because I need to show the regulator how this model is taking actions. Et cetera, right? You have risk departments and, you know, if the organization become large and complex like a bank, it's become more and more complex to bring AI to production, and you need more and more elements. Alright, so now we need explainability and confidence and out of distribution detection. And we need to monitor the model, and we need to make sure that the data is feeding in same from production and training. And we need to be able to work with the human in the loop and retrain the model, right? And how to keep these models on the rails, right? How to deal with maintenance? So many elements besides the model itself, right?

• 38:44Speaker 00

  Going back to what I told you in the beginning, shifting from a model to a system, right? We need so many different elements. And then how you scale that, that's a problem.

• 38:53Daniel Whitenack?

  It almost sounds like most of the work is after the initial model training. Who would have thought? The other thing I'm thinking about while you're discussing this is that it is necessary to bring all of those people together, like you're talking about. In particular, I think the infrastructure and software engineering and data science people, because software engineers, in terms of the stability that they think of, they think of it like, oh, I'm gonna write my unit test, right? If I change my code and it breaks the test, then I know that, you know, I broke my code. And I also have this, you know, table of, you know, calls that I'm testing against, and my API expects this data, and it either works or it doesn't. It gets that data or it doesn't.

• 39:37Daniel Whitenack?

  Whereas in AI, like you're talking about, you know, just testing that your model can take in a certain type of data and output the expected type of data, which is sort of what we would normally think of in testing, isn't enough, because the data and the distribution of data that you're feeding in could change the behavior wildly, right? So that monitoring piece, like the monitoring and testing piece, you sort of say those words to software and infrastructure people, and they think of one thing. But for AI, it is sort of that, but it's not exactly that. It's that and more, I guess, is what I'm after. Do you see it the same way?

• 40:16Speaker 00

  Yeah, totally. I think that every one of the problems that I mentioned are both software engineering problem, but also algorithmic problem. Take monitoring, for example, right? Every software engineer understands what monitoring is, right? That's something that they have been doing for 30 years, right? Monitoring for any system. You need to make sure logs, et cetera, and understand when it fails, et cetera. But it's such a complex research problem to monitor a model. How you make sure when your model is failing, if the data is changing, these are all research problems. If the statistic of the data is changing, if the model is suddenly not well-suited,

• 40:56Speaker 00

  where I'm exactly on the distribution versus the data that comes in, how I'm looking at that. The vectors that come out of the models and understand what's going on, right? Of the model or take that into, you know, the hot word today, monitoring is observability, right? Is bringing observation, what is happening, right? That's even more complex to understand what is happening and why, what is failing. Well, that's become even, you know, more complex. So people need to work together. There are many different elements to this problem. Software, research, data, infrastructure, et cetera. And my takeaway, that's what I have been doing in the last three years, is building

• 41:41Speaker 00

  a platform that is dealing with these elements in a unified way, right? I believe that you can generalize many of these problems in a way that will enable scale, right? It's required lots of work. The problems need to be generalized enough. You need to think about these technology elements across the pipeline of the system. But that can be done also in a small scale in an enterprise. But of course, my, you know, I have my own way.

• 42:09Speaker 04

  Well, I was gonna say, you've brought that up in your own way. Would you take a moment or two and, and tell us about the specific work that you've been doing in that area? I'd love to know how you are tackling these problems that we've been talking about for a while.

• 42:21Speaker 00

  Yeah, yeah, sure. Of course, you know, I, I won't be able to dive into all the fine details, but the idea of what Beyond Minds is doing in the last three years, we have been developing a platform, right? And this platform is not a developer tool. It's something that we use in order to provide solution to enterprises on scale, faster, better, and crossing and bridging this failure rate, right? And the components, the main components of this platform is a system that is wrapping the model, right? Trying to solve problems that are agnostic to the model as part, as past, as, as much as possible. Let's take out the distribution, for example. We have a component, that's all its work, right?

• 43:05Speaker 00

  Out of distribution is one method out of many others that we're using there to measure statistics and data and garbage in to make sure that garbage is not coming in. So we build an entire component. We call it an input gate. Yeah. And it's an entire research arena that is dealing with stability. And then monitoring and observability, right? That's another one that is. As much as possible model agnostic. Not all of these components can be totally model agnostic. Some of them need to be tweaked. For example, out of distribution detection, that's highly coupled with the data itself, right? I need to retrain that. But that's again, something that can be scaled. How to retrain, right?

• 43:49Speaker 00

  How to do that efficiently that I can bring solution to production in a matter of weeks, months. And I think today, probably 12 to 14 months. I think that's the average time to production in an enterprise. Our goal as a company should take that into a scale of weeks, right? 12 weeks will be amazing outcome for us. And that's enabled because of the platform, right? Building the model, that's we will have to do. That is customizable. But then the monitoring can come easily. How to combine human in the loop can come easily. Explainability for most problems that are required that can come easily. Stability can improve the model.

• 44:33Speaker 00

  I have a retraining in production. I have an entire monitoring and solution management toolbox that I were using. So I have this entire platform that is wrapping the model and then bring them to production.

• 44:47Daniel Whitenack?

  It sounds very similar to the conversation that we had with William Falcon around PyTorch Lightning in that one of the things he was observing as he was working with different companies and models is that developers were over and over again, fighting with the code that they were writing to deal with the hardware, right? Like multi GPUs versus single GPUs versus this GPU versus that. And so in PyTorch Lightning, the philosophy was to sort of decouple the model piece from that hardware stuff that everybody was trying to write. And that, you know, provided the boost that PyTorch Lightning is giving people. And it sounds like here, there are these other components, and actually a very large

• 45:30Daniel Whitenack?

  number of them, that data scientists are also, you know, spending their time writing code on over and over and over again. There's the stability, the monitor. The human in the loop piece, all of these pieces that promote stability and robustness. And they're having to do this over and over, so it's not efficient at all. So I definitely think that, you know, that mindset that you have at Beyond Minds is really interesting in that sense that it could provide that sort of similar boost in these other areas, which is really exciting.

• 46:02Speaker 04

  He's bringing, I mean, as I was listening to that, the thing that was going through my mind is he's bringing really good architectural decisions to kind of to everyone. And we have seen that often, you know, people in data science haven't necessarily had that exposure, you know, to the software side where that as it matured, as the field matured, that became part of it. So I love what you're saying because I think that really contributes toward the maturing of this industry that we're all in at this point.

• 46:30Speaker 00

  Yeah. Yeah. And, you know, I mentioned lots of research problem, but there are also many engineering problems there, right? Kubernetes and Docker and APIs and so many different elements, how to wrap models and deploy them efficiently and fast. And this is becoming a common practice more and more, but there are still some elements that are not that trivial.

• 46:56Daniel Whitenack?

  Well, I really appreciate you helping our listeners think through some of these things. I know my mind has been racing the whole time, thinking about like five different projects that I have going on and how these things map into those projects, like every single one of them, how these things map into that. So I know it's been really good, good in that sense for me, and I'm sure it will be for our listeners as well. We'll include show notes in the show notes, links to the talk and other things that you've put out. But thank you so much for joining us, Roy. It's been a pleasure.

• 47:26Speaker 00

  Thank you very much. It was a pleasure talking with you.

• 47:32Speaker 02

  Thank you for listening to Practical AI. We appreciate your time and your attention. If you enjoyed this episode, help us out by spreading the word. Think of a friend, think of a colleague, somebody who would benefit from listening to it, and send them a link. We'd really appreciate it. Practical AI is hosted by Chris Benson and Daniel Whitenack. It's produced by Jared Santo with music by Brake Master Cylinder. Thanks again to our sponsors, Fastly, Linode, and Launch Darkly. That's our show. We hope you enjoyed it, and we'll talk to you again next week.