Broccoli AI at its best 🥦

We discussed “🥦 Broccoli AI” a couple weeks ago, which is the kind of AI that is actually good/healthy for a real world business. Bengsoon Chuah, a data scientist working in the energy sector, joins us to discuss developing and deploying NLP pipelines in that environment. We talk about good/healthy ways of introducing AI in a company that uses on-prem infrastructure, has few data science professionals, and operates in high risk environments. Sponsors: Intel Innovation 2024 https://intel.com/innovation?regcode=CMCCHL&utm_campaign=Changelog – Early bird registration is now open for Intel Innovation 2024 in San Jose, CA! Learn more https://intel.com/innovation?regcode=CMCCHL&utm_campaign=Changelog OR register https://reg.oneventseries.intel.com/flow/intel/innv2024/InnovationReg?regcode=CMCCHL&utm_campaign=ChangelogMotific https://www.motific.ai/ – Accelerate your GenAI adoption journey. Rapidly deliver trustworthy GenAI assistants. Learn more at motific.ai https://www.motific.ai/Featuring: Bengsoon Chuah – GitHub https://github.com/bengsoon, LinkedIn https://www.linkedin.com/in/bengsoon, X https://x.com/bengsoonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: MLFlow https://mlflow.org/Prefect https://www.prefect.ioDuckDB https://duckdb.org/Agrilla https://argilla.io/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents, so you can launch your app near your users. Learn more at fly.io.

• 0:44Speaker 01

  What's up, friends? Intel Innovation 2024 is right around the corner. Accelerate the Future registration is now open, and it takes place September 24th and 25th in San Jose, California. This event is all about you, the developer, the community, and the critical role you play in tackling the toughest challenges across the industry. Ignite your passion for AI and beyond, grow your skills to maximize your impact, and network with your peers as they unleash the next wave of advancements in technology. Here's what you can expect. Understand the emerging innovation and trends in dev tools, languages, frameworks, and technologies in AI and beyond to empower you and the solutions you're building.

• 1:28Speaker 01

  Get in-depth technical experience doing hands-on workshops, labs, meetups, and hackathons to collaborate and solve problems in real time. You can explore featured partner and Intel solutions. They have partners there, startups there, customers there. And Intel is showcasing the latest in products, services, and solutions across keynotes, tech sessions, and the show floor to help you meet your development needs. Collaborate with experts, learn and have fun. Engage in interactive sessions to connect, get certified, gain unique ideas and perspectives, build long-lasting networks, and of course, have fun and get inspired. Hear from leading industry experts, technologists, startups.

• 2:10Speaker 01

  Entrepreneurs and fellow developers along with Intel leadership CEO Pat Gelsinger and CTO Greg Lavender as they take you through the latest advancements in technology. Don't miss this chance to be at the forefront of innovation. Take advantage of early bird pricing right now until August 2nd. Register using the link in our show notes or to learn more, go to intel.com/innovation. Once more, that's intel.com/innovation or go to the show notes and click that link.

• 2:46Daniel Whitenack?

  Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am founder and CEO at Prediction Guard. And this is a, this is a pretty special and, and fun episode for me because I get to kick back with an old friend of mine. We, we went to the same university, for those that haven't heard of it, Colorado School of Mines in, in Golden, Colorado. Of course, a shout out to all the ore diggers out there that are, that are listening. But yeah, we, we have, we have with us today Bing Sun Chuah, who's a data scientist and now, and working in the, in the energy sector.

• 3:28Daniel Whitenack?

  I was really fascinated to talk over the years with, with Bing Sun about, about all the things he's doing. And in particular, his kind of approach and learnings around active learning and NLP models. And yeah, I wanted to invite him on the show to talk through some of that and learn a little bit from him. So welcome to the show. How are you doing?

• 3:49Speaker 03

  I didn't think it's bothering me.

• 3:51Daniel Whitenack?

  Yeah, it's been a while since the days in Colorado School of Mines in, in Golden.

• 3:56Speaker 03

  Good old days.

• 3:57Daniel Whitenack?

  And now you're working as a data scientist in the energy sector and also working in, in Asia, which is super cool. I'm wondering if you could give us a little bit of a sense of some of the unique things about doing data science and machine learning type of things in the context of like the energy sector, in the context of. Like an actual enterprise, real world kind of situation because we talk a lot about. Recently, we've been talking a lot about all of these gen AI models and APIs and such, and that is super cool. But also, there's a lot of on the ground work going on in data science that maybe looks quite a bit different than that. So.

• 4:41Speaker 03

  Yeah, thanks. So, I mean, I, I work in, in the energy sector and it's pretty much a traditional type of a sector. A lot of the companies, as you go around, at least in Asia, or at least over here where I'm at. We do not actually even have things like cloud services or, you know, subscription and stuff like that due to different reasons and stuff. But at the same time, there's an appetite for machine learning, AI data and all of those things. You see people talk about Jenny as well. But I think the ones that I've noticed, at least for me personally, that has really brought a lot of values are kind of what you guys were talking about in the previous episode of a broccoli AI.

• 5:24Daniel Whitenack?

  Broccoli AI. I love it.

• 5:27Speaker 03

  Yeah, and not so sexy, but, but still really important, really brings value. Particularly, I guess what we're going to talk about is, is active learning in the context of the NLP. Natural language processing. And, and so I think that's a, it's a pretty exciting place to be in, to, to do, you know? I mean, it kind of translates into Jenny and to a certain degree. But at the same time, I think to at least the context that I'm in as well, in the, in the sector that I'm in. We do not have things like cloud services ready for us, right? And so you have to figure out ways to kind of bring that about in an on-prem server BM.

• 6:11Speaker 03

  So how do you work around that? And how do you actually bring in cloud native, modern? Technologies within a traditional kind of a structure. So, yeah.

• 6:22Daniel Whitenack?

  Yeah, and from at least my impression, even though there's sort of not this, whether it be for security reasons or legacy reasons or just connectivity, that you know, there's, there's not the type of connection to cloud services like you're talking about that others might be working with. But at the same time, at least my, my impression is that this sort of sector and maybe others, there's other related verticals where they have been sort of data driven to some degree for some time. And I don't know if you could speak to that, like the, the types of data that people, you know, have been processing or storing or are available in, in those contexts, but.

• 7:07Speaker 03

  Yeah, I mean, that's a good point because a lot of these traditional industries, at least, you know, in the energy sector that we see, we have sensors that are constantly flowing in data all the time. The data is there. I mean, we, we are collecting data, right? And then going a little bit deeper, then you find that a lot of us have been collecting a lot of unstructured data too. And so at least within, um, you know, where in my experience, at least what, what was happening was when I came in, I was pretty much the only data scientist and stuff. And so I had to make my existence justified in a sense. And so I knew that like I had to be something about, like bring value within this organization.

• 7:51Speaker 03

  And like I'll be able to have proof that like, hey, look, data science actually does work and does bring value. But the quickest, I guess, low hanging fruit that I've found within, at least in the context of where I'm at, is, is the whole unstructured data. So we have been collecting thousands and thousands, hundreds of thousands of unstructured data. But there has never been a way to really analyze that at scale. So people have been analyzing it, they've been able to do some sort of a human analysis on it. But there's never been someone who's able to like say, hey, what's been happening in the past ten years that have been collecting all this data, what's it telling you? Um, nobody has really been able to do that.

• 8:36Speaker 03

  So I thought, okay, you know, maybe that could be one thing that we could actually bring in is that like you have all this data that's ready for you. That has all the insight, that has all the information that is locked up, right? Waiting to be unearthed, pretty much. Waiting there for us to just extract it or mine it. Um, so I I I did a quick P O C for one of the departments, a company and said, hey, guys, like you've been doing a R B I, tableau of the I kind of thing. You've been able to do stuff with your structured data, right? And you've been able to plot it on a beautiful graphs and stuff that they've been able to analyze in that sense. What about all the instruction data that you've been collecting over the years?

• 9:20Speaker 03

  And they said, there's no way we could do it, right? I mean, like, there's no possible way for you to put it in R B I. And so I said, like, well, maybe we could explore ways that we could actually get into machine learning to actually help you to scale that analysis um from an open standpoint. Thankfully, they bought it and that's what we took off from there and it was pretty cool. I mean, it was a journey for sure, a journey to learn.

• 9:47Daniel Whitenack?

  Yeah. So like when you say unstructured data, uh, give people a sense of like the kinds of files or um not not maybe the specifics, but you know, I'm imagining a file store with some type of files in it that contain, yeah, something. Give people a sense of of that.

• 10:05Speaker 03

  Yeah, for sure. I mean, I guess in structured data, typically we think about it as like text, right? Um, it could be text or it could be something that's just out of structure.

• 10:13Daniel Whitenack?

  And then like Microsoft Docs or?

• 10:18Speaker 03

  So we have been storing all of those data within SharePoint, uh, Microsoft SharePoint, right? And so what I've seen is these unstructured data is usually, it usually comes in a tablet form, right? You have a table that is collecting all of the structured data, but alongside with it, there's always a comment or that additional things that you have to actually tell the story of what you're actually collecting. And um, those are the ones that I think it's always locked up. And you would see it's per, it's very typical in a, any kind of industry where you have a tablet data, data that collects, you know, sensor data and everything, or like reports of what's been

• 11:01Speaker 03

  happening. And those are like structured, but then there's also a column that would bring in some sort of remarks, you know, observation or actions taken, whatever it is.

• 11:13Daniel Whitenack?

  Comments.

• 11:13Speaker 03

  Comments, yeah. But um, more often than not, we just kind of like play through it and, and kind of like not really put too much attention to it. At least within the data set that we were looking at, um, we had found that there were a lot more insight in those data than the structured data that they've been collecting. Oh, I'm talking about like safety data. So we've been collecting like safety reports every single day, couple hundred sometimes, or tens them every day over the years. And so these data had some sort of a insight, right? And it brings in an insight of

• 11:52Speaker 03

  how is the safety condition, um, you know, operations. And so with that, like the people usually see at the categories or that, that, what the reports are being reported for. But then like when you look at the categories, you realize that sometimes it doesn't really jive with the stories that they're actually trying to bring in in the structured data. And so that's where the, the part where we felt like it's gonna bring in additional insight to the structured data that we see.

• 12:22Daniel Whitenack?

  Yeah. And so like, there's this, uh, we've talked a little bit about like the data that you was there, the potential insights in that around safety and maybe other insights. As you kind of came into this industry and kind of were getting an understanding of like, let's say that you built the coolest data science app that was out there and had a cool model in it to do some analysis, like what is the reality of how that would have to run? Like, what is in production mean for in your context?

• 12:57Speaker 03

  Yeah, so many things. I mean, just start from the data. We don't even have labeled data at the start. But say you want a classification app or model, you need to have some sort of a label, right? And we don't have that. And so you have to figure out ways to bootstrap your labeling process to start off and stuff. And then all the way down to like, of course, training the model is pretty easy these days, right? And but then you have to think about things like, hey, what does that look like to put it on the server? Most of our servers are running Windows server. And so I've had experiences putting production some apps on Windows server and it was painful.

• 13:39Speaker 03

  And so like, we have to figure out ways that like work with the ITs and stuff and said, hey, can you deploy a Linux server for us instead? And just work it out from there and set it up from there. That being said, like that, that's just the overall picture of it. And then you get into details of like, how do you actually store your model? You've got to have some sort of a infrastructure to kind of hold that, which, you know, in our case, felt like MLflow is pretty good model registry experimentation, tracking and stuff to keep track of what type of models that I'm using and stuff like that. And then so many things, honestly.

• 14:23Speaker 03

  And then like, how do you actually put it on an orchestration kind of a service?

• 14:30Speaker 03

  You could use cron jobs, but then, you know, it may not be so flexible, then you kind of need to work something out. And so you have to get some sort of orchestrator to spin it up and and have like make that a service for your infrastructure as well.

• 14:44Daniel Whitenack?

  I love this discussion because it, I think it fits the theme of the show so well around being practical and the fact that, yeah, I'm sure that there's actually a good number of listeners out there who are really wanting to do machine learning, AI, data science type of things. And they are sort of in a similar situation in their company. Cause actually I think probably more of the majority of companies are in this sort of situation than sort of infrastructure wise, extremely modern and just cranking everything out on on Kubernetes in the in the cloud and that sort of thing.

• 15:24Daniel Whitenack?

  So I, yeah, I love this. So you were talking a little bit about the sort of problem of, so you have have all of this tabular data with extra comments and unstructured data and, you know, certain things you want to do like extract insights or classify maybe some of the unstructured data, but then also nothing has been labeled over time. It's just unstructured data. So talk a little bit about that bootstrapping problem and how you've thought about that in terms of I've got all this stuff. I want to create a model, but I have no, no starting point.

• 16:01Speaker 03

  When we were going through that whole labeling process or data preparation process, it's pretty interesting because we really didn't have anything, no labelers or anything. And I didn't have a budget to get an external labeler.

• 16:16Daniel Whitenack?

  Just hire, hire thousands of, of people online.

• 16:20Speaker 03

  Right. I know I could, I could just do that maybe. But then again, even that, like I've thought of it. And our data is sensitive to start with, but at the same time, it's so nuanced. And it's so nuanced to the context of our company. And so a lot of data that we, that every company has is just so nuanced to their own context, you know? And at the same time, one of the nuances is this is the way that like these texts are being written. And so a lot of code switching happening, you know, which means code switching, which means like in Asia, a lot of times we speak in English, but we will. Kind of put in some, you know, native languages that we know or that we grew up with.

• 17:03Speaker 03

  And so it's just kind of like you go back and forth, back and forth, and it's just kind of a common thing in, especially in Southeast Asia. And um, so you can't just hire somebody online and then just kind of label it for you. Cause you just can't, I I don't even know how to bring in the context of these guys, right? And um, but thankfully, I mean, I would say that um, the part that really helped was. I happen to have a really good sponsor for the project. And these sponsors, um, they were super on board. They were not technical, um, they were SMEs, you know, in apartments. And they knew their stuff, but they know enough of machine learning and data and AI.

• 17:47Speaker 03

  That hey, it's, you know, kind of a model that predicts, not necessarily making a hundred percent accuracy. Yet go, and so they understand those nuances in essence, and so they kind of supported that. And understood the kind of uh, things that we have to go through. You know, as a practitioner, that we have to go through to kind of understand that part of it. Um, so that was really helpful for my part because having a good sponsor means you get really good support for the project. But that also means that um, because the product that I'm working on, the app that I'm working for on is for their people. You know, their sordinates and the people and people are reporting under them.

• 18:30Speaker 03

  And what happens is they said to them and say, hey, guys, this is your app. I want you to help things soon out with building this app. And so having the users themselves onboard from the foundational bootstrap level really helped us. So because we didn't have any labels, um, they had the guys to actually be the ones to label for us. They were the labelers and so the the users themselves were the labelers. So that was, that honestly, I was pretty really blessed to to even just have that that kind of worked out together. Um, I think that that kind of worked so much in my favor, yeah.

• 19:08Daniel Whitenack?

  Yeah, and in that labeling process, how did you develop your sort of set of instructions for like how you like explaining the problem to them or helping them define the problem and the categories, for example, in a classification model? How was it for you? Because I've also had the experience personally, probably, and been burned a couple of times where I'm like, oh, this problem makes sense to me. I set up the labeling thing. I release a bunch of labelers in there and either the instructions don't make sense, or I've biased this in some way, or, you know, likely because I, like you had mentioned, I wasn't super close maybe to the

• 19:49Daniel Whitenack?

  to the users in that situation. But yeah, any learnings from that experience?

• 19:55Speaker 03

  Think it's a lot of iteration with them. I had so many times like I would travel to see them. They work in our operations, so I would literally travel there to see them in person. And I said, I would, we would just go through, hey, these are the labels that we want to label. We kind of get a general idea of what they are. But when you get into the weeds of it, like when you get into the details of it, you're like, you would think in this situation, that label should be here, should be number one. But then now somebody else said, no, it's two. So the way I worked it out was there will always be contention. I noticed no matter how tightly knit your labelers or your team are,

• 20:39Speaker 03

  there will always be contention. And you just got to work around with it, at least in my experience. I just had to work around with it. And the way I worked around with it was I just kind of had a voting system. You know, and I set up an account. So the technical side of this is I I could have just given them Excel sheets and we could just label them on an Excel sheet, but I find that Um, you know, they are doing for me a favor kind of thing, but I want them to have a really good user experience. Instead of just going through Excel sheets and stuff. And so I I used to Argila back in the days and when they started and.

• 21:19Daniel Whitenack?

  Pre-hugging face days.

• 21:20Speaker 03

  Yeah, yeah, exactly. And and I noticed that Argila is amazing in the sense that like it allows you to set up different users. And you could, I I mean, even other kind of interfaces as I've I've used labels to as well, but Argila. Was able to, I could use the API and just kind of like set up each user, right? And then for each user, I would just kind of like sample the same data for them. Um, so I had to actually go through the first round, the same number of set of uh data for them to label, say. Five hundred of them, I think I remember. And they would all label within two weeks. And at the end of that two weeks, I'll collect them and I'll find which one are which are the most contentious ones.

• 22:03Speaker 03

  And so the ones that are the most contentious, the ones that have the least um percentage of the majority. I would pull it up and I said, hey, guys, what do you think about this? This is contentious for you guys. Why is it contentious? And you work it up from there, right? Because chances are you're going to see the same kind of a label again, or the same kind of a data again. And um if you talk it out, hopefully, when you see a similar thing and you say, hey, I actually, we already talked about this, we all agree that we've been able to go with this. And so that's first round, we had the same labels, uh the same data set for everyone. It's a bit inefficient to a certain degree, but I think it's important to actually get into that space. So you understand the contentions of each person.

• 22:47Speaker 03

  And then the second round, I have everyone just kind of like label that scale pretty much. And um yeah, we collect that from there pretty much.

• 22:57Daniel Whitenack?

  Interesting. Yeah. So you did um in this process, you did a initial sort of offline bootstrapping of of labels, right? And did that so like. Scale wise, like what sort of scale when you're solving? So here we're talking about an NLP problem, creating a classification model on some labels of this unstructured data. Of course, this would vary by domain, but sort of what scale of labels did you shoot for when you were doing that initial, trying to get to that place where you could start up and, and train your first model?

• 23:33Speaker 03

  We did just about 1800 to 2000 labels.

• 23:38Daniel Whitenack?

  Okay.

• 23:39Speaker 03

  Of rows of data, basically. And then we start training our first model.

• 23:43Daniel Whitenack?

  That probably means you're not training like a 400 billion parameter model with, with 1800 samples.

• 23:50Speaker 03

  You don't even have infrastructure to be able to train that.

• 23:54Daniel Whitenack?

  Yeah.

• 23:56Speaker 03

  No, we don't.

• 23:57Daniel Whitenack?

  So what does a broccoli AI model look like?

• 24:01Speaker 03

  I mean, this is all text, right? And so we were, we were going with like the simpler, simplest you can find on, on Hugging Face at that time. I think at the time, sentence transformers were really making it big, you know, for different reasons, like whether it's topic modeling or, or classification. And at the same time too, I remember they came up with the set fit model, which is fine tuning the sentence transformer, which was honestly revolutionary for me.

• 24:30Daniel Whitenack?

  Yeah.

• 24:30Speaker 03

  Amazing. And I thought it was amazing that like it's, you're able to do something that was meant for similarity, but then you could actually fine tune it for classification and, and with pretty good performance. And it's supposed to be something that is a few shot classification model, a few shot kind of a fine tuning. And so I thought 2000 should be enough for me to start somewhere, right? And in fact, when I trained that, I tried some other models, but I think sentence transformers were the ones that actually gave the best performance out of all. Um. Still wasn't that good, you know, uh, talking about like sixty something, seventy percent, anything in terms of F one score.

• 25:09Speaker 03

  But when I talked to my sponsors about this, I said, hey, guys, like, you okay with like me deploying this at like sixty, seventy percent? And I said, no, actually, that's fine, right? Because um, the objective for this was, number one, to bring visibility of these reports to the users, is one of the pain points that the they said was for us to be able to know what people have been reporting, at least in the past twenty four hours. They had to get on SharePoint and just different hoops and loops to try to find out, you know, filtering and stuff. But to be able to get that sent out in the email with the classification was already win.

• 25:50Speaker 03

  And so I thought, okay, let's do that, but let's not stop that, right? I mean, we should actually create a pipeline, and that's where the active learning comes in. And it really helped because uh, I'm glad that I actually used Arjila to start with the bootstrapping of uh, our data set. And having that Arjila, which means our users are already used to the interface, and they already have an account. And so I was able to kind of hack around with the Arjila as a Python API and basically I uh, was able to create a loop where pretty much what this model does every day, it will bring in the new data that people have been reporting for the last twenty four hours.

• 26:34Speaker 03

  And make some prediction on it at about sixty, seventy percent F one score, accuracy, whatever it is. And then send it out to the users, and these users will see it, and at the end of that email, they will say, hey, uh, I don't think this is that signal, it should be this signal, I want to give a feedback, and at the end of it, they're able to click on a link that brings them to their profile in Arjila that will allow them to give a feedback for the particular day's data set. And so over time, now it's in production every day, I would get, from time to time, I'd get people giving their feedback. And we've gotten like close to four thousand um, data sets now, labeled

• 27:16Speaker 03

  um, from this active learning, and so we're, so we will train the model periodically, not on what? I could, I could have done it automated, but I didn't really wanna like just, I didn't feel the need for it yet to put it on automation. But then at the same time, like, you know, you're just collecting and everything, and we're just training it from time to time, basically.

• 27:46Speaker 01

  Hey friends, Outshift, Cisco's incubation engine, merges innovation with the art of possible, a launchpad for transformative emerging tech. Outshift blends startup agility with corporate strength to develop next-gen technologies from the ground up in AI, quantum technologies, cloud native, and more. Their newest AI innovation, Motific, addresses a critical challenge in the rapidly advancing world of Gen AI. Bridging the gap between concept and deployment, this model and vendor agnostic solution supports the entire Gen AI journey. From assessment and experimentation, Motific accelerates deployment from months to days while safeguarding against Gen AI security,

• 28:30Speaker 01

  trust, compliance, and cost risks, all while empowering business function and IT teams to rapidly configure end-user assistance powered by organizational data. Motific provides advanced customizable policy controls to prevent unauthorized access to sensitive data and helps ensure compliance throughout the entire process. With deep visibility into operational and business metrics, Motific enables you to track ROI, optimize costs, and make informed decisions. By offering a centralized view, Motific deters shadow AI usage and empowers teams to innovate responsibly. So move beyond the traditional constraints of AI implementation,

• 29:15Speaker 01

  utilizing AI deployment that is both responsible and is revolutionary, ensuring your projects are not just quickly launched, but built on a foundation of trust and efficiency. Visit motific.ai, that is M-O-T-I-F-I-C dot AI.

• 29:53Daniel Whitenack?

  So BingSun, it's super interesting to hear kind of how you were able to engage the users of the application through this like reporting process essentially that they were, you know, had some of the right incentives in place to respond and to give you updated labels. And you mentioned also the model repository, saving models, getting them out with MLflow in the context of, you know, you deploying your model on prem, you updating the model, you just mentioned kind of retraining the model. What does that look like for you right now in terms of that cycle of when you would want to push out a new model after

• 30:36Daniel Whitenack?

  gathering this data? How you would judge that to be worthwhile or useful in any sort of testing that is relevant to that cycle of getting in new labels, retraining, you know, evaluating that sort of thing? What does that look like for you? And how do you kind of put in the right, or have you thought about the right metrics to understand when to update the model?

• 31:01Speaker 03

  At this point, honestly, we keep it simple. We just kind of like periodically do it at a cadence, you know, a couple of months or two. But I did think about like, what does it look like to actually measure the drift of the data and stuff like that of the model predictions? That could be one of the ways that we could do it too. But what I'm seeing is actually the model is doing its job fairly well, well enough to actually solve the business problem. And so we don't see a need to actually implement a more sophisticated monitoring unless we need to, you know? That's where we're at with it.

• 31:39Daniel Whitenack?

  Yeah. And when you push your model, sort of like you update it, you had mentioned the model repository. How are you shipping your model out to the application? Cause I think like you had mentioned, you know, you only have so many resources. I think there's, there's also a lot of people out there in your situation where I think it was Kristin Lum on a previous episode, she had talked about kind of that data scientist out there that is maybe one of very few or the only data scientists in a potentially a large organization and having to like do all of these things. They're not like an ML Ops person, they're not a model trainer, they're not a observability person.

• 32:24Daniel Whitenack?

  They're doing all of that, right? So there are limitations to, you know, how much sophistication you can put in place. And I think that like some people go way too far and they're like, oh, I'm going to implement all of this stuff. And it actually makes their life as a practitioner less happy than, than, than otherwise. So yeah, how have you found that balance? And like, what does it look like for you to do these cycles in, in terms of tooling? And the things maybe that you, like you say, you mentioned your thought at some point, maybe it's relevant to implement some of this observability stuff, but maybe not yet, or there's other priorities.

• 33:03Daniel Whitenack?

  So what does that look like for you in terms of how you decide what, what level of sophistication is right? And how you push things out and?

• 33:12Speaker 03

  I'm a hundred percent with that, honestly, because it is a matter of priority. My customers are happy, I'm happy and I'm not going to like change what's, you know, what's good. Or I don't want to break what's been working, right? So to speak, but that being said, like I, you know, when it comes to like all of that, I think I have a general idea of what would be the minimum thing. So now I'm working on some other things as well, like anomaly detection and stuff like that, which needs to be deployed. So having gone through that, that kind of like set up a, like a zero infrastructure for me to know what kind of infrastructure that I'm going to be looking for, for whatever else that I'm working on. And at the bare minimum, I think model registry is is super important.

• 33:54Speaker 03

  And being able to call the different versions that you've been training and being able to track that. And being able to call it through an API, through a function. You know, MLflow has this great Python connection with it. And so being able to do that is is amazing. I mean, it keeps my life sane, right? I don't have to like figure out where I stored my model, pretty much. So I would be doing exactly the same thing with whatever I'm working on next, which I I've since moved on from that project and I'm just kind of like maintaining it. That project's count now in maintenance and now I've moved on to a different project to solve in a different part of the business, you know, in that sense.

• 34:34Speaker 03

  But that project kind of set, like I said, set the foundation and knowing what kind of things that needs to be done. So, sorry, I'm I'm kind of going ahead of myself. So one is MLflow being the most important thing for me um in in terms of this sort of scenario. The other one is um orchestrator is also really important, having a really robust orchestrator. So for me, I think Prefect was perfect for me, you know, and I was able to do different things and stuff, yeah.

• 35:03Daniel Whitenack?

  The types of things that you're orchestrating are what types of things?

• 35:07Speaker 03

  You could do it real time with Prefect at the same time. You could also be listening and stuff, but at the same time, you could also just do running on schedule, calling different functions, um subfunctions and things like that. Um, so that was really cool to be able to have that. That's pretty much what we do right now. We're not really going into a real time monitoring yet. Until we do that, then we'll have I'll have to figure out something else more sophisticated.

• 35:30Daniel Whitenack?

  Yeah. Yeah, and and are you just shipping your models sort of as part of a Docker container or or something like that?

• 35:38Speaker 03

  Pretty much, yeah. We do use Docker containers and just so that we can keep it contained in that sense.

• 35:46Daniel Whitenack?

  Yeah, that's awesome. I think you had mentioned in one of our conversations, something about a DuckDB. Where does that fit into some of this?

• 35:55Speaker 03

  So the data, the raw data that we get from is from SharePoint. But if you have uh anyone who has any experience with SharePoint in terms of like wrangling and data stuff, it's so painful.

• 36:09Daniel Whitenack?

  Yeah.

• 36:10Speaker 03

  So I I thought that would be good to actually have some sort of a middle layer, mini like house. Or data lake kind of thing. And I don't want to bother my I T guys too much. So I thought DuckDB is a great thing for it, right? I don't need a V M for it and you can have an embedded SQL service that you can use. So that's being pulled every day, pulling the data into DuckDB. And DuckDB will be the one that actually cleans up the data, preparing the data to send it to the model. And that becomes like a pipeline for me to be able to work around the complexity of SharePoint, really. Let's see.

• 36:51Daniel Whitenack?

  Yeah, I personally found a lot of use for DuckDB even in the past. Yeah, even in the past year, on the, even on the more gen AI stuff where you're doing sort of like text to SQL or like queries and that sort of thing. And every company we're working with has different crazy sets of data or different configurations of this or that. And that layer of having a kind of unified analytics layer, but also not that sort of, you know, easy to pull in to Python, easy to spin up, easy to like test with locally and then like deploy with. Yeah, that's been really useful.

• 37:29Speaker 03

  I remember you talked about LandsDB, you know, for RAG and things like that. And it's the same thing. I love like embedded database. I think it just works well, you know, and like it's kind of scalable eventually, you know? And I think I really like that.

• 37:45Daniel Whitenack?

  I think there was one blog post I've always referred back to because I also went through the, you know, you and I were at Minds at the same time. And then like there was like data science hype and then there was like the big data period where everybody was in Hadoop and Spark and all this stuff. Which I know a good number of people still use Spark, but I, there's a blog post by the mother Ducker.

• 38:11Speaker 03

  Oh, yeah.

• 38:12Daniel Whitenack?

  Company. Yeah, but I think the title is Big Data is Dead or something. It basically goes through some of the discussion around like, hey, we all thought we had big data. But like the actual query problem, like the types of queries that we need to run. These aren't like big data problems. What's needed is different. So yeah, for those, shout out to whoever wrote that blog post because it was really, really good. If you ever want to come on the show and talk about it, that would be awesome. Yeah, well, as you kind of look back on this process and some of the things that you've learned, like what are you, what are you looking forward to in terms of like the future of the process of your own work or of the things you're learning?

• 38:53Daniel Whitenack?

  Or maybe like as you go into this next phase, it sounds like you're working on some new things. You'll want to reuse some of the tooling and kind of process that you have used. But you know, what's different or what are you excited about kind of for this next phase in light of what you've kind of learned over the past years?

• 39:12Speaker 03

  Generally speaking, I think ML Labs is just so nuanced, you know, in different contexts. Everyone has a say of what should be done. And I think if I learned something from this was nobody really knows everything. So you're going to have to figure out from there and you kind of take a risk on certain things that you decide in terms of your system design and stuff. What I'm excited for is actually to be able to take this and see what it looks like to for other things, right? And in other applications, like whether it's anomaly detection or whatever it is.

• 39:51Speaker 03

  In a broader sense, I think I'm excited to see things like embedded database, you know, getting more and more mainstream, especially in the context of LLM and GNI and stuff. I'd love to see that getting more and more mainstream as well. One of the things I'm always thinking about is scale is one thing because we, a lot of the applications that we talk about today, especially in the context of GNI, we always talk about like the bigger compute and bigger scale. I would love to see that getting smaller, which it is happening now, getting more accessible on different devices and stuff, being able to do more cool stuff.

• 40:34Speaker 03

  On device and the band stuff on there. Excited for that, too. Yeah.

• 40:38Daniel Whitenack?

  Yeah, I think there's a lot of people excited for that and sort of this new phase of AI where people talk about AI everywhere or the sort of thing, which in reality, you know, there's been machine learning and data science sort of everywhere for for some time, but that that sort of wave of these newer generation of models kind of being runnable in in more practical scenarios is is exciting. But um, yeah, thanks for thanks for joining Bean Soon to to talk about a little bit of your broccoli AI. It's been uh, it's been fun and uh,

• 41:14Speaker 03

  Love it. Thanks for indulging me.

• 41:15Daniel Whitenack?

  Yeah, yeah, you you and I can can hype the broccoli AI and I'm sure we can get Demetrius to help us hype it, too. Yeah. I don't know if he trademarked that term. He's got it in in his hype cycle now. So, uh, yeah, yeah. Thanks. Thanks so much for joining and I hope to talk to you again soon.

• 41:35Speaker 03

  Thanks. Thanks, Barry.

• 41:44Speaker 00

  All right. That is Practical AI for this week. Subscribe now if you haven't already. Head to practicalai.fm for all the ways and join our free Slack team where you can hang out with Daniel, Chris, and the entire change log community. Sign up today at practicalai.fm slash community. Thanks again to our partners at fly.io, to our beat freaking residents, Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.