Deploying models (to tractors 🚜)

Alon from Greeneye and Moses from ClearML blew us away when they said that they are training 1000’s of models a year that get deployed to Kubernetes clusters on tractors. Yes… we said tractors, as in farming! This is a super cool discussion about MLOps solutions at scale for interesting use cases in agriculture. Sponsors: Fastly https://fastly.com/ – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipChangelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Featuring: Alon Klein Orback – X https://x.com/GreeneyeAgMoses Guttmann – Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Greeneye https://greeneye.ag/Greeneye Blog https://techblog.greeneye.ag/blog/tag/clearmlClearML https://clear.ml/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  It's all about intuition, I guess, these days. Everything is changing so fast. It's very hard to actually try to predict something. And what we do is an iterative process. We try every time and then we refine. And that's exactly how we look at the field itself. So if we see a company or a specific field where we feel that what they need to do in order to solve their problem they're trying to approach is actually an iterative process where you have a model and you're constantly refining, rebuilding a better model, then this is a great fit for MLops. If you have enough automation, you can really accelerate the process.

• 0:40Speaker 01

  Hey, Jared here. One of the things we can count on in the software industry is change. The state of the art changes so fast, in fact, that keeping up can feel like a whole other job on top of your actual job. That's why we created Change Log Weekly. It's our totally free newsletter that we drop in your inbox each and every Sunday. We link to the latest news, the best articles, and the most interesting projects that you should be aware of. We also add a little commentary from us saying why something's important, pointing you to other instances of a trend, or just making a dorky joke to keep it lively. So if you haven't yet, I recommend subscribing to Change Log Weekly and help us help you keep up with the latest.

• 1:23Speaker 01

  Head to changelog.com/weekly and sign up today. Again, it's totally free and we never spam you. Yuck. One last time, that's changelog.com/weekly.

• 1:47Speaker 01

  Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science have been joined us at practicalai.fm/community and follow the show on Twitter, we're at Practical AI FM. Thank you to our partners at Fastly for shipping our pods super fast all around the world. Check them out at fastly.com.

• 2:20Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International. I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

• 2:36Chris Benson?

  I am doing well, Daniel. How are you today?

• 2:38Daniel Whitenack?

  Doing well, yeah. It's been an interesting couple of months and lots of new projects kicking off. So keeping busy, but we just came back from a little bit of vacation last week, which was nice.

• 2:49Chris Benson?

  You gotta tell everyone where you went now, now that you've actually brought that up.

• 2:52Daniel Whitenack?

  Okay, yeah. So we drove down to warmer weather. So we live in the Midwest of the United States and drove down to like Alabama, did some hiking, kind of back up through Mammoth Cave, which I learned is the world's largest cave network system thing. I don't know the proper terms, but yeah, that was fun. So it went underground for a bit and yeah, it was a, it was a good time. Yeah, and I'm, I'm pretty psyched for a couple of reasons for this show, Chris, because of course I always enjoy talking with you. But we've also got some familiar technology because we had, I don't know if you remember a while back, we had an episode, I think at the time it was called Allegro AI, which is ML Ops. They've since rebranded to Clearmell.

• 3:37Daniel Whitenack?

  And we've got Moses Gupta, who is Clearmell CEO and co-founder. But we've also got one of their partners, Green Eye. We've got Alon Klein-Orbach, who is the CTO and co-founder at Green Eye, which is an agricultural AI company. So this is going to be fun. We're going to talk about agriculture, AI, ML Ops, and get to chat with some, some old friends as well. So welcome, Alon and Moses.

• 4:04Speaker 04

  Thank you. Thank you for having us. Pleasure to be here.

• 4:06Daniel Whitenack?

  Yeah, it's great to revisit ML Ops with Clearmell. We had that show previously and of course, ML Ops is sort of like, even since we had that show, just kind of exploding as a topic that's on people's mind. What has that been like, Moses, in terms of just like this sort of meteoric rise? People caring about ML ops and how they actually practically do machine learning.

• 4:32Speaker 00

  So I think the market really matured in the last two years. I guess it's probably COVID accelerating the process where everyone is working remotely. So you have to have automated processes and you gotta log everything because you cannot call your colleagues every minute or so. And I think that the problems that we kind of discussed in theory two years ago became very dated a practical problems that companies and individuals run into on a daily basis. And now it's become a problem that it's if before it was it was nice to have. Now it became like a must for most companies. Back then, only a few understood the benefits and and kind of the need for for this

• 5:17Speaker 00

  very comprehensive approach where everything is streamlined. And I think now it's kind of common knowledge, probably not that common to actually implement, but at least the understanding is is there.

• 5:28Daniel Whitenack?

  Yeah, and Alon, in terms of your your company, which is sort of like has customers who are in agriculture vertical, but is very much like at its core an AI company from from my understanding. Like how I don't know the full history of Green Eye, so maybe you could give a little bit of that. But did you sort of like do a bunch of AI as Green Eye? And then kind of like come to the ML ops problems? Or from the beginning, was that something that you kind of needed and was problematic for you? So you kind of started started with that early on?

• 6:05Speaker 04

  Hey, it's a good question. We actually started from different goals or different objective of Green Eye and until we established what this Green Eye is about. So we started hacking about the laptop and training on a GPU on the laptop and stuff like that. And really understand there is no scale in it in many, many, many ways. And even add a server there and we needed to install CUDA and CUDA and using no app to keep the script running and really all the best worst practice that you can ever that you can do.

• 6:42Daniel Whitenack?

  There's no shame. We've we've all been there.

• 6:46Speaker 04

  Yeah, so we started with no ML ops at all, really no ML ops. And I think few years ago, my brother, one of my then told me, eh, try Docker. And I say, ah, no, no, no. And the rest is history. Three years later, today, we are completely Dockerized from end to end, Kubernetes all the way, cloud and edge. And this has changed everything because then we realized we can do a lot of ML ops all around and move stuff, so.

• 7:14Daniel Whitenack?

  Yeah, and maybe just stepping back in terms of green eye, I think, Chris, I don't know if you remember this, but I forget in what episode it was. I think it was one of our fully connected episodes, we were using the example of like spraying. Yep. In crop fields as an example of like this sort of scale between like compute, completely human manual process up to like automation. And how that's changed, it struck me, Chris, that neither of us are farmers and really know much about that process. So, Alon, this is very much like the world you live in. So maybe you could just step back and let us know a little bit about like AI in agriculture.

• 7:55Daniel Whitenack?

  Before we kind of talk about some of the other ML ops things that your company is doing along with Clearmel. Like what, what is AI in agriculture look like generally and how has that developed over time?

• 8:07Speaker 04

  Sure, so I must be honest, I'm not a farmer as well. I'm from the technical side, but I like what we are doing. So in the big image, I think you can divide the, the interest to two, the, the ones that gives tools to other farmers to get more information and more details about, about their field, the crop, the yield or anything like that. And you can divide the other group is the tools that make the decision by themselves. So you get a lot of true cool companies like in Israel, Terranis, Prospera, the other cool that doing a lot of intelligence in the field. They collect data, they analyze it and show the farmers insight.

• 8:52Speaker 04

  You get drones there, you get satellites there, you get the pivot and cameras and et cetera. And from the other end, you get. Decision making tools that driving the tractors autonomous sprayers like in the same domains that we are Blue River and John Deere position that was I think four years from now.

• 9:14Chris Benson?

  As a follow up to that, I'm curious, you know, we're so used to on the show kind of talking about these very technical topics. And yet, you know, the clientele that your company is serving is one that is getting into technology as we've talked about. But if you look at the broad history, hasn't really been something that we associate with high tech and all. What is the merger of something as cutting edge as deep learning ML ops, you know, on one side with farming on the other, you know, where that's making this massive transition. What's it like being in that space where you're presumably kind of tying together two very, very different worlds?

• 9:53Speaker 04

  We really like it. We are really a very devocable company. We have like chemistry, we have agronomics, we have data science, we have real time and we have everything for everything cloud and ML ops and business sizes, sprayer operators and being this spot that everything is connected, the technology is related to the field is we really like it. You need to have the business to run and to make sense from monetized perspective, but it's also nice to do something nice.

• 10:26Daniel Whitenack?

  Yeah, it's nice to apply AI, I'm sure, to a problem that we all have, which is we all need food. So I saw in one of the videos on your site, this sort of like, and this is where my knowledge of all the machinery and stuff, but I'm assuming this is like a spraying machine. I don't know if it's specifically for spraying, but it sprays. And then you've kind of got cameras. So I don't know if you could maybe just describe this sort of machine and like the arms of the machine and like, just so people have a visual of kind of like where your technology fits in.

• 11:00Speaker 04

  So maybe I restarted to understand the problem that we are solving. So imagine you have a garden and you grow vegetable there. And what you do in your free time, you are weeding, you are taking the weed out because they compete with your vegetable about resource, sun, water, and etc. And when you are getting bigger, you are starting to use mechanical tools. And when you get really big, like farmers, the Midwest, Nebraska and Iowa and all of this area, you started to put chemicals because you cannot control the size of farming. You put chemicals and you want to do it fast, so you don't put with a small tractor. You have dedicated sprayer, self-propelled sprayer, 66 metre long boom and it's a monster.

• 11:42Speaker 04

  You can walk underneath without need to bend under the sprayer. So it's a really big monster that you just drive to do as much as acres as possible in short time. And what we are doing in GreenEye, instead of assuming the worst-case scenario that every spot in the field is with there. So instead of doing that, we are putting sensors, cameras and our computers and nozzles and everything else we are using to spray only when you need to spray. So instead of putting 100% of chemicals over 5,000 acres field, you are putting 10% field.

• 12:27Speaker 04

  And so you're saving money and the world's saving chemicals and it's a win-win situation. So about your question, sorry, I forgot. So we have a big sprayer, we have cameras, each like three metres long and we are just filming the entire boom. We are putting the cameras looking a bit ahead, so we have time to process. Everything is done in real time, no connectivity at all to the cloud.

• 12:55Chris Benson?

  So Moses, you know, as I'm listening to Elon talk about his story and as we're doing this, I'm thinking back to kind of starting with these cutting-edge ML ops. When you're looking at the landscape on your side, as someone who's bringing this technology to bear in the marketplace, how are you evaluating different opportunities in industry? I mean, is it just that everything is open? Do you have a way of looking and saying, I see, I see an opportunity where this technology... In a particular industry is going to be very useful. How do you make those kind of judgment calls on how to engage?

• 13:32Speaker 00

  Good question. So first, it's all about intuition, I guess, these days. Everything is changing so fast. It's very hard to actually try to predict something. If I'm referring back to machine learning and what we do is an iterative process. And we try every time and then we refine. And that's exactly how we look at the field itself. So if we see a company or a specific field where we feel that what they need to do in order to solve their problem they're trying to approach is actually an iterative process where you have a model and you're constantly refining, rebuilding a better model. Then this is a great fit for MLops because that's basically means that if you have enough automation, you can really accelerate the process.

• 14:16Speaker 00

  If not, obviously you have to do the same process only manually, which time to market wise really increases the time for you from like the research phase to actually something that is working, where you have some alpha in the middle. And when you see a process where you can say, you know what? With a bit of automation, this model will really work. Like not 90% of the time, which means one out of 10 you fail, which is not like in theory 90% looks, sounds fine. But in practicality, this is not something you can actually sell. You think to yourself, okay, the only thing that I need is a bit for more information from the field itself. And then I can just refine the model, rerun it, get a better performance, and then just repeat the process.

• 15:01Speaker 00

  Then basically I'm golden. I can take it to different scenarios and get my model up and running. And every time we see one of those scenarios and we, that's this kind of a moment where we hear the bling. Okay, this is a perfect fit for automation for MLops as kind of a holistic approach.

• 15:37Speaker 04

  We are going to send in three, two, one.

• 15:42Speaker 05

  I'm Karhal Azou, host of Ship It, a show with weekly episodes about getting your best ideas into the world and seeing what happens. We talk about code, ops, infrastructure and the people that make it happen like charity majors from Honeycomb.

• 15:56Speaker 06

  We act like great engineers make great teams and it's exactly the opposite. In fact, it is great teams that make great engineers.

• 16:04Speaker 05

  And they farly were the founders of continuous delivery.

• 16:08Speaker 07

  Start off assuming that we're wrong rather than assuming that we're right. Test our ideas, try and falsify our ideas. Those are better ways of doing work and it doesn't really matter what work it is that you're doing. That stuff just works better.

• 16:20Speaker 05

  We even experiment on our own open source podcasting platform so that you can see how we implement specific tools and services within changelog.com, what works and what fails.

• 16:32Speaker 01

  It's like there's a brand new hammer and we grab hold of it and everyone gathers around, we put our hand out and we, we strike it right on our thumb. And then everybody knows that hammer really hurts when you strike on your thumb. I'm glad those guys did it. I've learned something instead.

• 16:47Speaker 05

  Yeah.

• 16:47Speaker 07

  I think that's a very interesting perspective, but I don't see it that way.

• 16:52Speaker 01

  Okay.

• 16:52Speaker 07

  It's an amazing analogy, but I'm not sure that applies here.

• 16:56Speaker 05

  Listen to an episode that seems interesting or helpful and if you like it, subscribe today. We'd love to have you with us.

• 17:28Daniel Whitenack?

  So Alan, following up on what, what Moses was just talking about with sort of where the value of ML ops really comes in with automation. You gave the example, sort of the concrete example of the spraying, detecting like places to spray within a field with this, you know, massive machine with, that has the sensors or the cameras on it. So like, how does the automation or the retraining of models that you're using, where does that come in? How often and what sorts of things are you automating in practicality?

• 18:08Speaker 04

  I think I can spend hours to answer fully of this question, but I must say that the things about MLops and ML in general, if you compare to understand a field in the industry like coding, deploying in servers and et cetera, there is no best practice yet. The entire industry, everyone invents something on his own and doing something else. We are start getting to some main path, but there is no best practice that everyone is agree on. So I think from our side, we had a lot of challenges. We have challenge of that and controlling the data. We have a massive amount of data, more than millions of images over the field.

• 18:52Speaker 04

  And one of the challenges is getting a model that was trained to the tractor. And before we had any automation, we did, okay, we trained the model and then we frisked it and we are using TensorRT and Onyx to TensorRT and stuff like that. And we did it manually, like each step at a time. I think like we got 10 models a year to the tractor. When we got automation, we have thousands of models to the tractors and rerun. And when a researcher finished training, there is a click and everything is done automatically, is convert, is get metrics there, to Krill or mail and what else is done.

• 19:37Speaker 04

  Is checking the converting if we didn't miss anything by the converter like performance. And we got the end of the file there to the tractor by a single click. So I think this is one example of automation that MLops really change it.

• 19:55Speaker 00

  Do I remember correctly that you're also running Kubernetes on the embedded devices?

• 20:00Speaker 04

  Yes, yes, we are running Kombinetis on the embedded devices. This is completely game change in our end. Because this is also a conversation different that I can speak hours about it.

• 20:13Daniel Whitenack?

  And that's when you say on the embedded devices, we're talking about like on the tractor.

• 20:18Speaker 04

  On the tractor, yeah, yeah. We have a few devices there and they are running K3S. It's the light version of Kombinetis. It's really nice because we can best practice from the cloud and we get best practice to the edge. And for example, if we want to use Clearmail from the edge or from the cloud, it doesn't really make us any different to us. So we know how to press the, move the secrets and how to use it. And we just do it in this sense, yeah.

• 20:49Speaker 00

  So you have full visibility to the tractors inside the same dashboard that you're developing in? And you have the entire cycle streamlined?

• 20:59Speaker 04

  So something like this.

• 21:02Daniel Whitenack?

  Good, good setup. I like it.

• 21:04Speaker 04

  There is a rancher and I can connect any running tractor around the world that is online now. And just doing even SSH to the machine, I can view it all. But this is more bigger one than the MLX and the ML Ops. And this is like, we call it Special Ops.

• 21:25Daniel Whitenack?

  Special Ops, I like that.

• 21:27Speaker 04

  We actually have dedicated team that's doing research around Special Ops. So we have ML Ops and Dev Ops and IoT Ops. Now there is FinOps. This is the team that moves one step ahead and doing a lot of checks. I tried to, to, to find when we started to use Clearmail, I searched Clearmail. Then I understand I need to search Allegro. And I didn't find the right point, how we got to know Moses and his guys. So I'm not really sure, but before we, we got to know him, we used group flow pipeline to do the runs and to do the metrics. And we got there to a big wall of complexity.

• 22:11Speaker 04

  And we shift the training. From cool profile lines to clear metal.

• 22:17Chris Benson?

  I'd like to ask a follow-up question about something you were saying a moment ago. And it's something kind of close to what I'm doing when I'm not podcasting. And that is when you talked about having Kubernetes in all the places, in this case on the tractor. And you talked about K3S. Can you tell us a little bit about, I'm a big advocate in Kubernetes in all the places at various scales as a setup. And so since in your use case, you have done that. I'd love to hear how you arrived at that and what benefit you think it's given you. Well, you know, why, why do that? Cause most people don't think about putting Kubernetes in all the places. They're running kind of in the cloud.

• 22:58Chris Benson?

  They don't have, they're not out on the edge yet, the way you are, where you're way out on the edge. And as someone who also in my day job works out on the edge, I'm curious what your thoughts are about how Kubernetes in all the places is a good model going forward.

• 23:12Daniel Whitenack?

  So Chris, you, you don't think people, when they think of the ideal deployment target for Kubernetes, they don't think immediately tractor?

• 23:20Chris Benson?

  I'm glad that he does, I'll say that.

• 23:23Speaker 00

  Should be right next to it.

• 23:26Speaker 04

  We are in sprayer world. It's a good question. We started, when we started containers, we started with no Kubernetes and we did like our own deployment system. And we fast, like in months, we, we had the wall of keep them alive, visioning and everything. And then we understand, okay, someone solved this problem, it's no way into. And then we got to Kubernetes on the cloud. And the day we still use containers, but with different orchestrator. We had the Azure IoT, if you know it. And we used it for a while and then we got another wall. Because Kubernetes is this concept of separation, virtual separation of pods. And this is amazing idea.

• 24:10Speaker 04

  And the other orchestrator didn't add it and we understand, okay, we need to change. And it was not easy because nothing is really ready to arm 64 bit. You'd be surprised that most of the libraries, maybe today there are some that are more, but most of the libraries back one year, two years from now. No vision for ARM 24 bits and it was like, there is no way that we are on the edge of the edge as a young company.

• 24:39Chris Benson?

  Do you think that Nvidia, having bought ARM recently, will have any impact on that? Not buying ARM, failing to buy ARM. Oh, you're right. Yeah, yeah.

• 24:50Speaker 04

  I think the regulation issues might affect it.

• 24:54Chris Benson?

  I forgot about that. That's a good point. Yeah.

• 24:58Speaker 04

  But definitely, I think we'll be here for the edge. And Kubernetes, as back to your question, help us a lot in, for example, when we have a new researcher or a new programmer or whatever, he does not do anything on his laptop. The laptop is only the gate to the pod that runs or on the tractor or on the cloud. And this is the workspace. So if we get to the tractor or in our subject, the research, a research come to the company, a new research, we just hired a new one and she's really good one.

• 25:40Speaker 04

  And she's got, just get, okay, you get access to, we still use Kupfla for a notebook server. So if she needs a workspace, she just click and get a new notebook. And she can use our tool, we use PyCharm for remote interpreter. And you just connect to the pods there and you get all the data. And you can play from your PyCharm, you can play from your notebooks. And this is possible, you can get it in different ways, but this is mainly possible. All the games of sharing and forwarding with Kubernetes is much easier. And this is true for the edge and for the clouds.

• 26:22Speaker 04

  So no one installing any dependency, I don't know if you play with Java home or something like that, or NPM install, or no one installing anything on his computer besides the ID. Our philosophy in general, everything containerized, no workspace to be installed in any way. And this is really up in specifically in MLab environment.

• 26:44Daniel Whitenack?

  So Alon, in terms of like the automation that you're talking about before, you mentioned this sort of information about like thousands, you're able to sort of do more than like thousands of models a year now versus like a sort of order of magnitude lower now that you didn't have automation. I'm wondering if you could talk a little bit more about like, for your case, it's, and I'm trying to think of like, maybe weeds sort of look similar. So like, what kind of needs to be updated so much throughout the sort of year for these like vision models or whatever models you're running? And then like, could you describe maybe a little bit more, you mentioned kind of bringing in new data, training the model, and then having

• 27:29Daniel Whitenack?

  the pipeline to push it out to the tractor and deciding when and when not to do that? It'd be interesting to hear about the sort of when and when not question in terms of like, what you test within like your MO Ops to determine when you push something out and how you do that?

• 27:46Speaker 04

  It's a good question. I will start with the first one about the models and the numerous models. I think in the end, well, from my experience, I might be wrong, but having one model to rule them all or something like that, it's not enough. You got the vision of YOLO, Retinonet, or any other model, and you need to put more effort to solve a real world problem because you have very lot of variables, a lot of variance in the real world, and you need to combine classical vision and deep learning one. So I think in the end, we have metrics on the

• 28:28Speaker 04

  cloud for the models, for the big models and so on. But at the end, we want to have metrics on the devices, on the trucks, the runs themselves. So we keep constantly testing ourselves on the real environment, how we are doing. Also in the terms of performance, not only metrics, performance, speed wise, the cycle clock, how fast we can go. Today, we can go about 6 meter per second.

• 28:58Speaker 04

  You guys speaking mile per hour. It's about twelve, around this area, mile per hour. So this is a really big factor for us. Besides that, we have different crops, different geography and everything changed. The landscape changed, the weather changed, the sunlight changed. It's completely different game to play in Israel, for example, and the Midwest. The farmers changed, some used tilt and some in Midwest, they mostly stopped using tilt. They just keep the crops, the old crops there and letting the ground do their magics and just sitting above the old crops.

• 29:40Speaker 04

  So everything is changed and we need to react to those changes. So, and this is for the first one. And the second one, I think that every tractor run that we are doing or any other way that we are getting data, we try to get as much as variant as possible. So retraining our model by one or two or even, I don't know, five hundred images is not, it's not a game. It won't change a lot for the model. We have a lot of them, so we try to understand when few models don't agree with each other or something like that or tricks like that, to understand when there is an information that is interesting to rerun and train on it.

• 30:25Daniel Whitenack?

  So are you logging your sort of data preprocessing and data set like creation and your training runs in the, in the sort of ML Ops and kind of building, training off of certain, certain triggers or something that you, that you have set up or how does that work?

• 30:44Speaker 04

  So one of the challenges of ML Ops is reproducibility. I think this is a really hard one to get right. You get code versioning and then you get dependency. And okay, let's say you solve it with Git and Docker, but then you get data versioning. And then in all of that, you need some system that will take everything from every place you need and then you need to push it and. Just click play and rerun it. So reproducibility is really hard. And if you did like, I don't know, half a year ago, you did something good and you want to go back to it. It's really hard. So we are trying to log as much as possible

• 31:26Speaker 04

  from the system perspective and from the training and the research perspective. What's nice about ClearML, that we are using it not only from MLops, we are using it as a dashboard in general. So we're just pushing everything that we want to use as metrics and show stuff there. So from this perspective, we just log everything possible. And if it's visible, we can use ClearML for it. But also, we want to push our limits and to run faster and faster. And if we run faster, we can do even more stuff. Today, we detect weed, but our mission is to spray less, grow more.

• 32:05Speaker 04

  So we want to do fungal size and pesticide and fertilizer and etc. So we need more compute power all to be better in what we are doing and saving computer power for different tasks. So we try to log everything and be better in what we are doing.

• 32:29Speaker 00

  Quick question. You mentioned retraining models. So do you have like a model per tractor or scene?

• 32:35Speaker 04

  No, no, no, it's not tractor per scene. But we retrain a model like, okay, there's different reasons why to do it. Fix, maybe we want to improve it in different variants of the appearance of the backgrounds or anything like that. Or we want to make sure that we got, I don't know, a new weed that we don't know or we are not familiar with. So instead of doing a zero shot that we are kind of doing or one shot that we are doing, so we can stop the system for doing this for a specific weed that. Became more common. But in general, we are playing with a lot of tools.

• 33:20Speaker 04

  We are trying to get in line with the best practice in the industry. And we are also experimenting. It's not just for the experiment, but it's for the research to be better. So in this sense, we might be running the same train. Or if we want to verify that we got the same result, like a real research. More, not to read, it's not academic, you know, but in this sense.

• 33:48Daniel Whitenack?

  OK, well, given the fact that Alan, that you're training so many models, you're updating a lot of models. It sounds like there's a lot of like training scenarios that you're encountering. You're kind of doing this at scale. And you've been sort of partnering with Moses and his team to do this. I'm curious, actually, from Moses, from your perspective, sort of looking back on like the things you've been trying to enable with Clear ML and seeing like someone use it at a larger scale like this. What are some of the things that you kind of like, you thought were going to be important and ended up being important in terms of like the things you're tracking or the features that you've enabled?

• 34:33Daniel Whitenack?

  And maybe what are some things that like, or maybe you didn't expect? And now you're thinking about differently than you when you started things out?

• 34:40Chris Benson?

  As a follow on to that, what insights are growing in your head?

• 34:45Speaker 00

  I'll try to cover everything. I'll probably forget, so just remind me. OK, I'll start where, so I think that Alan's team were the first to say, guys, we want better connectivity with Kubernetes. And the reason I remember is we all started this discussion. A lot of our features are actually driven by the community. And Alan and his team started from the open source and kind of graduated in a way. Basically, they just said, we're sick and tired of maintaining our own servers. Plural, they had many, and they just said, it's not worth it. Just go do that for us. And we had multiple conversations even before. So we try to keep a very active stack channel. And GitHub. So we actually, this is how we develop features, right?

• 35:27Speaker 00

  Basically, people would say, hey, I want to build something and then it's just a crazy idea. And then you try to think about, okay, maybe this is doable. It kind of makes sense. And if it does, then we try to figure out first how to hack it. So someone can, you know, continue with with their day job and kind of build on top of it. And then try to realize, is there a way to actually structure into the platform itself? And if there is, we try to figure out a way to actually put it in there and see if there's traction. And one of the things that I remember that Alon Sim were the first to do was to better connect the orchestrator with the Kubernetes cluster. Basically, when we started developing it, it was like, I don't know, a long time ago. Kubernetes was not a thing.

• 36:11Speaker 00

  So containers were, but Kubernetes was not. It was just before Google just released Kubernetes as an open source solution. Before it kind of killed Docker. So we started with Docker as kind of a bare metal. So we said, okay, fine, we'll have the orchestrator that will just pull jobs, set up the container and then run everything inside the container. And they did that and it's great. But the resource management or allocation of Kubernetes is terrific. So these guys came and they said, look, guys, we have a Kubernetes cluster. And we like the idea of your orchestrator. So basically Clearmail Orchestrator will do, think of it as a dynamic Docker file in a way. I'm oversimplifying. Base Docker image with the ability to kind of control that kind of a Docker file you need to do in runtime.

• 36:55Speaker 00

  But and then also introduce some caching. Bottom line, you do not have to have like a container per job. Just to accelerate, because when you streamline a process, you cannot have like every step. Containerized, you end up with thousands of containers that no one will know who is using and no one will delete because someone might be using. And basically, yeah, you got the idea. Anyhow, so they, they, they said, okay, we, we love Kubernetes because it allows us to schedule resources very easily. But then when the resources schedule, we want this dynamic approach and obviously visibility, which is always, obviously hard with Kubernetes. We also don't want our users, like the data scientists, developers. To have actual access to the Kubernetes cluster because well, no.

• 37:38Speaker 00

  So I think that was the first time we developed the, what we now call the Kubernetes glue. Basically, it kind of converts a job from a ClearML into a Kubernetes job. Basically trying to figure out whether this job can actually be executed on Kubernetes. Give you kind of better visibility into the cluster itself. So users can basically push jobs into what we call a queue, which is think of it as in Kubernetes terminology. It's basically like the template YAML that you'll be using for that specific job. Only you have a priority queue on top of it. So it kind of implicitly holds the setup itself, which is kind of resources, et cetera. But also priority on top and then use that in order to schedule. Use Kubernetes as basically your resource scheduling, which is it is terrific for.

• 38:23Speaker 00

  But it's, it's lacking the scheduler itself, like order and priority, et cetera. This is exactly what the glue itself adds. And now obviously it, it solves for the, the problem of making sure that the end users, meaning the data scientists will have access to the Kubernetes cluster, right? So that was a first feature that we added just because of them. And this is how we heard on, so you guys are running Kubernetes on the edge device? And we were amazed that someone is trying to do that.

• 38:51Chris Benson?

  So I feel completely obligated to throw yet another buzzword and just ask you about if you have an opinion. And that is, that is.

• 38:58Daniel Whitenack?

  Don't go blockchain, Chris.

• 39:01Chris Benson?

  Well, there, that's a yet another one right there you throw out there. No, I'm just going to stick with kind of microservice architecture. When you were talking about, you know, all the containers out there and, and managing that. And as people kind of are moving more and more into microservice architecture over time and segregating off all their functions. And yet trying to keep them together, does ClearML as a platform have an opinion on that in any way? Or do you not care? Are you agnostic about, you know, where do people end up? So if I throw the word microservices at you, what do you say?

• 39:32Speaker 00

  Okay, that's terrific. I love it because microservices is in Kubernetes was invented basically to manage them. But the idea behind a microservice, it's alive, it's production ready, and it has to be stable. Like, the default of Kubernetes is if it fails, restart it because you had a good reason to put it there. This is not what is going on with AMLOPS. If it failed, then it'll continue failing. Like, just drop it. That's the default. That's the total opposite of microservices. And that's basically our approach. Our approach is use Kubernetes for what it's good for. So you probably have another cluster doing whatever microservices that you're running,

• 40:12Speaker 00

  which is terrific. But for the AMLOPS perspective, use Kubernetes as a resource scheduler more than anything else. It's basically the opposite of the default of Kubernetes. And then I guess the bridge is serving models, which is actually a microservice, but you want that elasticity because you still want to be able to change it without building new dockers all the time. You actually want that to be in-flight model upgrades, Canary, etc. That you probably want to control from outside, from like an ELB perspective. So this is kind of the bridge between the two, at least from our perspective.

• 40:48Daniel Whitenack?

  So if I'm getting sort of just like stepping back and thinking, I'm kind of trying to connect some of the things Alon you've said in terms of how things are working on your end. You sort of have data coming in off of the tractors, coming into various, like maybe data processing jobs, which might be queued up on a queue which runs through ClearML. That might sort of lead then into like model training jobs, which I agree. So I love your illustration, Moses, about like you expect a lot of model training jobs to fail. So like Kubernetes, like spinning up a service in Kubernetes to run a training is sort of like the opposite of like a lot of what they had in mind. But anyway, so you sort of spin up these jobs in a queue.

• 41:33Daniel Whitenack?

  So from the data scientist perspective, you're basically just saying, hey, I want to use this data to train a model. Put it in a queue, it runs the training and finishes or not. But then like that model then, which has like a version, it's tied to the data, then sort of gets shipped out. Does it get, in your case, does it get shipped out kind of like Moses is saying to a service that's running in your like edge K3S cluster? Like as a rest service for what's going on on the tractor? Or how does that piece work?

• 42:04Speaker 04

  It's a good question. I think we can see two paths to start the training. One path is from the data, as you spoke, and then the other path is for the researchers who want to test the code or new experiment, new model, or et cetera, and want to fire a training. So when we started, let's take the example for the researcher. He just go in his ID and he just connect to the remote workspace that is on Kubernetes that runs Groupon, Jupyter and so on, and he's just playing there. And he just click remote executed. This runs to Moses servers and tells the ClearML system to speak, to log everything.

• 42:47Speaker 04

  Like, okay, I want to use this container and this is the changes I did. Like Moses says, it does, it does not create new containers for every step. So it just keeps the changes. Moses, if I'm saying something wrong here, feel free to fix me. And then it goes to Moses servers and then from there it's going back to our servers as our agent there that Moses spoke about. This is the glue and this start the training. So this is the training and the training continuously reports to ClearML, to the main server, what's the status and at the end, what's the metrics? And once we have metrics, we can decide one of the two.

• 43:29Speaker 04

  If we are free a new version for testing on the edge or we are stopped there and just keep this track and moving on.

• 43:37Daniel Whitenack?

  And the inference you're running on the Kubernetes cluster as a service or as like on the K3S as a rest service?

• 43:46Speaker 04

  So this is tricky one. We have something in the box that is not in production. Like when I'm saying production, like internal production, a research that we are planning to have K3S inside a pod to have. A special environment for the developers that is separate, like a cluster in this cluster, something like that, but let's keep this aside. We are using at the moment KF serving to serve the model internal. So our metric system, just when we get a new model, we just serve it. And the metrics just being API rest service locally. And this is something like a micro service that they're using. One is responsible entirely. It does not know anything about the metrics itself.

• 44:31Speaker 04

  You can just probe it and to get result for each one of what you want to do. And there was another tool that is doing the metrics and it just probe the service and accumulate and then reports metrics to the Clearmail service. So we have three different service. Clearmail is not micro, but the other two are. And I think this is the cycle. We have a human decision in between. So we don't do the entire cycle for every model and not for every data.

• 45:03Chris Benson?

  So I have a final question that's addressed to both of you in turn. I want to start with Alon. I'm actually letting Moses cheat and hear what Alon's answer is. Yay. So Moses, I'm throwing you a bone on that one. So here's the question. As you are thinking about kind of these amazing uses of technology, both from the technology creator and from the technology implementers perspectives. And you were thinking about what's next. What are you wanting to do next? You've made it this far. You've had tremendous success and you've got to have something that when you go to bed at night, you're going, maybe I could do that. Alon, I'd first like to hear for you as the implementer in a very specific use case,

• 45:48Chris Benson?

  what you're thinking and then Moses, hint, hint, now that you've heard his question, you know, you can, you can answer yours as well. So Alon, I'll throw it to you first.

• 45:56Speaker 04

  So I have a many thing in my mind, a lot of imagination before I go to sleep. Sometimes I see bounding boxes and just seeing them and see them and see them. Sometimes I see different stuff, but yeah, in a larger scale, I think still in green eye perspective. We can do, and we are about to do the both that I said in the industry about decision making and helping the farmer, because we are already scanned the field and we have byproduct of lots of data and high quantity of data, high resolution data from the field. So our goal is to do both.

• 46:38Speaker 04

  And we have lots of idea that in the pipeline to also, one, to get, to make the decision ourself. And the other one is to help the farmer get better decisions for different stuff that's not related to spraying at all.

• 46:53Speaker 00

  That was a good answer. Moses, how about you? So two things I can expose here. So one, we are working now, this is public and one of the repositories, on the new version of the serving solution. Basically, we're not fans, we're not fans of KF Serving as an infrastructure, because it's very hard. If you have a single model, if you're not changing it, that's fine. But if you're constantly changing them, it's not easy. Adding preprocessing is not easy. Metrics, everything is hard. So together with a lot of people from the community, we redesigned the Clearmall Serving. So now it's internal testing and it's very, very nice.

• 47:34Speaker 00

  Basically, you can add the preprocessing without even code, deploy it, have it out of scale on your Kubernetes. It's basically building a serving service that is flexible and that you can change online, which is terrific. This is what we want from these type of services. So this is something that we're working on. And I'm hoping that we will be able to release before the end of next month. So I think we have a talk in GTC. So before the talk, that's a deadline, basically. We will have to release it, which is, it's always good to have a deadline. And the other thing that we're working on, and this is really research, we're still trying to wrap our heads around how to actually solve it. And that's coming directly from Alon, actually. So he was one of the guys that said,

• 48:19Speaker 00

  "Oh, I really want that." And he's not the only one. So we always want to make sure that we're not building for Very specific problem that it's it's actually a wide thread problem. And that problem is I have a lot of data stuck there there in the backend of ClearMeL, which basically means like multiple databases. I just want to be able to query more deeply, like create a dashboard. Like the data is there. I know I put it there. Now I I want to have better interface for the database without actually accessing the database. Like this is doable, but probably kind of too risky. So we we're trying to think how to hack together like a dashboard solution on top of it to allow you to better

• 48:59Speaker 00

  create better visibility for an entire process, because this the the entire idea is to make this entire Amel Ops holistic approach basically means that if you have data, you should be able to use it in whatever step you're along the way of of developing your your product. And this is something that is making but will be find its way out there very soon, I'm hoping.

• 49:19Daniel Whitenack?

  That's awesome. I'm super excited to explore those things. I'm a fan, so pretty excited to hear about the serving and and the other things coming along. And I appreciate both of you being willing to kind of talk through. First of all, give us an update on what is now ClearMeL, which previously we talked about as Allegro and all the great things you're doing, but also in the in the context of this use case with GreenEye. So I'm really impressed with what each of you are doing. And yeah, thank you both for for joining.

• 49:49Speaker 00

  Thank you. Thank you.

• 49:54Speaker 01

  All right, that is Practical AI for this week. If this is your first time listening, subscribe now at practicalai.fm or just search for Practical AI in your favorite podcast app. We're in there. And if you're a long time listener, please do share the show with your friends. It is the best way you can help Practical AI succeed. Thanks again to Fastly for shipping our shows super fast all around the world to Break Master Cylinder for the Beats and to you for listening. We appreciate you. That's all for this week. We'll talk to you again next time.

• 50:54Speaker 06

  Game on!