MLOps is NOT Real

We all hear a lot about MLOps these days, but where does MLOps end and DevOps begin? Our friend Luis from OctoML joins us in this episode to discuss treating AI/ML models as regular software components (once they are trained and ready for deployment). We get into topics including optimization on various kinds of hardware and deployment of models at the edge. Sponsors: Changelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with extended episodes, make the ads disappear, and increment your audio quality with higher bitrate mp3s. Let’s do this https://changelog.com/++! Ship It! https://changelog.com/shipit – A podcast about getting your best ideas into the world and seeing what happens. Listen to an episode that interests you and subscribe today https://changelog.com/shipit. Featuring: Luis Ceze – LinkedIn https://www.linkedin.com/in/luis-ceze-50b2314, X https://x.com/luiscezeChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: OctoML https://octoml.ai/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  You know, the tools for model deployments today are largely not super accessible to data scientists. They're more accessible to folks that are machine learning infrastructure, to say machine learning engineer. And I feel like with the right tools, you should be able to get a data scientist to export, you know, their model into a well-defined container that contains a model that then you can hand off to an existing DevOps team and IT infrastructure. They should not be specialized to machine learning. That step from a model to the thing that the existing DevOps teams and IT infrastructure could use, that can be automated.

• 0:45Speaker 01

  Hello, friends. Jared here to tell you about Change Log++, our membership program for those of you who want to directly support our work. Your++ membership gets you closer to the metal with extended episodes, makes the ads disappear, and takes our audio to the next level with higher bitrate MP3s. You can join today at changelog.com/plusplus.

• 1:18Speaker 01

  Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join us at practicalai.fm/community and follow the show on Twitter. We're at Practical AI FM. Thank you to our partners at Fastly for shipping our pods super fast all around the world. Check them out at fastly.com.

• 1:51Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International. I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

• 2:06Chris Benson?

  I'm doing very well, Daniel. It's a beautiful spring day here in Atlanta and we are going to have a good time for the next hour or so.

• 2:13Daniel Whitenack?

  Yeah, definitely. You know, it's interesting, the topic of ML Ops has increasingly come up on the show and we've had different takes on the topic and I'm really excited today to have a sort of different perspective on that. And so, welcome back to the show, Luis Ceze, who is CEO of OctoML. Welcome back, Luis.

• 2:35Speaker 00

  Thank you, Daniel. Thank you, Chris. It's great to be back here. I had a lot of fun, you know, almost a year ago now.

• 2:41Daniel Whitenack?

  It was almost a year. It's crazy.

• 2:43Speaker 00

  Lots happened since then and right now it's also a beautiful spring day in Seattle as well, so.

• 2:48Daniel Whitenack?

  Yeah, it was about a year ago, we talked through some things about Apache TVM and OctoML. Do you want to give just like a quick update on maybe like the Apache TVM world and then maybe, and then maybe circle back over to OctoML and like what's been happening with OctoML in that meantime?

• 3:07Speaker 00

  Yeah, absolutely. So on Apache TVM, lots of progress there. The community kept growing nicely and steadily with a lot of fantastic people doing work on machine learning systems, compilers and so on. So TVM, there's a lot of progress on automation and better performance. We call performance automation to make it easier to get to high-performance machine learning code in different hardware. We held TVM, our TVM conference in December last year, the largest ever. We had just about 1,600 registrants and 1,700.

• 3:38Daniel Whitenack?

  That's awesome.

• 3:38Speaker 00

  And 700 people actually attending live and then more folks that consume the content after that. And it was really nice to see contributors to TVM, but also folks from the general, you know, machine learning acceleration, the community and hardware vendors participate and cloud providers participate and so on. Also related to Apache TVM, we, we announced the TVM Unity Effort.

• 4:01Speaker 00

  Which is essentially an effort on bringing together all the key threads in performance automation, extensibility and so on, and integration with the rest of the ecosystem, you know, front and center on TVM. So it's our, you know, view there is really not what we call not too opinionated on how you actually get a model to run well on a higher targets about how do you actually enable people to do what they want productively, including using other pieces of the ecosystem. So yeah, TVM is moving along really, really well. So it's great to see. And now on the OctoML side. So since May last year, we more than doubled. The team is about 130 people now. We've made significant changes to our platform, to the SaaS platform that is

• 4:44Speaker 00

  uses TVM as one of its key components to automate the process of deploying machine learning models. And we recently released also a private accelerated model hub, which is a set of models that are pre-accelerated to a bunch of different hardware. So folks can see, you know, the power of the platform in automating the process of getting models from the hands of data scientists into deployable artifacts. Yeah. And we also formed a lot of former partnerships with with key hardware vendors like AMD, ARM, Qualcomm and cloud providers like Microsoft Azure.

• 5:19Daniel Whitenack?

  Yeah, that's awesome. You sort of mentioned this idea of the of the hub. We've definitely seen like model and data hubs sort of like just grow huge over the past year. Probably one of the things over the past year that we've really seen explode is like, you know, hugging face with, you know, 30,000 plus models now and other sort of hub environments. So yeah, it's interesting to hear how that that kind of idea is impacting a lot of, whether it's people trying to optimize models for certain hardware or people just trying to try out things and that's and that sort of thing. Has that impacted the the types of clients and customers that are coming into OctoML? Because like in general, they can access models much

• 6:04Daniel Whitenack?

  quicker and then they realize, oh, these are slow.

• 6:08Speaker 00

  Right, yeah, no, great point. They absolutely has affected not so not just customers that comes our way, but also, you know, I would say the entire ecosystem, you know, because the way I see the maturation of these model hubs is that it's much easier for folks to find models to start from, or even find models already, do what they need to do and just get them to deployment. So they're really making it all easier for folks to get to a working model that does what they need to do, which means that a lot of the action now shifts to how do you get these models to production, to add value as part of an application, right? I know it's great to see our friends at Hugging Face making incredible progress in the marketizing, you know, creating new machine learning models and creating communities around it. And our point of view on the model hubs is to complement that.

• 6:53Speaker 00

  So we're not talking about not a place where people come and find, you know, new models like Hugging Face and we find those models to create other models. It's more about here's some popular models that people can come and see pre-accelerated to a bunch of different hardware targets and see how they compare across, you know, edge devices and cloud instances and so on.

• 7:11Daniel Whitenack?

  Yeah, and maybe as a reference, I like to bring this up occasionally in the podcast because a lot of people are focused on that sort of training new models side and maybe like coming up with cool demos and such. But, you know, the bulk of what happens in industry in terms of how you run models is inference, right? So have you seen people like come in, they're really excited about like the demo that their data scientist, you know, created, but then just really blocked on that sort of, like what is the, what is the typical kind of process and viewpoint that you see come into OctoML?

• 7:53Daniel Whitenack?

  Where they maybe, do they already know what they want to do and they're just really blocked on scaling that up? Or is it, is it something else?

• 8:01Speaker 00

  Yeah, no, the part of the flow that we cover is you have a model that you want to deploy and then you have to navigate all of the path from a model to, you know, what kind of hardware you're going to deploy it on, you know, and for that hardware, what kind of libraries and compilers and tools you should use to actually arrive at a deployable. Artifact and even just extracting the model from what the data scientist produce, you know, from model to working piece of software is something that takes manual work and then we automate that. Right. So, but now this might be a good moment for us to step back and talk a little bit about ML Ops. Right. So, you know, thinking about the entire flow from data to a deployed model that's actually adding value to, to a business or adding value to a user, there's several steps, right?

• 8:42Speaker 00

  So you curate the data to create training data sets and then you think about model architectures. And, you know, you train models or you do some architecture search to find us right architecture for the model that you want. And then after that, you need to, uh, once you arrive at a model that has the right statistical properties for what you want to do, you need to turn that into a working piece of software that you can deploy. Right. So, and that step is very labor intensive, right? You have to extract code from, you know, it could be a jumbo mass of Python code to go and extract a model that you can put in a box with a clean interface, right? So you have to extract that.

• 9:17Daniel Whitenack?

  That terrible button in your notebook that's like export this notebook to a Python script.

• 9:23Speaker 00

  Yeah. And then from there to a working piece of software to go and deploy is, uh, is a lot of work. And then you have to go and optimize, make sure it has the right performance properties such that it has the right latency in case it's interactive or it has the right throughput in case it works in batch. All the way to, if you're going to deploy it in the cloud, make sure that you find the right cost effectively of doing so with the rights, you know, and also have the right reliability and the right, the expected behavior in deployment. Right. So in our opinion, by the way, is that model creation and model training does have its special place in the flow. I can, I can understand why in ML Ops, people, you know, think about those steps. But I think everything else comes after that, like how do you, you know, process a model into, how to put a model in a container?

• 10:06Speaker 00

  And how do you monitor that model in deployment? How do you build C I C D integrations and so on? All of that should just be DevOps, you know, people are building and calling that ML Ops as well. I feel like it creates a lot of confusion because the way I think about machine learning models today, and if you really zoom out a million feet is they are an integral component of any intelligent application today, which is pretty much any application that we're excited about today is, you know, can be called an intelligent application. Right. So it has, you know, natural user interface, can recognize your voice, recognize gestures. It has rich media and it has machine learning components as an integral part of them. But machine learning models are not treated as any other piece of software. They're treated as this special thing that's just hard to deploy, hard to integrate and so on.

• 10:48Speaker 00

  And we need to get past that, I think, to improve the cadence of innovation of these intelligent applications. So people don't have to treat machine learning models in any special way. This colors a lot in how I and we see at OctoML, the value that we can add is really enable folks to treat machine learning models as if you were any other piece of software, right?

• 11:08Chris Benson?

  You have no idea how relieved I am to hear you say that because that's like a huge hot button issue for me. I mean, that's like, Daniel's heard me rant about this repeatedly over time is, you know, the model's just part of the software at the end of the day. You know, that's, it's a model that's not, is not usable. So, you know, as you're looking at deploying it out, as maybe ML Ops kind of, you know, outgrows its diaper and gets into the big boy pants of DevOps and actually like becomes part of the real world around it and usable. How is that changing? You know, we talked to a year ago and kind of had some of the same conversation, but I know this is a very fast moving ecosystem and the evolution

• 11:51Chris Benson?

  at it. So can you talk a little bit about, you know, like, you know, I shudder to say it, is ML Ops is kind of growing up and is hopefully it gets further and further recognized and integrated into DevOps. How do you see that evolving over time into maturity? What does that look like to you?

• 12:07Speaker 00

  Great question. So I'd say that one thing that's changing fast that I think people are starting to understand or have some common view of what ML Ops is, because if you ask 10 people what ML Ops in this space, what ML Ops is, you're probably gonna get 12 answers, you know, so because some people are gonna have multiple answers to that too, right? So I think there's clarity being brought there on what is it that should be have, should have a different name. And, you know, if you allow me to be cynical for just a second, sometimes it's people like giving names to things because it makes it easier for you to grab attention from the investment community, you know, from investors. It's easier to grab attention from folks that are more on the, hey, the cool technology side and let's give a name to something that looks different. But I feel like sometimes you give a new name to something that already exists.

• 12:51Speaker 00

  That you can just do better can cause a lot of confusion, right? So, and I think that there's some maturity just that's starting to happen on what people call ML Ops. And I'm glad to see that most of it is going more and more towards, you know, how you deal with data and how you create models, because everything else I think should just be called DevOps, Chris. Shouldn't even be called ML Ops.

• 13:11Chris Benson?

  We're building solutions, aren't we? We're building big solutions to solve real world problems. And these are all parts of that solution.

• 13:19Speaker 00

  That's right, exactly. So, and then to answer your, I completely agree with that 100%. And to answer your question more, more directly, what I see changing even in that space is a couple of things. Like one, you know, let's say a year ago, there's a lot more emphasis on end to end fully integrated platform, like platforms like you get Sage Maker or you get, you know, Azure ML or some other big tools. But now I think there's a lot more attention to best in class for each one of the steps in this flow and have clean, clean integration points, right? So, you know, have tools to deal with data that has a clear integration point with how you move on to, to, to the training step, to how do we move on to say network architecture search, to how do you package the model and how do you actually monitor the model in deployments?

• 14:02Speaker 00

  And our, where we sit in that flow is again, and we have an API that allows you to, you know, we take a model as input and we produce your deployable artifact. There's a lot of evolution going on. And I can't talk about all the details because otherwise my marketing folks are going to be mad at me, but you're going to hear more about what we're up to soon. But basically, in a word, we're all about automation. So automating the manual steps of getting, getting working software model from, from like, you know, your, say your Jupiter notebook, right, your Python scripts, and, you know, put into a container that you can deploy, right? So that's highly specialized to the hardware target. But once we do that, if you put this in the right format, you should be able to use your regular DevOps flows, right? So if you have the right API, you can use GitHub Actions, for example, to do C I C D on your model.

• 14:46Speaker 00

  As you change your model, you run through this flow. And then if you put in the right container format, you can use existing microservices to serve your model, right? So, and then if you also put this, the right hooks for monitoring, you can collect data from these, from these deployed models and put it in, in, say in a data dog way of visualizing your data. You can put views on top of that to look at model behavior. So that means that I think a lot of the work being done. E-model monitoring today is really, really important, but I think it's less about the ML ops part of it of like, how do you collect the data? But it's more about how to abstract it away and find higher level behaviours for models that you should, you should go and debug because those things are different than how you debug software today, right? So sorry, there was a very long answer to your question, Chris. I don't know if it was an answer, it was more like a

• 15:31Speaker 00

  quick tangent there, you know.

• 15:39Speaker 04

  We are going to ship it, Chris.

• 15:43Speaker 05

  I'm Karhal Azou, host of Ship It, a show with weekly episodes about getting your best ideas into the world and seeing what happens. We talk about code, ops, infrastructure and the people that make it happen like charity majors from Honeycomb.

• 15:57Speaker 06

  We act like great engineers make great teams. And it's exactly the opposite, in fact. It is great teams that make great engineers.

• 16:05Speaker 05

  And Dave Farley, one of the founders of Continuous Delivery.

• 16:09Speaker 04

  Start off assuming that we're wrong rather than assuming that we're right. Test our ideas, try and falsify our ideas. Those are better ways of doing work. And it doesn't really matter what work it is that you're doing. That stuff just works better.

• 16:22Speaker 05

  We even experiment on our own open source podcasting platform so that you can see how we implement specific tools and services within changelog.com, what works and what fails.

• 16:33Speaker 01

  It's like there's a brand new hammer and we grab hold of it and everyone gathers around. We put our hand out and we, we strike it right on our thumb. And then everybody knows that hammer really hurts. When you strike on your thumb, I'm glad those guys did it. I've learned something instead, yeah.

• 16:48Speaker 04

  I think that's a very interesting perspective, but I don't see it that way.

• 16:53Speaker 01

  Okay.

• 16:53Speaker 04

  It's an amazing analogy, but I'm not sure that applies here.

• 16:57Speaker 05

  Listen to an episode that seems interesting or helpful. And if you like it, subscribe today. We'd love to have you with us.

• 17:16Daniel Whitenack?

  So one thing I was thinking about, Luis, as you were talking about this sort of transition from ML Ops to DevOps is just, I guess, the team dynamics that are at play here and the sort of human dynamics. And I'm just thinking, like, in my own experience with the teams that I've been on and working on AI models and different applications, there is this real sense that, like, you can do a lot. Like you say, GitHub Actions, I love GitHub Actions, I do so much with that. But like, that sort of like onboarding into that for someone coming from like a PhD scientist route and they like have like no idea this thing

• 17:57Daniel Whitenack?

  exists, right? And as soon as they find out, like, they can, you know, they're smart, they they can grab onto it and use it and figure out ways to use it, or things like Datadog, like you were talking about, other other things out there. How much of this sort of confusion in the terminology and the workflows here is is caused by this sort of mismatch of what teams are aware of and their sort of spheres of knowledge versus like actual functionality differences?

• 18:27Speaker 00

  Yeah, I know, this is a great question. So I would say that, you know, a key aspect in the in the human and team dynamics today is that you have folks that create models, typically data scientists, or some people call them ML engineers or data engineers. And then once they arrive at a model, they hand off to a team, if the company is big enough, right, that goes and, you know, turns that into the deployable thing, right? So and that team is still special compared to the DevOps teams. Okay, so these are folks that are more sophisticated engineers and understand machine learning, understand the tool involved and then put the work and then put that in an art, in a in a format that you can actually go and and deploy. And he writes that, you know, the tools for model deployments today are largely not super

• 19:09Speaker 00

  accessible to to data scientists per se. They're more accessible to folks that are machine learning infrastructure, or to a machine learning engineer. And I think this can also change, by the way. I feel like With the right tools, you should be able to get a data scientist to export, you know, their model into a well-defined container that contains a model that then you can hand off to an existing DevOps team and IT infrastructure that should not be specialized too much in learning. Right? So I think that step from a model to the thing that the existing DevOps teams and IT infrastructure could use, that can be automated. So, and I think there's a lot of work to make that automated because it involves human engineering today, but fundamentally it can be automated. And once you do that, I feel like you make both things productive, right?

• 19:54Speaker 00

  Data scientists can focus on making the models and then the DevOps teams can focus and continue making their applications run well and integrate models if you were any other piece of software and bring best practices to machine learning deployment, right? So.

• 20:11Chris Benson?

  It's funny, when you were, when you were talking about that a little bit, I was thinking you're solving two problems. You're solving the problem that you're describing, but you're also solving the problem that these, these data and development capabilities and organizations, it sounds like high school, you know, you have the jocks and you have the, the nerds and you have different groups socially that are doing stuff and you're going to automate the whole thing and it's going to bring everybody together, which is a good thing. It's a, it's a good thing in this world to do that. So I'm curious, as you are bringing everyone together, how does that change the dynamics for organizations working for these different individuals that have different functions right now? And they're off in their little silo and they're trying to interact. As you get those automations, it sounds like it'll make it more efficient.

• 20:56Chris Benson?

  How does that look to you? What kind of workflow are you striving for as you, as you achieve this?

• 21:01Speaker 00

  Good question. And again, this is all happening and maturing fast, right? So our current view is that with the right automation, again, you could have the folks creating models do the best job they possibly can in creating the models that have the right properties, right? So, and then on the, on that DevOps side, they can focus on continuing deploying models and deploying software the way they, they do. And I think that the dynamic is going to change that in organizations wouldn't have to go and look for people that are actually specializing both because that's the reality today. So this team. Between you know folks that create models and folks that run the regular DevOps. These are people that understand, again, as I said before, understand machine learning. They understand the tooling around, you know, all the the guts of say TensorFlow and PyTorch. And what are the right libraries to use depending on the hardware target? What are the right compilers to use?

• 21:46Speaker 00

  For example, should you use TensorRT for Nvidia? Should you use TVM in case you have, you know, broader options of hardware? And then you have to do performance evaluation to make sure that you're getting the right performance from your model running on the on the chosen hardware. Often they have to make decisions about the procurement. So if you're going to deploy it in the cloud at reasonable scale, chances are that your model is going to have a line item on the budget because it's going to cost a lot of money to run at scale. So you have to go and understand what are the cost implications. And these are the kind of things that none of these teams are used to. I just like it's not their strength. So DevOps don't understand enough machine learning to do that. And then data scientists don't understand enough of the systems aspects to do that themselves. So I think that the dynamics that's going to change is, you know, if we automate this right,

• 22:28Speaker 00

  you know, companies wouldn't have to go and look for this kind of people that are specialized, that understand this intersection, right? So and I hope this is going to make, uh, it's going to make it even more accessible for users to put models into production because they don't have to go look for people like that. They don't have to have the expert, the systems expertise coupled with machine learning to make use of machine learning.

• 22:49Daniel Whitenack?

  Yeah, this point about automation is a really key one. And I'm trying to find like other, you you may have some examples outside of the machine learning world that where there's reasonable parallels. But I was just thinking about like cyber, something like cybersecurity or something like that. Like at a certain point, I think if you look at kind of the history of how things developed, you really like had to have very specialized people having very specialized knowledge to understand like what sorts of vulnerabilities are in my software cybersecurity wise. And to some degree, like those sorts of people are still very valuable and they they have their place. Then you come along and there's like now systems, like I was just looking at one called Sneak,

• 23:33Daniel Whitenack?

  where it's like you can just run this automation suite on your software and figure out all the various vulnerabilities and like. Like the open source packages and the dependencies that you're importing and like where you're doing something wrong in terms of exposing this or that and like that allows any kind of person that can stick that in a kind of DevOps workflow to be able to like enable better security for their application, maybe not prevent everything, right? But you certainly can do much better. And I wonder if, I don't know if that's a good parallel for what you're talking about here where some of these things are still at that stage where you really have to know about all of these granularities, but as automation kicks in, a lot of those things are going to be taken care of.

• 24:16Speaker 00

  Yeah, no, that's a fantastic analogy, really fantastic analogy, Daniel. So yeah, I was thinking when you describe it, I was thinking about sneak exactly, right? So because, you know, a lot of the tools like sneak can, you know, plug into your DevOps workflow and whenever there's new code committed, you're going to kick off your, you know, static analysis to find for vulnerabilities. You're going to make sure that all the open source code used there has actually been vetted and has checked all of the security properties, right? So and that automation, I think, brought, you know, a significant progress in producing secure codes. The parallel there is valid. The only thing that I think is still different is that you have very, very different kind of people, like people writing writings, regular code today, they're subject to the flow like sneak would use to go analyze the code are

• 25:02Speaker 00

  typical software engineers, right? So and then you have some, you may not need a security engineer, but even if you did a security engineer, it's still kind of things like a software developer, except that, you know, they know what all the best practices for security are, right? So the kind of automation that we are talking about in machine learning, I think it's a, it's deeper and different because of the following. So first of all, the, the difference between somebody who can create models and somebody can write system software and then can write software to go and deploy is so much wider than what you typically have in between folks that worry and don't worry about security today. And then second, the kind of automation that's needed, it's still pretty deep, right? So as you're saying, like, you know, if you're going to export your model from a Jupiter notebook to turn that into

• 25:47Speaker 00

  you know, workable piece of software that you can go and deploy requires a lot of manual software engineering that hasn't been automated yet. Right? So there's still a lot of work to do. And I think that kind of automation is not quite as well defined, I would say, and as, as, you know, clean as the automation required going and analyze code for security vulnerabilities. So.

• 26:09Chris Benson?

  As you're talking about this, I'm kind of trying to visualize the description. And I'd like to ask you, can you give me kind of a concrete example of how this is evolving now? That either would have been much harder like a year ago when we spoke last, or at least, or maybe even not possible in the sense of, given the constraints that most organizations are dealing with, it could be anything, but what's a typical use case that you feel that you're enabling at this point going into it?

• 26:38Speaker 00

  Good question. So let me, let me give you one specific example. Suppose that you have a computer vision module in your application today, because you're going to go and verify whether images don't have anything inappropriate in them when you upload to a blog interface, let's say, okay? So the way you do that today, you have to, the way it happened in the past, like I'd say a year ago, is to go and maybe you find a model, you're going to probably put quite a bit of work on this model to make sure that it's actually classifying appropriate content correctly. And so this is all done by the data scientists, data engineers, I would say, and machine learning creators. Let's, let's use that label now. And then, once you, once you're right at the model, you have two options. You just go and say, you know what? Let's just use the regular, say, PyTorch or TensorFlow serving mode,

• 27:23Speaker 00

  and just hope that that's fast enough. If it's not fast enough, then you're probably going to hire a consultant to go and help you. Okay, so you're probably going to hire, you know, go hire folks to do it, to hire. And now, I think with tools that, you know, like, you know, TVM, some things that we're building, and then some other folks that are in this space as well, I'd say are essentially enables one to take that model and go through not just the default path and package with existing libraries that are not optimized for the hardware, but help you choose, you know, all right, if you're going to deploy in an Intel processor, what is the right libraries to use? If you're going to deploy in Nvidia, should you be using a TensorFlow compiler to generate a more performant version? And wrappers around it to go and run that for you and more easily produce a higher performance, you know, outputs.

• 28:07Speaker 00

  Even in the last year, that's already is getting a lot easier. But even then, after that, you still have to get that output and put an interface around it that has just the right API for you to integrate in your application. And that's what We firmly believe that we can automate even that, really just go from you upload your raw PyTorch or TensorFlow model to the service and then you get a package ready to be deployed with the right interface that you define, right?

• 28:32Daniel Whitenack?

  Yeah, this might be like a sort of off the wall question, but I know also, Luisa, you do some teaching and lecturing and other things. And I'm wondering, like, in a lot of the workshops, even like the ones that I've taught at various places, a lot of the focus is on the sort of model creation pathway. And I find like increasingly the people that I interact with in industry, once they're sort of in a position, are really like not even aware of some of those components that you just talked about, right? Like model optimization, like the different ways of serializing models, the different ways of serving models, like batch inference or other ways of applying models.

• 29:16Daniel Whitenack?

  And yeah, I'm just wondering if you have any, you know, thoughts on, is that true across the way that we're kind of bringing up the next generation of machine learning practitioners? And are there ways that we can maybe shift the balance a little bit?

• 29:34Speaker 00

  That's another great question. So you're right that, you know, a lot of the way that folks are learning about machine learning today and getting started with it, you know, they're not thinking about model optimization deployment. Because, you know, luckily, we actually create pretty good tools to arrive at models and test them and make sure that they have the right accuracy, it has the right properties. But chances are that most of those models created wouldn't actually see deployment. They wouldn't make it to deployment, right? So it's good for people to learn, but chances are they wouldn't be deployable. And honestly, I wanted to stay that way even for the sophisticated users. So let me talk about the other users. These are folks have already done this for a while and now they are improving models in a significant way because they're doing things that haven't been done the way they wanted before, right?

• 30:16Speaker 00

  Today, people already have to start worrying about performance too early. They have to look, okay, if I make this change in my model, am I going to be able to deploy it? You know, if I get deployed, should I deploy in these CPUs or GPUs? And so like, people should not worry about this. I want human power, like I want human creativity, ingenuity to go into making a better model and not have to worry about systems aspects too early because that's going to constrain the way they see the model, right? It's as if like as you're designing, you know, a new feature for a car, you're all thinking about all the sort of different. Different ways is going to be used like too early, then you don't, you're not innovative enough because you get, you get constrained by practicalities before you're able to unlock your creativity, right? So I like what, if you don't mind, I just go on a, on a quick side commentary.

• 31:00Speaker 00

  It's like, I like where hugging face is going, where it makes it very easy for folks to go, start from an existing model, make, you know, modifications from a foundation model, right? They talk about, and then specialize specific use cases. And then you don't have to worry about, you know, some of the systems details yet. When you actually put into deployment, chances are you're going to have to, to worry about it because performance is going to come and, and bite you if you don't. But at the conceptualization stage and model creation stage, I don't think people should worry about details of how these models are going to be implemented, right? So I wanted, I wanted to stay that way. But I wanted to stay that way and still allow these models to actually be deployed, right?

• 32:00Chris Benson?

  So I'd like to follow up. It's actually kind of combining what you were just addressing with, with Daniel's previous question a little bit, because the two of you together really got me thinking about a different way in that sense. And that is, you know, even now with all the model creation and all the detail on that, and yet that's not an industry, you know, we're, we're doing, we're using foundation models and there's a lot of transfer learning. And typically, students aren't doing this off the bat. They're, they're kind of going through it the hard way to, to teach the stuff. But, but then they get into industry and they're struggling because they only have a narrow focus of the picture. So as you're doing this work and as Daniel brought that point up,

• 32:42Chris Benson?

  I'm wondering, does it make more sense than to, by using these kind of tools, to get more of an end-to-end learning process going? And you're, by providing the right ecosystem and tooling as you're doing, you're essentially saving them from, from wasting what creativity they have on the wrong thing because you're kind of helping them through that. But at the same time, you're crucially helping them understand the full end-to-end workflow on how to get it out there. Is that the way of teaching AI going forward in your view?

• 33:17Speaker 00

  Yeah, so I'm so glad you brought this up because when I said that they shouldn't worry about all the practicalities of deployment, I meant that it shouldn't be constrained, but it should still have an idea of what would it take to take their models into production. And the more automated we make that, the more in the loop of model creation you can put it. For example, like succeeding where we want to be on taking models, you know, fresh models and deployable artifacts, evaluated benchmark and so on. If you make that fast enough and productive enough, you can actually put this in the active loop of as people are developing models, they can just go and try them out, right? So but not like the way it is today where I have to think about is it a CPU or GPU? What kind of GPU and then go and benchmark? Like if it's just completely automatic and seamless, you can always get for all the versions of your models as it evolves.

• 34:00Speaker 00

  You can see how, you know, how well this model will do in production and in a very in various scenarios. And just during model creation, right? So think of it as an outer outer loop of what they call network architecture search. Even outer loop from that is like for all the candidates models for what I'm thinking about doing. Just give me an idea of how well this this way of thinking about the original model would do in production. If you don't mind me add one more thing, that's not related to what you talked about, Chris, but it made me think of another thing that we've been observing that I think is interesting. Even though we're talking about a model, say model A or model B and how to take that model to production. The reality is, as we're talking about, models are integral part of applications, right? So of bigger applications. And it's not just a single model, by the way. It's all an ensemble of models.

• 34:43Speaker 00

  You have computer vision, you have language and you have like decision trees and you all combine this ensemble of models that they might talk to each other directly because there's a data flow from model A. Output of model A could go to input of model B. But also, even if they don't, chances are they're actually in a package running some machine in some container in the cloud that interact with each other because of performance. So like these are all system aspects that we have to worry about because in the end, you're going to have to package all of these into modules that it can actually deploy, right?

• 35:11Chris Benson?

  That's a great point you're making.

• 35:13Speaker 00

  The ensemble of models is something that's important, right? So, yeah.

• 35:16Daniel Whitenack?

  As you're talking about these things and, you know, going from this sort of ensemble and systematic thinking and these things interacting with each other, I'm wondering about some other jargon that has come up on the show in the past and is, I think, related to some of this automation and workflow sorts of things. And that's this sort of low-code, no-code type of stuff. There's increasingly this messaging around like, we're making AI and machine learning systems, low-code, no-code, et cetera. And, you know, I have maybe my own sort of opinions on where we're at on that spectrum and where things could go or might

• 35:57Daniel Whitenack?

  go. I was wondering, you know, maybe your perspective on that and like, how far can we push the automation aspects and the things that are, where are the machine learning engineers and the DevOps engineers really going to sink their teeth into the lower level like code? And where are those opportunities really for the like low-code and automation pieces that are maybe a little bit hyped at the moment?

• 36:24Speaker 00

  Yeah, so good, good points and juicy, juicy topics. So I would say that for model creation, I see a path to low-code, no-code. And it's kind of going that way, at least low-code, I can see, I can see a path because there's ways of you, you know, defining the data set, you know, how partition it, there are tools now to better label the data and create that. It just all has very little to no-code. And then when you think about classes of models, you can imagine it just being a high-level choice and not a programmatic thing that you write in a piece of code. And I, I see a path there. But now from that to deployment, I have a harder time. I can see, you know, some code, I don't call it low-code, because you still have to figure out like, okay, what is the API? Again, going back to the thing that we keep repeating every five minutes, like models being part

• 37:07Speaker 00

  of applications, right? So, you know, they still have to define an API is going to call that model. So, and then to be part of the rest of the application. So that part, I still feel like, again, maybe the rest of the code is already, we succeeded and all of that is already low-code and no-code. I don't know if that's going to be the case. But you have to have very, a very well-defined API for that to work and that involves some significant amount of code fundamentally in my view. And then there's some some other aspects that when you're actually taking a model to to deployment that you have to think about that. You know, I think are important, like, for example, is it latency sensitive or is it throughput? Like, for example, is it, you know, are you going to care about how how long each prediction each inference takes?

• 37:48Speaker 00

  Or you're going to care about, oh, if I do over all this piece of bu- bucket of data that you're going to have the right throughput, right? So and these are things that now you have a lot deeper systems thinking there that even if it's a low code, it still requires a lot of deep understanding of what you're doing that in the by that time you're going to require specialized people. It doesn't really matter what's low code or no code, right? So.

• 38:09Chris Benson?

  I got a follow up and and that would be, you know, if we go back to addressing kind of the low code, no code approach to model creation and or more specifically optimization, you know, there are a lot of kind of known things and you could actually, you know, you can train a model to do the optimization along those lines and so and we're seeing that.

• 38:27Speaker 00

  Great to see autonest auto ML works finally happening right here.

• 38:30Chris Benson?

  Yeah, I'm I'm wondering if, you know, one of the challenges is is we're addressing deployment and recognizing that, you know, people focused for a while on model creation way more than than getting the model out there in a in a usable fashion and that we're and that we're getting mature about that now. Your organization and there are others out there too that are thinking very deeply about this. Do you think that there's an opportunity for maybe low code, no code approaches? Once that we arrive at more kind of standardization, right now it's still very early. It feels like in terms of of different approaches. So there's a real custom feel to how you're going to deploy, especially when you combine it with all sorts of edge targets. And with edge being a catchall phrase, that could be almost anything.

• 39:15Chris Benson?

  And so there's a ton of specificity to your target at this point. Do you think over time, as those get categorized and kind of best practices emerge, that there might be more opportunities or do you think that's still going to be a challenge, given some of the API concerns and such that you mentioned in your previous answer?

• 39:32Speaker 00

  Yeah, I would say that once, again, once how the model is used in an application as well, defined in APIs is settled and we're talking about more the evolution of the model, I can see, I can see that being very low, you know, low to no code. Because by that time, we've already seen a path to deployment. If you find the box where it fits in and then I can see the model evolving, having very little code for the model, for the model updates. And to your point specifically, like if you're going to deploy it on the edge and you have to see how it works on a wide variety of devices. I also see a path to automation to say like, for this model that you created, it's going to work on 85% of the phones, for example. I think we, you know, going where automation is going, you know, having the ability to benchmark it across all sorts of scenarios

• 40:16Speaker 00

  where the model is going to be deployed and validate that it's going to work across the set of devices where you care about. I can imagine a feedback loop with the model creator that says like, okay, for these decisions that I've made, it's going to work on these classes of devices, you know. It's a great point, like it made me visualize that in a clean way. I can, I can see that happening. But again, only after you've defined where the model fits in the larger application, right? So.

• 40:38Chris Benson?

  I'm relieved to hear that because I work in an industry where absolutely everything that we target would be on the edge.

• 40:44Speaker 00

  Oh, yeah, great. Yeah, yeah.

• 40:46Chris Benson?

  And so if you hadn't given me some hope right there, I was going to start crying on you on that. So thank you. I appreciate that.

• 40:51Speaker 00

  I always like to step back and think about models being part of a bigger application. You know, that big application was written by someone. I had to put a lot of things by teams, right? They had to think clearly about where is the model going to fit, right? So once that is defined, I can see a lot of, a lot of the rest being, being automated and being very low, uh, low to no code, right? So I think that, you know, one, one, you know, way of actually summarizing this, if we nailed how to turn models, trained models, right? And by that point, you can do this in low to no code into like this agile, performant, reliable pieces of software that you can integrate to our application. Once we nailed that automation, you know, everything's going to get easier, in my opinion. You know, managing applications and also creating better models.

• 41:35Speaker 00

  Because then you have separation of concerns, the way that I think needs to be done here, given that machine learning folks are going to think, creators are going to think very differently than the hardcore software engineers that are on the other side of the application building, right? So we'll make sure it stays that way.

• 41:51Daniel Whitenack?

  Yeah, so maybe as you look at, hopefully, that future, what is your sense of like, over the next, you know, the next year, when we have you back on the show a year from now? What are those things that you would really hope that like, are maybe enabled that aren't at the moment? And but are sort of achievable within that kind of timeframe?

• 42:17Speaker 00

  Yeah, so I would say, again, being able to get a model that was freshly created by, you know, a data scientist, a machine learning, without thinking about systems, systems aspect of deployment, you know, having their models, benchmark, benchmark, knowing how it runs, you know, know how much it costs, they're gonna click deploy, it's gonna produce a box that the DevOps folks can go and deploy into the application. I think we have line of sight to that. And I hope next year, in the show here, we're gonna be talking about all the ways that people are using, are using that. But now, if you ask me to think about, what about five years out? I think what's interesting to think about here is that, how do you even stop thinking about what's edge? What's what's in the edge? What's in the cloud? What runs where? And just think about, hey, I wanna solve this problem that involves machine learning.

• 43:02Speaker 00

  And yes, so users are on the edge. Computers are in a building somewhere, spread all over the world, right? So, and I think a systems, an application creator, like, I don't wanna use the word developer. An application creator here should say, should be able to specify, here's what they want to do. And then the system should automatically figure out, okay, so we need this kind of models. And this model, this one should run in the edge. This one should run in the cloud. This one should might run in a cellphone-based station. And automatically split what should run where? That should be done automatically. Because again, that's an optimization problem that, once you define the constraints, you should be able to place the right piece of your model in the right physical place automatically. And if we nail the automation that we just talked about,

• 43:45Speaker 00

  from a model to the thing that runs in the hardware really well, that part is already also done, right? So now we can go and work on a higher level problem. Like, how do you break down your model into the pieces of, what's in the edge? What's in the infrastructure? What's in the cloud? Just the right way. I also think that that should be possible, right? So because, hey, we have machine learning designing chips today. That's pretty hard, right? So you have machine learning designing, you know, much better ways of doing, you know, power management in large-scale data centers. These are all low-level things that we used to do by hand or with heuristics, right? And we abstracted that away. As this goes up and up the stack, I'm very excited about the future of creating exciting applications without having to worry about all the design constraints that we have to worry about today.

• 44:30Daniel Whitenack?

  Awesome. Yeah, well, I'm very excited about that that future. And as always, it's a it's a pleasure to talk to you, Luis. And I'm really excited by the things that you and your team at Octomel are doing. Make sure and listeners, make sure and check out the show notes for some links to some really great stuff, both for TVM and for Octomel. And yeah, thank you again for taking time, Luis. It was a pleasure.

• 44:54Speaker 00

  Thank you, Daniel. Thank you, Chris. Always a pleasure. And again, looking forward to coming back and talking about all the other new stuff the next year. So thank you. You guys are fun.

• 45:02Daniel Whitenack?

  Bye.

• 45:03Speaker 00

  Bye.

• 45:12Speaker 01

  All right, that is Practical AI for this week. If this is your first time listening, subscribe now at practicalai.fm or just search for Practical AI in your favorite podcast app. We're in there. And if you're a longtime listener, please do share the show with your friends. It is the best way you can help Practical AI succeed. Thanks again to Fastly for shipping our shows super fast all around the world, to Brake Master Cylinder for the Beats, and to you for listening. We appreciate you. That's all for this week. We'll talk to you again next time.