MLOps is alive and well

Worlds are colliding! This week we join forces with the hosts of the MLOps. Community podcast to discuss all things machine learning operations. We talk about how the recent explosion of foundation models and generative models is influencing the world of MLOps, and we discuss related tooling, workflows, perceptions, etc. Sponsors: Fastly https://fastly.com/ – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Changelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with extended episodes, make the ads disappear, and increment your audio quality with higher bitrate mp3s. Let’s do this https://changelog.com/++! Featuring: Demetrios Brinkmann – X https://x.com/DpbrinkmMihail Eric – Website https://www.mihaileric.com, GitHub https://github.com/mihail911, X https://x.com/mihail_ericChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: MLOps. Community https://mlops.community/ Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Well, welcome to a very special episode. I'll say this is an episode of a podcast, actually of two podcasts, Chris. Uh huh. This is a different kind of episode than we normally do because we're privileged to be joined by Demetrius and Mikhail from the ML Ops Community Podcast. Welcome, guys.

• 1:06Speaker 02

  Yeah, we're hiding over here in the shadows.

• 1:09Daniel Whitenack?

  In the ML Ops Community, the shadows of the ML Ops Community.

• 1:15Chris Benson?

  We don't really have a guest this week. We have four hosts, is what we have.

• 1:20Chris Benson?

  This will be interesting.

• 1:21Speaker 04

  There's a lot of chefs in the kitchen right now. Yeah.

• 1:23Chris Benson?

  That's right.

• 1:24Daniel Whitenack?

  Which is obviously what listeners want more of.

• 1:30Speaker 02

  We're just going to answer each question with more questions.

• 1:33Chris Benson?

  I don't know if we have the hosts with the most, but we have the most hosts.

• 1:38Daniel Whitenack?

  There we go.

• 1:40Speaker 02

  The real question is, who's going to ask the first question?

• 1:42Daniel Whitenack?

  Oh, I was going for it, but now you can. But technically, that was a question? I'm not sure. So maybe I'll answer your first question with my first question. ML Ops and community. I love everything you all are doing with the community over there. Of course, the podcast, which, you know, we'll link to and the meetups and the content that you're putting out there. My sort of basic question, though, is like, we have heard ML Ops however many times on this podcast, Practical AI, and we've heard even conflicting sort of statements about what that even is. So like, you're branded the ML Ops community. So from

• 2:25Daniel Whitenack?

  your perspective, like, what does that mean? Who's in the community? How is it defined? All that stuff.

• 2:31Speaker 04

  So at a really high level, ML Ops, it's an abbreviation of machine learning and operations. And one really quick, easier way to think about what ML Ops does, it's fundamentally about the question of how do we take machine learning that you typically see in like research and deliver that in the real world. So until relatively recently, I want to say the last few years, machine learning was starting to get really hot. Maybe like it really kicked off in like 2012, 2013, right, is when we had this resurgence of deep learning that really picked up and people got excited about image recognition systems, speech, you know, ASR systems. And for a long period of time, a lot of machine learning, even cutting edge machine learning, still happened in the lab, right, in the research lab at other

• 3:14Speaker 04

  big institutions or even just at actual academic institutions. And so it was around, you know, I want to say around the time that TensorFlow came out, when we started really getting this effort to standardize the processes of going from something that you see in like research, new models, new AI systems, and try to actually put it out in front of people, right? How do we get these systems to the point where they can start delivering real value to real people? And ML Ops is pretty much a set of frameworks and tools and systems that people developed that are based a lot in software engineering as well as DevOps to make that translation process from research into production much easier. And so it's really just a set of like principles and techniques and tools that a lot of the communities developed to make that translation possible.

• 3:58Daniel Whitenack?

  Yeah, I think around the time like we started this, Chris, I think one of our feelings was, and now we're, you know, a good ways in, but I still feel this way that like, It's not an easy leap to sort of just start learning like, oh, I'm going to do the fast AI course or something. And then like, okay, let's say you get a position as a data scientist, or like whatever the title is. And like, okay, whatever is happening like in that position, like, there are overlaps between what you learned in like a fast AI or whatever. Like, certainly this like wonderful stuff. But like, it is only like a very small component of a

• 4:39Daniel Whitenack?

  larger like system of like practicalities that are not that easy to access and learn about.

• 4:46Speaker 02

  Totally. And they're still trying to standardize like, the other pieces, depending on your use case, ML Ops might be one thing or another thing to you. It's really not super clear right now. Like, if you're using deep learning, that's one type of ML Ops that you're doing. And if you're doing a recommender system, and you like to play in the trees, and the decision trees, those random forests, then it's a different type of ML Ops that you're doing.

• 5:14Chris Benson?

  So like, I think we should dive, as we go here, we should dive into the weeds on what those are. But I actually want to throw a hand grenade into the conversation and say like, DevOps or DevSecOps, depending on where you're coming from, you know, and like, the differences and ML Ops, how does it integrate? Is it some of the same? This is the show where we can clarify all of these things between us here. We should just dive in.

• 5:40Daniel Whitenack?

  By DevOps, do you mean a Docker, Chris?

• 5:42Speaker 02

  Oh, yeah.

• 5:45Daniel Whitenack?

  What do you, what do you mean by DevOps?

• 5:48Speaker 02

  Something you put in the container?

• 5:49Chris Benson?

  Well, once again, I mean, I don't want to put my, my definition. These are the things that have ops at the end that everyone is talking about. If you are out there in the world and you're doing models of various types, it might be deep learning, might not be deep learning, you know, Dimitri's those points. But, you know, it also integrates in with software.

• 6:10Chris Benson?

  Where does one stop and the other start? You know, if we can start sorting through what we think these things are, I think that would be useful because you put a bunch of data scientists and software people in a room and they still today don't necessarily know what each other means when they're having these conversations.

• 6:27Daniel Whitenack?

  I aired a bit of my dirty laundry and my pet peeve of like people saying a Docker or Dockers, I deployed a Docker. But yeah, I think that's like the minds that you hit when you put a software engineer and a data scientist together.

• 6:42Speaker 04

  I think it touches on a really astute point, which is that the definitions are a little fuzzy. That they are, to be perfectly honest, they're very fuzzy in terms of where, where does the domain of one begin and the domain of another? Like, how do we cross these territories? Where are the fiefdoms? Where are the the boundaries set up? And fundamentally, ML Ops is, has been and is a very interdisciplinary field. And so maybe one of the easiest visuals that we like to sort of discuss when we try and explain what ML Ops is, is like, imagine like a three circled Venn diagram where one is data science, the other is software engineering, and the third is DevOps. And ML Ops sits kind of squarely right in that little intersection area of those three circles. Because a lot of the stuff that makes ML Ops and the way people use it

• 7:26Speaker 04

  does touch on pretty much every single one of those three circles. There's modeling, right? There's fundamentally, you're building models, your training systems. You have these data concerns that really falls out of the data science category. You have software engineering, right? Because you're trying to actually build out good systems that have to be engineered. You have to write code. It has to be well-tested, unit-tested code. And then the DevOps is really about how do you streamline that process of going, you know, doing this once, putting it in a Docker, and then doing it multiple times, and then just repeating that over and over and over and doing it systematically so that, hey, I have 100% ideally uptime on the machine learning systems I put out into production. And that's really the DevOps component. And so really, it borrows a little bit from every single one of those circles.

• 8:07Speaker 02

  I see what you did there. And just shout out like to Docker Compose because you can do so much with that. Like, let's just not forget the unsung hero of this whole thing. And give a huge shout out to Docker Compose. Or the bash script.

• 8:24Speaker 02

  The true OG hero. True, yeah, exactly. The honest young hero in reality. But there's, you know, like Mikhail was saying too, I think that it's not only one person that should be tasked with this ML Ops title. And a lot of times, you can get yourself into a lot of trouble if you're trying to build out something that is going to productionize your machine learning and you expect your data scientist to be able to do everything, or you have a very naive sense of what it

• 9:05Speaker 02

  takes to actually productionize machine learning and not just like productionize it in a way where you get a model out, but really make a process out of it. And I really enjoy an idea, which is ML Ops is not just putting a model out there once, it's the act of N plus one. So ML Ops is a machine learning model in production, but not just one, N plus one. And that's when you get into, how are the processes around this? And what do we really need to be thinking of when we're productionizing our machine learning?

• 9:44Chris Benson?

  I have a question for you, just straight off of that. Can you talk about, like, when you're talking about getting that out in that N plus one model, how is ML Ops in that capacity, like, kind of late in the process there? As you're kind of getting it out into wherever your environment, the cloud, or wherever it is that you're putting the model, where does that differ from the DevOps world? Because it's easier to see it at the front end, because you're kind of starting with two different things that you're trying to put out there in the world, you know? In one case, you're starting by training a model of some type. In the other case, you're working on software and all that and you're, you know, you say, okay, we're ready to go. But somewhere down those pipelines, they start looking very similar in a lot of ways. How do you look at that?

• 10:28Chris Benson?

  How do you differentiate what those are? Do they merge? Do they not merge? How do you see it?

• 10:33Speaker 02

  So it's the fun one that we talk about a lot in the ML Ops community, right? Like, isn't ML Ops just DevOps and? The dirty secret, I think, is that, yes, it is just DevOps, but then you sprinkle a little bit of data on it. And so magically, it's not DevOps. And I think Michael can really like add a lot of great insight to this. But we were just talking about this, probably like a few hours ago when we met. And it was very much like ML Ops is more about software engineering than it is about the modeling piece, I think. Like, it's much easier for someone who knows about software engineering to get into ML Ops

• 11:18Speaker 02

  than it is for like a machine learning researcher to get into ML Ops. And that's what I think we've seen over the years. And we really have recognized in the past couple of years that there's been a very clear engineering discipline that comes with ML Ops. As opposed to, in the beginning, it wasn't so clear. It was like, oh, data scientists should be doing this. And now it's a little bit like, hmm, maybe it's not the easiest thing in the world to teach data scientists how to code.

• 11:50Chris Benson?

  You know, it's funny that you say that. And I'll leave the company unnamed, but a few years ago, I worked at a different place. And I had a team for a first AI team, and I had a team of data scientists. And I was like the only one to come from kind of a software development background. And everybody else was kind of straight data science. And that was what we found. It was a struggle a little bit to try to figure out how to create that ML Ops line. And I was like, there's an entire engineering skillset that I think people come into it without realizing it upfront. I think there's a, it's like an insight that they have, you know, that you just stated, that it takes them a little while to realize that.

• 12:32Speaker 04

  And the interesting thing is, when we look at how ML Ops, the evolution of ML Ops over the last few years. Because it seems to have been born a little bit in the wrong way in that data scientists started learning these principles and these software engineering practices. Right. So we had these data sciences like, oh yeah, like unit testing for data. That's super novel. And then the moment you had any person that came from DevOps or traditional software engineering, start entering these conversations. They're like, what are you guys talking about? Like, this has been a thing for like decades. Like, what do you mean testing your model code? What do you mean testing like different parts of your software engineering stack? Like, what do you mean using a Docker? Like, what do you mean by like making it seem like this is some profound insight you guys have really found? And yeah, in a way, there was, I don't want to say there was tension, but there was definitely like a

• 13:15Speaker 04

  WTF, what are you guys talking about? Like, this is not, this is not, especially the guys make it seem.

• 13:21Speaker 02

  Yeah, you have the different sides of the spectrum, right? Where coming at it from this engineering point and they're like, this is just DevOps. And then you have the data scientist coming at it from the opposite end of the spectrum where they're saying like, wow, this is so much different than anything that's ever been out there before. And so we need to build a whole new discipline around it.

• 13:45Daniel Whitenack?

  Yeah. Also, I think that the part of the confusion is like, just like everything else, like labeling and naming. Like some of, I think there's been data scientists and researchers over time. They're like, yeah, this sounds great. Like machine learning plus ops. And what that ends up being is experiment tracking, right? Which actually is something that also is not new because like people have been doing this in high performance computing for like decades and decades, right? But this sort of solution around like, let's say weights and biases, ClearML. Like, I love these tools. We use ClearML.

• 14:26Daniel Whitenack?

  Like, the value I get out of that is experimenting at scale, tracking experiments, making sure I know like what assets are what, like maybe logging data sets and like input, output, all of that stuff. But that's all like experiment tracking and production of the model.

• 14:45Daniel Whitenack?

  I sort of tend to think about ML Ops as like everything after that, right? Like you have a model, that's all great. Like whatever happened before, even if you want like lineage about what data was input to your training, which output what model, like that's all tracked, that's great. There's still some huge hurdles to get over in terms of like this model being called from within a software application that has like real users on the other end and like all of the potential implications around that. So in my mind, that sort of distinction is really confusing for people. And I don't know, maybe you can comment on this because it's maybe something that's come up on your podcast. I'm not sure.

• 15:30Daniel Whitenack?

  Have you also seen that confusion of like Ops versus experiment tracking? And the fact that a lot of data scientists might, might think there, it's like, uh, I think that word that you're saying doesn't mean what you think it means or like whatever.

• 15:48Chris Benson?

  Little princess bride in there?

• 15:50Daniel Whitenack?

  Yeah, exactly.

• 15:51Chris Benson?

  I think that word doesn't.

• 15:54Speaker 02

  That's so classic. It reminds me of the meme too, where you have the little girl who's standing in front of the burning house. And it's like, "Worked fine in my Jupiter notebook, it's an Ops problem now."

• 16:09Daniel Whitenack?

  I love Jupiter and all that, but when I, like when someone told me there was like an export to Python script function and Jupiter notebooks, like it absolutely frightened me to like no end. Like I'm like, "Why does this feature exist?" This is like, I guess I understand like where it could have come from, but it just like frightens me that maybe it's that evidence of like, you know, the disconnect between, oh, I wrote some code and did this thing between that and like software development and integration. I think it's evidence of that gap for sure. I've kind of been perusing your goods on, on your podcast, which is awesome. You know, various inflammatory titles such as "Airflow sucks for ML Ops."

• 16:55Speaker 02

  We learned from you guys though. Let me just tell you that I learned from you on the title or the naming conventions when Luis was on and it was like, "ML Ops doesn't exist" or something like that, or "ML Ops is a lie."

• 17:08Daniel Whitenack?

  Oh, yeah.

• 17:09Speaker 02

  I was like, these guys went and did it. Oh, man, like my lifeblood.

• 17:16Speaker 02

  It's amazing that we went from that title to having the ML Ops community on the podcast. That's why, yeah, I knew we needed to have a conversation because it was like, all right, I like it. And I think most people that if anyone knows me, they know that the most critical of ML Ops is myself. And I feel like I kind of have to be. And that's why I love when Luis says that kind of stuff. And we went and we grilled him hard when he came on our podcast too, because it was very much like, okay, what do you mean by that? And why is that? And I'm very much trying to see at every juncture, what is going on with ML Ops? And is there something new?

• 18:01Speaker 02

  Like my new favorite thing to ask people is, is ML Ops going to get hadoop? And that is like with ChatGPT or with foundational models, is ML Ops going to become obsolete? Or it's going to become something that old legacy companies use? And so that's my big question right now and what I love thinking about and those thought exercises that are going through my head. And so I almost feel like the pressure of needing to do that.

• 18:36Chris Benson?

  By the way, you realize that there's a very, very current version of hadoop, you know, that a certain large company has a big fear of. Right now, I think the big fear is instead of hadoop, would it be Googled?

• 18:48Speaker 02

  Yeah.

• 18:48Chris Benson?

  Because of ChatGPT, you know, you've seen all the stuff in the last few days about, you know, is the search algorithm gonna be gone? Is it done for?

• 18:58Speaker 02

  Done for? I've got a bit of a contrarian stance on that.

• 19:02Chris Benson?

  Okay, go for it.

• 19:03Speaker 02

  When I see that coming up a lot, I'm just kind of like, there's no way it's going to be replaced. Because first of all, I said it once and I'll say it again on this podcast, like my New Year's resolution was to be as confident as ChatGPT.

• 19:20Chris Benson?

  I saw that. I saw that.

• 19:23Speaker 02

  Because let's be honest, like it's the amount of bullshit that it spews out is incredible. I want that kind of confidence when I'm spewing bullshit at people, you know, like give me that, please. And so that's the first thing, like you don't know if you can trust it. But the other thing is there's a lot of stuff on there. Like I just see them as two completely separate uses and it's going to definitely take parts of what you would Google and that is for a good cause because a lot of the stuff that you do Google. Right now, it seems like the user experience isn't that good. And so I would like to have a better experience when I'm doing that. Like, for example, if I just want to know the recipe of whatever my favorite vegan bean burger.

• 20:08Speaker 02

  I can ask chat GPT and I don't have to get like this verbose, fully SEO optimized with a ton of ads and pop ups and everything. That user experience is horrible and so please like, don't make me have to go through that. But

• 20:22Chris Benson?

  I'm feeling some bro love here, by the way, as a vegan. I'm just saying, thank you for using that example.

• 20:26Speaker 02

  All right, there we go, there we go, nice. But yeah, then there's going to be a lot of other use cases where it just isn't the right medium, in my opinion. I don't know about you all, but that's my take on it.

• 20:39Daniel Whitenack?

  Just bringing it back to the ops perspective here, it's this weird like, because even with like fine tuned or like transfer learned models where you have this like foundation model type of workflow, generally up until recently, the thought process was, okay, well, I'm still performing a task. And that task, like I can generate like a table of tests for that task, right? And like minimum functionality, I can put that as part of like my automation for when I release this model. Like however you do that, whether that's like, you know, whatever ML ops tools you want to use for that. And now you've sort of got this, like, everybody's thinking about these generative models and like,

• 21:24Daniel Whitenack?

  okay, well, if I'm completely open domain, how, how do I mitigate risk in those situations with these sort of open domain models? And I think There is, so I think there are ways, and I think that some of the same workflows will apply, but I do think it does kind of tweak your mindset a little bit to think about some of these things. I don't know if that's, that's something that you all have been thinking about with the kind of most recent wave of like generative models.

• 21:55Speaker 04

  It's interesting because they do change the paradigm quite a bit in a lot of ways. There was a hackathon, so I'm, I'm based in the Bay Area and there was a hackathon, it was like last week, hosted by Scale, where people were just hacking on different things related to AI, but a lot of them naturally, because gen AI, generative AI is a hot thing to do, there were a lot of applications built around generative AI use cases. And one of them was actually, someone tried, basically built an entire backend using nothing but LLM calls, like they were basically just making calls and updating literally data structures in the backend, fully using generative models. Naturally, this wasn't like amazing, right? I mean, in the sense, it was a little bit hacky to try and like update dictionaries or, you know, like literally objects via these API calls using things like Codex and whatnot.

• 22:40Speaker 04

  But it does force you to ask this question of like, well, okay, now it's maybe not great, maybe now it's a little bit brittle, but we see this out, like, let's like, you know, imagine for a second what 10 years from now looks like. I mean, the 60s, computers took an entire room, right? And there was only IBM that had them, but here we are with them like literally on my wrist, you know, or in my pocket. And so it's not unreasonable to expect that like, some of the ways we even do some of these workflows are going to be touched by this, right? It's going to change how we operate with our software engineering systems and by extension, by our machine learning systems. I mean, what can be done away with once we have the only interaction being like language to language style, you know, kind of translations?

• 23:19Daniel Whitenack?

  I also wonder if there's like a meta layer here where like part of the question is, how do we test and like set up the ops around large language models and generative models? I think that's one question that can be asked.

• 23:36Chris Benson?

  And are those the same or are they different? Is one of the, I was gonna ask that anyway. How do they relate? How are they not the same?

• 23:43Daniel Whitenack?

  Sure, and how can one feed into the other, right? Like, what you were talking about just now, Mikhail, is that there's this idea of like, how could generative models help me with my ML Ops, right? Like, let's say I'm trying to put a model into production, and I'm trying to test that well, or like, I'm like, searching through logs and other things, and trying to parse that out. Like, if I can ask an agent to like, "Help me do some of those things", which, to be honest, are fairly predictable, if I've seen them before. Yeah, it's very meta, I know, but there's like, the one side, like, "How do I put these things into production?" And, as you say, Chris, like, "Are there differences with that?" And the second place, like, "Well, could I actually like,

• 24:28Daniel Whitenack?

  bring those around the other side, and help them do my... Have those models help me do my ML Ops tasks, right?"

• 24:35Speaker 02

  So, I am 100% in the same boat as you. I know the CEO of U.com, he posted on Twitter, like, "What is the best use of a large language model in your mind?" Of course, I instantly thought, like, "YAML fluency?"

• 24:56Speaker 02

  That would be incredible if it could do that! Like, let's just be honest, how much time would it save if it could just go and set up my Kubernetes cluster for me? And I've talked to a lot of smart people about that, and I think a lot of people are like, "No, that's not gonna happen." Like, "That's too far." But I also am a little bit like, "Yeah, well, there's a lot of other stuff that we thought wasn't gonna happen, and it's happening right now, so..."

• 25:25Chris Benson?

  I don't think that's too far. I, like, that is my pain point. Like, we're this far into Kubernetes, and that still hurts. Like, if you're not just using someone else's implementation, you're gonna set up your own cluster, it's still painful. And like... That this is known stuff. It's just a pain in the butt. That's a great thing, right? I don't think that's too far at all, and I want it now.

• 25:47Speaker 02

  Yeah, exactly. Maybe it's just like a fine tune away. Let's be honest. Maybe that's all it is. And somebody needs to think about that. I'm sure I'm not the only one that has thought about that. And so, but I'll let Mr. Eric chime in too. I know he has some strong thoughts and I'll like preempt his thoughts on this with his idea of an incredible app on top of generative models was stealing somebody else's IP. You want to tell us about what you've created, Mr. Eric?

• 26:19Daniel Whitenack?

  Oh, that sounds perfect. And by the way, just disclaimer, I'm not implicating myself in any of the following conversation.

• 26:29Speaker 02

  This is not hacking advice.

• 26:31Chris Benson?

  Oh goodness, this show just took a turn.

• 26:34Speaker 04

  Oh boy. I didn't realize I was gonna be basically implicating myself criminally by coming on this podcast. But we're gonna edit this out, right? This is edited out.

• 26:43Daniel Whitenack?

  Sure, sure. We'll say that now, but we didn't sign a contract of any type, so.

• 26:49Chris Benson?

  And there are no intelligence agencies listening, I promise.

• 26:52Speaker 04

  Absolutely not, right? Absolutely not. Yeah, I mean, I guess I'll maybe spend a little bit of time talking about the use case that I think Demetrius is referring to, which is a little bit of a toy use case. We didn't do it because we were trying to get sued. That wasn't the goal, but hopefully it should spur a little bit of creativity.

• 27:10Chris Benson?

  What a start.

• 27:13Speaker 04

  Well, okay, maybe, maybe like 10% we were trying to get sued. No, the use case to give an idea of like what has become possible outside of, you know, yes, the really practical use cases, generating YAML, generating unit tests. I think these are all in a very, on a very serious note, these are things that are going to be possible. I actually don't see this being as too far off. And I don't, you know, not even like a decade out, I see this being like the next few years. Being able to spin off things like that is just gonna be totally feasible. You know, this is like a language-based kind of generation, generative modeling. The other one that we haven't really touched on as much, which is also really capturing people's interest, has been more of the image-based, right, like vision-based, whether that's a single static image, or that's an entire video, or in some cases even audio, just like different modalities beside pure text.

• 27:59Speaker 04

  And so, one that has certainly become really interesting because of the rise of things like Dali from OpenAI, and then Stability, you know, Stable Diffusion, has these been these incredible photorealistic images and different styles that you can just prompt with literally human text, right, which has never been possible before, or at least not at this level of quality. And so, one of the things that my co-founder and I actually

• 28:20Speaker 02

  Here it comes.

• 28:21Speaker 04

  Yeah, that we were working on was this little toy application that we called Rick and Mortify. And the basic use case was, we were big fans of the Rick and Morty TV show, and, you know, we love it, we think it's super, like, it's very great, great, great content.

• 28:37Speaker 02

  That did not stop a cease and desist letter from coming.

• 28:42Speaker 04

  There is a backstory here that I found out after we released the application, which if you guys are curious, I'll go into, but we were really trying to test this hypothesis of like, okay, you have these vision-based models that are incredible, you have these language-based models that are incredible, how can we like merge them to actually try and do something between, like kind of at the intersection of them two? And so what we came up with was like, well, can we personalize episodes of Rick and Morty? Like, if I, as a superfan of the show, want to imagine a new episode, if I provided a premise, if I provided, you know, a set of characters that I wanted, maybe myself eventually, but let's just start with the basic characters that are in the show already, Rick, Morty, Summer, you know, Poopy Butthole, Mr. Me-Six, like, all these folks. That is an actual character name, I just want to point out for the audience, I did

• 29:27Speaker 04

  not just make up a name to try and be profane, that is a character, Mr. Poopy Butthole.

• 29:35Chris Benson?

  I'm not sure we've ever had that word on the show before, actually.

• 29:40Speaker 04

  I didn't know it was a word until the show. So, you know, we're both in this, we're all new to this game. What we essentially built out was you could come into this application, you could provide a premise of what you would like the episode to look like. So Rick and Morty go to the Practical AI Podcast and have a great conversation about generative AI. And then you pick your characters and using a combination of vision-based generative systems, Stable Diffusion, DALL-E, and then the GPT-3s, right, that, you know, these kind of latest generations of GPT models, we were able to generate not only the visuals, we're able to generate like a script of effectively a storyboard for a new episode. This is just like a first use case, like, well, we're literally getting new episodes, you know, maybe there's like five, 10 frames of this episode.

• 30:20Speaker 04

  But you're seeing some flavor of a plot with dialogue, with accompanying visuals, and you know, sure, it's a little bit rough. But extend this out to the future, right? Like, what happens then? Like, this has become literally a fully fledged episode. And that's where this could become, right? And so, no cease and desist yet. But people did play with it, people had a lot of fun with it. We're not making any money on it, so, you know, don't worry, Dan and Justin, like, it's, it's just for, it's Fan Art Friday. We're just trying to show our appreciation for the show.

• 30:49Chris Benson?

  You raised something, it's kind of funny, because when, just in day-to-day conversations about AI topics, and like, new model comes out, and you see all the media stuff where people are just kind of bashing it and telling you all the problems with it and everything. And to your point, you just said this, that's why I bring it up. It's like, but think of what we can do with this tomorrow. Like, today we have this, and it's imperfect. And yesterday, we didn't have any such thing. Think about if today we have the imperfect thing, tomorrow's could be pretty amazing. And to your point there, like, what you're doing there is like, way out there, but it's not far from being very mainstream.

• 31:27Speaker 02

  So my question about that is, like, you have the training data and you have the base show that a human made, right? And so, without that, like, I'm trying to extrapolate 10 years in the future, and if it's just going to be us remaking a bunch of shows that we made back when we used to do it all by hand? Or is it, like, that's, that's kind of my question there, right?

• 31:54Chris Benson?

  I think that's a generative function right there. I don't think you have to start with where you were in the sense of part of that function at the very front end of that workflow is going to be generating different possibilities that are not directly linked to the training data and then like carrying it from there. I think, I think it's the future of entertainment.

• 32:25Speaker 00

  Hello friends. This is Jared here to tell you about ChangeLog++. Over the years, many of our most diehard listeners have asked us for ways they can support our work here at ChangeLog. We didn't have an answer for them for a long time, but finally, we created ChangeLog++, a membership you can join to directly support our work. As a thank you, we save you some time with an ad-free feed, sprinkle in bonuses like extended episodes, and give you first access to the new stuff we dream up. Learn all about it at changelog.com/plus. You'll also find the link in your chapter data and show notes. Once again, that's changelog.com/plus.

• 33:10Speaker 00

  Check it out. We'd love to have you with us.

• 33:21Daniel Whitenack?

  What I think is kind of disconcerting is the right term. I don't know the time scale on this or whatever, but like imagine like all these data sets around language models especially are scraped from the internet, right? And like a lot of these image data sets are scraped from the internet, right? So there's a proliferation of these models and the internet is being filled with computer-generated content, right? So the next scraped versions of the internet, like the next Common Crawl, the next whatever, like what proportion of that data set is coming out of generative AI models? Now, I think there's interesting things going on, of course, with like detecting what is

• 34:05Daniel Whitenack?

  AI-generated and what isn't. I think- I've seen like what OpenAI came out with for their own system and of course, like you say, Chris, everybody has like criticisms of that already, but there's like many other people exploring this as well, like GPT-0 and like other things that are exploring like how do we determine what is AI-generated and not? And so that's where like, I don't know what the implications of that are for these large-scale data sets down the road, but that's sort of where my mind is going more so than, oh, should we be populating the internet with a lot of this generative art and other things, even if it's terrible? Well, I'm having a lot of fun with it right now, but there's sort of second-order effects,

• 34:50Daniel Whitenack?

  I guess.

• 34:51Chris Benson?

  No worries then, man.

• 34:52Daniel Whitenack?

  Yeah, well, I'll be dead by that time, I don't know, maybe?

• 34:56Speaker 04

  No, no, no, no, this will be soon. It's a really interesting thought experiment, and again, one that is closer than I think we believe. This idea of what happens when the bulk of these data sets are actually generated by systems, and I can see, you know, some positive effects and some that you would be like, okay, this might be a questionable downside. On the positive side, you know, these models, like the generative ones, especially the language variety, are like very good, I mean grammatically correct, right? They're like very semantically good in terms of what they output. Does this mean that now the bulk of the data that's being trained on is just going to be like way higher quality writing than you would typically find in typical Reddit posts, right? Or something just in the deepest corners of the web? And then now if you're training on data that's significantly better, you have this compounding

• 35:41Speaker 04

  effect of like, well, now the data I'm training on is better, so the model's only getting even better at writing of some kind, and then we mix it in with some other more diverse writing, and then is it going to continue to compound on itself in terms of quality? The downside, I mean, one you could hypothesize is that like these models are only able to generate certain kinds of distributions of data, right? Only certain kinds of things they can write about or talk about. So now when you're just injecting all of these training sets with like a very skewed distribution of topics, of ideas on these topics, et cetera, how do you ensure that actually you're still giving your model enough of a versatility in what it sees, these next generations that are then trained on these data sets, to ensure that it's still a general purpose model, right? Like what if 90% of the content that's put out there is just really bad marketing copy? You know, Facebook ads or something?

• 36:25Daniel Whitenack?

  You overfit.

• 36:26Speaker 04

  Yeah, it'll overfit exactly to that, and then what happens to these next generation systems? They actually might be hurt in the long run.

• 36:32Speaker 02

  Yeah, it's like we're in the golden era.

• 36:33Daniel Whitenack?

  Yeah, I think that you can have an ML Ops community, in my opinion, for a long time because there's such a wide variety of problems that people are still dealing with. So on this like very far end, we're talking about generative AI. Being part of like my day job as part of an international NGO, like, we're dealing with problems still like where, hey, there's no internet in this place and like we're running our model on like an Android tablet or like whatever. Like, there's this range of like, okay, what's the problem that happens when I scrape the whole internet again, all the way down to like, how do I run this small

• 37:12Daniel Whitenack?

  model on an Android tablet? And I don't actually see that changing for some time. Yes, the world's like changing a lot, but there's still like such a wide variety of issues and I think fun challenges to like wrestle with around this concept of like operations plus AI and machine learning. And so, yeah, it is sort of hard to define in that sort of way, but it's also really exciting because of the wide variety of things that you could be involved with. Like, as a DevOps person or software engineer or data scientist, like, there's plenty of problems that have to do with like, running even like a much older

• 37:55Daniel Whitenack?

  language model on your phone and like other problems that have to do with like various different scales or different modalities of data, all these things. It's, yeah.

• 38:07Chris Benson?

  And that goes back to kind of in the pre-show that our listeners were not there for, we kind of talked a little bit about that level of diversity. Like, how do you guys see that? So, I mean, Daniel, you've kind of talked about almost an extreme case of kind of edge concept, you know, being your target, you know, and having this slow Android thing. And yet, we're used to being just like gluttonously resourced in the cloud, you know, everything you could possibly want there. Ramp it up, ramp it up. You've got all the stuff you want there. And you have this incredible diversity. So, for you guys thinking about ML Ops, you know, so much, and how do you deal with that? Like, when you go from big company to one person struggling to work at all, how does ML Ops look when

• 38:52Chris Benson?

  you're talking about diversity of use cases plus diversity of users and that you're serving? How do you make it a thing? You know, how do you keep it all together there?

• 39:01Speaker 04

  A few, a few answers to this question, actually. So the first part of the diversity of use cases. And in a sense, we've gone a little bit backwards from something we said before. We're on the one hand, we're like, oh, generative systems are just going to do it with all of this. And then now we're back at like, well, but it's, there's like enough stuff that we still need to solve that it'll probably be around.

• 39:19Daniel Whitenack?

  Both and.

• 39:20Speaker 04

  We're not quite there yet. Exactly, exactly. And to that point, I actually, I was actually asked this in a podcast once a few weeks ago, where I was asked like, what do I believe will be the position, like the role that will exist 10 years from now? And I was asked, like, will it be like a data scientist, a prompt engineer, an ML ops engineer? Like, where would I put my money? And I still answered ML ops engineer, like machine learning engineer, you know, like kind of in that, in a similar category. Because I do believe that these same problems will persist, whether or not therefore, old school decision tree based models, discriminative models that we used, you know, maybe five, 10 years ago, even pre-deep learning, or these new GPT 500, whatever, will, will come later, models, stable diffusion 10,000.

• 40:03Speaker 04

  You know what I mean? Like, the same problems will persist. How do we, how do we operationalize it? How do we make it scalable? How do we keep uptime on models so that people can interact with this? These are all questions that machine, you know, that ML ops is fundamentally trying to solve and address. And whether or not you're using Kubernetes today, or some prompt engineering based Kubernetes tomorrow, or, you know, you know, make it even more concrete, like airflow today versus like an airflow for prompt engineering, right? Which is what people are actually developing today. The same sort of principles and the same concerns are going to apply. And so I don't see that going away anytime soon. As long as there's a machine learning system, as long as, you know, we, I would assume all of us here in this room, in this virtual room, anyone who's listening here believes that AI is going to be the future.

• 40:48Speaker 04

  It's going to be here for decades. Then the same questions will still have to apply. And so that's like the first part I want to just like throw out there is, is there's, ML ops is here to stay. Whether or not it's Kubernetes or whatever comes after Kubernetes, we hope it comes soon. We hope it comes soon.

• 41:07Speaker 04

  Someone needs to get it soon.

• 41:09Daniel Whitenack?

  I have a tip there that I'll provide later. Yeah, I don't know if you all have seen what Eric Bernadson is doing with Modal and Modal Labs. I've been playing around with that recently. I've been pretty floored by it. But anyway, that's a whole nother side topic and episode, which hopefully we can have soon.

• 41:26Speaker 04

  Totally. But I think maybe the second part of the question, which was, how do different organizations think about maybe the ML ops question? And something that we do want to address here, which is, it does depend. It really does depend in terms of the maturity over the organization and budget, time, et cetera. These are all different axes that fundamentally define how a team or a business should think about its approach to ML ops. And there's different axes that we can go into, exactly what they are. But, you know, open source versus not open source, like are you going to use something? You're going to stitch together a bunch of open source tools or are you going to use SageMaker out of the box? How much money do you have? Can you get by with just spinning everything up on your own? All these different axes, different organizations have to think through.

• 42:11Speaker 04

  And it becomes not a one size fits all. It really is like, it's a function of all these different parameters to really tailor the right solution to the organization.

• 42:21Speaker 02

  And there you go. You didn't realize that Mr. Michael has a little consultant in his blood.

• 42:28Speaker 04

  Do I get to put another plug here? Another shameless plug? Is that what this is? No.

• 42:32Daniel Whitenack?

  As many as you want. Yeah, as many as you want.

• 42:37Speaker 04

  That is the most consulting answer you can possibly give to any question is like, well, for this rate, I can tell you more details on what comes after.

• 42:45Speaker 02

  Yeah, exactly. You want to go at it. But I think what I just will add a quick piece to that, which is there was a time in the ML ops community that like a week wouldn't go by in our Slack where someone would not share the "you are not Google" blog post. And it's like the amount of people that try and go at it and try and get that. Especially because Google puts out so much great thought leadership on ML Ops. And they have the zero, level zero, level one, level two, or they have the like ML test score, all that stuff. And people think that straight out from zero to one,

• 43:27Speaker 02

  you need to be creating everything automated. It needs to be like the most high performant, bulletproof system that you can think of. And it was just setting up a lot of people for failure. And I think we've moved past that because I haven't seen the You Are Not Google blog post being shared as much in the community, which makes me think people recognize and they're a little bit more self-aware when they're trying to create their systems at their jobs.

• 43:57Chris Benson?

  You don't think they're just off in the corner crying?

• 44:00Speaker 04

  Or they just don't think being Google is a good thing anymore, just objectively given, give it.

• 44:06Speaker 02

  Yeah, that might be it too.

• 44:07Daniel Whitenack?

  I think I was listening to, I don't know if you all listen to the Indie Hackers podcast, but a recent episode, like I think they made this good point. They were talking about like, because they talk about a lot about like bootstrap startups and stuff like that. And, and one of the points they made was like, you know, whenever like Basecamp was around or like starting up in that sort of thing, like they made such inflammatory statements about like, like taking venture capital is stupid. Like, why would you ever do like, there was a need for that voice to be in there, right?

• 44:42Chris Benson?

  That voice is still there with the founder, you know?

• 44:45Daniel Whitenack?

  Yeah. But now like you hear less of that, I think, and it's sort of like normalized in that like there's still different perspectives, but there's not like as much like it's become more normalized to have these like more nuanced discussions. I think with ML Ops, there's sort of like, you know, we've all made our sort of inflammatory like podcast titles, like whatever it was, ML Ops is dead, or you know, I forget what it was. And so there is probably like still this, we're kind of feeling out where like the normal is and where things settle down to and providing like a balanced perspective where, yeah, like.

• 45:26Daniel Whitenack?

  You probably shouldn't be doing ML Ops the same way Google is, but you also should probably be doing ML Ops. It's just like, where on that spectrum do you land? And what type of tools make sense for you?

• 45:40Chris Benson?

  You know, in real life, I don't think it's you probably should not be. I think it's you can't do ML Ops the way Google does. I mean, just from a resource standpoint, you know, most companies don't have that team available and that set of tools. And, you know, I like the fact that we're talking it in a more realistic context here. You know, for the vast majority of us out there that are not, you know, accessing the best of the best in all the categories. It's not possible for many of us. We have to settle for something that's doable. Kind of going to Demetrius's point. You know, you have to find that level that you can do it. You can sustain it and yet it's still incredibly productive, even if it's not the Google version.

• 46:20Speaker 02

  Yeah, and one thing I'm fascinated by, just because there is almost this open source, you've got one person trying to hack something together and looking out there and seeing what's on the market that they can get for a price point between free and cheap. And then you've got the Googles that have built everything and have so much time and ability to do that. And in between those two points, you have a lot of companies that popped up. And they popped up in the last like three to five years. And as Michael was saying, we came at ML Ops from this data science perspective.

• 47:03Speaker 02

  And so I think in the beginning, a lot were trying to cater to them. And then some were like, whoa, wait a minute. There's like platform engineers. And so they try to cater to them. And then you're seeing now.

• 47:17Speaker 02

  I almost wanted to like change my title for what I do on LinkedIn as I'm just gonna be like the uh, I ride hype waves because ML Ops was a complete hype wave and we felt it and especially like, I just got lucky because the pandemic hit and right when it hit, I was working for a company that was trying to sell ML Ops tools. And that company went out of business, but I was in the ML Ops world. So I figured I would start this Slack community and then it took off. Right? And so I was able to ride that hype wave and we really felt it over the last two, three years. And now it's like, all right, now there's this generative AI hype wave. And so if you talk to VCs and the ones that poured

• 48:01Speaker 02

  a boatload of cash into the ML Ops market, they're now like, yeah, that's kind of not really that big of a deal anymore. We're not going to follow up on all those investments. What we're going to do is invest in the next AI tool. And so I love thinking about that and how now what is the new hype cycle of the AI and generative AI large language models, like where's that going to play out? You know, we've been talking about it. There's a lot of potential out there. But where are we going to go with that? And is it like just a bunch of money pouring into it? And who knows what's actually going to happen?

• 48:39Chris Benson?

  It's kind of funny when you say that and it's like there's an oversimplification, you know, when the market observes these things. You know, and you know, ML Ops, well, now we're past that, you know, that's been solved, you know, we're gonna web three, you know, whichever hype cycle it is. But I think when you look at what's happening, it takes all these things to have gotten, you know, right now we're saying generative because that's kind of the sexiest part of the puzzle. But it's not just generative. It's the fact that you have ML Ops now that's matured a little bit and is supporting all that. You have the large language models that people like Daniel have been working on. If you have all the transformers that are well established, if you didn't have all of those components,

• 49:22Chris Benson?

  the current hype that's now being attributed to generative would not be happening. And so it's once again an oversimplification by the market on the sexy piece. But it's that whole ecosystem that's evolved over the last few years that enabled all that to happen. And so we're seeing a really cool moment, definitely. But it's really the fusion of it all as opposed to just being generative. Um, you couldn't have generative today without that.

• 49:49Speaker 04

  I think that's such a fantastic point. Like, I just want to like sit on that a little bit longer because when the GPT three paper came out, this was, I guess, toward the end of twenty twenty, maybe middle to end of twenty twenty. You know, this is like long thing that, you know, I remember reading and the most interesting, in my mind, achievements of that whole system was not even anything about modeling, right? Like fundamentally, the building blocks of the AI architecture, if you want to call it that, was just, we've been using these systems for years in some sense. It was really the fact that they had built this incredibly good software infrastructure to build out, train large systems at this scale over this many GPUs at this latency

• 50:32Speaker 04

  to make sure that, you know, the updates for the gradients could happen fast enough that this wasn't going to take fifty years to train. And they did that all, which fundamentally was really like an ML Ops challenge. I mean, at its core.

• 50:42Chris Benson?

  Agreed.

• 50:43Speaker 04

  You know, being able to archive that kind of a system is the complexity. It's not the fact that there was this new scientific achievement that we really came up with. It was really like an engineering achievement. And so in that way, it totally was a layering on of things that we had seen before. But, you know, now they're like old hat, right? Like now we have open source repos that can literally approximate the same effect, like deep speed from Microsoft has become really fast and then being very used. You can train large systems at that scale. But the people that pioneered those systems really had to solve some ML Ops challenges in their own right.

• 51:13Chris Benson?

  It's all right there. They haven't gone away. You know, it's still right there. It's a team effort for all of those different parts, constituent parts to put the whole together. But at this moment, uh, the word generative is the front man, apparently, you know, as we record this today. It's pretty amazing that it's that ML Ops take that has put that all together and kept it together, kept them in lockstep so that you can create the new things of today.

• 51:39Daniel Whitenack?

  Yeah. I mean, another example of that is like all that we're hearing about about language model chaining and that sort of thing like. This is like, if you look at each step of that process, right, Langchain or whatever it is, those are things we've been talking about for a long time. Like, there is an operational burden, though, to like chain these things together and make them work well, like in concert, right? That's really like one of the fundamental things that we learned, like when we talked about ChatGPT, right? Like, the like, the language model existed, reinforcement learning existed, human feedback existed, like, all of these things existed, like, the chaining of them together in a certain workflow is the real

• 52:21Daniel Whitenack?

  interesting piece. So I do think that's really exciting, where a lot of those sort of chaining operations and like bringing things together in unique ways, that is a lot more possible because the tooling has gotten better. It doesn't like move us past ML Ops. Actually, it just like, we use ML Ops slightly differently. And if anything, I think it becomes more crucial because there's so many moving pieces, right?

• 52:47Chris Benson?

  Shoulders of giants all the way through.

• 52:49Daniel Whitenack?

  Yeah.

• 52:49Chris Benson?

  You combine that with the Einstein turtles all the way down kind of thing, it's shoulders of giants all the way down. It's...

• 52:54Speaker 04

  Yeah. The one thing to add maybe to that is, very often when I have described sort of the progression of ML Ops over time, like, people often liken it, use the Gartner hype cycle, right, to describe trends in technology, right? You have a kind of the initial hype, then the waiting, and the kind of the trough of just disillusionment, and then this gradual climb upwards, right? I've often said that right now where we are, we're at that gradual climb upwards. Like, the ML Ops systems and the technologies developed are like standardized, and they're becoming more commonplace, and people are using them. But the thing is, that linear climb upward, while it is where a lot of value is extracted, is not really what a VC would look for. They would want this. They want something that's like exponential in growth. And so they're never gonna ride a hype train that's like a linear climb upwards, even if that's where people are actually deriving value. They're

• 53:39Speaker 04

  like, well, what about, how do we get to the next thing in the next Gartner hype cycle, where there's the next big inflection point up that we need to ride? Because that's where you get real 100x gains, if you know, that's the kind of thing they're looking for.

• 53:49Chris Benson?

  You know, and ironically, the place where you find all that value there after that initial hype is from the trough of disillusionment, you know? That's where you kind of go, now I understand. Now I know what we really need to do. And you get that really good growth after that.

• 54:05Speaker 04

  That's a great point. That's a great point.

• 54:07Daniel Whitenack?

  Yeah.

• 54:07Speaker 02

  This has been amazing.

• 54:08Daniel Whitenack?

  Yeah, this has been awesome.

• 54:10Speaker 02

  I have not needed to carry the conversation at all, so I appreciate you all doing the heavy lifting for me. Usually I am constantly thinking about that and you made it very easy today, so I appreciate that.

• 54:26Chris Benson?

  This was a fun conversation.

• 54:27Daniel Whitenack?

  Yeah, this is awesome.

• 54:28Speaker 02

  We should do this more often.

• 54:29Daniel Whitenack?

  Yeah, we should. We should circle back and see if any of the things that we said actually are true next year, which maybe some of them, I'm going for like 25% if I get there, I'm good.

• 54:42Speaker 04

  Well, I didn't put my money on any of these predictions, so I don't really care.

• 54:46Daniel Whitenack?

  Right. You've already explained that you're not culpable or legally on the hook for anything.

• 54:53Speaker 04

  Rick and mortified, it's just a toy project. Don't worry about it. I come in peace.

• 55:00Chris Benson?

  I keep waiting, you know, because we can see each other on video. I know it's an audio recording, but I keep waiting for the feds to bust in behind you on your screen there, you know, you know, the door goes flying back.

• 55:11Daniel Whitenack?

  It's a different sort of BBC kid moment.

• 55:16Speaker 04

  Yeah, and then my mic just goes out and it's like, well, he'll be back probably, maybe.

• 55:20Chris Benson?

  He'll be back. That's right. But then Demetrius would have to carry the conversation.

• 55:25Speaker 02

  Yeah, that would not be good. I'd be gone looking for you.

• 55:29Daniel Whitenack?

  Well, next year, we'll see if this or any of the other things come true on Practical AI Plus ML Ops community. Definitely check out the show notes for those Practical AI listeners out there. We're going to include links to all the great things going on in the ML Ops community, Slack channel, podcast, events, newsletter, all sorts of amazing stuff. And yeah, thanks, guys. It's been awesome to have you on the show.

• 55:56Speaker 02

  Likewise.

• 55:56Speaker 04

  Thank you. Pleasure.

• 56:06Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Chang Talk podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.