YOLOv9: Computer vision is alive and well

While everyone is super hyped about generative AI, computer vision researchers have been working in the background on significant advancements in deep learning architectures. YOLOv9 was just released with some noteworthy advancements relevant to parameter efficient models. In this episode, Chris and Daniel dig into the details and also discuss advancements in parameter efficient LLMs, such as Microsofts 1-Bit LLMs and Qualcomm’s new AI Hub. Sponsors: Changelog News https://changelog.com/news – A podcast+newsletter combo that’s brief, entertaining & always on-point. Subscribe today https://changelog.com/news. Fly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Sentry http://sentry.io/events/launch-week/ – Launch week! New features and products all week long (so get comfy)! Tune in to Sentry’s YouTube https://www.youtube.com/c/Sentry-monitoring/featured and Discord daily at 9am PT to hear the latest scoop. Too busy? No problem - enter your email address to receive all the announcements (and win swag along the way). Use the code CHANGELOG when you sign up to get $100 OFF the team plan. Typesense https://cloud.typesense.org/?utm_source=changelog – Lightning fast, globally distributed Search-as-a-Service that runs in memory. You literally can’t get any faster! Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: YOLOv9: Yolov9: Learning What You Want to Learn Using Programmable Gradient Information https://artgor.medium.com/paper-review-yolov9-learning-what-you-want-to-learn-using-programmable-gradient-information-8ec2e6e13551Yolov9 Object Detection with Programmable Gradient Information (PGI) and Generalized Efficient https://medium.com/ai-trends/yolov9-object-detection-with-programmable-gradient-information-pgi-and-generalized-efficient-4fa3352409ccYolov9: A Comprehensive Guide and Custom Dataset Fine-Tuning https://www.datature.io/blog/yolov9-a-comprehensive-guide-and-custom-dataset-fine-tuningYOLOv9 SOTA Machine Learning Object Detection Model https://encord.com/blog/yolov9-sota-machine-learning-object-dection-model/YOLOv9 https://docs.ultralytics.com/models/yolov9/Unleashing the Power of YOLOv9 https://www.linkedin.com/pulse/unleashing-power-9-yolov9-gurneet-singh-wcrrc/YOLOv9 with NNCF and OpenVINO https://www.linkedin.com/posts/yurygorbachev_yolov9-nncf-openvino-activity-7168875232626163712-3k6pArXiv:2402.13616 https://arxiv.org/abs/2402.13616Parameter efficient LLMs: Hugging Face Paper page, 1-Bit LLMs https://huggingface.co/papers/2402.17764ArXiv paper: “The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits” https://arxiv.org/abs/2402.17764Qualcomm AI Hub https://aihub.qualcomm.com/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents, so you can launch your app near your users. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another Fully Connected episode of the Practical AI Podcast. In these Fully Connected episodes, Chris and I keep you fully connected with everything that's happening in the AI and machine learning world. We'll take some time to dig into the latest news articles and releases from the AI community and hopefully share some learning resources that'll help you level up your machine learning game. My name is Daniel Whitenack. I am the founder and CEO at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris? Doing great, Daniel. How's it going?

• 1:26Daniel Whitenack?

  It's going great. I'm spending a few weeks in the UK, which is a lot of fun, and have got enough sleep to not be jet lagged quite as much, so that's encouraging. Okay, so we have a transatlantic podcast going here today. Exactly, worldwide. That's right, across the pond. Practical AI Worldwide, 21st Century Incorporated. I don't know. We need rebranding. Exactly. Yeah, yeah. Well, Chris, one of the things I was going through, I don't know how often people are flying these days, but one of the things that stood out to me as I took my flight across the pond was...

• 2:09Daniel Whitenack?

  Now, when you board at least some flights, you don't even give them your ticket, right? You just go up and there's a little, I guess you would call it a kiosk, a little edge device that takes a picture of your face and matches it, I assume, with what was your scanned passport, which you scanned at the time of check-in and you board your plane, of course. And it was really, really fast as well. And the same thing happened, you know, crossing into the border, into the UK. As long as you have a certain passport, you just go up to the little machine and scan your passport and then it takes your picture. And I'm assuming, I could do a little bit of research. I'm assuming what's happening under the hood

• 2:53Daniel Whitenack?

  is that it's matching your actual facial features up with the image on your passport and computing some score of shadiness or something like that, or risk associated with you not being the person in the... But I was amazed at how fast it was. And I'm assuming, I could be wrong. I'm assuming maybe some of that's running at the edge, not reliant on an internet connection to do that facial recognition. I'm not sure if you, if you know or if you've had also this experience, Chris?

• 3:27Chris Benson?

  I don't know what they're using algorithmically, but I definitely partake of the technology. It's an area that I forgo privacy and always buy my way into expeditious processing. So, yes, I'm curious.

• 3:39Daniel Whitenack?

  Well, I don't know in that case if you have a choice. Maybe there is an opt-out situation or something. I'm not sure. But it's pretty cool that some of this technology is being applied at the edge and in a very seemingly efficient way, such that you could use it on a mass scale like that. Or I don't know if you'd consider that a mass scale, but it's definitely in use for many. You know, there's a huge flood of people going through those stalls and the computation happens very quickly and reliably enough to make a judgment.

• 4:14Daniel Whitenack?

  In the midst of all the hype around generative AI, one of the things that stood out to me over this last news cycle, Chris, was the release of YOLO V9. So we're on the ninth iteration of this YOLO model. Did you happen to see any of the videos of YOLO 9 in action, Chris?

• 4:37Chris Benson?

  I haven't seen the YOLO 9 one, but I'm kind of stunned. You know, when you think about it, YOLO has been around a long time, it was occurring to me. Because we actually had some conversations about YOLO back in the very first days of this podcast, which has been, you know, closing in on six years now. So it's, uh, V9 is a long time coming. Uh, and we haven't really gone back and touched such models in quite a while. It's, we're long overdue.

• 5:01Daniel Whitenack?

  Yeah, yeah. So as everyone is freaking out and enjoying the hype over large language models and other generative types of models, Sora and all the things coming out. In the background somewhere, there's these amazing computer vision people that are just really cranking and innovating on, actually at the architecture level of neural networks in really interesting ways. So it might be good to set a little bit of background for this. Chris, you mentioned, um, we've been kind of talking about YOLO for some time. So if people just search for YOLO, Y-O-L-O, object detection, you'll see, uh, you know, a huge set of articles and GitHub

• 5:45Daniel Whitenack?

  and everything about YOLO. YOLO actually kind of made a splash because it processed entire images in a single pass for object detection and bounding box detection. So if you think about, if you've ever seen one of those videos of like a street with a bunch of people walking around and cars and dogs and shops and scooters and whatever.

• 6:10Chris Benson?

  With their boxes around them?

• 6:11Daniel Whitenack?

  Yeah, and they have their boxes around them and they, they're labeled person or whatever. That's likely YOLO. So what happens is that single image in a YOLO model goes into the model and then out. Comes the bounding boxes and the actual classification of those bounding boxes, which is interesting because previous models, previous to YOLO, I'm still sure some models do this in a multi-stage way, which is more computationally expensive. So they actually take multiple passes through a model or multiple models to compute both the bounding boxes and the classes.

• 6:48Chris Benson?

  Yeah, I remember way back when we were first starting and I was at a different employer. I was at Honeywell, leading AI there at the time. I remember just as YOLO 2 came out, we were using that for a couple of projects that we were working on way back in the day, but that seems, I mean, that's like before dinosaurs were in the earth by AI standards. But yeah, way back.

• 7:10Daniel Whitenack?

  Yeah. And I think even we had a podcast episode, maybe about Fast R-CNN or whatever it's called, the fast version of R-CNN.

• 7:21Chris Benson?

  Good memory.

• 7:22Daniel Whitenack?

  Yeah.

• 7:22Chris Benson?

  Good memory.

• 7:24Daniel Whitenack?

  That one's cool. I mean, that one, I think how that one worked was you pass your image in and then it detects the bounding boxes of objects. And then in a second pass, it then classifies each kind of subsection of the image as its class, which also is very effective, but it's less efficient computationally than the YOLO kind of single pass thing. And as you mentioned, there have been multiple versions of this. So between YOLO and now version two, version three, all the way up to version nine. And each version of these in some ways has, and not just in a kind of more train with more data way.

• 8:08Daniel Whitenack?

  They've actually made kind of very significant discoveries and improvements in neural network architecture, training methodologies, this sort of thing that has led it to be kind of the go-to solution for at least real-time object detection in images, which is why you see all these videos of the bounding boxes around, around people and such.

• 8:31Chris Benson?

  They've at least gotten the visual bit a little bit nicer than they used to. We had the big clunky boxes overlaying everything. It was.

• 8:38Daniel Whitenack?

  Correct.

• 8:39Chris Benson?

  Yeah.

• 8:40Daniel Whitenack?

  Well, the V9 version of the project, which dropped, at least if the date on the archive article link is right, that would have been the 21st of February of 2024 as we're recording this. So not that long ago, but it was developed by an open source team and kind of built on, on top of a codebase from Ultralitics Yolo version 5. And it's released, I believe, under the GPL 3 license is the code that they released. But the, it seems like what they focused on with Yolo V9 was continued focus on efficiency to where like

• 9:24Daniel Whitenack?

  you can do realtime object detection, meaning like as the frames of a video are coming in, you can process those in, in realtime with the model. So efficiency is, is really key in these types of applications. And then they focused on one of the fundamental challenges of, of deep learning models, of these deep neural network models, which is called the information bottleneck principle, which happens because, especially as you kind of propagate, if you think about a neural network, what it is, is a big data transformation, right? You take a bunch of matrix data in the front end, maybe representative of an

• 10:06Daniel Whitenack?

  image, and that gets processed through successive layers of processing. And then out the other end comes maybe these indication of classes or other things. And the information bottleneck principle talks about the errors or the lack of information or the loss of information that you lose as you process an input through the successive layers of the feed forward process of that, of that neural network, which in some ways can be addressed by having bigger networks and more data. Maybe you're less prone to these informational problems, but it's more of a problem when you're dealing with these

• 10:48Daniel Whitenack?

  very efficient lightweight networks, like the Yolo networks, because you have less layers to deal with and you don't want to lose any information that might be relevant to the classification of the outputs.

• 11:01Chris Benson?

  I noticed within Yolo's nine docs, they talk about also reversible functions as well. Does that feed into, no pun intended, does that feed into the ability to not lose data by reversing that feed forward through a function backward? How does, how do you see that's utility?

• 11:19Daniel Whitenack?

  Yeah. So the interesting way that they dealt with this, or kind of addressed this, at least in this version of the model, is something that they're calling programmable gradient information, or PGI. And the PGI portion of their research and advancement relies on a couple of things. But one of the main things is this focus on, again, improving the informational efficiency of the network. And one of the ways that they've done this is with what they call an auxiliary reversible branch. And this gets to these reversible functions that you mentioned.

• 12:02Daniel Whitenack?

  So the concept of a reversible function, for those that maybe that's new to them, means that the function and the inverse of the function can transform data without the loss of information. And so, again, there's that loss of information piece there. And so it's a little bit hard to describe this on the podcast without having a whiteboard or a visual. But if you think about this PGI functionality that they've added into the network, it's kind of like they're bolting on this auxiliary reversible branch, which helps deal with this information loss as gradients are calculated during the, the training process. And so during the training process, this reversible

• 12:46Daniel Whitenack?

  branch helps not lose that gradient information as during the forward pass and during the calculation of the updates of the weights of the model. And that helps it be very efficient during the training process, but it's called auxiliary, which is key because you can actually unbolt it and take it off for inference, which means, I think, part of the problem in the past with these reversible branches and efforts at this were

• 13:15Daniel Whitenack?

  Helped with the information loss, but it also decreases the efficiency in terms of computational efficiency of the model during inference.

• 13:23Chris Benson?

  I'm going to throw a question at you, and I realize this is not your thing. But just in case, is you're using a reversible function in that programmable gradient information process that you're talking about? And in a normal feed forward network, you're maintaining the weights as they're going through and those are changed? And are you reversing functions to maintain that back in the same space to where you're actually maintaining a new weight? And you're keeping that gradient information for maybe future feed forward passes? Or do you have any sense of what the purpose of that is?

• 13:56Daniel Whitenack?

  Yeah, I think that, so definitely we'll link some of the papers and the explanations in the show notes. So feel free to look at that for accurate information and let us know if we get it wrong. But yeah, I think that the idea is that, and the reason why this is especially useful in the training side of what they're trying to do, and it's kind of unbolted during the inference side, is that during the training time, it's really crucial that as you're calculating the updates to your weights, you can do that in a very informationally accurate, precise

• 14:36Daniel Whitenack?

  manner, especially for these lightweight networks, which have fewer parameters to train. And so maintaining that information, especially as you're calculating updates based on the gradients, is really important.

• 14:50Chris Benson?

  Gotcha.

• 15:11Speaker 00

  Shipping Quality Software in Hostile Environments. Luca Claderic writes, quote, "I once had the opportunity to work for a startup that had fallen from tech debt into tech bankruptcy."

• 15:23Speaker 03

  Bankruptcy, Michael, is nature's do-over. It's a fresh start. It's a clean slate. Like the Witness Protection Program. Exactly.

• 15:32Speaker 00

  Although we managed to get it back on the right track, it made me rethink the concept of tech debt and how we ship software, especially in hostile environments, end quote. He goes on to tell this true story in great detail, which is horrifying, yet echoes so many of our experiences. Here's just one of the many horror scenes Luca describes. Quote, "There is also a handcrafted build server, a Jenkins box hosted in the office, but no record of how it's provisioned or configured. If something were to happen to it, the way you build software would just be lost. Each job on it is subtly different, even for the same tech. You have an Android source code that you build three instances

• 16:12Speaker 00

  out of, but each of them builds in a different way," end quote. This is a solid essay, replete with warnings and a plea at the end to ditch the tech debt concept altogether. You just heard one of our five top stories from Monday's Change Log News. Subscribe to the podcast to get all of the week's top stories and pop your email address in at changelog.com/news to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com/news.

• 16:51Daniel Whitenack?

  We talked a little bit about Yolo version 9's programmable gradient information. I had to remind myself, PGI, programmable gradient information. The other piece of the architecture, and I think this is just really interesting. You've sort of got all of this going on on the LLM side where things are getting very interesting ways to fine-tune and preference tune and all these families of models on the computer vision side, man, They're really, really thinking deeply about the architectures going into these models, which have made them so, so efficient. The other thing that kind of is a combination of things that have come in the past that they're utilizing

• 17:35Daniel Whitenack?

  in this Yolo V9 is a generalized ELon architecture. So this is kind of a progression of a couple of things that have been in Yolo models in previous generations, but they've combined them in kind of a unique way. This stands for Generalized Efficient Layer Aggregation Network, or GELON. And this combines a couple of things from previous generations of Yolo and from things like CSP net. This has to do with how features are aggregated and gradients are aggregated through the model in a very efficient way, again, leading to a very parameter-efficient model,

• 18:20Daniel Whitenack?

  meaning a smaller set of parameters in Yolo V9 will have similar performance to maybe models with many more parameters. So this leads to the efficiency overall.

• 18:34Chris Benson?

  It's pretty interesting. They talk about being able to adapt to a much wider range of applications without sacrificing speed or accuracy. Is that a form of fine tuning the model? Or something that they're doing ahead of time that you're then fine tuning on top of that?

• 18:50Daniel Whitenack?

  At least how I read some of that flexibility in was, yes, there's kind of a parameter efficient. This is a parameter efficient setup for fine tuning maybe to a variety of types of scenarios, or even training a new model from scratch in an entirely new domain and doing that very efficient. And some of the things that I've seen, you know, people have already quantized this model using things like OpenVINO, which is very popular for these kind of edge vision cases, and running this very efficient, so real-time object detection on even desktop or laptop CPUs.

• 19:34Daniel Whitenack?

  So the new architecture developments are both geared towards, yeah, that efficiency, but also squeezing every ounce of performance out of parameter-efficient models, both in terms of training and flexibility across different use cases.

• 19:49Chris Benson?

  Yeah, I think there's great applications for this on the edge where you're not in a one of the giant clouds with essentially, if you're willing to pay for it, infinite compute available to you, whether it be training or for inference, either way. So the fact that this can run on just about anything, I mean, back in the early days, we could do Yolo V2 on smaller equipment, but it was, it didn't run smoothly. You'd have points where that it would overwhelm the computational cycle. And so it's nice seeing something like this has come this far. It's quite an open source library.

• 20:22Daniel Whitenack?

  Yeah, and there's a link that we'll add into the show notes, which includes a notebook for running Yolo V9 in a Colab notebook, even like I say, on CPUs. So in terms of the efficiency, one of the things that I saw was Yolo V9 operates with 42% fewer parameters and 21% less computational demand than Yolo V7, yet it achieves comparable accuracy. So, you know, it was already fairly accurate, right? And kind of an industry standard, but now with much fewer parameters. And I think that that is definitely a trend that we've

• 21:05Daniel Whitenack?

  been seeing, not only in computer vision, but in other cases where you see things like O-Lama or other things that Llama CPP that are allowing you to run large language models on a variety of hardware, including just on your, on your local laptop and, you know, quantization type of libraries like Bits and Bytes and Optimum and BigDL and these libraries that allow you to run maybe 7 billion parameter large language models or other generative AI models, but in lower precisions so that you can run them on a variety of

• 21:46Daniel Whitenack?

  hardware, optimize them for a variety of hardware. We also had Neural Magic on the show a little while back now, and who has a set of libraries for optimizing models to run on CPUs. And yeah, so there's a lot of kind of precision and quantization that can happen even on top of the use of these parameter efficient models. One of the interesting things also that I saw this last news cycle, which at least in the circles that I run in with large language models, people were talking about a lot, which is this release from Microsoft or a paper from Microsoft that I think is titled something like

• 22:27Daniel Whitenack?

  "The Era of 1-Bit LLMs," which is interesting because, you know, a lot of people have talked about going from maybe float 32 to float 16 and 8 and 4-bit precision, that sort of thing. And this kind of brings in this idea of 1-bit LLMs with this architecture, BitNet. And so I found it interesting that we got both YOLO-V9, but now comes on the LLM side, this 1-bit architecture. And it seems like a similar thing is happening. I don't know if you remember back when we were talking about R-CNN and some of the larger computer vision models. We've seen the progression to more and more parameter efficiency

• 23:12Daniel Whitenack?

  and flexibility across deployment scenarios. And now we're seeing that maybe in a more rapid way with LLMs and this, you know, 1-bit LLM, but also all the other quantization and that sort of stuff that we've seen on the generative side.

• 23:29Chris Benson?

  Do you have any sense from an application standpoint, like where you might go with these 1-bit LLMs? Like, what are some of the use cases that come to mind for you?

• 23:37Daniel Whitenack?

  Yeah, I think it's interesting. So this 1-bit LLM that was released, they talk about it having similar performance to a model of the same parameter size, but more computational efficiency. Because of course, these parameters are bits. They're actually not just 0 and 1. We can talk about that here in a second. But more computational efficiency. So I think that this is really interesting for cases where you do want to run maybe an LLM on an edge device in a scenario like, think about disaster relief and you have a device out in the field that's giving help to...

• 24:19Daniel Whitenack?

  First responders or something, giving them information or processing information from training documents or something. And you're using an LLM to provide answers. It's likely very spotty internet connection in that case. And so having something that could run on device in a variety of scenarios would be quite relevant, or maybe. So one scenario would be lack of connectivity. I think another scenario would be very latency sensitive scenarios where you want a response very quickly. You don't want to have to rely on network overhead or things going out of a network that you're operating in for security reasons. That sort of thing might be a good use of these.

• 25:03Chris Benson?

  Yep, that sounds interesting. They have a term in here that I'm curious about. Referring to bit net, they talk about it being a 1.58 bit LLM. And Hugging Face in their paper notes that all large language models are 1.58. Do you have any comment about that, what that means?

• 25:22Daniel Whitenack?

  The reality is, if you, I think they talk about this in the paper, if you go down to a truly one bit LLM, each weight of your model is either zero or one, right? Then, yeah, you would expect to lose a lot of information that might be important. And so they make a slight compromise in here. Maybe it's unfair to call it a compromise. They, they make an astute conversion from bytes, in other words, zero to one or bits, to what they call ternaries. So these are basically triplets of, or three bits together.

• 26:04Daniel Whitenack?

  So you have, for example, a weight could be minus one, zero, one, or something like that. So you've got three numbers that represent certain information. And that's where they kind of get this 1.58 bit.

• 26:19Chris Benson?

  Gotcha.

• 26:19Daniel Whitenack?

  So this is also why it's kind of, they release this new type of architecture that processes these ternary bits or ternaries. These combinations of three bits. And that's presented in the Microsoft paper. But yeah, I think this is only the kind of latest, I think we'll see. My prediction would be that we'll see many more. Things like this where people are trying to be parameter and compute efficient with large language models.

• 26:52Chris Benson?

  We've seen models getting more and more efficient and more compact over time. And as we're looking at so many smaller, very capable models being used out on edge devices, do you envision something like this where they're really targeting efficiency in terms of being able to do that in something like small electronics? Or is that a little bit overly ambitious for where this might take us in a reasonably foreseeable future?

• 27:20Daniel Whitenack?

  Yeah, it's actually a good question because one of the things that we saw also, I don't know if it was this week, but recently at least, was Qualcomm's announcement and release of a huge number of, I forget how many, a whole bunch of models on what they're calling the Qualcomm AI Hub for models that run on-device on their Snapdragon processors and other things at the edge on small devices. So these wouldn't be like the small devices of like a microcontroller or something like that. There's still a good bit of power in these processors.

• 28:01Daniel Whitenack?

  But it is super interesting that Qualcomm has made the effort to make these types of models, whether that be object detection or large language models or other things available in optimized forms to run on very small devices. And I think it's a trend that we'll keep seeing.

• 28:34Daniel Whitenack?

  It seems somehow like in computer vision, it took maybe, what, five years? We've been doing, five or six years we've been doing this podcast. And over that time, we've seen computer vision models shrink down and down and become faster and more parameter efficient. It almost seems like that's happening much faster on the large language model side and generative model size. It's like shrunk from five years to one year where a lot of that's coming out for on-device usage.

• 29:07Chris Benson?

  When we and the rest of the change log team are looking at what content to bring on to the show, there are various guests and there are all sorts of topics and advancements going out. It's become quite challenging to narrow it down to just what we can cover in these shows. And largely, that's because of what Daniel was just saying, that tremendous acceleration in the advancement of this technology is very hard to keep up with and report on, especially trying to figure out what folks are most in need of hearing or being pointed to. So on any given week, which of the dozens of things that are happening, do you want to do?

• 29:45Daniel Whitenack?

  And I would say for those out there listening, like in this episode, we've talked a lot about parameter efficient models and whether it be the Qualcomm AI models or the one bit L L M's or YOLO and running these on device and at the edge, it might be natural to think, oh, the news cycle is totally switched to local models, running all the models locally, and that'll solve all the problems. And I think the reality is in the future, it's going to be kind of both and, right? You're not going to serve, let's say that you integrate a model into some social media application or whatever mobile application, or you're

• 30:28Daniel Whitenack?

  serving a web app and it's got some AI integration or something like that. It's very unlikely, I think, that you're going to want to serve up millions and millions of requests using only local models. And in the same way, if you've got an enterprise batch use case, right, and you want to process 1.5 million documents through a large language model, you likely don't want that running on your Mac M2 or something like that. Like, that's not the deployment strategy for that scenario. But yet, you will see a lot of models running at the edge or locally. And I think the reality is that we'll go into kind of a both and

• 31:10Daniel Whitenack?

  sort of scenario where, yes, a lot of things you'll be able to run locally. But the same as like, I mean, you can run a lot of software locally, but it doesn't mean that you're also not running software in the cloud. You know, AI is just a new layer in your kind of software stack. So we're going to run it locally. Yeah. And we're going to run it in the cloud.

• 31:30Chris Benson?

  That's exactly right.

• 31:33Chris Benson?

  That was where I was gonna go anyway. You just hit it. And that was, it's following the maturity trend of software. And just as we have huge software systems that you can only run in the cloud and are massive scale, and you have apps on your phone, and you have also very small microelectronics, which have even smaller software functions on them, integrated in, maybe in the BIOS, all these different areas. And we're seeing models doing the same thing. So, one of the things that we're often asked to address, and we have done repeatedly over the years, is what's the current way to do training and deployment? And I think, to your point, Daniel, there are now, now that we're maturing rapidly

• 32:15Chris Benson?

  in this industry, there are many ways. And there's not one right way to do it anymore. It's kind of figuring out your use case, figuring out what mixture of different model types need to contribute into that, and what the architecture for all those models, and how they communicate through the software, and what hardware is available to them. So, it's become quite complicated. There's no longer the way, you know, to borrow the Mandalorian saying, it's now, it's now many ways. Do you have any thoughts on how people might approach that? How do you think about it when you're doing things in Prediction Guard, and trying to help your customers move forward?

• 32:53Daniel Whitenack?

  Basically, you kind of have to split things up a little bit by stage of your project, and also the use case that you're considering. So, what I mean by stage of your project is, I really encourage people, like especially if they have a generative AI use case, the best thing you can do to get a sense of, like, let's say that I want to summarize news articles related to stocks that I want to trade on, you know, or something like that. The very best thing you can do is not jump right to, okay, I'm gonna fine tune a model for that, or spin up some crazy GPU infrastructure, or something like that.

• 33:35Daniel Whitenack?

  The best thing you can do is just get some off-the-shelf models. And if you want to either run them, the Easiest cloud way to run those would be to run them, you know, if they're small enough in just a Colab notebook or a hosted notebook environment like that, that's more than enough to figure out if, if they're going to work for your use case, right? Or if you want to go the more local deployment route, there's things like I already mentioned, you know, of course, if you want to run Yolo, that's, that's easier now than ever. And there's quantized versions of that that you can run on, on a CPU even, you don't need even a special type of hardware. But then for the generative side of things,

• 34:17Daniel Whitenack?

  there's things like Ollama and LM Studio and Llama CPP and these things that will allow you to prompt models and figure out if they'll work for your use case locally. So that's kind of exploration stage. Then you have to decide, okay, well, if this project is a work project, right? I figured out maybe that I can prototype this and figure out it might work. Then you kind of have to play through the scenarios in your mind. Well, oh, if this is a mobile app and I'm processing customers' private data, maybe it makes sense to try to run a model at the edge in my mobile app on their device, you know, Qualcomm AI model from their AI hub on their

• 35:00Daniel Whitenack?

  mobile device, and that would be really good. But if it's a web app application and there's not as aggressive of a security posture, probably you want to figure out how you're going to run and host that model in a way that makes sense to you, even from a public endpoint that's just a product, like, you know, Together AI or Mistral or, or something like that. Or you're going to figure out how to run it in a secure local environment with either a product that can host that model in a secure environment in your own cloud or in your own network or your own kind of self-deployment of that model using.

• 35:42Daniel Whitenack?

  Things in your cloud infrastructure like SageMaker and AWS or other things like that.

• 35:48Chris Benson?

  Yeah, that's increasingly, it's becoming part of the software and your larger architecture. As you, you know, we've seen in, you know, the recent couple of years, especially the strong rise of ML Ops, you know, which kind of corresponds to DevOps in terms of deployment and all those things. Do you tend to think of it in more of an integrated way? Or do you still, at this point in time, as we're in 2024, think of it as separate approaches, you know, from the software? How do you parse those two sides of that coin?

• 36:19Daniel Whitenack?

  It's interesting. I think I, at least in my own mind, I tend to separate them out, maybe depending on some of what's involved in a project. So if it's the use of a pre-trained model, I think the burden is a lot more in kind of the traditional DevOps monitoring, testing, uptime, automation, deployment, that sort of thing. Because likely you're just interacting with the model via an API, like you would integrate any other API. Now, there's certain things that can help you, like versioning prompts and testing for model drift and/or data drift and that sort of thing. But that's not so dissimilar, I would

• 37:03Daniel Whitenack?

  say, to your traditional software development. Whereas if you are really have a unique scenario and you're fine tuning a model for a unique scenario, you're likely going multiple iterations on curating your data set, on training your model, on evaluating your model, on versioning your model, releasing it in your model servers, updating it with new data that comes in. And I think some of that specific ML Ops type of software will likely appeal to the people that are doing that process, which are usually data scientists and not, not software engineers.

• 37:43Daniel Whitenack?

  And versioning your model, versioning your data, evaluating your model, and the way that those systems are set up, like weights and biases. Or clear ML and these types of things are quite useful in terms of versioning your model out when you're training it like that. So I think MO Ops is alive and well, but I also think that with the rise of this kind of API driven AI development, a lot of that does or can fit into more of the DevOps side of things.

• 38:18Chris Benson?

  Yeah. When you're using an API that somebody else is hosting, maintaining his fine tuned, all that, you're basically using it as a service like any other service that would not be AI. And so you just treat it as an API along the way.

• 38:31Daniel Whitenack?

  Yeah, yeah. And where that's maybe slightly different is you are getting kind of some variability out of that API, both in terms of performance and latency, which are maybe common across software projects, but also in terms of the performance output of the model, especially if you're using like a closed model product, like an OpenAI or Anthropic or something like that. They're making improvements to their underlying model under the hood all the time. And it is really more of a product. It's not just you're hitting the model. There's layers around the model, which are product layers that can influence the behavior of that model. I mean, you just kind of look at what's happened with Gemini over the past three or four weeks.

• 39:16Daniel Whitenack?

  We don't need to, you know, get into all of the details of that. If people want to look it up, they can. But I think a lot of those issues that that product had were actual product issues that were at the product layer surrounding the model, not performance necessarily or biases in the actual model, but in the filters around the model and how things are modified in and out of the model. And so that actual product that you're interacting with can really cause small changes in how things go into the model on the product level can make huge changes in the quality of the outputs of the model.

• 39:56Chris Benson?

  That sounds like some pretty good practical AI advice right there. I think for me at least, that very much helps me to kind of contextualize the different things that we may be doing at work for myself. And we're making choices and decisions and how we're going to tackle different problems. So I appreciate you sharing that guidance there.

• 40:16Daniel Whitenack?

  Yeah, and I guess we're talking about the ML Ops side of things and we've talked about practicalities of deployment schemes and quantization and all of that this episode. And in terms of a learning resource for people, if they want to dive into some of this, there's a lot of great ones out there. One, one is to follow the ML Ops Community Podcast, which is a podcast that Chris and I love and have collaborated with over time. Demetrius, shout out to the great things you're doing.

• 40:49Chris Benson?

  Funniest guy in AI.

• 40:51Daniel Whitenack?

  Yeah, check out everything that they're doing over there. I also ran across this Intel ML Ops Professional Certification. From Intel, there's, if you just search for Intel ML Ops Certification, this is totally free as, as far as I can tell. There's seven modules and eight hands-on labs and talking about software solution architectures for machine learning and AI, API and endpoint design, principles of ML Ops, optimizing the full stack. So it really seems to be a good, a good set of things to look at if you're wanting to think

• 41:32Daniel Whitenack?

  more about the practicalities of these deployments and other things.

• 41:37Chris Benson?

  All right. Sounds good. Well, thanks for sharing your wisdom again today. Really good episode. I'm gonna, I guess I'll see you in the UK for the next few weeks to come.

• 41:48Daniel Whitenack?

  Sounds good. Yeah, thanks, Chris. We'll see you soon.

• 41:50Chris Benson?

  See you later.

• 41:59Speaker 00

  All right. That is Practical AI for this week. Subscribe now if you haven't already. Head to practicalai.fm for all the ways and join our free Slack team where you can hang out with Daniel, Chris, and the entire change log community. Sign up today at practicalai.fm/community. Thanks again to our partners at fly.io, to our beat freaking residents, Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.