Large models on CPUs

Model sizes are crazy these days with billions and billions of parameters. As Mark Kurtz explains in this episode, this makes inference slow and expensive despite the fact that up to 90%+ of the parameters don’t influence the outputs at all. Mark helps us understand all of the practicalities and progress that is being made in model optimization and CPU inference, including the increasing opportunities to run LLMs and other Generative AI models on commodity hardware. Sponsors: Fastly https://fastly.com/?utm_source=changelog – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFly.io https://fly.io/changelog – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog https://fly.io/changelog and check out the speedrun in their docs https://fly.io/docs/speedrun/. Featuring: Mark Kurtz – LinkedIn https://www.linkedin.com/in/markkurtzjr, X https://x.com/markurtz_Daniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Neural Magic https://neuralmagic.com/SparseML https://neuralmagic.com/sparseml/SparseZoo https://sparsezoo.neuralmagic.com/Neural Magic Scales up MLPerf™ Inference v3.0 Performance With Demonstrated Power Efficiency; No GPUs Needed https://neuralmagic.com/blog/neural-magic-scales-up-mlperf-inference-performance-with-demonstrated-power-efficiency-no-gpus-needed/Deploy Optimized Hugging Face Models With DeepSparse and SparseZoo https://neuralmagic.com/blog/deploy-hugging-face-nlp-and-cv-models-for-fast-inference-with-deepsparse-pipelines-and-sparseml/SparseGPT: Remove 100 Billion Parameters for Free https://neuralmagic.com/blog/sparsegpt-remove-100-billion-parameters-for-free/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:07Speaker 00

  Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

• 0:43Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist building a tool called Prediction Guard. And I am not joined today by my co-host Chris, but I am joined by an amazing guest who is an expert in all things model optimization and efficiency and running on CPUs, which is super exciting. I've got Mark Kurtz, who's Director of Machine Learning at Neural Magic. Welcome, Mark.

• 1:14Speaker 02

  Thank you, Daniel. Thanks for having me on.

• 1:16Daniel Whitenack?

  Yeah, yeah, of course. So let's maybe just start out with a kind of like state of model optimization right now. So first off, could you kind of describe like when you're talking about model optimization or that set of tooling, what do you mean by that? And like, how does that fit within maybe the things that a data scientist or an AI person would want to do?

• 1:45Speaker 02

  So whenever we're looking at model optimization, we're usually focused on a few different techniques. But the ultimate goal is to make the overall model smaller and faster, right? Neural networks known to be very large models, especially compared to more traditional machine learning. And it turns out the size of those models is the important part in terms of exploring a large dimensionality of a space. But it actually doesn't- Doesn't use all of those pathways at inference time. So what we specialize in is specifically pruning where we're going to remove connections within that network. Quantization, where we're going to reduce the precision of those connections in the network. So going from, you know, the typical FP32 down to N8.

• 2:27Speaker 02

  And then additionally, distillation, where we're taking larger models and trying to teach a smaller model to mimic the capability and the functionality of that larger model. So it's kind of, you know, overall a high level. And yeah, it's a, it's a very exciting space right now. It's a kind of exponential in terms of the number of research papers that are constantly coming out on a topic. Everybody's very excited about sparsity specifically mainly because you can turn these large models and get rid of up to 95, even 97% of the weights are actually useless in these. Obviously you can use that for a lot of efficiencies around performance and energy. And that's specifically where we've been focusing in at Neural Magic and what I've been focusing in on my work.

• 3:12Daniel Whitenack?

  Awesome. Yeah, that's, I definitely have felt this problem. So I'm sort of asking this question maybe for others out there that, that maybe haven't felt this problem as much. Why is it important to like make models smaller or make them more efficient? Like, how does that fit within what enterprises or like even users running smaller applications? Like, why is that important for people, I guess is the question?

• 3:41Speaker 02

  Generally, there's going to be two cases that we're looking at in terms of deployment. One would be an embedded space where we're running on the edge and trying to work there. So generally you want real time latency and optimizing the accuracy as best as possible. So if you're using an object detection model, you want to make sure that, you know, for example, you're on a security camera, trying to draw object detection and make sure that you know when a person walks into frame and whether or not that's alarming or not versus a dog or something like that. So in general, on that edge application, what you can do is use a larger model, remove a lot of the pieces from that larger model. So you can keep the accuracy. Of a larger model, but take up the space of the smaller model.

• 4:23Speaker 02

  So significant improvement in terms of accuracy on that edge device. While still maintaining, you know, the constraints that were set for you in terms of memory and latency that you need to request back with. And then the second one would be on the server side. And that's generally where we're looking at, you know, more throughput based applications and potentially also latency if they're shipping the data up to some server to be processed either on NLP or a computer vision. But overall there, what we're looking at is especially whenever people get into larger deployments on ML and neural networks. The cost significantly shifts not from training, but to deployment.

• 5:08Speaker 02

  You're for a lot of larger enterprises that are actively deploying. Eighty, ninety percent of their costs is purely in deployment on these machines. So what you can do is take the exact same model that you have. Reduce again, the amount of a compute that you need to run it. So that one will run faster, but two, ultimately what that means is that it's going to run significantly cheaper. Right. And we have a cost savings on the order of, you know, ten X, twenty X, even larger if you're really trying to specialize and optimize. So it can be a significant reduction once you're at that scale. I would say definitely if, you know, you don't have anything deployed yet. Don't worry about optimizing the model, worry about getting a use case that works and something that you can prove out.

• 5:51Speaker 02

  As soon as you go into deployment, model optimization is a great thing to start. Because it's essentially just free performance that's left on the table. That can significantly affect your bottom line.

• 6:02Daniel Whitenack?

  We've mostly been talking about kind of like model size and optimizations. And I do want to get sort of down and get into the nerdy stuff around like how some of this works. But before we do that, I'm also curious about this element of deployment on GPUs versus CPUs. It seems like some of what's indicated, at least in like the tooling that you're building is like the potential. To take a large model which might require a GPU at inference time and potentially run that on cheaper commodity hardware that only has a CPU, maybe doesn't have a GPU.

• 6:43Daniel Whitenack?

  Like, what is the state of that now and like, how far can you push that? Or maybe also like, how could people best think about that in terms of like, when and when that might not be possible, I guess?

• 6:57Speaker 02

  As you said, we specialize almost entirely on CPU performance. And in that, actually, our latest ML inference results on ML perf has have come out. So in that, we show that we're running faster than T4s and A40s and things like that on this commodity CPU. So server-based CPUs, stuff that you have in your laptop, desktop, things like that. And it's very surprising that what's thought of as these little CPUs can outperform the GPU. And we see this generally across every domain that we've tackled. And that's been across image classification, object detection, excuse me, segmentation.

• 7:40Speaker 02

  And now we're working in the NLP and NLG space and actively coming out with that. But overall, we're seeing the same use case where these models are over parameterized. We can take away a lot of that compute. And what that means is that you can actually get the CPU and the GPU about equivalent in terms of compute throughput. Because with the sparsity and the dynamic setup of CPUs, we can run and skip all those zero multiplications, right? So significant reduction in compute, they're about even. But then the CPU has a unique cache hierarchy, which means that we can reuse that cache more often than what you can get on a GPU. L1 and L2 being extremely quick, faster than GPU's main memory, and L3 being about equivalent.

• 8:23Speaker 02

  So overall, what we do on our performance optimization is skip all the compute to get even and then use that cache hierarchy as efficiently as possible on the CPUs. So we can get faster memory access than even you can get on a GPU. And we pay a little bit more by doing a little bit more compute by doing that. But overall, it works out that you can actually beat the GPUs and that setup with just pure software.

• 8:45Daniel Whitenack?

  I anticipate this is maybe a question that you get sometimes, but it's like you hear so much about the necessity of GPUs for running these large models. Do you find generally practitioners are just unaware of this like possibility of running these large models on CPUs? And like, how has that been for you and those that you work with that are actually doing this amazing work and have these awesome tools? Like, how has that been in terms of overcoming that barrier of perception that people have?

• 9:21Speaker 02

  It was a barrier that we had, especially early on, a couple of years back, where it took a lot of convincing to even get in the door to talk to anyone because they just didn't believe what we were saying. Now it's gotten to be quite a bit better, especially with the newer software that's been pushed out of the newer chipsets on CPUs. They're getting a little bit more even in terms of GPUs, so it's, you know, within a stone's throw to try and match GPUs. So people are a little bit more accepting of it. But yeah, whenever we show them the numbers, generally the first reaction is, well, let me try that on my hardware. Because you guys gotta be doing something weird. Let me try to do that. Let me replicate that.

• 10:02Speaker 02

  And as soon as I do, then the next question is, okay, well, how do I do this to my model? And that's usually where the tricky part comes in, is model optimization hasn't always been the easiest thing to do. It can take a lot of research to enable new architectures and things like that. But that's what we've been also specializing on at Neural Magic, is making all the research that we're doing, being able to put that into open source, and also building out a SaaS platform on top of it, so everyone can easily play with hyperparameters and get something that is consumable. But I would say that's probably been the biggest gap in terms of trying to get people off of GPUs on the CPUs, is the model optimization that needs to take place first to be able to run faster than the GPUs.

• 10:46Daniel Whitenack?

  You talked a little bit about sparsity, which I want to dive into over time, and I want to get to those tools and the open source stuff. You mentioned like these large models, and people are probably used to hearing these numbers, you know. 3 billion, 7 billion, 13 billion, like however, and up from there like these models have in terms of numbers of parameters. Could you describe a little bit what you mean when you say 90 to 95% of these connections? Or, you know, maybe less than that, but a high percentage in some models have no impact on the actual forward pass or inference in the model. Could you describe a little bit more by what you mean by that?

• 11:27Speaker 02

  Yeah, definitely. And I'll, I'll take two steps to doing that. One is just covering kind of the 90, 95% class, at least where we've been able to get to on those. And the second is looking specifically at large language models. So for the first one, whenever we're looking at getting rid of 95% of the weights, let's take ResNet-50 as an example. This is our toy benchmark model. This is essentially what hits, uh, we prove out all of our technology on, because it's a common feature in MLPerf and for most performance tasks. So what we can do coming in is looking at those convolutional layers. It has, I forget how many million parameters within it, but it's definitely not the 3 billion, 7 billion or up on top of that.

• 12:07Speaker 02

  But within that, we can actually zero out. So what we're doing is taking all, imagine taking all those parameters, dumping them into a giant array. And we're just going to zero out the ones that are not important. And figuring out the ones that are not important is part of the research. The easiest assumption is just saying that the weights that are the largest are the ones that you want to keep. So the ones that are furthest from zero are the ones that you want to keep. Generally, you can think of this in two ways. One is that as the model is training and being regularized, the weights that don't matter are going to move towards zero. And then the other thing is during that forwards pass, the weights that are higher magnitude have more an effect on the output. Right? And everything else is going to be noise in between.

• 12:49Speaker 02

  So we're able to essentially get rid of just, or whenever I say get rid of, I mean, setting those parameters to zero within 95% of them. So you're left with 5% of your weights that are non-zero. And that's actually all that you need to preserve the accuracy on ImageNet for ResNet-50, for example. And some quick kind of intuition. In terms of how I've been able to think about this and why it works and things like that. You can see as we increase the size of our dimensionality in our optimization space. What we're doing is, and there's a few research papers out on it, they're able to connect more of the local men's. Right? So the optimization process will slowly converge further and further down because more of the local men's are connected.

• 13:34Speaker 02

  Generally, though, there's only a few of those pathways that you actually need to connect those local men's. So all that we're doing is we're following down that most optimized pathway and removing everything else around us in terms of that dimensionality. So it's kind of one of those things that as you're training, it's slowly selecting the weights that matter that gets you down to that local men. And there's very few. So the important part was that large dimensionality of the optimization space, but not every direction mattered. Right? So then we can get rid of it. And then diving in on the L L M side and large language models. We actually have a recent paper that came out from one of our principal research scientists, Dan Alastar, called sparse G P T.

• 14:15Speaker 02

  And that's where we're looking at taking O P T and Bloom models all the way up to one hundred and seventy five billion parameters. And be able to optimize those and remove as many weights as possible, all in this case in one shot. So just using the model without any retraining, we're able to get rid of around sixty percent of the weights without doing anything. And there's a new paper out of Cerebrus, actually, that was looking at the L L M story. And they're able now to get to eighty percent sparsity on these L L Ms with retraining. So that's kind of the research direction that we're headed down now is proving out how optimized we can make these models. Because there's also a lot of interesting stuff that happens with the large language models.

• 14:57Speaker 02

  Specifically because it's generating one token at a time, very latency bound. And that means that it's a lot of memory access to load those weights. So if you can quantize those and then get rid of half of them, you're already at, you know, anywhere from a four to six X speed up. Just on your inference times. And that's generally where we're, uh, where we're focused and looking at currently to try and get those L L Ms. Run faster. The other thing to call out for those two is, you know, seven billion parameters and 175 billion parameters. Those don't fit in a single GPU. So now you have, you know, clusters of GPUs to serve one model. And a lot of that compute is just completely wasted because all that it's going to is trying to maximize the memory on these GPUs.

• 15:41Speaker 02

  For CPUs, you can throw a few terabytes on there and it works out fine. So that's the other thing to call out with the other LLMs in terms of GPU versus CPU.

• 16:01Daniel Whitenack?

  This is really interesting, Mark. I want to follow up on what you were just talking about, which I think is a really, it's sort of a subtle point, but it's really interesting in that I think if I understood you right in what you're saying, like, let's say that I have one of these large models, 175 billion parameters or whatever, and even for inference, I have the necessity to have multiple GPUs just to load that model into the memory of the of the cards. Whereas on a CPU, you can have terabytes of memory. What I'm assuming is like you could load that in as long as you're able to execute it quickly, which I guess is the other piece. So,

• 16:41Daniel Whitenack?

  am I right? You sort of have to have both like the ability to load it into memory. And you have a bit more space in that on the CPU side, but then you also have to be able to execute it very quickly, which I guess is like why you would think about both space and sparsity. Is that an accurate way to put it?

• 17:03Speaker 02

  As you said, you know, you have a total spacing you take up and then a minimum latency that you want to respond to the user out, right? And that's going to set the constraints for your hardware. And for CPUs currently, at least for the smaller models, you can get to a usable speed on those. If you see in like llama.cpp, they're doing N4 and things like that on smaller models and they're usable, but they're less accurate. So what we're trying to do and what we're actually working on right now is making sure that we can get that GPU class speed while maintaining the large memory advantage of CPUs. So you can deploy this 175 billion parameter model on something local and you don't have to worry about data privacy, anything like that. It's just there, working and available and highly accurate for you.

• 17:46Daniel Whitenack?

  I definitely heard people that I've talked to who have tried various optimization techniques and have maybe been dissatisfied with the performance hit that they're getting, not in terms of compute, but in terms of like actual model performance or accuracy or whatever. Does that performance hit often come about because of the quantization? That's maybe part of the optimization techniques? Or are there multiple sources of that performance hit? How should people think about that?

• 18:21Speaker 02

  I think the biggest thing there is honestly just the amount of choices that people have to apply and not knowing when to apply them. Because generally for quantization, for example, you can apply quantization to pretty much anything at N to 8 for both activations and weights and have it recover. But there are definitely cases where, for example, we've been quantizing efficient nets quite a bit on our image classification side. There is one or two layers in some of these that are extremely sensitive, for whatever reason, to quantization that you can't quantify those. So removing those, then you get 100% recovery, right? So it's a lot of these kind of little things that researchers know intuitively

• 19:03Speaker 02

  in terms of having used this constantly, what will work, what won't? But that's not really coded into software anywhere, right? To make it easy for people to use. So generally they'll go through, try and quantize, and there's no feedback loop, there's no methodology. It's just, hey, I was able to apply it in one shot, but it lost 5% accuracy. Talking about quantization. But if you do a quantization or a training scheme, generally you'll recover all of that back and it generally works completely for that. Same thing on pruning. Pruning, you'll definitely see more of a drop and it's much more of a requirement to do training aware on the pruning side, at least to get to really high sparcities. But you definitely will see this kind of the choices that are made in the hyperparameters

• 19:47Speaker 02

  that are chosen. Those can significantly affect the recovery and the quality. So generally, I'd say, you know, if they were seeing drops in performance, it's primarily because of those choices and those issues and just the wide breadth that's available right now and not knowing how to narrow it down. And that's, you know, what we're actually working on. Does that make sense?

• 20:05Daniel Whitenack?

  Yeah, yeah, that's good for people to... Myself, I want to develop a little bit more intuition around these things. Because like you say, sometimes you're just like, oh, here's the command that I run on the command line. And like, I get this file out that is smaller, right? But I don't have a great intuition about like, similar to hyperparameter tuning, right? Like, that takes time to figure out, like, okay, how should I think about changing my learning rate if this happens? Or if that happens, that sort of thing. You mentioned one thing, which I think would also be good to kind of clarify and help people understand is, like, training-aware optimization versus just non-train, I guess, optimization?

• 20:50Daniel Whitenack?

  Or I don't know what the counter to that is. Could you talk about those, like, how they're differentiated? Some people might guess what that means, but like, how are they differentiated and how does that work out in practice in terms of how you would optimize a model?

• 21:04Speaker 02

  Technically, we have three categories generally available. One, and the two you're going through is one, training-aware. And then we have post-training or one-shot, which are kind of interchangeable. And then we additionally have on sparse transfer, which is something that we've been pushing a lot because the research has worked out quite a bit for it. So I'll cover all three of those in a little bit more depth. So for training-aware, what we're doing is taking the exact same model that you're wanting to deploy in the exact same dataset it was trained on, and continuing the training process further. So while we're continuing that training process, we could be continuing it. This is where the hyperparameters come in. But generally, it'll be about half the time that it originally took to train it. We'll train it for that much longer.

• 21:47Speaker 02

  And as we're doing that training, we're iteratively pruning away, or we're applying Quantization or both. And the reason we're continuing that training is because as we're iteratively applying these optimizations, we're slowly moving the model away from its local min, right? And it has to adapt and adjust back. So by slowly doing that and training over some time, we allow small jumps that the optimizer can recover from and adjust the remaining weights for, rather than doing it all at once. And the all at once piece is where we get into post-training in one shot, where we're not going to try and retrain the model at all. We're going to take a small calibration data set. And then we're going to use some heuristics or some heuristics or algorithms

• 22:30Speaker 02

  to figure out using that calibration data set, how to optimize that model and remove weights or quantize. So the most common case would be static quantization. We're using a calibration data set to figure out the activation ranges for each layer, right? And once you have the activation ranges for each layer, you can set up a simple quantization scheme to say, given that it's going from this layer is going from negative six to six, now I need to fit that range into an intake scale of zero to two fifty-five, right? And be able to map that in. So that would be a simple post-training or one-shot application.

• 23:08Speaker 02

  And then the final one is sparse transfer, which works exactly the same as transfer learning or fine tuning. It's just we're starting with a sparse model to run through. And that's a lot of what we've pushed up in neural magic into our sparse zoo. Are these open source sparse models that we have, you know, sparse birds and ResNet fifties and YOLO v5s, things like that. And you can just take those, plug in your data set and transfer over to it. So the sparsity math stays in place. And it just adjusts the remaining weights to fit your data set. We have a few papers out on that as well. That shows that sparse transfer works just as well as regular transfer.

• 23:49Daniel Whitenack?

  Yeah, that's really interesting. I guess this would somewhat depend to, like, if I'm just thinking of like the average practitioner out there, right? Probably in a lot of the space that I work in, in like the larger language model area, then like I'm not going to be able to retrain one of these large models on the original dataset, right? Or even half of that, or for half of the epochs or whatever. But these other things are certainly things that I do all the time, right? Like fine tuning, transfer learning. So it's cool to understand that there's options out there.

• 24:29Daniel Whitenack?

  Am I correct in assuming that, like, you mentioned the zoo, the sparse zoo that people can find on your website and we'll link in here too, that like researchers, your team, practitioners, whoever are the people out there are also putting in work to actually release some of these sparse models publicly to the community so that I can take those and then maybe do fine tuning on that? Or maybe it's just good enough from the what's released in the community? Could you tell us a little bit about that community and like what's being released?

• 25:04Speaker 02

  So on the open source side, pretty much everything that we have on the sparse zoo currently has either been from our lab, we have a few that are from Intel's lab up as well, and some Hugging Face examples and things like that. Primarily because a lot of the sparsification research is all built around a few models like ResNet-50, BERT, and things like that, and they don't expand it out past those models to prove out their algorithms. So we have the best of those. And then, yeah, that's exactly what our team is working on. And well, we get some community contributions in every once in a while for sparse models that people have generated or transferred, right? So our goal is to be able to push these up so that, as you said, anyone can come in

• 25:46Speaker 02

  from the community and be able to pull those down and get value out of those models. So you can think of them as sparse foundational models rather than the dense foundational models.

• 26:10Daniel Whitenack?

  Mark, in addition to the sparse zoo, which is really cool, I know that Neural Magic is producing some pretty interesting and useful tooling otherwise as well in terms of actually doing some of this optimization themselves. And I see certain things, Deep Spars, SparsML. Could you describe a little bit about if I'm a practitioner, I have a model and I want to do optimization, what does that look like for me right now with the tooling that's available?

• 26:43Speaker 02

  So we have SparsML, which is our open source model optimization framework. It's built primarily on top of PyTorch. And then we have integrations with Torchvision, Hugging Face, Yolo, Ultralyst, Yolo V5, pretty much all the common repos that most people are using, we've already integrated with. So you can use our integrations and just plug in your model and go along with that. And then the other part is that we have recipes, and I'll go through recipes in a second. But we have those kind of pre-coded integrations, or you can create your own integration. Usually, it only takes a few lines of code. We've done all the hard work in terms of making sure that whenever we want to optimize a model, that you just have to wrap the optimizer in PyTorch, essentially wrap the model and the optimizer, which is what our code handles, and then it's

• 27:27Speaker 02

  going to go through and handle the optimization pass that. So there's no coding really from your side, or from the practitioner's side on implementation. And then the other part is then coming up with an optimization recipe that people want to use. And what that's going to lay out is saying that I want to prune from this epoch to this epoch, for example, and then apply quantization and target these layers and at this sparsity level, things like that. We have automated ways to generate those as well as examples on the sparse zoo and in other places. That's generally what it would look like is get up and running. We definitely recommend checking out the sparse zoo first, seeing if there's anything that you can do to just transfer the model onto your dataset, because those are the quickest and fastest ways. And then otherwise, if you do

• 28:11Speaker 02

  have a specific model architecture that you're looking at, then you can start going down this. Integration pathway and generating your own recipes. And that's the current state that we're at. The other thing that I wanted to call out is that we're working on a SaaS platform right now to make all of this more intuitive. So you have a UI to be able to predict where that model is going to end up before you start optimizing it. And then additionally actively benchmark it across your different deployment scenarios. So this is called Sparsify. We've had an old kind of alpha state for a while that was downloadable. And we're actually going through alpha testing right now. So anyone that is interested in trying that out, definitely reach out. We're going through alpha currently, looking to go to beta in probably next month, two months. And then GA following up after that.

• 28:55Speaker 02

  So definitely check that out as well. And yeah.

• 28:58Daniel Whitenack?

  Yeah, that's great. Yeah, I love how you're thinking towards usability as well around these things. Cause I do see this as something that kind of blocks people on optimization a lot because they get, they get stuck, like you say, it's like, well, what, what recipe do I use here? It seems like there's a million options, like where do I start? So that's really awesome to hear. Just so people can understand, like, there's options available in the sparse zoo. I'm kind of scrolling through there now. There's a lot, even ones that I know that I use, like Distilbert, which is fine tuned on squad. Like I use that all the time for question, answer.

• 29:39Daniel Whitenack?

  And apparently I should use the sparse one because yeah, that would help out a lot, both in terms of, of compute and speed. Let's say that a model is not there. And in particular, like people of course are interested in like all of these things being rapidly released all the time. So one of the things I know that I've seen in optimization platforms over time is like, it's hard to maybe support new architectures as they come out. So how are you all approaching that? And like, what is the state of like being able to be flexible with these optimization schemes for a variety of architectures?

• 30:20Speaker 02

  Looking at the base framework that we have pushed out in Spark ML, for example, everything's implemented such that it's supposed to be able to run with any model architecture. So there's no real big assumptions on there, other than you have convolutions, and you have linear layers inside of your model somewhere that we can target for optimizations. So everything's set up very generically, which means that whenever there is a new architecture that comes out, or new weights, things like that, it's going to take some time for us to tackle that. And that's if it gets on to the list of top used models, things like that. But that is the nice thing about having an open source community, is that people are welcome to come in. The tooling and the framework there should

• 31:02Speaker 02

  work out of the box with everything. And they're more than welcome to be able to commit or push up wherever they're working on. And we have an active Slack community and GitHub community for people to come in. Our engineers are actively on that. They can come in, look at it, and easily get support on any issues they're running into.

• 31:22Daniel Whitenack?

  Awesome. Yeah. And we'll make sure for the listeners who are wanting to get plugged into this, we'll make sure and include the links to the Slack group and GitHub in our show notes. So make sure you visit there and get plugged in and start optimizing your models. As we kind of get a little bit closer to the end here, I'm wondering about a couple of things. One is, I know that, like you're saying, you're actively involved in research in this space. What trends are you seeing in research around optimization? And in particular, like, what are the directions that your team is sort of excited to go into in the near future in terms of the research side of this?

• 32:05Speaker 02

  So I would say there's two big trends right now. One is around the focus on post-training. Our principal research scientist, Dan Alstar, he came, him along with his lab, came up with this algorithm called OBCOBQ, which actually we have a webinar which we'll have already aired by the time that this comes out. It's airing tomorrow. But that algorithm specifically is where a lot of Efforts going around, which is requiring as little data as possible and no retraining and trying to increase sparsity as much as possible. So that's one of the key things that people are trending down is trying to do that.

• 32:44Speaker 02

  The second one I would say is a large push around quantization in terms of getting to lower bits. And that's something that's been around for a while, but it's something that's becoming more and more prevalent as we're looking at the larger models. Mainly because their execution time is mainly dominated by trying to pull in these large matrices of weights. So there's been a big push there now around trying to get down to, you know, pass N A down to N four, N three, N two quantization. On these active representations, I'd say that's the other trend. The final kind of bonus trend that I throw in there, because I said too, the third one would be more research around sparse training.

• 33:25Speaker 02

  So specifically trying to figure out how to start with an unoptimized and untrained model. And be able to make it sparse from the start and then keep sparsity throughout training. Because generally what we do to guarantee accuracy, we'll start from a dense converge model. And then iteratively print on top of that, which adds training time, right? So now there's a lot of research going into trying to figure out how quickly the model can be pruned. And then be able to carry that over as it's training. So so that's where the other big, big piece is. And all three of these are definitely active areas that we've been investing heavily in, especially looking at the generative AI space now going through that.

• 34:09Daniel Whitenack?

  Yeah, yeah, I know it must be a crazy time for you all, just like it is a crazy time for everyone. But yeah, I think this is a really important piece of it. I know one of the trends we've even talked here on the show about, it seems like a lot of people are talking about like serverless deployments of of machine learning, deep learning models. And I know a lot of the issues related to that and things that people are dealing with is cold start time and loading models into memory. I don't know if that's impacted you all at all, but it seems definitely relevant. Like if you're going to run your model serverless, you probably want it as small as possible, I would imagine.

• 34:49Speaker 02

  Yeah, absolutely, absolutely.

• 34:52Daniel Whitenack?

  So you mentioned where people can find out about Neural Magic on Slack, on GitHub. I would really encourage people to do this. As we close out here, what are you kind of personally excited about during this? I mean, it's like I say, it's a crazy time for everyone right now with generative AI and the way things are trending. What's exciting to you right now about the AI community and certain things you're seeing? What do you see as kind of positive trends, I guess?

• 35:22Speaker 02

  The part that I'm most excited about is that generative AI space specifically in being able to augment humans. Obviously, there are a lot of privacy concerns and data concerns and bias issues and things like that in this, which I don't want to see, you know, LLMs deployed everywhere become a default response for like Google Search or something like that. But it is really exciting to see, even in my day to day, starting to use these actively to augment what I'm doing around content generation and framing and things like that. So it's one piece that I'm really excited for. And with the work that we're doing on Neural Magic, we're especially looking at these because, one, we want to see that continue to grow to open source.

• 36:05Speaker 02

  And I think that's been the other push that's been really big and really exciting to see is that whenever GPT-4 came out, it was completely privatized. They put out, you know, a little white paper on it that had no details about it at all. A lot of data concerns and things like that within that. But the open source community has already released, I mean, I can name probably 10 models so far that have been released since then that are chat GPT-like or GPT-4-like. So it's really exciting to see that. I think the next stage from those open source models is going to be making them runnable anywhere, right? So you don't need this big GPU cluster farm to get something that is usable. And that's where we're really looking at going. We're actively working on the LLM deployment issue right now

• 36:47Speaker 02

  and hope to have something out in the, you know, next few weeks, the next few months that people can start actively using, download it, they can run it anywhere they want on any CPU, and it'll be just as fast as GPUs.

• 36:59Daniel Whitenack?

  Cool. Yeah, well, keep us posted. I know I'm personally interested in that one. So yeah, thank you so much for joining us, Mark. This is a really fun conversation. I love getting into the weeds of these practicalities because this is a topic where people get stuck a lot is on the deployment side and the optimization side. So yeah, thank you for all that you and your team are doing at Neural Magic in this area. And yeah, keep up the good work. We're excited to see it. So thanks for joining.

• 37:32Speaker 02

  Thanks, Daniel. It's great talking with you.

• 37:42Speaker 00

  Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already. And if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changetalk podcasts. Check out what they're up to at fastly.com and fly.io. And to our beat freak in residence, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.