Stellar inference speed via AutoNAS

Yonatan Geifman of Deci makes Daniel and Chris buckle up, and takes them on a tour of the ideas behind his amazing new inference platform. It enables AI developers to build, optimize, and deploy blazing-fast deep learning models on any hardware. Don’t blink or you’ll miss it! Sponsors: RudderStack https://rudderstack.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog – Smart customer data pipeline made for developers. RudderStack is the smart customer data pipeline. Connect your whole customer data stack. Warehouse-first, open source Segment alternative. SignalWire https://signalwire.com – Build what’s next in communications with video, voice, and messaging APIs powered by elastic cloud infrastructure. Try it today at signalwire.com https://signalwire.com and use code SHIPIT for $25 in developer credit. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFeaturing: Yonatan Geifman – Website https://www.yonatangeifman.com, GitHub https://github.com/geifmany, X https://x.com/geifman_yChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Deci https://deci.aiAn Introduction to the Inference Stack and Inference Acceleration Techniques https://deci.ai/resources/blog/inference-stack-and-inference-acceleration-techniquesDeci and Intel Collaborate to Optimize Deep Learning Inference on Intel’s CPUs https://deci.ai/resources/news/deci-intel-optimize-deep-learning-inference-cpusDeciNets: A New Efficient Frontier for Computer Vision Models https://deci.ai/decinetsWhite paper https://info.deci.ai/hubfs/resources/deci-white-paper-autonac.pdfUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  What we are doing and specializing is kind of the model design approach, where you need to select the right model that is optimized both for the data to reach the accuracy and also for the hardware to reach the latency. And by optimizing the model specifically for the inference hardware that you're looking to use, you can get significant boost in the performance compared to having kind of an open source model or off the shelf model that you just took from an academic paper or from GitHub or something like that.

• 0:32Speaker 01

  Big thanks to our partners Leno, Fastly and Launch Darkly. We love Leno, they keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com. And get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com. This episode is brought to you by our friends at RudderStack. And we're calling all data engineers to check out RudderStack Cloud and start building smart customer data pipelines. RudderStack is warehouse-first, no more silos. RudderStack builds your customer data lake on your data warehouse, not theirs, enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API-first. RudderStack can be easily integrated into your existing development processes.

• 1:16Speaker 01

  And because they're open source, you can see all their code, so you don't have to worry about vendor lock-in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. RudderStack is free up to 500,000 events and pricing scales transparently from there. Learn more and get started at rudderstack.com. Again, rudderstack.com. That's R U D D E R S T A C K dot com.

• 1:50Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and Slack with us around various topics of the show at change.com/community and follow us on Twitter. We are at Practical AI FM.

• 2:15Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I'm joined as always by my co-host, Chris Brinson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

• 2:31Chris Benson?

  I am doing very well. It's dog days of summer as we record this and just try not to melt.

• 2:37Daniel Whitenack?

  Yeah, yeah. It's hot like you're sitting right on top of a bunch of GPUs or something.

• 2:42Chris Benson?

  That's exactly right.

• 2:44Daniel Whitenack?

  Yeah. Well, speaking of hardware, today's episode is kind of connected to that. Really excited to have Jonathan Geifman with us, who is CEO and co-founder of the deep learning company, Desi. Welcome, Jonathan.

• 3:00Speaker 00

  Hi, thank you for hosting me. How are you?

• 3:02Daniel Whitenack?

  Yeah, doing wonderful. So I know a lot of what your company does is related to productionizing models, optimizing models. There's a bunch of things that you do related to optimizing inference. And I'd love to dive into all of that as time goes on, but maybe let's just start out by talking about why should we care so much about inference? So a lot of effort and I think screen time, if we want to put it that way, in terms of Twitter and other places, is put on the training side of things and scaling up training. But why do deep learning and AI practitioners need to be concerned about inference?

• 3:43Speaker 00

  So actually, the first thing to be concerned with is the training. But after you finish building a model, then you need to deploy it somewhere. It could be in the cloud or in the edge. And when you're going to deploy it and do serving or inference at scale, you must be concerned or think about how this model is going to perform in terms of latency or throughput? What is the operational cost of this model? Or are you actually able to deploy it on the edge device that you want to use in order to serve the inference of that model? And it's kind of the second phase of the development cycle after you reach the accuracy. But you must think about it at the beginning when you choose the model in order to know

• 4:24Speaker 00

  that you are going to finish the project with a model that meets the SLA of going to production in your environment with your characteristics of performance that you are looking for in production. If you think about it, you'll see that a model is trained once or once in a while, but the inference workload is huge because it's kind of linearly scales with the amount of data in production. So if you think about an organization that, for example, develops a self-driving car, you'll think about the number of data scientists that doing the training work, building the models, each one of them needs some amount of GPUs in order to do his work. But when you deploy these algorithms, these are deployed among huge amounts of cars.

• 5:07Speaker 00

  So in order to optimize the performance, you're more concerned about what type of hardware you'll need to fit into the car compared to how many GPUs you'll need to have in the cloud for your data scientists to build those models. So this is kind of the proportions. The training is kind of linearly scaled by the number of data scientists, while the inference is scaled by the amount of data, this scale of the deployment that you're going to use. And you must think about the performance when you want to go to production at scale.

• 5:35Daniel Whitenack?

  Yeah, and as a practitioner, you mentioned thinking about inference upfront, maybe even before training, thinking about what models you might even be able to use in production. As you're working with different companies and different teams of data scientists, do you see people running into this challenge a lot where they're really happy with their model that they spent, you know, a lot of time and effort training and they just are completely blocked in taking the model to production? And how often do you see that? And what are the main reasons like why they're not able to do inference with that model? Is it a latency thing? Is it a memory constraint thing? What are some of the challenges you see?

• 6:15Speaker 00

  So it is really divided into several problems, but there is a simple principle that you can always take a larger model and get better accuracy. And it's easy to get better accuracy with larger models, but this is only happens if you don't need to think about production. When you're thinking about production, you understand that those models are not production-ready, let's say. For example, if you want to deploy them in the edge, you have several constraints like the memory and the latency that you want to run probably in real time or giving some reasonable response time for the model. But in the cloud, you're more concerned about the cloud cost of serving this at scale, the amount of hardware that you will need to orchestrate in order to serve your users or your demand for inference. And this is kind of the things that you need to take into account

• 7:00Speaker 00

  while you develop your model. Because as I said at the beginning, going larger is easier to get accuracy, but it's not easier to get deployed. So we see companies all the time getting some huge models, trying to feed them into small devices for edge inference, or companies that just understand that they took the largest model that they can have and get the right accuracy. But then they're starting to scale and see the costs of running this inference workloads. And then they're trying to think about, okay, how can I reduce those GPUs costs or the CPU cost in running production in the cloud? And this happens all the time. So our approach is think about these problems at the beginning. While you start development, choosing the right architecture at the beginning

• 7:43Speaker 00

  instead of kind of having a trial and error iterations when you get to the accuracy. But then you understand that you need to change the model in order to get it productized.

• 7:51Daniel Whitenack?

  Yeah, and I actually have a follow up on that, mainly because I only have a certain sphere of experience and I only know what my organization does and what some other organizations do. Just from my own curiosity, how often are you seeing people use GPUs on the inference side versus just CPUs? Do you see that being done more and more? Or, you know, what's the majority of inference use cases that you're seeing? Is it inference on GPUs or inference on CPUs? Maybe specifically thinking about cloud deployments. Edge deployments might be somewhat differently, but thinking about cloud deployments at organizations that you work with, is it often inference on the GPU or is it often inference on a CPU?

• 8:35Speaker 00

  I think that it really depends on the task that you're trying to do the inference and what are you trying to achieve? I think that we were talking about video analytics workloads. You must use GPU. You have a lot of data. You want to process the data. They're images in high resolution. You need a GPU performance. If you're talking about having some queries of NLP model. You usually find those deployed on CPUs, but it doesn't matter because both of them are getting expensive when you get to scale. So if you look about, for example, prices on the cloud for having a four-core CPU and a T4 GPU, it's approximately the same. So it's not like the problem is only the prices of GPU. Also, the computer, the CPU is getting expensive when you have to run in

• 9:20Speaker 00

  large workloads with large clusters with multiple nodes and cores.

• 9:24Chris Benson?

  Could you talk a little bit, as we've kind of touched on edge specifically, could you touch on what some of the challenges you see about deployment for, to the edge and the inference that's associated with that are even beyond just the GPU-CPU consideration? That's certainly something that I'm involved a lot in and more and more people are now having to deploy to edge in all sorts of different use cases. And I think we're pretty accustomed at this point to thinking about cloud-based deployment because that's matured a lot faster. But as more and more organizations are involved in edge deployments, I think they're trying to explore their way through that. Do you have any guidance for that?

• 10:01Speaker 00

  So first of all, there is kind of a jungle of edge hardware types. There are a lot of types of hardware that you can use at the edge, and it really depends on the application. It could be a mobile phone that if you deploy a mobile app with deep learning in it, you will find out that your users are spread across something like 10 or more types of hardware from iOS with the A and E of Apple and Samsung devices with the Qualcomm Snapdragon and hardware types like that. So first of all, you need to understand what hardware your users or you are going to deploy on. That's the first task. Then you need to understand kind of what is the software stack that is best to use for that type of hardware. If we're talking about Apple devices and iOS, we have the Core ML,

• 10:44Speaker 00

  but most of the other types of hardware will probably be better running with TF Lite or those frameworks that are optimized for edge inference. And above all that, you need to understand kind of what is the limitations of the hardware that you are going to use. For example, memory constraints, memory bottleneck of loading the weights, loading the data and stuff like this. And the performance, what are you going to get if you will run, for example, an object detection model? On a jet zone, how many frames per second are you going to get? How many video streams can you put on that jet zone? And this is kind of something that if you get to the accuracy, you'll finish building a model and only then we'll measure it on the device that you're looking to deploy on.

• 11:26Speaker 00

  You'll get back to square one to redesign the model in order to get the SLA, the latency that you're looking for at the edge. And those are things that you need kind of to have an holistic approach and to see how you solve all of them together. It's kind of a multi-constrained optimization where the accuracy, the latency and the model size are things that you need to consider together when building an edge AI application.

• 11:50Daniel Whitenack?

  And that maybe leads me into another question about that. I do want to get into the specific like methods and technology that you've been involved in developing, but it sounds like you're also sort of suggesting a different kind of workflow that people can have in their mind where as you contrasted to like, oh, I'm going to build my model and the environments in which I'm training my model and testing it are totally different from those where I'm going to deploy it. How can people like adjust their workflow to maybe, is it a matter of always making sure that you have a testing cycle where you're testing on the hardware that you are targeting in the end? Or how can that be integrated into data scientist workflows in a better way?

• 12:35Speaker 00

  So we are kind of pushing to hardware in a loop development approach when you are taking the inference hardware in the development stage very early in the model selection stage where you're considering some models usually based on some open source repositories and academic papers and you have to measure them at the beginning. This is kind of the first step in understanding if you're going in the right way. After that, you need to understand or think how can you bring that model that meets the SLA to the accuracy that you're looking for. So this is kind of an opposite approach. Then first reaching the accuracy and then reaching the latency. And this is for edge applications with constraints about the latency and the model size, that those are

• 13:19Speaker 00

  need to be considered at the beginning and not at the end.

• 13:22Daniel Whitenack?

  Makes sense. I am wondering maybe if you can just give like a broad strokes sketch of what people, like, let's say that they get to a point where they have the model that they want, and it's not quite optimized for the hardware target that they have in mind. Maybe there's too much latency, or they need to shrink the model or something. What just sort of generally are people trying out there in terms of methods for optimizing their models for inference on certain hardware?

• 13:56Speaker 00

  There's something we call the inference stack, where at the bottom, we consider the hardware itself. On top of that, we have a layer that we call it the drivers and the graph compilers. So for example, the open Vino for Intel CPU or the TensorRT for GPUs. On top of that, we have like open source methods like pruning and quantization. Pruning is a method to reduce and eliminate unneeded neurons and connections in the network. And quantization is representing the weights and the activations of the network in lower bit representation, like eight bit or something like that. So those all are kind of open source and public methods and techniques in order to build more efficient inference.

• 14:37Speaker 00

  On top of that, what we are doing and specializing is kind of the model design approach, where you need to select the right model that is optimized both for the data to reach the accuracy and also for the hardware to reach the latency. And what we understand today is that different hardware type prefer different types of model. I will give a simple example. If you think about GPU which has parallelism capabilities, it will prefer large, wider layers with fewer layers in the network because it can parallelize the layer itself, but it cannot parallelize between layers. In contrast, in CPU, there's low parallelism capabilities and you will probably prefer having narrow layers with less neurons,

• 15:22Speaker 00

  but you can have more layers. And this is kind of a general idea, how the inference speed could be affected by the model structure. And by optimizing the model specifically for the inference hardware that you're looking to use, you can get significant boost in the performance compared to having kind of an open source model or off the shelf model that you just took from an academic paper or from GitHub or something like that.

• 16:04Speaker 04

  Signal Wire is real time video tech to help you create interactive video experiences previously not possible. It gives you access to broadcast quality, ultra-low latency video that's proven and trusted by Amazon, Ring Doorbell, Zoom and others. See why the future of video communication is being built on Signal Wire. They have easy to deploy APIs, SDKs for the most popular programming languages and expert support from the OGs of software-defined telecom tech. Try it today at signalwire.com and use code AI for $25 in developer credit. Just visit signalwire.com. That's signalwire.com and use code AI to receive that 25 bucks.

• 16:48Speaker 04

  Once again, that's signalwire.com, code AI.

• 17:03Daniel Whitenack?

  So Janathan, I want to maybe get into some of those design aspects that you were talking of. And I find it fascinating that you're sort of diving into this area of thinking about what sorts of layers run best on what sorts of hardware. Has that research around that topic, has that been going on for some time? And you were able to sort of build on that? I'm curious because hardware is progressing so rapidly, even in the terms of accelerators like GPUs and VPU's and TPUs and all sorts of use. All of that's advancing very rapidly. So it sounds like you have some intuition around what runs best on what hardware.

• 17:44Daniel Whitenack?

  But is that rules that you're encoding into your methods in order to optimize models for certain hardware? Or what's going on there? I know you've got this sort of architecture search concept.

• 17:56Speaker 00

  Yeah, there are many types of hardware and we cannot have rules for every type of hardware. Designing neural architecture is very complicated because it's kind of a composition of multiple layers with multiple types and sizes. So searching among neural architecture is a large search space with a very complicated search. So we can't have kind of rule of thumb for this hardware use that operation and for this hardware use this operation. What we are doing, we are employing neural architecture search that is hardware aware in the sense that it's connected to the hardware and optimizes the structure of the network to both have better accuracy on the data and better latency on the data on the hardware.

• 18:37Speaker 00

  And this is something that we must connect to the specific hardware in order to get that because you can't really model the hardware and understand how a specific neural network will behave on that hardware. So you can think about it as a hardware in the loop, neural architecture search, where we have an automatic search algorithm that search among hundreds of thousands of neural architecture and find the best one that kind of sits on the sweet spot between the accuracy and the latency on the given hardware while searching on that space is very complicated because the latency can be measured or estimated by using the hardware. But in order to understand the accuracy of a given architecture, you must connect the data and train it.

• 19:21Speaker 00

  Candidate architectures could be very expensive. So the trick here, or the secret sauce here, is how can we do it efficiently? How can we scan 100,000 neural architectures and find the best one for your operational point in terms of latency, accuracy, trade-off? And this is kind of what we do with our proprietary neural architecture search algorithm that is called OTNAP, Automatic Neural Architecture Construction.

• 19:47Chris Benson?

  Could you take us through maybe kind of an example, just to make it very tangible for the practitioners that are listening to this in terms of how the practitioner is going to use neural architecture search to do that, maybe ideally with an edge use case? Just to give a sense of how you're approaching it and what that feels like if you're targeting inference on the edge ultimately?

• 20:06Speaker 00

  Yeah, so we have a collaboration with Intel that we recently published in MLPerf last September or something like that about a performance booster that we have made to image classification model, ResNet-50 for a MacBook laptop, for example. We've done that for several types of hardware, but let's take, for example, the edge use case with the MacBook Pro. So we're having the baseline model. We have three inputs to the algorithm of the neural architecture search. We're having the baseline model, which is, in this example, was ResNet-50. We're having the hardware itself and the data. The data was imagenet in the MLPerf benchmark.

• 20:44Speaker 00

  And the OTNAP algorithm kind of searching what types of changes or how can it change the original architecture of ResNet-50 in order to get a better architecture that preserves the accuracy of 76% of accuracy, for example, and minimize the latency or maximize the attribute. And it is very interesting to see that, first of all, it replaces some of the operations from dense convolution layers to depth-wise and some other variants of convolutional layers that are, let's say, less memory bounded because the cache memory also limits the inference speed of the network. Second thing that we can learn from that idea...

• 21:28Speaker 00

  Is that some layers are more important than others. So for example, if you think about what size to have on each layer in the network, we have some understanding that the initial layers that are doing the feature extraction in the image classification example are more important than the later layers in the network. So putting kind of most of the computation at the beginning of the network seems to have better accuracy preserving properties. This is kind of what we see when we observe on the results of that run of AutoNAS on this example. And we end up with a model that is faster in something like 3X compared to the baseline and having the same accuracy.

• 22:13Speaker 00

  So this is kind of how we take an algorithm that is fully automatic and try to do a postmortem to understand what happened there, why the output of these algorithms look like that, why the initial layers haven't changed or almost haven't changed, but the later layer have changed significantly, replaced to other types of layers, their size was smaller, and some other changes that we observe on the result of the algorithm.

• 22:40Daniel Whitenack?

  So I want to actually get back to that later because it's really fascinating that you can sort of use these tools to learn more about the types of things that work well, architecture design wise on certain hardware. That's really interesting. But first, just to sort of bring that example into focus, it sounds like you had this base ResNet model. I'm just thinking about the sort of inputs, outputs of the automatic network architecture search. Like if I'm a practitioner and maybe I have my own custom model for object detection or a custom model for speech recognition or whatever it is, and I've trained that, then in terms of doing this automatic neural architecture search.

• 23:22Daniel Whitenack?

  I'm assuming one input to that is the serialized version of my model in whatever format it's in. You also mentioned that a dataset was input to that. Maybe you could give some details on that. So why is the dataset input? Is that so that you can make sure that you're not optimizing just for the hardware, but you're also optimizing to make sure that performance doesn't degrade in terms of prediction performance? And then also, do I need to have access to the specific hardware that I'm targeting? So I need this dataset? And then do I need the specific hardware that I'm targeting and run the automated neural architecture search on that hardware? Or how does that work?

• 24:05Speaker 00

  Yeah, so you got it right. Let's formulate the equation that the neural architecture search is trying to solve. We are talking about minimize the latency of the model on a given hardware, subject to getting an accuracy that is above the given threshold. So the latency of the architecture or the model can be measured without training in most of the cases on the hardware. So we don't need the data to understand what is going to be the latency of ResNet-50 on a CPU of Intel. But the accuracy is data dependent. And if we want to put that constraint, and obviously we want to put that, we need to have the data and verify that the model that is selected by the minimization problem of

• 24:48Speaker 00

  the latency still meets the accuracy requirements. Because if we'll not put the accuracy constraint, we'll end up with a model with one neuron that predicting nothing. So this is kind of the composition between the latency that is measured on the hardware and the accuracy that is measured on the data.

• 25:04Chris Benson?

  Could you talk a little bit more about the output path on that in the sense of, let's say that you have a model that's doing object detection, or it could be anything really, and you want to put it on maybe a platform that's on the edge that has a bunch of sensors, maybe a bunch of cameras pulling in. When you're going through the training process, how are you accounting for that variability out there? Kind of going back to if you're not targeting the hardware in the architecture search that you're going to deploy to, how do you all account for that? How do you say, "Oh, there's a very unique configuration for my output target, my deployment target." How do you approach that?

• 25:47Speaker 00

  Are you asking what happens when we don't know the hardware in production?

• 25:51Chris Benson?

  Well, yeah, I guess just like if you're targeting a particular environment that may be customized fairly significantly for deployment. As more and more practitioners are now kind of getting out of the data center and they are putting things on drones and they're putting things into automotive being a big one, obviously. Anything like that that's out on the edge and kind of has a custom environment, what does that look like from a practitioner perspective? Kind of out of the theory and into the hands on.

• 26:17Speaker 00

  So we prefer to connect to the actual hardware that the model is going to be deployed on. So for example, if it's Jetson, we connect the exact Jetson model to the neural architecture search. If we don't know that, we need to have some proxies. So a good proxy might be the number of floating point operation. But we know that this is not such a good proxy and some other methods like pruning target this metric. But this metric correlates to latency only on CPUs. So having proxies there, it's not the right approach in our perspective. Measuring the metric that really matters on the actual device is the way to go in my perspective.

• 26:58Speaker 00

  Like measuring the actual latency, measuring the actual throughput on the device is the way to go. In order to understand the exact performance that you're going to see in production. Cause I can show examples where I cut the flops in factor of two and the latency is getting slower. On GPU, on Jetson, or on some types of devices. So these proxies are very problematic today. And I think that this is kind of the interesting part of doing hardware where neural architecture search compared to to other compression techniques like I mentioned as pruning. That reduces the number of flops or the number of parameters or any proxy for the size or the complexity of the network.

• 27:39Daniel Whitenack?

  So Jonathan, as you're working in this space, one of the big things that I always start thinking about when I think of optimizing networks in this way is that there's just so many different types of layers and custom layers that people are using. And new stuff coming out all the time. So what has it been like sort of maintaining your search space over time? And growing that space to sort of include new things as they're coming out? How do you approach that as an organization and figure out how to expand that search space and what to include and what not to include?

• 28:16Speaker 00

  So that's a good question, because the research field of deep learning is progressing very fast. And we have a team of researchers that's sitting on the latest academic papers that propose all those new layers and those new operators, and kind of reproduce all these models to understand which type of layers, which types of techniques are worth adding to the search space of the neural architecture search, and which are not. And actually, the result is very interesting. In most cases, for example, if we'll take the computer vision domain, the basic operators that are well-known for the last five years or something like that are performing the best.

• 29:00Speaker 00

  So some of the tricks that we see now are not improving so much compared to using those blocks and operators that built ResNet and MobileNet and EfficientNet and those networks. So having the right composition of operators is more crucial than having all those fancy tricks that showed up in the last two or three years. This is something that we see. It's quite general claim. There are some cases that we see things that were adding to the search space, but in general, I would say that it's not so easy to beat a ResNet model that is quantized and use the graph compiler like TensorRT or something like that.

• 29:41Speaker 00

  And you need to work hard in order to beat it. So this is kind of how we see all the advancement. Of course, we have other advancement in other fields like training tricks, optimizers and stuff like this. And we have to be on the front line on all of those. And this is something that we are working really hard in order to reproduce all state-of-the-art, all the types of model, all those new models that just announced, and having the results and the operators in our search space.

• 30:10Daniel Whitenack?

  And a follow up on that, I guess, which is related to that approach is on the one side, you have all of these different types of architectures. On the other side, you have all of these different tasks being solved with deep learning models and AI models. So I'm curious, as you've experienced this over a number of years now, is it harder to optimize the inference of certain tasks versus other tasks? So maybe like NLP tasks versus computer vision tasks versus audio tasks versus, you know, maybe it's time series modeling or something. Are there certain domains of AI tasks that are harder to optimize than others?

• 30:52Speaker 00

  So at the moment, we are mostly focused on computer vision and NLP. And in those domains, we see that the principles that we are using that are machine learning based are walking across the board. Yes, I can tell that there are some tasks in a domain that are a little bit more complex. For example, semantic segmentation networks are more complex than classification networks. And they have to preserve the information along the network in order to do kind of image to image tasks. But also on those type neural architectures, we can optimize them. And the principle is very simple. Most of the networks and the new fancy algorithms are kind of built on top of three components. One is

• 31:35Speaker 00

  the stem, is the few layers connected to the input. The second component is the backbone, and the third component is the prediction block. In most cases, 80 to 90% of the compute is happening in the backbone. And usually the backbone are just a bunch of convolutional layers in the case of computer vision. And by optimizing that significant part of the network, which is similar across classification, semantic segmentation, and object detection, you can get that boost of performance that we are looking to have in all tasks.

• 32:09Chris Benson?

  As we're talking about this, I'm just visualizing everything in my head. As you get to the output and you've targeted a particular deployment target and accounted for the hardware and what the capabilities are. I'm curious, how does your platform integrate in with whatever DevOps pipeline a practitioner might have put into place? What is a typical scenario for actually pushing the deployment out to the hardware that it's going to run on for inference look like?

• 32:33Speaker 00

  So we look at our platform as an end-to-end platform from development to production. We develop two production tools, one of them called Inferi and the second is called RTIC. Inferi is lightweight edge inference engine that could be integrated to a monolite application easily. And RTIC is a containerized inference server. So if you'll take that solution, it could be easily deployed by a DevOps with the model inside, fetch from the model repository that we provide as part of our SaaS offering and kind of serve the model in a standardized API that contains all the packages, libraries, environment details

• 33:14Speaker 00

  that you need in order to go over Kubernetes for inference and scale. Sometimes we see companies that already have their own infrastructure and don't want to change their existing infrastructure. And we provide them with the specific model, just the exact model that ran the optimization, the model after the optimized model in their format. We support all the types of format from Onyx to TensorFlow, PyTorch, Keras and all of these frameworks that you can run inference on. So this is kind of the two ways to get DSE optimized model to a production environment, either by our deployment tools or getting the model and using your existing stack.

• 33:53Daniel Whitenack?

  I'm just kind of browsing around on some of the information about DSE and super fascinating. One of the things that I see is this idea of DSE Nets, which you share about and share some of the sort of successes that you've had taking this approach in various domains for various types of models. Could you just share a few success stories in terms of what you've been able to achieve performance-wise with this approach?

• 34:21Speaker 00

  Yeah, so DSE Nets is a good example for that. We took a few well-known tasks like image classification, object detection and semantic segmentation. And we've taken the most famous open source data set, for example, ImageNet, COCO, and data sets like that, and we built kind of a catalog of pre-optimized model for each and every hardware. And what we are doing, we are kind of plotting what we call the Efficient Frontier, chart with the latency on the X axis and the accuracy on the Y axis, and plotting all the models that we know, the ResNet, the Yolo, and those models on those charts, and putting those destinets or deci-dets for detection,

• 35:05Speaker 00

  and kind of seeing the, what we call the Efficient Frontier, how those models reach better accuracy and better latency, and dominates this trade-off between accuracy and latency. And now we provide those destinets for data scientists in order to try feed them to their specific data, fine-tune them for their specific data, and having kind of a pre-optimized results of OTANAG that is ready for use immediately.

• 35:31Daniel Whitenack?

  And I'm wondering, as you plot out this landscape, I love how you term that like the efficiency landscape. That's a really cool way to think about space. As you plot this out and explore that space yourself, I'm wondering, one way to think about what you're doing and how you've expressed it is, I'm a data scientist, I've trained my model, now I run it through auto neural architecture search and get out my, you know, better model faster and more efficient for the architecture while still performing well for prediction. But I'm wondering if this sort of cycle, as you do that more and more, you start sort of building some intuition as a data scientist or a practitioner to like, start with a better model in the first place.

• 36:14Daniel Whitenack?

  Like, if I look at your plots of the efficiency landscape, can I learn some things about maybe, maybe I start with better models in the first place rather than sort of relying as much on the neural architecture search? Do you think there's that sort of feedback and that learning that can happen from like, what architectures are learned by the automatic neural architecture search?

• 36:38Speaker 00

  Yeah, absolutely. We share some information, intuition behind those architectures that found for a given hardware. For example, one of the things that we see, that those architectures are faster but having more parameters, for example. So this is kind of an intuition that we see around these models that can be used to understand that. We not always look to smaller models, but for more faster model and accurate model. We provide those models as a starting point. So for example, if you're considering taking ResNet 54R I or EfficientNet B0, you can take the corresponding Dessenets model and start from that and tweak that

• 37:18Speaker 00

  for your application, whether you're doing object detection, classification or anything like that. And this is kind of giving the ability to use a NAS-produced model compared to having a model that is off the shelf for general use. So for example, Efficient Nets is a result by Google from two years ago or something like that that are supposed to be efficient, but it appears to that Efficient Nets are not so efficient for GPUs, even when they are over quantization and graph compiler like TensorRT. So having a pre-optimized model for the given hardware that you're going to use is very crucial.

• 37:57Speaker 00

  And in our example of Dessenets, we show that you can have a model that is something like three times faster than EfficientNet B0 for a Jetson GPU while having even better accuracy. So this is kind of a result that you can take off the shelf. Instead of using EfficientNet B0, you can take that model and train it to your application, build on top of that some other prediction heads, some other tasks that you want to solve with that backbone and get an autonomous result without running all the neural architecture search for that specific task.

• 38:36Chris Benson?

  That's very cool. This is so interesting as you were talking about model optimization and the things that Dessen has done with it. What are you envisioning as you guys are looking into the future at this point? What kinds of things are aspirational for Dessen in terms of where you want to take the platform and what you envision will be the next thing in model optimization that you'd like to implement?

• 38:58Speaker 00

  I think that we feel at a good point in the model optimization space that now we are seeing that we need to expand to the whole development lifecycle. So after you have the data, we look on controlling all the training optimization and deployment of deep learning model on our platform. Whether you're using those destinets or using some off-the-shelf models and kind of having a full workflow of development, optimization and deployment based on our platform. That's because we understand that there is kind of a triangle that we can draw that is on one edge, we have the model, on the other one, we have the data and on the last one, we have the hardware.

• 39:43Speaker 00

  And this is kind of a combined optimization that every data scientist need to understand how they solve that. And we are kind of providing the tools to solve this, optimize this triangle. And for now, we are mostly focused on the model side. But in the future, we'll be also focused on the data and the hardware side in terms of not having them fixed, but having some techniques for data enrichment, data augmentation, self-supervised learning, having some hardware recommendation system, maybe having some FPGA capabilities and having our hardware that is optimized for the given model. And this is kind of the far future about how I see optimization in its full.

• 40:26Daniel Whitenack?

  That's awesome. Well, I'm really excited that we got to talk through this because I know I learned a lot about this neural architecture search and the things that you're doing. So really impressed with where you're headed with this and appreciate you taking time to join us and chat with us about it.

• 40:43Speaker 00

  Sure, thank you very much. It was great talking to you and I look forward to to the episode.

• 40:52Speaker 04

  Thank you for listening to Practical AI. We have a bundle of awesome podcasts for you at changelog.com, including our brand new show, Ship It with Gerhard and Lazo, a podcast about getting your best ideas into the world and seeing what happens. It's about the code, the ops, the infra and the people that make it happen. Yes, we focus on the people because everything else is an implementation detail. Subscribe now at changelog.com/shipit or simply search for Ship It in your favorite podcast app, you'll find it. Of course, the galaxy brain move is to subscribe to our master feed. It's all changelog podcasts, including Practical AI and Ship it in one place. Search Change Log Master feed or head to changelog.com/master and subscribe today.

• 41:35Speaker 04

  Practical AI is hosted by Daniel Whitenack and Chris Benson with music by Brickmaster Cylinder. We're brought to you by Fastly, Von Starfleet, and Linote. That's all for now. We'll talk to you again next week.