Apache TVM and OctoML

90% of AI / ML applications never make it to market, because fine tuning models for maximum performance across disparate ML software solutions and hardware backends requires a ton of manual labor and is cost-prohibitive. Luis Ceze and his team created Apache TVM at the University of Washington, then left founded OctoML to bring the project to market. Sponsors: O'Reilly Media https://www.oreilly.com/changelog – Learn by doing — Python, data, AI, machine learning, Kubernetes, Docker, and more. Just open your browser and dive in. Learn more and keep your teams’ skills sharp at oreilly.com/changelog https://www.oreilly.com/changelogSnowplow Analytics https://snowplowanalytics.com/ – The behavioral data management platform powering your data journey. Capture and process high-quality behavioral data from all your platforms and products and deliver that data to your cloud destination of choice. Get started and experience Snowplow data for yourself at snowplowanalytics.com https://snowplowanalytics.comChangelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Luis Ceze – LinkedIn https://www.linkedin.com/in/luis-ceze-50b2314, X https://x.com/luiscezeChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Apache TVM https://tvm.apache.orgOctoML https://octoml.aiOctoML: Automated Deep Learning Engineering with Jason Knight and Luis Ceze https://softwareengineeringdaily.com/2021/02/09/octoml-automated-deep-learning-engineering-with-jason-knight-and-luis-cezeUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  There's literally billions of ways in which you can compile the same model on the same hardware target. The question is, how do you pick the fastest one? You cannot try them all because they'll take too long. So you have to use intuition. That's what software engineers do. We replace that intuition with machine learning-based optimizations that learn about how the hardware behaves in the process of optimization and uses those models of how the hardware behaves to tune and search the right way of optimizing and compiling your model to the hardware target.

• 0:31Speaker 01

  Big thanks to our partners, Leno, Fastly, and Launch Darkly. We love Leno, they keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com. And get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com.

• 0:50Speaker 01

  This episode is brought to you by our friends at O'Reilly. Many of you know O'Reilly for their animal textbooks and their conferences, but you may not know they have an online learning platform as well. The platform has all their books, all their videos, and all their conference talks. Plus, you can learn by doing with live online training courses and virtual conferences, certification practice exams, and interactive sandboxes and scenarios to practice coding alongside what you're learning. They cover a ton of technology topics: machine learning, AI, programming languages, DevOps, data science, cloud, containers, security, and even soft skills like business management and presentation skills. You name it, it is all in there. If you need to keep your team or yourself up to speed on their tech skills, then check

• 1:34Speaker 01

  out O'Reilly's online learning platform. Learn more and keep your team's skills sharp at o'reilly.com/changelog. Again, o'reilly.com/changelog.

• 1:51Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM.

• 2:17Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International. And I'm joined, as always, by my co-host, Chris Benson, who is a Principal Emerging Technology Strategist at Lockheed Martin. How are you doing, Chris?

• 2:35Chris Benson?

  I'm doing very well. A little bit chilly here in Atlanta. I don't know what's going on with that. Mid-May, we're supposed to be sweltering heat.

• 2:41Daniel Whitenack?

  Yeah, it's a little bit chilly here as well. I don't know. What's chilly for Atlanta, though? Depends on your reference frame.

• 2:48Chris Benson?

  I don't know. I'm guessing it's like 60 out there or something like that, you know?

• 2:51Daniel Whitenack?

  OK, that sounds pretty delightful to me, actually.

• 2:56Chris Benson?

  I mean, if it's not 85 by now, it's just not normal, so.

• 3:00Daniel Whitenack?

  Yeah, well, what are you keeping busy with these days?

• 3:03Chris Benson?

  Just lots of technology stuff at work, having a good time with it. And I'm super excited to pause in my day job so that we can have a good conversation about AI ML.

• 3:13Daniel Whitenack?

  Yeah, yeah. And specifically, I know in my job recently, we've been working on a few different deployments of the same model, one to like an edge device, which is disconnected from the internet, one cloud deployment, and then another one for an on-prem system. And that carries with it, of course, all sorts of joy in terms of fitting into resource constraints, but also optimizing a model for different deployments. And I think that's definitely one thing that, I don't know if anyone ever pitched being a data scientist to me, but I don't know if that was part of the pitch, like optimizing for a certain hardware and all that. It's cool, but it's definitely sometimes laborious and hard. I don't know if you've run into this in your past.

• 3:57Speaker 00

  Yeah, you have to be a data scientist and a software engineer to do that, right? So.

• 4:02Chris Benson?

  You know what struck me as you were saying that, Daniel? The thing is, is I'm kind of doing a similar thing. But being in the defense industry, we have a completely un-software-like name for it. We call that Joint All-Domain Operations. Nobody outside this world would ever, they're like, that's software? Really? That's AI, really?

• 4:19Daniel Whitenack?

  Wait, what acronym is that?

• 4:21Chris Benson?

  Yeah, that's what we call it. We call it JADO. JADO. We call it JADO and stuff. But you'd never know. I don't think anyone outside my industry would ever associate JADO or Joint All-Domain Operations with software and Kubernetes and AI deployment.

• 4:35Daniel Whitenack?

  I would have no idea what that meant if you if you gave those words to me. But I think we'll both be enlightened today quite a bit on this subject. I'm really excited because we have Luis Cezé with us, who is co-founder and CEO of OctoML and professor at University of Washington. Welcome, Luis.

• 4:54Speaker 00

  Thank you. Thanks for having me in the show. It sounds like you guys are fun to talk to. Can't wait for the conversation here. So, and it's a nice sunny day in Seattle, by the way.

• 5:03Daniel Whitenack?

  We don't take ourselves too seriously, trust me.

• 5:05Speaker 00

  Oh, good. I love that. That makes even even better. Just to finish the weather theme of the conversation, it's a nice sunny day in Seattle in the 70s, you know, so I'm wearing shorts and shirts here.

• 5:16Daniel Whitenack?

  What's up with that? He's in Seattle and he's having the great weather and, you know, Seattle's famous for rain and we're here in these other parts of the country and it's just kind of eh. Isn't it, though? I mean, Luis, correct me if I'm wrong. I mean, summer times in the Pacific Northwest are quite delightful, aren't they? It's just like the winter, like rain and fog and... That does sound nice. Yeah.

• 5:35Speaker 00

  Yeah, that's great. Yeah, winters are long, but the summers are awesome. So there's some balance in the universe and nature here, especially for a native Brazilian like me that matters, you know, so.

• 5:46Speaker 00

  Happy a nice summer, yeah. Well, thanks again for having me on the show. It sounded like from your adventures with machine learning model deployments, we're gonna have a lot to talk about here, so.

• 5:54Daniel Whitenack?

  Yeah, for sure. So before we get into all that, maybe you could just give us a bit of a background on yourself and how you got interested in this topic and ended up working in the areas that you're working now, from your moment of birth right up until this recording.

• 6:08Speaker 00

  Exactly. Great, perfect. So as I was mentioning before, I grew up in Brazil. And the funny thing is, I like to joke that I'm in the 20th year of a three-month internship in the US. I was in school in Brazil. I was recruited by IBM Research, this is in the early 2000s, to work on the Blue Gene Project. There was a three-month internship that became a year-long co-op that doing this and next. I went to grad school in Champaign-Urbana, Illinois, close to where you are. And then after that, came to University of Washington as a professor. And next thing I know, 20 years have gone by. So anyways, at IBM, I worked on hardware software co-design. We were part of this team building the Blue Gene Machine, the first Blue Gene Machine. And this was like hardware and software design for, you know, very high-performance molecular dynamics simulation.

• 6:50Speaker 00

  And a lot of it, like if you look at the kind of workloads that we ran there was high-performance linear algebra, right? And then you have HPC systems with all sorts of internal communications, reductions, pairwise communication, and so on. So after IBM, I went to grad school. In grad school, I did work on hardware and compiler support for speculative parallelization, essentially making it easier for folks to write parallel code by not having to prove that the code is actually parallel. And then I came to UW. At UW, I started in my faculty career, started working on, you know, hardware software design for emerging applications. So I had done a bunch of work on approximate computing. Essentially, the idea is to take advantage of the fact that some applications don't have to have perfect execution to have a meaningful and useful answer.

• 7:33Speaker 00

  So, right, simulations are like that, there's multiple valid outputs. And of course, machine learning is a huge workload in that category. That's how it started, actually. Started working on hardware and software optimized for machine learning for energy efficiency, better performance. And then about six years or so ago, I started collaborating with folks in the core machine learning group here at UW. Carlos Gaston, who's a friend of mine, also Brazilian, you know, Brazilian transplant to the US. Although he's Brazilian, Argentinian, transplanted to the US and, you know, has a long history in machine learning systems. And we were chatting six years ago or so, there was already a growing set of machine learning models that people are interested in, and a growing set of hardware targets. You know, it's kind of crazy to think about that, you know, six years ago, GPUs were just

• 8:17Speaker 00

  starting to get popular for machine learning. It feels like that has always been the case, but, you know, six years ago, that was just like, kind of like picking up, right? And people were starting to think about it.

• 8:26Chris Benson?

  A lot's happened since then.

• 8:27Speaker 00

  I know, yeah. So then now we had those CPUs, you know, GPUs are picking up, people started talking about that PGA's they had for a while, but at that time was getting heated again. And then people started talking about building accelerators. So we were wondering like, whoa, kind of interesting that there is this growing set of models already starting to fragment ecosystem with TensorFlow. PyTorch doesn't quite exist yet, but there was already some fragmentation going on and it grew inside of hardware targets. So now imagine this cross product of models, frameworks and hardware targets. It's already becoming a pretty complex cross product. So we started thinking about, oh, wouldn't it be nice if we had a common intermediate representation? It allows you to do high level model optimization, do and generate specialized code for that model on specific hardware targets.

• 9:10Speaker 00

  So, and T N G Chen, who's an incredible machine learning systems researcher, was Carlos's grad student at that time and Carlos and I started co advising T N G. And that's, that was the genesis of, you know, let's look at machine learning model optimization and compilation as a way to actually optimize and deploy machine learning models. And I had other grad students there, Terry Moreau, who was working on specialized architectures for machine learning, got interested as well. So he quickly became part of the mix and started working on FPGA backends for machine learning models as part of this mix. So that's how I got into that. In the end, everything boils down to high performance linear algebra, say for machine learning. And I had a history with that at IBM and everything kind of came together.

• 9:50Speaker 00

  Approximal computing, high performance linear algebra and machine learning in this, in the genesis of the T V M project, so.

• 9:57Daniel Whitenack?

  Awesome. And maybe we could just sort of take a step back. You talked about like machine learning compilers, we talked about intermediate representation. Maybe for that person out there that has taken the Coursera course and they went through the Coursera course on deep learning, something and hey, it never talked about machine learning compilers. They learned maybe how to do a training thing and they created an image classifier. And then they learned how to do an N L P thing and do autocomplete or something. And then they end the course and they know how to use TensorFlow and do some data preprocessing stuff. And maybe even a little bit about GPUs, but they never heard this machine learning compiling. Could you just sort of?

• 10:39Speaker 00

  Yeah, absolutely.

• 10:40Daniel Whitenack?

  Yeah, where does that fit in?

• 10:41Speaker 00

  Yeah, I'm going to translate all of that. You're basically saying, wow, I'm really missing the audience here. Like, let me translate all of that here. So, all right, when you write a typical machine learning model. By the way, I'm really glad that machine learning tools are becoming more and more accessible, more and more people. Can build meaningful and useful machine learning models, either starting from an existing one or tweaking and creating new ones. I'm so glad that the tools are getting much better for that. But now, when you create a machine learning model, though, in the end, you need this model to actually run well on whatever hardware device on them to run, right? So today, this is typically done by having a set of libraries that implements parts of your model, like different layers will have different, often hand-coded implementations. And then your high-level framework like TensorFlow, PyTorch

• 11:26Speaker 00

  just gets your model and calls these hand-toed libraries that either Nvidia provided, or AMD, or Intel, the hardware vendors typically provide those libraries. And then the frameworks stitch everything together to produce the thing that runs on your hardware, right? So, which is all well and good. It's nice and easy as being, you know, being able to build pretty complex systems that way. Except that now, as we build richer and richer and more complex machine learning models that really need to make the most out of the hardware they're deployed to, and which is great. Like, so it's kind of crazy that you can build gigantic language models or very complex computer vision models that do a ton of computation. That's really pushing at the limit of how fast you could run, you know, just by writing on Moore's Law, right?

• 12:11Speaker 00

  So you need to be able to squeeze much more performance out of that. So that's where a machine learning compiler comes into play, right? So when you write a piece of code in a language like C, for example, you run through a compiler and you run it on your hardware. You don't even think about, yeah, you run through a compiler, you run it. But for machine learning, we don't do that, right? So you just interpret the model. What a machine learning compiler does is essentially treats this process of going from your model to what runs on your hardware as a compiler problem. What do you mean by that? I mean by translating your model into a representation, as we call it, intermediate representation, that enables optimizations of your model. For example, you could Fuse a layer with the next one, say a fully connected layer, which would be matrix multiplication followed by convolution that you chose in your model.

• 12:56Speaker 00

  Potentially, you can fuse them, treat them as a unit and generate nice new fresh code that specializes to your model to run on your hardware target. So and then as a machine learning compiler, what you get, as an end user, the benefit that you get is your model becomes a highly optimized executable for your target hardware. And the difference of performance can be huge, like 2, 3, sometimes 30, 50x better performance than your stock high-level framework execution time and speed, right? So does that make sense? Essentially, you treat that, basically what you take for granted in writing regular software, you know, does that for you in machine learning, right? So produces a fresh, highly optimized binary for your model.

• 13:37Daniel Whitenack?

  Yeah, and that makes a lot of sense. And it sounds like a lot of what we're talking about here is performance in the sort of maybe latency and resource consumption type realm. When we're thinking about compiling, is that mostly what we're concerned with? Or do some of these compilations, like for example, does the actual performance of like the model's predictions, does that ever come into play when you're doing this sort of optimization as well?

• 14:06Speaker 00

  Great question. So performance here is ambiguous. For a computer systems person, performance typically means how fast it runs. But for a data scientist, performance also means how good is statistical properties of your model, right? So by and large, machine learning compilers do not change the accuracy of your model. There are optimizations, for example, you can do automatic quantization. You can quantize parameters in your model, it can change the behavior of your model. In that case, you do trade off some accuracy for better execution time or for better system performance, right? But for the most part, in fact, what we focus on in Apache TVMR transformation optimization do not change the accuracy of your model at all. Even though we do support quantization and so on, by and large, the way it's used is it just compiles your model faithfully to run without any changes in accuracy, right?

• 14:51Speaker 00

  So and to tie back to your comment earlier of deploying multiple hard, say, you know, deploying on the edge, getting the right performance and using the right amount of resources that fits in your heart is something that's an extremely laborious task. Right, so if you have a model that's going to run on Raspberry Pi-like device, say it's computer vision, and the first version of your model runs at half a frame per second, right, and uses too much memory and doesn't leave any other compute resources for the other things you need to run on your device, you can't really deploy it, right? So, and our experience has been with several users and customers is that the process of getting a model that is ready from a data science point of view, from the accuracy

• 15:32Speaker 00

  point of view, to be able to be deployed, it could take weeks to months of hardcore software engineering work, and that's what we want to automate with Apache T V M. And all the machine learning compilers do similar things, but you know, Apache T V M is just especially good at doing that because it uses essentially machine learning for machine learning. The process of translating your model to an executable, like a deployable artifact, it's a search problem. So there's literally billions of ways in which you can compile the same model on the same hardware target. The question is, how do you pick the fastest one? You cannot try them all because they'll take too long. So you have to use intuition, that's what software engineers do. We replace that intuition with machine learning-based optimizations that learn about how the hardware

• 16:17Speaker 00

  behaves in the process of optimization, uses those models of how the hardware behaves to tune and search the right way of optimizing and compiling your model to the hardware target.

• 16:31Speaker 01

  This episode is brought to you by Snowplow Analytics. Snowplow is the behavioral data management platform for data teams. Maximize the value of your behavioral data using Snowplow Insights, a managed data platform that's built on leading open source tech, leveraged by tens of thousands of users. Capture and process high-quality behavioral data from all your platforms and your products, and deliver that data to your cloud destination of choice. When marketing needs to make data-informed decisions, when product needs next-level understanding, and when analytics needs rich and accurate data, Snowplow is the solution for data teams who want to manage the collection, processing, and warehousing of data across all their platforms and products.

• 17:12Speaker 01

  Get started and experience Snowplow data for yourself at snowplowanalytics.com. Again, snowplowanalytics.com.

• 17:34Daniel Whitenack?

  So Luis, I'm curious, there's one of the hardest problems in running this podcast, I think, is just the wide variety of jargon that is used throughout all sorts of different areas of ML. And I know that on this podcast, we've talked before about model serialization. So we've got something like an Onyx project, for example, where, you know, maybe I could quote "save" a model to Onyx and maybe like load it into PyTorch or a PyTorch model into TensorFlow. It's this sort of overlapping serialization format. So as people are thinking about kind of saving and serializing their models, where does maybe this compilation fit in in terms

• 18:16Daniel Whitenack?

  of the developer workflow? Like, let's say that I have my model created, or I've trained it, let's say I'm in TensorFlow, and I've decided on a way to serialize it. So I've created a, I've output a file that corresponds to my serialized model. What happens next? What's the workflow look like after that in terms of Apache TVM?

• 18:37Speaker 00

  Yeah.

• 18:37Daniel Whitenack?

  And the compilation process?

• 18:39Speaker 00

  Great question. So there's many ways in which Apache TVM ingests your model. One is exactly what we talked about. So there's a front end to Apache TVM that ingests a model that has been serialized into Onyx, right? So you just import the model, and then you specify the harder target, and then you wait for a little bit, and then you get your artifact ready, your executable for your model, and packaged in various ways. But there are also ways in which you can call TVM directly from the code that specifies your model. So you can do that from TensorFlow, from PyTorch, from MXNet or Keras, and so on, that you do, essentially, it imports TVM, you load your model graph into the TVM representation, then you choose your harder target, and then you compile, right? So there's just two ways

• 19:23Speaker 00

  in which you can interface with TVM, either via in a serialized model, or by just embedding a few lines of code to call TVM. So in the flow from data to your deployed model, right? You create the data, curates the data, create a training set. Model architecture, sometimes it's a little bit of architecture search. You arrive into your model architecture, you train it. And then you're able to actually test it, validate your model, then you're happy with it. So at that point, your model has the right statistical properties. You know, it does what it's supposed to do and what you want to do. You want to make it run fast, right? So now you have a trained model. At that point, that's what you hand off to, say, Apache TVM. Either via Onyx, serialized models, or via calling directly the stack. And then you specify your hardware targets and then you click compile, right?

• 20:06Speaker 00

  So you call compile, that's when CVM does its high level and low level optimization magic. And also uses this machine learning for machine learning engine to tune the code for your specific model on your hardware target.

• 20:17Chris Benson?

  So I'm curious, I got maybe several questions that I'm going to try to combine into one a little bit. And you can segment them any way you want. If you go through that optimization process, A, what does that output look like? How is that different from the model before you took it through the optimization process? In terms of how you approach inference on it? And what are the limits in terms of that target architecture that you're trying to hit? You mentioned the Raspberry Pi and there's, this day and age, there are tons and tons of kind of low capability or low power targets that you might want to run a model that otherwise would have been impossible. Can you describe kind of what that looks like after that optimization and what the limits are on it?

• 20:59Speaker 00

  Yeah, great. Okay, so let me tell first what it looks like. So the output is really a executable. It's just an executable code for your model. That includes, you know, your model, it's a super model plus a runtime. Like the runtime would be like a support for your model that's tuned for that harder target. With CVM, you get a custom packaged binary for your model. The way you call it, you load that binary and then there's an API to call. It could be a shared library. For example, one way of packaging this whole thing up is this dot SO library or a DLL in Windows where your model is just. Fresh executable code with an API that you call to do inference on. Does that answer your question? This is what it looks like as the output. And then there's many ways of making it standard. Like you can put in a Python wheel and have Python bindings for it. There's many ways to make it easy to call it.

• 21:44Speaker 00

  Yeah.

• 21:45Chris Benson?

  Yeah, it does. Like as a follow up, and I'm going to say something extreme. As we're building bigger and bigger models and we're taking, you know, things like GPT three and things that are, that are very large. And we have this once upon a time unrealistic expectation of putting them into places that you would be like.

• 22:01Daniel Whitenack?

  I think Chris wants to run GPT three on his smart watch. You are closer than you realize on that one. So.

• 22:08Speaker 00

  So absolutely, no. And we had people asking about this. Yeah, there's a limit to that, right? So you can't get around physics, right? So even if you compress maximum, if it doesn't fit in the memory that you have, it just doesn't fit.

• 22:18Chris Benson?

  Understood.

• 22:19Speaker 00

  Or if he uses, you know, so much computer, it's going to take too long to do the inference. You know, that is not going to be a useful output, right? And of course, in the process of searching what hardware makes sense, you can use a tool like the optimizer. Let me just put a quick plug in here. So optimizer is a software as a service platform that OctoML built that uses Apache TVM as its engine. It is like a very easy to use, you know, TVM as, as you can tell, business conversation is likely to be a more sophisticated stack for, you know, more general data scientists, right? So, and general data scientists already have enough to worry about making models that do the right thing is what they should focus on. With the optimizer, we are raising level abstraction to match that. So you can upload your model as a nice graphical user interface where you upload your model,

• 23:04Speaker 00

  it tells you what the layers are, you tell what the input layer is, you can click on the harder targets that you want, or you can have it choose for you by running across them all and see which ones it runs best at. So you can get the highest throughput per dollar, or you can hit the, you know, we do support Raspberry Pis now. So you said this is run Raspberry Pi. If it doesn't, it's just going to say, hey, it couldn't run this model there. So then you know that even doing all this magic, it does not run on the harder target, right? Anyway, so back to what you're asking. So now on the limits, how do we know? We can get around physics, right? So there's only so much compute resources that you can afford in an edge setting. But there are plenty of new techniques that actually can twist and turn your model to make it fit. One, for example, specifically for language models that you brought up, something that we've done quite a bit of work on is support for sparsity, right?

• 23:47Speaker 00

  So these language models are giants, but there are a lot of zeros in it. So as we all know, lots of zeros, easy to compress, right? So and if you're going to multiply something by zero, you don't even need to multiply it, right? So and if it's a bunch of zeros, why are you going to write a bunch of zeros in memory? Just say like, hey, you know, only write the non-zero values. Sure, you can do that. And that actually goes a long ways in using less memory and using less compute. I would go as far as saying that we can quite run GPT-3-like models on CNE device, but we're getting close to running pretty large models because of good sparse computing support and also, you know, other forms of compression and customization that makes big models fit in edge devices.

• 24:25Daniel Whitenack?

  So Just to follow up on the inference side of things. So maybe this is part of what you're building with OctoML, because like you were saying, Apache TVM, maybe it's a lower level thing for real experts. OctoML maybe is some

• 24:39Speaker 00

  More accessible, yeah.

• 24:41Daniel Whitenack?

  Has a bunch of convenience built in, yeah. So I'm curious on that inference side, maybe you could contrast the two. Like if I compile a model with Apache TVM, you mentioned sort of Python wrappings around that output model, and maybe there's other language wrappings. Is that as simple as sort of importing a Python library and then importing your compiled model and running an inference? Or what other sort of workflow changes might you have to do to run Apache TVM compiled model in terms of the Pythonic ways that people sort of are used to doing things?

• 25:16Speaker 00

  No, it's exactly what you said. It's two lines of code. So the results in Python will, you import it and you call inference on it, and that's it.

• 25:23Daniel Whitenack?

  That's awesome.

• 25:24Speaker 00

  Yeah. And that's the experience that we offer with the optimizer. And then the optimizer also has an API call, by the way. So if you want to embed the model optimization compilation to your workflow, you can use the optimizer. Give a hard sell, I'm just saying it's super easy like TVM itself. You have to set up your task framework, you have to spin up some instances to run this auto tuning and this, you know, optimization tuning component, you have to collect the data for the machine learning model based optimizations and so on. And it's work, it pays off, but it's work I have to put upfront, right? So with the optimizer, you can replace that with two lines of code in the API. You upload the model via the API and then you specify your harder targets.

• 26:06Speaker 00

  And then you start the optimization process and then when it's ready, you can download the resulting we all that you can import into your workflow, right? So you can do all of that either via the web interface or via this API.

• 26:19Daniel Whitenack?

  Awesome. Yeah. So I know I'm super interested in OctoML and I know that you're kind of in in beta now. It doesn't have such a long history, so maybe you could tell us, you know, how that came about? If it was sort of natural, feeling some of these pains of the lower level things, you know, expertise in in Apache TVM and the desire to have that be more automated and have some convenience around it. Could you give us a little bit more of the motivation there and your thought process with OctoML?

• 26:46Speaker 00

  Yeah, absolutely. Yeah. So maybe I can do a quick once upon a time on OctoML as well. So we had worked hard on Apache TVM, you know, started as a research project and got a bit of traction. It would be deployed in, you know, Amazon, Facebook, Microsoft and so on. And when running one of our conferences, we're like, oh, wow, there's a lot, quite a bit of interest here. But the folks that were attending were really, you know, machine learning researchers, engineers, or, you know, folks that have data science and machine learning experience as well as software engineering experience. We, we said, oh, it's, it's cool. We want more people to use it. And of course, we're going to continue catering and growing the open source community. So that's why we formed OctoML to invest in Apache TVM and grow the community, grow the ecosystem, work closely with hardware vendors to enable more hardware targets on TVM.

• 27:29Speaker 00

  TVM today, you know, has a pretty broad set of hardware targets, like it has, you know, service CPUs from Intel and AMD and ARM CPUs and mobile and server, right? So Nvidia GPUs, ARM GPUs, Intel GPUs, AMD GPUs, AMD APUs. We have FPGA support, have a broad set and a lot, and a lot of these were done with us in partnership with hardware vendors. And some hardware vendors that go and put the work into the open source package. And the reason that I keep emphasizing open source here is that I don't think you'll be sustainable. To have a project of the diversity that TVM needs, or any sort of machine learning compiler needs to thrive, because you want diversity of models that it supports, frameworks, and harder targets. The only way you make the diversity manageable,

• 28:14Speaker 00

  in my opinion, is by having an open and welcoming community that even competitors can collaborate, kind of like Linux. You know, so Linux is a great story of success there. We want to, and that's what we want with TVM. So, and what we wanted to do with OctoML is enable more harder targets and work with harder vendors, but then also make the power of TVM and machine learning model optimization and compilation accessible to the broadest set of users possible. Right? And the way we approach that is by building a high-level service that is, makes it very, very easy and fully, you know, so natural that you'd never choose not to use it. Right? So, and that's really the goal of the optimizer. And we made it into a SaaS offering because machine learning moves fast, as you know, right? So, models change fast. There's new harder targets almost every other week. Right? So, the best way to do this

• 28:59Speaker 00

  is actually package that into a service, because then we can take care of all of the, you know, complicated lower-level things that you, you don't want your data scientist spending time on.

• 29:08Chris Benson?

  So, you've got me pretty excited about it as we've gone through this. And how, with different people having their practical workflows that they're using, and you know, whether it's TensorFlow or PyTorch, and they have a deployment string attached to it. And I know that you've mentioned it targets several frameworks. We've talked about, you know, the, the kind of the two biggies and stuff. Can you talk a little bit about how, first on the Apache TVM side, and then talk about where we optimally can use OctoML? But try to, give me a sense, a practical sense of, how do I get that into my work stream? What do I need? What are limitations? What should I avoid? What should I do? I'm just trying to get a sense of how do I get started? If I'm listening, that you've sold us on what to do here and rubber meets the road kind of moment.

• 29:50Speaker 00

  Yeah, absolutely. Yeah. So, if you are so inclined and you, and you, and you want to do some software engineering adventures, you should definitely go to tvm.ai. There's plenty of tutorials there on how you can get started. You know, there's a TensorFlow example. There's a PyTorch example. There's a Linux example. And you can just go there. And that's how you use the open source offering there. Right? So, now, there's many ways in which you can. And use open source Apache TVM, right? So you can use just as a compiler, know what we call auto tuning. Auto tuning is the machine learning based magic that I told you about that searches for better implementations. For that, you have to do a little bit more work, right? So you have to set up a benchmark infrastructure. It could be just your machine, but then it takes a little bit longer because it needs to run a bunch of experiments. It's one of those tutorials for that.

• 30:34Speaker 00

  If you have more experience with the lower level parts of your framework and you're ready to use, you know, a stack like Apache TVM, I'll start with that route, try and run through the examples there. Now, if you have some work to do and you want to get your model ready quickly and you want to enjoy some automation, you can use the optimizer because it's a full SaaS offering. Then what you do today, the way you support it is you serialize your model to Onyx, you upload the model. And then, once you upload it, there's all sorts of hardware targets that you can click as a checkbox for Nvidia GPUs, Intel CPUs and Raspberry Pi, you can choose the ones that you want. And then you click optimize and then you got a notification when your workflow is done. You get all the performance compares across all the hardware targets and across even different ways of compiling your model, so.

• 31:19Chris Benson?

  So between the optimizer and the, you know, having the baseline open source project, you can kind of choose the level of abstraction that you want to get into for what your workflow is. So you have, you have choice that way, right?

• 31:32Speaker 00

  You do have choice, yes. You have choice of how you want to take it into that. You can go through the Apache TVM routes, you know, then you do it and we'll support that. There's a thriving community that will help you. Now, if you want, really want to get started from day zero and not have to worry about that, then you go to the optimizer, either using the web interface or the API. And then you can use our support and you also, you get access to our ready to go machine learning models for the hardware targets that you care about.

• 32:07Speaker 04

  ChangeLog++ is the best way for you to directly support Practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of Practical AI into the future. Simply follow the ChangeLog++ link in your show notes, or point your favorite web browser to changelog.com/plusplus. Once again, that's changelog.com/plusplus.

• 32:53Daniel Whitenack?

  So Luis, this is super interesting. What I'm wondering, one of the things that we've mentioned a few times is Onyx, and I think that sounds like some of what you're sort of centralizing around with OctoML is Onyx maybe as a recommendation, since you've mentioned it a couple of times. Could you maybe just give those who aren't familiar a brief definition of what we're talking about with Onyx? And also, maybe from your perspective, as someone maybe not working on Onyx day to day, but working on something that depends on that, how you see that that project progressing and the momentum with that?

• 33:28Speaker 00

  Great, great question. I want to emphasize that even though you use Onyx several times, you know, we do support directly, you know, if you go through TensorFlow or PyTorch and so on. And there's no, I do tend to like what Onyx aims to do, because Onyx is just a way of representing your model. Essentially, at the high level, it's a description language. So you can, you have your model, you build it in memory, you just specify in whatever framework you like to use, and you can, you want to store it, right? So you need to describe it somehow. So Onyx is an agreed upon way by, you know, multiple players in the space that this is a good way in which you can describe a machine learning model. That includes the computational structure, your layers, what each, you know, operator does, as well as all of your parameters gets, you know, serialized in one single package, right? So.

• 34:12Speaker 00

  Phoenix is evolving and it has its ups and downs. And I think right now, you know, people are getting more excited and they're so excited about it. And then there's this, I think it's an uptick now. I'm sure there will be other model description language is an exchange format, right? So that would pop up and we are ready to support those as well. I do think that it's good to have at least one format of storing models that is generally adopted, like widely adopted, because then you know, if you keep your models that way, chances are that the software components that you need in your workflow will support your model, right? So.

• 34:44Chris Benson?

  This field is evolving so rapidly right now. And you have not only each frameworks kind of way of doing things, but like I'm looking through the tvm.ai website that you referenced and going through and there's like the get started with TVM. And there's so many different options and I can't help but wonder you're covering so many architectures. And with all those changes happening and, and this is happening at light speed all the time. We're constantly getting bombarded with new things. And I know that I, as a practitioner, struggle to keep up at times with all the things. How do you do that? How do you keep the project going, keep it current, keep all these new things coming out? I'm assuming you don't sleep.

• 35:21Speaker 00

  Yeah, it's a great, a great question. I do, I do sleep and thank you. One thing that makes me sleep. Yeah. I, I, I have to keep my natural neural network working. The way we do that is by having good sleep and a glass of wine on Fridays, you know, so.

• 35:34Chris Benson?

  Glass of wine, a very important part of that too.

• 35:35Speaker 00

  Very important, yeah. Absolutely. So the answer is how do we keep up? It's really having a strong community and nurturing it and being player in it, encouraging more folks to participate. And looking back, I mean, I'm very grateful to the community and I think we were lucky to have been involved in helping catalyze that community because somehow luckily TVM was able to capture the interest of folks that build the frameworks, folks that build models, because new classes of models. Say when you were recurring neural networks, we had to go support that in TVM. Once you have dynamic models with dynamic shapes, all these things that you don't need to know it is, but essentially different aspects of your model that make them more powerful and more general needs to be supported in TVM. All of that were actually contributed by community members and we help make that happen.

• 36:20Speaker 00

  We, we put a lot of work ourselves too. But then the harder vendors, you know, so the harder vendors are the ones that actually really feel the pain, to be honest, of this growing complexity of the ecosystem that you put very well, Chris, you know? So harder vendors today, they have to write libraries, low level libraries that are tuned. For their harder targets, for each one of the major operators in these models, every time models change, have to go and tune it again. So they're always like having to update. That's not sustainable, right? So that's why TVM automating a lot of that made it very attractive for them to contribute to TVM. And they wanted to be open source because they also want to enjoy the simplification effect that the community has.

• 37:00Speaker 00

  So since we incubated Apache TVM into the Apache Software Foundation, there was even more interest and industry became more comfortable in contributing because now there is professional independent governance of the project. Because before was, you know, a few grad students and a couple of folks in, you know, some contributors sitting in a room or, you know, sitting in a virtual room or folks at the University of Washington too. Anyway, so there was a long answer to your question, but basically it is by having a open source community and having the writing center, technical incentives for folks to contribute to it. That's how we deal with the growing diversity, so.

• 37:36Daniel Whitenack?

  I'm curious more about that open source side. So could you give, maybe there's people out there listening that are working on what they feel like might be the next really cool AI practitioner tooling or data science developer tool or something, they want to get this project out there. They want to have it be an open source project and get other people involved. Do you have any tips for those sorts of people out there that are working on tooling, working on new things in terms of helping them understand how they might foster a community around those things and maybe get a little bit of momentum going? What are some of maybe the key points with that?

• 38:16Speaker 00

  That's a great question. I'll say first recruit early users and truly listen to them and make them feel like they're part of your adventure here and then you're helping them succeed and their success is your success, right? So let's say like we got lucky and we were fortunate that we had early users that were very involved in giving us feedback and we, you know, by showing that you care about their feedback and implementing quickly and then that catalyzes the process, right? So kind of like have to treat them as customers paid with love, right? So they're giving you feedback and you respond to that by making their lives easier, right? So that's the first thing, you know, really treat your community as well as possible and respond fast. And then second, you know, whenever picking the general theme, try hard and find what are all the other open source tools or maybe not open source tools that exist there that

• 39:01Speaker 00

  are very adjacent to what you do. And you know, have a lot of clarity on what is your differentiation there, you know? So what kind of new problem are you solving? How do you communicate that? And if it's related to a research project, it might be a little bit easier because you write a paper about it. And you know, people read the papers like, oh, you're solving a cool problem. They come and take a look at your work, right? So those are two things, you know? Clear differentiation and then recruiting users as early as possible. So you can iterate fast. And if you solve their problem, chances are they're going to tell their friends and colleagues and they start using it too. That's how you catalyze it, so.

• 39:34Chris Benson?

  I'm kind of curious, kind of in the same strain of that last question. You're affecting so many other communities out there that, that, you know, maybe commercially based with a hardware vendor. There's a lot of communities involved in the targets that you're, that you're compiling to. And I'm, I can't help but wonder, how do you see those types of communities getting involved? Because you are essentially a pretty significant influencer in how those targets get used. Because if the compiling is working, if it's really awesome, that benefits them in a big way. How do they choose to engage you? Do you need more engagement from those target communities to do better? And what kind of value can they add?

• 40:15Chris Benson?

  I just, as you're hitting Raspberry Pi, I would imagine the Raspberry Pi community would have to be keenly interested in working with you on this.

• 40:22Speaker 00

  Yeah, it's a, it's a great question, Chris. So ARM is already fully brought into the TVM ecosystem. They built their CPU, GPU and NPU compilers on top of TVM. They're a very active contributor to the open source community and they work closely with us at OctoML as well. So some of the big players, of course, like for example, let's say Nvidia, right? Nvidia has a very mature, probably the most mature system software stack for machine learning on their hardware. And arguably, that's a good chunk of their success. And even Nvidia is interested in working with TVM. You see some commits from them, you know, we support Nvidia pretty well. And the point I wanted to make here is the following. For hardware that's really popular, the community is going to do anyways, even if the hardware vendors themselves are not involved. So we have really good support for Nvidia because we did a bunch of work.

• 41:06Speaker 00

  The community did a bunch of work because Nvidia hardware matters. If there's other we have, we have very good performance on Nvidia hardware because of that. And the point here is not to just say compete with cuDNN and TESR RT. Of course, they know their hardware, they do well in many cases. We do really well in some other cases that was not in their radar. And we want to offer users a choice of which one to use and sometimes even choose it automatically for them. So in CVM, there's something called best of both worlds. When you process a model through CVM, you can choose to either use cuDNN or CVM native code and it picks the best of each, each part of your model. And you compose this for the end user. What you care is your model runs fast, right? So the point here I wanted to make is that for this big hardware, the company supports it. For emerging hardware vendors, honestly, I don't know if they have an alternative.

• 41:50Speaker 00

  The alternative is to build everything in-house. So think of it like, let's say you are a new AI chip company and you have a great idea for a hardware mechanism that's going to make a certain class of models run really well. And that's your business. Are you going to cater to this set of users? So it's not building your hardware, then you go and look at the ecosystem and you say, oh, now I have to support PyTorch, TESR, Flucarism, Max, I have to support this type of models. You start looking at everything that needs support to connect with the rest of the ecosystem. It's a daunting task. So that's a huge incentive for them to come and use CVM because if they support a very clean code generation interface that we made very easy for new hardware vendors to come and be part of the ecosystem. I don't think they'll find a more compelling alternative, to be honest, because you know, how are you going to spin up a team of 15 compiler engineers

• 42:35Speaker 00

  to go and build your own internal compiler? That is just hard to imagine, right? So basically, what I'm trying to say in a long way is that the way we made it easy to add new hardware targets and the fact there's a community around it and the fact there's a lot of momentum already sells itself to the new hardware vendors. And we see that by having hardware vendors come wanting to work with us to do some enablement work for them directly. We do some of that for some hardware targets. But a lot of times you just see hardware vendors going directly to the new, smaller emerging ones going directly to the Apache CPM. So does that answer your question, Chris, or yeah?

• 43:07Chris Benson?

  No, it does. That was a really good answer, actually. That satisfied it. Thank you.

• 43:11Daniel Whitenack?

  Yeah, definitely. I agree. I have sort of a strange question as we get more towards the end here. What else out there sort of across the AI industry? Do you have your eyes on in terms of things that really excite you in terms of where the industry is going? Or maybe particular groups that are innovating or particular technology that you have your eye on? What else do you have your eye on as you're looking kind of towards the future of the AI industry?

• 43:41Speaker 00

  Yeah, well, there's just so many things. Let me start with the shorter term, then it goes longer term. So in the shorter term, I think it's really exciting what's going on in doing more and more hardware aware network architecture search. There's some companies doing it, but essentially as you evolve, how your model looks like with network architecture search, you do that in a hardware way. I think it's super important because it's complementary to everything that we talked about here, right? So what we talked about here is having a model that is ready to be deployed, is going to compile it. But now having the model in the first place to actually be better to do for your hardware is great. So that's one of them. The other one is more automations in the data management side. For example, I love what those narco.ai folks are doing, which essentially, as if you've heard of them,

• 44:22Speaker 00

  but they essentially have tools that enables you to construct synthetic data sets in a programmatic way that essentially automates a lot of the work that's required for you to start building a data set to train models. And then on the hardware side, what I think is happening right now that's really exciting is just to see more and more reconfigurable architectures coming into the mix, right? So I'm sure you've heard of CPUs, GPUs, right? And CPUs like accelerators. You've probably heard of FPGA's as well, which essentially hardware fabric that you can program very much the same way that you design hardware. But you know, it's not quite as fast as a truly application specific chip, but it's pretty general. And then you can do a lot of meaningful things with it. I find it exciting that those FPGA's are getting more and more tuned for machine learning.

• 45:07Speaker 00

  So Xilinx has offerings that way. You know, Altera is having, you know, enriching their FPGA's with more blocks that are more suitable for new models. I find this exciting in the short term. So now, in the medium to long term, I just love what's going on in between machine learning and life sciences, you know, so just seeing machine learning enabling, you know, very large scale genomics studies that creates crazy amounts of data to go and make sense of data that is incredibly complex, because nature is a complex beast, right? So, and then also using machine learning to design systems, you know, so people using machine learning to design molecular systems, people using machine learning to design aircraft, people using a reverse engineer, like, what was it again, reverse design, where you give the properties and you synthesize something that has those properties using machine learning.

• 45:52Speaker 00

  These are all things that I find really, really exciting to think about, because machine learning itself is also a system. So, using machine learning for machine learning improvements is pretty interesting. We do that to some extent, but I feel like we're just scratching the surface, right? So, you can use machine learning to design machine learning chips, right? You can use machine learning to optimize machine learning training systems. And when you close the loop there, that's when you should embrace, embrace and let it evolve, right? So.

• 46:20Chris Benson?

  I remember a while back, we had some guests from Intel, and they were talking about just what you just said, using machine learning for chip design. And yeah, the sky's the limit, that you can use it for so many different things at this point.

• 46:30Speaker 00

  Yeah, absolutely. And machine learning itself, going back full circle now, you know, as a workload, it's so tolerant to, we call noisy execution, right? So by that, I really mean, if you have flaky hardware, and then there's just no way around it, as you go to two nanometer technologies, you know, which, I don't know if you heard, IBM just announced they're getting, you know, two nanometer process, kind of making a lot of progress with it. It's kind of crazy to think about, right? And it's likely to be very, very noisy and have very flaky transistors. And the way we make use of that is not just by doing the typical system design of layering, error correcting nodes, we should do that too. But with machine learning, you can use that directly, because it's so tolerant to noisy execution, that there's many interesting possibilities there. So for better energy efficiency, and so that machine learning wouldn't get

• 47:14Speaker 00

  as much bad rap by using a whole lot of energy, right? Which I'm sure you've heard that before, right?

• 47:19Chris Benson?

  Well.

• 47:22Speaker 00

  That's a whole topic of whole other conversation, you know, so.

• 47:24Daniel Whitenack?

  Yeah, and I hope we can have that conversation sometime. Well, Luis, I appreciate you taking time to help Chris and I fully optimize our discussion on the podcast to maximize interest. It's been, it's been great. I've really enjoyed it. And we'll have links to TBM and OctoML and all the things in our show notes. So definitely encourage our listeners to check those things out. I know I'll be playing around a bit after the episode. So thanks so much, Luis. Really appreciate you taking time.

• 47:56Speaker 00

  Thank you. You guys are really, really fun. I can't wait to hear your other episodes for other topics as well. So you and Chris are really fun. Thank you.

• 48:06Speaker 04

  Thank you for listening to Practical AI. We appreciate your time and your attention. If you enjoyed this episode, help us out by spreading the word. Think of a friend, think of a colleague, somebody who would benefit from listening to it and send them a link. We'd really appreciate it. Practical AI is hosted by Chris Benson and Daniel Whitenack. It's produced by Jared Santo with music by Breakmaster Cylinder. Thanks again to our sponsors, Fastly, Linode, and Launch Darkly. That's our show. We hope you enjoyed it and we'll talk to you again next week.