LLMs Everywhere: Running 70B models in browsers and iPhones using MLC — with Tianqi Chen of CMU / OctoML

We have just announced our first set of speakers at AI Engineer Summit https://www.ai.engineer/summit! Sign up for the livestream or email [email protected] http://[email protected] if you’d like to support. We are facing a massive GPU crunch https://twitter.com/tqchenml/status/1689397314576662528. As both startups and VC’s hoard Nvidia GPUs like countries count nuclear stockpiles, tweets about GPU shortages https://twitter.com/abacaj/status/1687946803377782784?s=20 have become increasingly common. But what if we could run LLMs with AMD cards, or without a GPU at all? There’s just one weird trick: compilation. And there’s one person uniquely qualified to do it. We had the pleasure to sit down with Tianqi Chen https://www.linkedin.com/in/tianqi-chen-679a9856/, who’s an Assistant Professor at CMU, where he both teaches the MLC course https://mlc.ai/summer22/ and runs the MLC group. You might also know him as the creator of XGBoost https://www.nvidia.com/en-us/glossary/data-science/xgboost/, Apache TVM https://tvm.apache.org/, and MXNet https://mxnet.apache.org/versions/1.9.1/, as well as the co-founder of OctoML http://octoml.ai. The MLC (short for Machine Learning Compilation https://mlc.ai/) group has released a lot of interesting projects: * MLC Chat https://testflight.apple.com/join/57zd7oxa: an iPhone app that lets you run models like RedPajama-3B and Vicuna-7B on-device. It gets up to 30 tok/s! * Web LLM https://webllm.mlc.ai/: Run models like LLaMA-70B in your browser (!!) to offer local inference in your product. * MLC LLM: https://github.com/mlc-ai/mlc-llm a framework that allows any language models to be deployed natively on different hardware and software stacks. The MLC group has just announced https://twitter.com/junrushao/status/1689321439625883648?s=20 new support for AMD cards; we previously talked about the shortcomings of ROCm, but using MLC you can get performance very close to the NVIDIA’s counterparts. This is great news for founders and builders, as AMD cards are more readily available. Here are their latest results on AMD’s 7900s vs some of top NVIDIA consumer cards. If you just can’t get a GPU at all, MLC LLM also supports ARM and x86 CPU architectures as targets by leveraging LLVM. While speed performance isn’t comparable, it allows for non-time-sensitive inference to be run on commodity hardware. We also enjoyed getting a peek into TQ’s process, which involves a lot of sketching https://twitter.com/FanaHOVA/status/1689680222726283264: With all the other work going on in this space with projects like ggml https://github.com/ggerganov/ggml and Ollama https://ollama.ai/, we’re excited to see GPUs becoming less and less of an issue to get models in the hands of more people, and innovative software solutions to hardware problems! Show Notes * TQ’s Projects: * XGBoost https://www.nvidia.com/en-us/glossary/data-science/xgboost/ * Apache TVM https://tvm.apache.org/ * MXNet https://mxnet.apache.org/versions/1.9.1/ * MLC https://mlc.ai/ * OctoML https://octoml.ai/ * CMU Catalyst https://catalyst.cs.cmu.edu/ * ONNX https://github.com/onnx/onnx * GGML https://github.com/ggerganov/ggml * Mojo https://www.modular.com/mojo * WebLLM https://webllm.mlc.ai/ * RWKV https://github.com/BlinkDL/RWKV-LM * HiPPO https://arxiv.org/abs/2206.12037 * Tri Dao’s Episode https://www.latent.space/p/flashattention#details * George Hotz Episode https://www.latent.space/p/geohot#details People: * Carlos Guestrin https://guestrin.su.domains/ * Albert Gu https://twitter.com/_albertgu Timestamps * [00:00:00] Intros * [00:03:41] The creation of XGBoost and its surprising popularity * [00:06:01] Comparing tree-based models vs deep learning * [00:10:33] Overview of TVM and how it works with ONNX * [00:17:18] MLC deep dive * [00:28:10] Using int4 quantization for inference of language models * [00:30:32] Comparison of MLC to other model optimization projects * [00:35:02] Running large language models in the browser with WebLLM * [00:37:47] Integrating browser models into applications * [00:41:15] OctoAI and self-optimizing compute * [00:45:45] Lightning Round Transcript Alessio: Hey everyone, welcome to the Latent Space podcast. This is Alessio, Partner and CTO in Residence at Decibel Partners https://decibel.vc/, and I'm joined by my co-host Swyx, writer and editor of Latent Space. [00:00:20] Swyx: Okay, and we are here with Tianqi Chen, or TQ as people call him, who is assistant professor in ML computer science at CMU, Carnegie Mellon University, also helping to run Catalyst Group, also chief technologist of OctoML. You wear many hats. Are those, you know, your primary identities these days? Of course, of course. [00:00:42] Tianqi: I'm also, you know, very enthusiastic open source. So I'm also a VP and PRC member of the Apache TVM project and so on. But yeah, these are the things I've been up to so far. [00:00:53] Swyx: Yeah. So you did Apache TVM, XGBoost, and MXNet, and we can cover any of those in any amount of detail. But maybe what's one thing about you that people might not learn from your official bio or LinkedIn, you know, on the personal side? [00:01:08] Tianqi: Let me say, yeah, so normally when I do, I really love coding, even though like I'm trying to run all those things. So one thing that I keep a habit on is I try to do sketchbooks. I have a book, like real sketchbooks to draw down the design diagrams and the sketchbooks I keep sketching over the years, and now I have like three or four of them. And it's kind of a usually a fun experience of thinking the design through and also seeing how open source project evolves and also looking back at the sketches that we had in the past to say, you know, all these ideas really turn into code nowadays. [00:01:43] Alessio: How many sketchbooks did you get through to build all this stuff? I mean, if one person alone built one of those projects, he'll be a very accomplished engineer. Like you built like three of these. What's that process like for you? Like it's the sketchbook, like the start, and then you think about the code or like. [00:01:59] Swyx: Yeah. [00:02:00] Tianqi: So, so usually I start sketching on high level architectures and also in a project that works for over years, we also start to think about, you know, new directions, like of course generative AI language model comes in, how it's going to evolve. So normally I would say it takes like one book a year, roughly at that rate. It's usually fun to, I find it's much easier to sketch things out and then gives a more like a high level architectural guide for some of the future items. Yeah. [00:02:28] Swyx: Have you ever published this sketchbooks? Cause I think people would be very interested on, at least on a historical basis. Like this is the time where XGBoost was born, you know? Yeah, not really. [00:02:37] Tianqi: I started sketching like after XGBoost. So that's a kind of missing piece, but a lot of design details in TVM are actually part of the books that I try to keep a record of. [00:02:48] Swyx: Yeah, we'll try to publish them and publish something in the journals. Maybe you can grab a little snapshot for visual aid. Sounds good. [00:02:57] Alessio: Yeah. And yeah, talking about XGBoost, so a lot of people in the audience might know it's a gradient boosting library, probably the most popular out there. And it became super popular because many people started using them in like a machine learning competitions. And I think there's like a whole Wikipedia page of like all state-of-the-art models. They use XGBoost and like, it's a really long list. When you were working on it, so we just had Tri Dao, who's the creator of FlashAttention on the podcast. And I asked him this question, it's like, when you were building FlashAttention, did you know that like almost any transform race model will use it? And so I asked the same question to you when you were coming up with XGBoost, like, could you predict it would be so popular or like, what was the creation process? And when you published it, what did you expect? We have no idea. [00:03:41] Tianqi: Like, actually, the original reason that we built that library is that at that time, deep learning just came out. Like that was the time where AlexNet just came out. And one of the ambitious mission that myself and my advisor, Carlos Guestrin, then is we want to think about, you know, try to test the hypothesis. Can we find alternatives to deep learning models? Because then, you know, there are other alternatives like, you know, support vector machines, linear models, and of course, tree-based models. And our question was, if you build those models and feed them with big enough data, because usually like one of the key characteristics of deep learning is that it's taking a lot [00:04:22] Swyx: of data, right? [00:04:23] Tianqi: So we will be able to get the same amount of performance. That's a hypothesis we're setting out to test. Of course, if you look at now, right, that's a wrong hypothesis, but as a byproduct, what we find out is that, you know, most of the gradient boosting library out there is not efficient enough for us to test that hypothesis. So I happen to have quite a bit of experience in the past of building gradient boosting trees and their variants. So Effective Action Boost was kind of like a byproduct of that hypothesis testing. At that time, I'm also competing a bit in data science challenges, like I worked on KDDCup and then Kaggle kind of become bigger, right? So I kind of think maybe it's becoming useful to others. One of my friends convinced me to try to do a Python binding of it. That tends to be like a very good decision, right, to be effective. Usually when I build it, we feel like maybe a command line interface is okay. And now we have a Python binding, we have R bindings. And then it realized, you know, it started getting interesting. People started contributing different perspectives, like visualization and so on. So we started to push a bit more on to building distributive support to make sure it works on any platform and so on. And even at that time point, when I talked to Carlos, my advisor, later, he said he never anticipated that we'll get to that level of success. And actually, why I pushed for gradient boosting trees, interestingly, at that time, he also disagreed. He thinks that maybe we should go for kernel machines then. And it turns out, you know, actually, we are both wrong in some sense, and Deep Neural Network was the king in the hill. But at least the gradient boosting direction got into something fruitful. [00:06:01] Swyx: Interesting. [00:06:02] Alessio: I'm always curious when it comes to these improvements, like, what's the design process in terms of like coming up with it? And how much of it is a collaborative with like other people that you're working with versus like trying to be, you know, obviously, in academia, it's like very paper-driven kind of research driven. [00:06:19] Tianqi: I would say the extra boost improvement at that time point was more on like, you know, I'm trying to figure out, right. But it's combining lessons. Before that, I did work on some of the other libraries on matrix factorization. That was like my first open source experience. Nobody knew about it, because you'll find, likely, if you go and try to search for the package SVD feature, you'll find some SVN repo somewhere. But it's actually being used for some of the recommender system packages. So I'm trying to apply some of the previous lessons there and trying to combine them. The later projects like MXNet and then TVM is much, much more collaborative in a sense that... But, of course, extra boost has become bigger, right? So when we started that project myself, and then we have, it's really amazing to see people come in. Michael, who was a lawyer, and now he works on the AI space as well, on contributing visualizations. Now we have people from our community contributing different things. So extra boost even today, right, it's a community of committers driving the project. So it's definitely something collaborative and moving forward on getting some of the things continuously improved for our community. [00:07:37] Alessio: Let's talk a bit about TVM too, because we got a lot of things to run through in this episode. [00:07:42] Swyx: I would say that at some point, I'd love to talk about this comparison between extra boost or tree-based type AI or machine learning compared to deep learning, because I think there is a lot of interest around, I guess, merging the two disciplines, right? And we can talk more about that. I don't know where to insert that, by the way, so we can come back to it later. Yeah. [00:08:04] Tianqi: Actually, what I said, when we test the hypothesis, the hypothesis is kind of, I would say it's partially wrong, because the hypothesis we want to test now is, can you run tree-based models on image classification tasks, where deep learning is certainly a no-brainer right [00:08:17] Swyx: now today, right? [00:08:18] Tianqi: But if you try to run it on tabular data, still, you'll find that most people opt for tree-based models. And there's a reason for that, in the sense that when you are looking at tree-based models, the decision boundaries are naturally rules that you're looking at, right? And they also have nice properties, like being able to be agnostic to scale of input and be able to automatically compose features together. And I know there are attempts on building neural network models that work for tabular data, and I also sometimes follow them. I do feel like it's good to have a bit of diversity in the modeling space. Actually, when we're building TVM, we build cost models for the programs, and actually we are using XGBoost for that as well. I still think tree-based models are going to be quite relevant, because first of all, it's really to get it to work out of the box. And also, you will be able to get a bit of interoperability and control monotonicity [00:09:18] Swyx: and so on. [00:09:19] Tianqi: So yes, it's still going to be relevant. I also sometimes keep coming back to think about, are there possible improvements that we can build on top of these models? And definitely, I feel like it's a space that can have some potential in the future. [00:09:34] Swyx: Are there any current projects that you would call out as promising in terms of merging the two directions? [00:09:41] Tianqi: I think there are projects that try to bring a transformer-type model for tabular data. I don't remember specifics of them, but I think even nowadays, if you look at what people are using, tree-based models are still one of their toolkits. So I think maybe eventually it's not even a replacement, it will be just an ensemble of models that you can call. Perfect. [00:10:07] Alessio: Next up, about three years after XGBoost, you built this thing called TVM, which is now a very popular compiler framework for models. Let's talk about, so this came out about at the same time as ONNX. So I think it would be great if you could maybe give a little bit of an overview of how the two things work together. Because it's kind of like the model, then goes to ONNX, then goes to the TVM. But I think a lot of people don't understand the nuances. I can get a bit of a backstory on that. [00:10:33] Tianqi: So actually, that's kind of an ancient history. Before XGBoost, I worked on deep learning for two years or three years. I got a master's before I started my PhD. And during my master's, my thesis focused on applying convolutional restricted Boltzmann machine for ImageNet classification. That is the thing I'm working on. And that was before AlexNet moment. So effectively, I had to handcraft NVIDIA CUDA kernels on, I think, a GTX 2070 card. I have a 22070 card. It took me about six months to get one model working. And eventually, that model is not so good, and we should have picked a better model. But that was like an ancient history that really got me into this deep learning field. And of course, eventually, we find it didn't work out. So in my master's, I ended up working on recommender system, which got me a paper, and I applied and got a PhD. But I always want to come back to work on the deep learning field. So after XGBoost, I think I started to work with some folks on this particular MXNet. At that time, it was like the frameworks of CAFE, Ciano, PyTorch haven't yet come out. And we're really working hard to optimize for performance on GPUs. At that time, I found it's really hard, even for NVIDIA GPU. It took me six months. And then it's amazing to see on different hardwares how hard it is to go and optimize code for the platforms that are interesting. So that gets me thinking, can we build something more generic and automatic? So that I don't need an entire team of so many people to go and build those frameworks. So that's the motivation of starting working on TVM. There is really too little about machine learning engineering needed to support deep learning models on the platforms that we're interested in. I think it started a bit earlier than ONNX, but once it got announced, I think it's in a similar time period at that time. So overall, how it works is that TVM, you will be able to take a subset of machine learning programs that are represented in what we call a computational graph. Nowadays, we can also represent a loop-level program ingest from your machine learning models. Usually, you have model formats ONNX, or in PyTorch, they have FX Tracer that allows you to trace the FX graph. And then it goes through TVM. We also realized that, well, yes, it needs to be more customizable, so it will be able to perform some of the compilation optimizations like fusion operator together, doing smart memory planning, and more importantly, generate low-level code. So that works for NVIDIA and also is portable to other GPU backends, even non-GPU backends [00:13:36] Swyx: out there. [00:13:37] Tianqi: So that's a project that actually has been my primary focus over the past few years. And it's great to see how it started from where I think we are the very early initiator of machine learning compilation. I remember there was a visit one day, one of the students asked me, are you still working on deep learning frameworks? I tell them that I'm working on ML compilation. And they said, okay, compilation, that sounds very ancient. It sounds like a very old field. And why are you working on this? And now it's starting to get more traction, like if you say Torch Compile and other things. I'm really glad to see this field starting to pick up. And also we have to continue innovating here. [00:14:17] Alessio: I think the other thing that I noticed is, it's kind of like a big jump in terms of area of focus to go from XGBoost to TVM, it's kind of like a different part of the stack. Why did you decide to do that? And I think the other thing about compiling to different GPUs and eventually CPUs too, did you already see some of the strain that models could have just being focused on one runtime, only being on CUDA and that, and how much of that went into it? [00:14:50] Tianqi: I think it's less about trying to get impact, more about wanting to have fun. I like to hack code, I had great fun hacking CUDA code. Of course, being able to generate CUDA code is cool, right? But now, after being able to generate CUDA code, okay, by the way, you can do it on other platforms, isn't that amazing? So it's more of that attitude to get me started on this. And also, I think when we look at different researchers, myself is more like a problem solver type. So I like to look at a problem and say, okay, what kind of tools we need to solve that problem? So regardless, it could be building better models. For example, while we build extra boots, we build certain regularizations into it so that it's more robust. It also means building system optimizations, writing low-level code, maybe trying to write assembly and build compilers and so on. So as long as they solve the problem, definitely go and try to do them together. And I also see it's a common trend right now. Like if you want to be able to solve machine learning problems, it's no longer at Aggressor layer, right? You kind of need to solve it from both Aggressor data and systems angle. And this entire field of machine learning system, I think it's kind of emerging. And there's now a conference around it. And it's really good to see a lot more people are starting to look into this. [00:16:10] Swyx: Yeah. Are you talking about ICML or something else? [00:16:13] Tianqi: So machine learning and systems, right? So not only machine learning, but machine learning and system. So there's a conference called MLsys. It's definitely a smaller community than ICML, but I think it's also an emerging and growing community where people are talking about what are the implications of building systems for machine learning, right? And how do you go and optimize things around that and co-design models and systems together? [00:16:37] Swyx: Yeah. And you were area chair for ICML and NeurIPS as well. So you've just had a lot of conference and community organization experience. Is that also an important part of your work? Well, it's kind of expected for academic. [00:16:48] Tianqi: If I hold an academic job, I need to do services for the community. Okay, great. [00:16:53] Swyx: Your most recent venture in MLsys is going to the phone with MLCLLM. You announced this in April. I have it on my phone. It's great. I'm running Lama 2, Vicuña. I don't know what other models that you offer. But maybe just kind of describe your journey into MLC. And I don't know how this coincides with your work at CMU. Is that some kind of outgrowth? [00:17:18] Tianqi: I think it's more like a focused effort that we want in the area of machine learning compilation. So it's kind of related to what we built in TVM. So when we built TVM was five years ago, right? And a lot of things happened. We built the end-to-end machine learning compiler that works, the first one that works. But then we captured a lot of lessons there. So then we are building a second iteration called TVM Unity. That allows us to be able to allow ML engineers to be able to quickly capture the new model and how we demand building optimizations for them. And MLCLLM is kind of like an MLC. It's more like a vertical driven organization that we go and build tutorials and go and build projects like LLM to solutions. So that to really show like, okay, you can take machine learning compilation technology and apply it and bring something fun forward. Yeah. So yes, it runs on phones, which is really cool. But the goal here is not only making it run on phones, right? The goal is making it deploy universally. So we do run on Apple M2 Macs, the 17 billion models. Actually, on a single batch inference, more recently on CUDA, we get, I think, the most best performance you can get out there already on the 4-bit inference. Actually, as I alluded earlier before the podcast, we just had a result on AMD. And on a single batch, actually, we can get the latest AMD GPU. This is a consumer card. It can get to about 80% of the 4019, so NVIDIA's best consumer card out there. So it's not yet on par, but thinking about how diversity and what you can enable and the previous things you can get on that card, it's really amazing that what you can do with this kind of technology. [00:19:10] Swyx: So one thing I'm a little bit confused by is that most of these models are in PyTorch, but you're running this inside a TVM. I don't know. Was there any fundamental change that you needed to do, or was this basically the fundamental design of TVM? [00:19:25] Tianqi: So the idea is that, of course, it comes back to program representation, right? So effectively, TVM has this program representation called TVM script that contains more like computational graph and operational representation. So yes, initially, we do need to take a bit of effort of bringing those models onto the program representation that TVM supports. Usually, there are a mix of ways, depending on the kind of model you're looking at. For example, for vision models and stable diffusion models, usually we can just do tracing that takes PyTorch model onto TVM. That part is still being robustified so that we can bring more models in. On language model tasks, actually what we do is we directly build some of the model constructors and try to directly map from Hugging Face models. The goal is if you have a Hugging Face configuration, we will be able to bring that in and apply optimization on them. So one fun thing about model compilation is that your optimization doesn't happen only as a soft language, right? For example, if you're writing PyTorch code, you just go and try to use a better fused operator at a source code level. Torch compile might help you do a bit of things in there. In most of the model compilations, it not only happens at the beginning stage, but we also apply generic transformations in between, also through a Python API. So you can tweak some of that. So that part of optimization helps a lot of uplifting in getting both performance and also portability on the environment. And another thing that we do have is what we call universal deployment. So if you get the ML program into this TVM script format, where there are functions that takes in tensor and output tensor, we will be able to have a way to compile it. So they will be able to load the function in any of the language runtime that TVM supports. So if you could load it in JavaScript, and that's a JavaScript function that you can take in tensors and output tensors. If you're loading Python, of course, and C++ and Java. So the goal there is really bring the ML model to the language that people care about and be able to run it on a platform they like. [00:21:37] Swyx: It strikes me that I've talked to a lot of compiler people, but you don't have a traditional compiler background. You're inventing your own discipline called machine learning compilation, or MLC. Do you think that this will be a bigger field going forward? [00:21:52] Tianqi: First of all, I do work with people working on compilation as well. So we're also taking inspirations from a lot of early innovations in the field. Like for example, TVM initially, we take a lot of inspirations from Halide, which is just an image processing compiler. And of course, since then, we have evolved quite a bit to focus on the machine learning related compilations. If you look at some of our conference publications, you'll find that machine learning compilation is already kind of a subfield. So if you look at papers in both machine learning venues, the MLC conferences, of course, and also system venues, every year there will be papers around machine learning compilation. And in the compiler conference called CGO, there's a C4ML workshop that also kind of trying to focus on this area. So definitely it's already starting to gain traction and becoming a field. I wouldn't claim that I invented this field, but definitely I helped to work with a lot of folks there. And I try to bring a perspective, of course, trying to learn a lot from the compiler optimizations as well as trying to bring in knowledges in machine learning and systems together. [00:23:07] Alessio: So we had George Hotz on the podcast a few episodes ago, and he had a lot to say about AMD and their software. So when you think about TVM, are you still restricted in a way by the performance of the underlying kernel, so to speak? So if your target is like a CUDA runtime, you still get better performance, no matter like TVM kind of helps you get there, but then that level you don't take care of, right? [00:23:34] Swyx: There are two parts in here, right? [00:23:35] Tianqi: So first of all, there is the lower level runtime, like CUDA runtime. And then actually for NVIDIA, a lot of the mood came from their libraries, like Cutlass, CUDN, right? Those library optimizations. And also for specialized workloads, actually you can specialize them. Because a lot of cases you'll find that if you go and do benchmarks, it's very interesting. Like two years ago, if you try to benchmark ResNet, for example, usually the NVIDIA library [00:24:04] Swyx: gives you the best performance. [00:24:06] Tianqi: It's really hard to beat them. But as soon as you start to change the model to something, maybe a bit of a variation of ResNet, not for the traditional ImageNet detections, but for latent detection and so on, there will be some room for optimization because people sometimes overfit to benchmarks. These are people who go and optimize things, right? So people overfit the benchmarks. So that's the largest barrier, like being able to get a low level kernel libraries, right? In that sense, the goal of TVM is actually we try to have a generic layer to both, of course, leverage libraries when available, but also be able to automatically generate [00:24:45] Swyx: libraries when possible. [00:24:46] Tianqi: So in that sense, we are not restricted by the libraries that they have to offer. That's why we will be able to run Apple M2 or WebGPU where there's no library available because we are kind of like automatically generating libraries. That makes it easier to support less well-supported hardware, right? For example, WebGPU is one example. From a runtime perspective, AMD, I think before their Vulkan driver was not very well supported. Recently, they are getting good. But even before that, we'll be able to support AMD through this GPU graphics backend called Vulkan, which is not as performant, but it gives you a decent portability across those [00:25:29] Swyx: hardware. [00:25:29] Alessio: And I know we got other MLC stuff to talk about, like WebLLM, but I want to wrap up on the optimization that you're doing. So there's kind of four core things, right? Kernel fusion, which we talked a bit about in the flash attention episode and the tiny grab one memory planning and loop optimization. I think those are like pretty, you know, self-explanatory. I think the one that people have the most questions, can you can you quickly explain [00:25:53] Swyx: those? [00:25:54] Tianqi: So there are kind of a different things, right? Kernel fusion means that, you know, if you have an operator like Convolutions or in the case of a transformer like MOP, you have other operators that follow that, right? You don't want to launch two GPU kernels. You want to be able to put them together in a smart way, right? And as a memory planning, it's more about, you know, hey, if you run like Python code, every time when you generate a new array, you are effectively allocating a new piece of memory, right? Of course, PyTorch and other frameworks try to optimize for you. So there is a smart memory allocator behind the scene. But actually, in a lot of cases, it's much better to statically allocate and plan everything ahead of time. And that's where like a compiler can come in. We need to, first of all, actually for language model, it's much harder because dynamic shape. So you need to be able to what we call symbolic shape tracing. So we have like a symbolic variable that tells you like the shape of the first tensor is n by 12. And the shape of the third tensor is also n by 12. Or maybe it's n times 2 by 12. Although you don't know what n is, right? But you will be able to know that relation and be able to use that to reason about like fusion and other decisions. So besides this, I think loop transformation is quite important. And it's actually non-traditional. Originally, if you simply write a code and you want to get a performance, it's very hard. For example, you know, if you write a matrix multiplier, the simplest thing you can do is you do for i, j, k, c, i, j, plus, equal, you know, a, i, k, times b, i, k. But that code is 100 times slower than the best available code that you can get. So we do a lot of transformation, like being able to take the original code, trying to put things into shared memory, and making use of tensor calls, making use of memory copies, and all this. Actually, all these things, we also realize that, you know, we cannot do all of them. So we also make the ML compilation framework as a Python package, so that people will be able to continuously improve that part of engineering in a more transparent way. So we find that's very useful, actually, for us to be able to get good performance very quickly on some of the new models. Like when Lamato came out, we'll be able to go and look at the whole, here's the bottleneck, and we can go and optimize those. [00:28:10] Alessio: And then the fourth one being weight quantization. So everybody wants to know about that. And just to give people an idea of the memory saving, if you're doing FB32, it's like four bytes per parameter. Int8 is like one byte per parameter. So you can really shrink down the memory footprint. What are some of the trade-offs there? How do you figure out what the right target is? And what are the precision trade-offs, too? [00:28:37] Tianqi: Right now, a lot of people also mostly use int4 now for language models. So that really shrinks things down a lot. And more recently, actually, we started to think that, at least in MOC, we don't want to have a strong opinion on what kind of quantization we want to bring, because there are so many researchers in the field. So what we can do is we can allow developers to customize the quantization they want, but we still bring the optimum code for them. So we are working on this item called bring your own quantization. In fact, hopefully MOC will be able to support more quantization formats. And definitely, I think there's an open field that's being explored. Can you bring more sparsities? Can you quantize activations as much as possible, and so on? And it's going to be something that's going to be relevant for quite a while. [00:29:27] Swyx: You mentioned something I wanted to double back on, which is most people use int4 for language models. This is actually not obvious to me. Are you talking about the GGML type people, or even the researchers who are training the models also using int4? [00:29:40] Tianqi: Sorry, so I'm mainly talking about inference, not training, right? So when you're doing training, of course, int4 is harder, right? Maybe you could do some form of mixed type precision for inference. I think int4 is kind of like, in a lot of cases, you will be able to get away with int4. And actually, that does bring a lot of savings in terms of the memory overhead, and so on. [00:30:09] Alessio: Yeah, that's great. Let's talk a bit about maybe the GGML, then there's Mojo. How should people think about MLC? How do all these things play together? I think GGML is focused on model level re-implementation and improvements. Mojo is a language, super sad. You're more at the compiler level. Do you all work together? Do people choose between them? [00:30:32] Tianqi: So I think in this case, I think it's great to say the ecosystem becomes so rich with so many different ways. So in our case, GGML is more like you're implementing something from scratch in C, right? So that gives you the ability to go and customize each of a particular hardware backend. But then you will need to write from CUDA kernels, and you write optimally from AMD, and so on. So the kind of engineering effort is a bit more broadened in that sense. Mojo, I have not looked at specific details yet. I think it's good to start to say, it's a language, right? I believe there will also be machine learning compilation technologies behind it. So it's good to say, interesting place in there. In the case of MLC, our case is that we do not want to have an opinion on how, where, which language people want to develop, deploy, and so on. And we also realize that actually there are two phases. We want to be able to develop and optimize your model. By optimization, I mean, really bring in the best CUDA kernels and do some of the machine learning engineering in there. And then there's a phase where you want to deploy it as a part of the app. So if you look at the space, you'll find that GGML is more like, I'm going to develop and optimize in the C language, right? And then most of the low-level languages they have. And Mojo is that you want to develop and optimize in Mojo, right? And you deploy in Mojo. In fact, that's the philosophy they want to push for. In the ML case, we find that actually if you want to develop models, the machine learning community likes Python. Python is a language that you should focus on. So in the case of MLC, we really want to be able to enable, not only be able to just define your model in Python, that's very common, right? But also do ML optimization, like engineering optimization, CUDA kernel optimization, memory planning, all those things in Python that makes you customizable and so on. But when you do deployment, we realize that people want a bit of a universal flavor. If you are a web developer, you want JavaScript, right? If you're maybe an embedded system person, maybe you would prefer C++ or C or Rust. And people sometimes do like Python in a lot of cases. So in the case of MLC, we really want to have this vision of, you optimize, build a generic optimization in Python, then you deploy that universally onto the environments that people like. [00:32:54] Swyx: That's a great perspective and comparison, I guess. One thing I wanted to make sure that we cover is that I think you are one of these emerging set of academics that also very much focus on your artifacts of delivery. Of course. Something we talked about for three years, that he was very focused on his GitHub. And obviously you treated XGBoost like a product, you know? And then now you're publishing an iPhone app. Okay. Yeah. Yeah. What is his thinking about academics getting involved in shipping products? [00:33:24] Tianqi: I think there are different ways of making impact, right? Definitely, you know, there are academics that are writing papers and building insights for people so that people can build product on top of them. In my case, I think the particular field I'm working on, machine learning systems, I feel like really we need to be able to get it to the hand of people so that really we see the problem, right? And we show that we can solve a problem. And it's a different way of making impact. And there are academics that are doing similar things. Like, you know, if you look at some of the people from Berkeley, right? A few years, they will come up with big open source projects. Certainly, I think it's just a healthy ecosystem to have different ways of making impacts. And I feel like really be able to do open source and work with open source community is really rewarding because we have a real problem to work on when we build our research. Actually, those research bring together and people will be able to make use of them. And we also start to see interesting research challenges that we wouldn't otherwise say, right, if you're just trying to do a prototype and so on. So I feel like it's something that is one interesting way of making impact, making contributions. [00:34:40] Swyx: Yeah, you definitely have a lot of impact there. And having experience publishing Mac stuff before, the Apple App Store is no joke. It is the hardest compilation, human compilation effort. So one thing that we definitely wanted to cover is running in the browser. You have a 70 billion parameter model running in the browser. That's right. Can you just talk about how? Yeah, of course. [00:35:02] Tianqi: So I think that there are a few elements that need to come in, right? First of all, you know, we do need a MacBook, the latest one, like M2 Max, because you need the memory to be big enough to cover that. So for a 70 million model, it takes you about, I think, 50 gigahertz of RAM. So the M2 Max, the upper version, will be able to run it, right? And it also leverages machine learning compilation. Again, what we are doing is the same, whether it's running on iPhone, on server cloud GPUs, on AMDs, or on MacBook, we all go through that same MOC pipeline. Of course, in certain cases, maybe we'll do a bit of customization iteration for either ones. And then it runs on the browser runtime, this package of WebLM. So that will effectively... So what we do is we will take that original model and compile to what we call WebGPU. And then the WebLM will be to pick it up. And the WebGPU is this latest GPU technology that major browsers are shipping right now. So you can get it in Chrome for them already. It allows you to be able to access your native GPUs from a browser. And then effectively, that language model is just invoking the WebGPU kernels through there. So actually, when the LATMAR2 came out, initially, we asked the question about, can you run 17 billion on a MacBook? That was the question we're asking. So first, we actually... Jin Lu, who is the engineer pushing this, he got 17 billion on a MacBook. We had a CLI version. So in MLC, you will be able to... That runs through a metal accelerator. So effectively, you use the metal programming language to get the GPU acceleration. So we find, okay, it works for the MacBook. Then we asked, we had a WebGPU backend. Why not try it there? So we just tried it out. And it's really amazing to see everything up and running. And actually, it runs smoothly in that case. So I do think there are some kind of interesting use cases already in this, because everybody has a browser. You don't need to install anything. I think it doesn't make sense yet to really run a 17 billion model on a browser, because you kind of need to be able to download the weight and so on. But I think we're getting there. Effectively, the most powerful models you will be able to run on a consumer device. It's kind of really amazing. And also, in a lot of cases, there might be use cases. For example, if I'm going to build a chatbot that I talk to it and answer questions, maybe some of the components, like the voice to text, could run on the client side. And so there are a lot of possibilities of being able to have something hybrid that contains the edge component or something that runs on a server. [00:37:47] Alessio: Do these browser models have a way for applications to hook into them? So if I'm using, say, you can use OpenAI or you can use the local model. Of course. [00:37:56] Tianqi: Right now, actually, we are building... So there's an NPM package called WebILM, right? So that you will be able to, if you want to embed it onto your web app, you will be able to directly depend on WebILM and you will be able to use it. We are also having a REST API that's OpenAI compatible. So that REST API, I think, right now, it's actually running on native backend. So that if a CUDA server is faster to run on native backend. But also we have a WebGPU version of it that you can go and run. So yeah, we do want to be able to have easier integrations with existing applications. And OpenAI API is certainly one way to do that. Yeah, this is great. [00:38:37] Swyx: I actually did not know there's an NPM package that makes it very, very easy to try out and use. I want to actually... One thing I'm unclear about is the chronology. Because as far as I know, Chrome shipped WebGPU the same time that you shipped WebILM. Okay, yeah. So did you have some kind of secret chat with Chrome? [00:38:57] Tianqi: The good news is that Chrome is doing a very good job of trying to have early release. So although the official shipment of the Chrome WebGPU is the same time as WebILM, actually, you will be able to try out WebGPU technology in Chrome. There is an unstable version called Canary. I think as early as two years ago, there was a WebGPU version. Of course, it's getting better. So we had a TVM-based WebGPU backhand two years ago. Of course, at that time, there were no language models. It was running on less interesting, well, still quite interesting models. And then this year, we really started to see it getting matured and performance keeping up. So we have a more serious push of bringing the language model compatible runtime onto the WebGPU. [00:39:45] Swyx: I think you agree that the hardest part is the model download. Has there been conversations about a one-time model download and sharing between all the apps that might use this API? That is a great point. [00:39:58] Tianqi: I think it's already supported in some sense. When we download the model, WebILM will cache it onto a special Chrome cache. So if a different web app uses the same WebILM JavaScript package, you don't need to redownload the model again. So there is already something there. But of course, you have to download the model once at least to be able to use it. [00:40:19] Swyx: Okay. One more thing just in general before we're about to zoom out to OctoAI. Just the last question is, you're not the only project working on, I guess, local models. That's right. Alternative models. There's gpt4all, there's olama that just recently came out, and there's a bunch of these. What would be your advice to them on what's a valuable problem to work on? And what is just thin wrappers around ggml? Like, what are the interesting problems in this space, basically? [00:40:45] Tianqi: I think making API better is certainly something useful, right? In general, one thing that we do try to push very hard on is this idea of easier universal deployment. So we are also looking forward to actually have more integration with MOC. That's why we're trying to build API like WebILM and other things. So we're also looking forward to collaborate with all those ecosystems and working support to bring in models more universally and be able to also keep up the best performance when possible in a more push-button way. [00:41:15] Alessio: So as we mentioned in the beginning, you're also the co-founder of Octomel. Recently, Octomel released OctoAI, which is a compute service, basically focuses on optimizing model runtimes and acceleration and compilation. What has been the evolution there? So Octo started as kind of like a traditional MLOps tool, where people were building their own models and you help them on that side. And then it seems like now most of the market is shifting to starting from pre-trained generative models. Yeah, what has been that experience for you and what you've seen the market evolve? And how did you decide to release OctoAI? [00:41:52] Tianqi: One thing that we found out is that on one hand, it's really easy to go and get something up and running, right? So if you start to consider there's so many possible availabilities and scalability issues and even integration issues since becoming kind of interesting and complicated. So we really want to make sure to help people to get that part easy, right? And now a lot of things, if we look at the customers we talk to and the market, certainly generative AI is something that is very interesting. So that is something that we really hope to help elevate. And also building on top of technology we build to enable things like portability across hardwares. And you will be able to not worry about the specific details, right? Just focus on getting the model out. We'll try to work on infrastructure and other things that helps on the other end. [00:42:45] Alessio: And when it comes to getting optimization on the runtime, I see when we run an early adopters community and most enterprises issue is how to actually run these models. Do you see that as one of the big bottlenecks now? I think a few years ago it was like, well, we don't have a lot of machine learning talent. We cannot develop our own models. Versus now it's like, there's these great models you can use, but I don't know how to run them efficiently. [00:43:12] Tianqi: That depends on how you define by running, right? On one hand, it's easy to download your MLC, like you download it, you run on a laptop, but then there's also different decisions, right? What if you are trying to serve a larger user request? What if that request changes? What if the availability of hardware changes? Right now it's really hard to get the latest hardware on media, unfortunately, because everybody's trying to work on the things using the hardware that's out there. So I think when the definition of run changes, there are a lot more questions around things. And also in a lot of cases, it's not only about running models, it's also about being able to solve problems around them. How do you manage your model locations and how do you make sure that you get your model close to your execution environment more efficiently? So definitely a lot of engineering challenges out there. That we hope to elevate, yeah. And also, if you think about our future, definitely I feel like right now the technology, given the technology and the kind of hardware availability we have today, we will need to make use of all the possible hardware available out there. That will include a mechanism for cutting down costs, bringing something to the edge and cloud in a more natural way. So I feel like still this is a very early stage of where we are, but it's already good to see a lot of interesting progress. [00:44:35] Alessio: Yeah, that's awesome. I would love, I don't know how much we're going to go in depth into it, but what does it take to actually abstract all of this from the end user? You know, like they don't need to know what GPUs you run, what cloud you're running them on. You take all of that away. What was that like as an engineering challenge? [00:44:51] Tianqi: So I think that there are engineering challenges on. In fact, first of all, you will need to be able to support all the kind of hardware backhand you have, right? On one hand, if you look at the media library, you'll find very surprisingly, not too surprisingly, most of the latest libraries works well on the latest GPU. But there are other GPUs out there in the cloud as well. So certainly being able to have know-hows and being able to do model optimization is one thing, right? Also infrastructures on being able to scale things up, locate models. And in a lot of cases, we do find that on typical models, it also requires kind of vertical iterations. So it's not about, you know, build a silver bullet and that silver bullet is going to solve all the problems. It's more about, you know, we're building a product, we'll work with the users and we find out there are interesting opportunities in a certain point. And when our engineer will go and solve that, and it will automatically reflect it in a service. [00:45:45] Swyx: Awesome. [00:45:46] Alessio: We can jump into the lightning round until, I don't know, Sean, if you have more questions or TQ, if you have more stuff you wanted to talk about that we didn't get a chance to [00:45:54] Swyx: touch on. [00:45:54] Alessio: Yeah, we have talked a lot. [00:45:55] Swyx: So, yeah. We always would like to ask, you know, do you have a commentary on other parts of AI and ML that is interesting to you? [00:46:03] Tianqi: So right now, I think one thing that we are really pushing hard for is this question about how far can we bring open source, right? I'm kind of like a hacker and I really like to put things together. So I think it's unclear in the future of what the future of AI looks like. On one hand, it could be possible that, you know, you just have a few big players, you just try to talk to those bigger language models and that can do everything, right? On the other hand, one of the things that Wailing Academic is really excited and pushing for, that's one reason why I'm pushing for MLC, is that can we build something where you have different models? You have personal models that know the best movie you like, but you also have bigger models that maybe know more, and you get those models to interact with each other, right? And be able to have a wide ecosystem of AI agents that helps each person while still being able to do things like personalization. Some of them can run locally, some of them, of course, running on a cloud, and how do they interact with each other? So I think that is a very exciting time where the future is yet undecided, but I feel like there is something we can do to shape that future as well. [00:47:18] Swyx: One more thing, which is something I'm also pursuing, which is, and this kind of goes back into predictions, but also back in your history, do you have any idea, or are you looking out for anything post-transformers as far as architecture is concerned? [00:47:32] Tianqi: I think, you know, in a lot of these cases, you can find there are already promising models for long contexts, right? There are space-based models, where like, you know, a lot of some of our colleagues from Albert, who he worked on this HIPPO models, right? And then there is an open source version called RWKV. It's like a recurrent models that allows you to summarize things. Actually, we are bringing RWKV to MOC as well, so maybe you will be able to see one of the models. [00:48:00] Swyx: We actually recorded an episode with one of the RWKV core members. It's unclear because there's no academic backing. It's just open source people. Oh, I see. So you like the merging of recurrent networks and transformers? [00:48:13] Tianqi: I do love to see this model space continue growing, right? And I feel like in a lot of cases, it's just that attention mechanism is getting changed in some sense. So I feel like definitely there are still a lot of things to be explored here. And that is also one reason why we want to keep pushing machine learning compilation, because one of the things we are trying to push in was productivity. So that for machine learning engineering, so that as soon as some of the models came out, we will be able to, you know, empower them onto those environments that's out there. [00:48:43] Swyx: Yeah, it's a really good mission. Okay. Very excited to see that RWKV and state space model stuff. I'm hearing increasing chatter about that stuff. Okay. Lightning round, as always fun. I'll take the first one. Acceleration. What has already happened in AI that you thought would take much longer? [00:48:59] Tianqi: Emergence of more like a conversation chatbot ability is something that kind of surprised me before it came out. This is like one piece that I feel originally I thought would take much longer, but yeah, [00:49:11] Swyx: it happens. And it's funny because like the original, like Eliza chatbot was something that goes all the way back in time. Right. And then we just suddenly came back again. Yeah. [00:49:21] Tianqi: It's always too interesting to think about, but with a kind of a different technology [00:49:25] Swyx: in some sense. [00:49:25] Alessio: What about the most interesting unsolved question in AI? [00:49:31] Swyx: That's a hard one, right? [00:49:32] Tianqi: So I can tell you like what kind of I'm excited about. So, so I think that I have always been excited about this idea of continuous learning and lifelong learning in some sense. So how AI continues to evolve with the knowledges that have been there. It seems that we're getting much closer with all those recent technologies. So being able to develop systems, support, and be able to think about how AI continues to evolve is something that I'm really excited about. [00:50:01] Swyx: So specifically, just to double click on this, are you talking about continuous training? That's like a training. [00:50:06] Tianqi: I feel like, you know, training adaptation and it's all similar things, right? You want to think about entire life cycle, right? The life cycle of collecting data, training, fine tuning, and maybe have your local context that getting continuously curated and feed onto models. So I think all these things are interesting and relevant in here. [00:50:29] Swyx: Yeah. I think this is something that people are really asking, you know, right now we have moved a lot into the sort of pre-training phase and off the shelf, you know, the model downloads and stuff like that, which seems very counterintuitive compared to the continuous training paradigm that people want. So I guess the last question would be for takeaways. What's basically one message that you want every listener, every person to remember today? [00:50:54] Tianqi: I think it's getting more obvious now, but I think one of the things that I always want to mention in my talks is that, you know, when you're thinking about AI applications, originally people think about algorithms a lot more, right? Our algorithm models, they are still very important. But usually when you build AI applications, it takes, you know, both algorithm side, the system optimizations, and the data curations, right? So it takes a connection of so many facades to be able to bring together an AI system and be able to look at it from that holistic perspective is really useful when we start to build modern applications. I think it's going to continue going to be more important in the future. [00:51:35] Swyx: Yeah. Thank you for showing the way on this. And honestly, just making things possible that I thought would take a lot longer. So thanks for everything you've done. [00:51:46] Tianqi: Thank you for having me. [00:51:47] Swyx: Yeah. [00:51:47] Alessio: Thanks for coming on TQ. [00:51:49] Swyx: Have a good one. [00:51:49] This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:09Alessio Fanelli?

  Hey everyone, welcome to the Late in Space podcast. This is Alessio, Partner and CTO at Resident Decibel Partners, and I'm joined by my co-host Swyx, Writer and Editor of Late in Space.

• 0:20Speaker 01

  Hey, and we are here with Tianqi Chen, or TQ as people call him, who is Assistant Professor in ML Computer Science at CMU, Carnegie Mellon University, also helping to run Catalyst Group, also Chief Technologist of OTO ML. You wear many hats. Are those your primary identities these days?

• 0:41Speaker 02

  Of course, of course. I'm also a very enthusiastic open source, so I'm also a VP and PRC member of the Apache TVM project and so on. But yeah, these are the things I've been up to so far, yeah.

• 0:53Speaker 01

  You also created Apache TVM, XG Boost and MXNet. And we could cover any of those in any amount of detail. But maybe what's one thing about you that people might not learn from your official bio or LinkedIn on the personal side?

• 1:07Speaker 02

  Let me say, yeah, so normally, I really love coding, even though I'm trying to run all those things. So one thing that I keep a habit on is I try to do sketchbooks. I have a book, like real sketchbooks, to draw down the design diagrams. And that sketchbooks, I keep sketching over the years, and now I have like three or four of them. And it's kind of usually a fun experience of syncing the design through and also seeing how open source project evolves and also looking back at the sketches that we had in the past to say, you know, oh, these ideas really turn into code nowadays.

• 1:43Alessio Fanelli?

  How many sketchbooks did you get through to build all this stuff? I mean, if one person alone built one of those projects, it would be a very accomplished engineer, like you built like three of these. What's that process like for you? Like, is the sketchbook like the start? And then you think about the code or like?

• 1:59Speaker 02

  Yeah, so usually I start sketching on high-level architectures and also in a project that works for over years, we also start to think about, you know, new directions, like of course generative AI language model comes in, how it's going to evolve. So normally, I would say it takes like one book a year, roughly at that rate. It's usually fun to, I find it's much easier to sketch things out and then give the more like a high-level architectural guide for some of the future items, yeah.

• 2:28Speaker 01

  Have you ever published this sketchbooks? Because I think people would be very interested, at least on a historical basis, like this is the time where XGBoost was born, you know?

• 2:36Speaker 02

  Yeah, not really. I started sketching like after XGBoost, so that's a kind of missing piece. But a lot of the design details in TBM are actually part of the books that I try to keep a record of.

• 2:49Speaker 01

  Yeah, we'll try to publish them and publish something in the show notes. Maybe you can grab a little snapshot for visual aid.

• 2:56Speaker 02

  Sounds good, yeah.

• 2:57Alessio Fanelli?

  And yeah, talking about XGBoost, so a lot of people in the audience might know it's a gradient boosting library, probably the most popular out there. And it became super popular because many people started using them in like a machine learning competitions. And I think there's like a whole Wikipedia page of like all state-of-the-art models that use XGBoost. And like, it's a really long list. When you were working on it, so we just had TreeDow, who's the creator of a flash attention on the podcast. And I asked him this question, it's like, when you were building flash attention, did you know that like almost any transformer-based model will use it? And so I asked the same question to you when you were coming up with XGBoost, like, could you predict it would be so popular? Or like, what was the creation process? And when you published it, what did you expect?

• 3:40Speaker 02

  We have no idea. Like, actually, the original reason that we built that library is that at that time, deep learning just came out, like that was the time where AlexNet just came out. And one of the ambitious mission that myself and my advisor, Cal Scestran, then is we want to think about, you know, try to test the hypothesis, can we find alternatives to deep learning models? Because then, you know, there are other alternatives like, you know, support vector machines, linear And models and of course tree based models. And our question was if you build those models with and feed them with big enough data. Because usually like one of the key characteristics of a deep learning is that are taking a lot of data, right?

• 4:22Speaker 02

  So we will be able to get the same amount of performance. That's a hypothesis we're setting on our test, of course. If you look at now, right, that's a wrong hypothesis. But as a byproduct, what we find out is that, you know, most of the gradient boosting library out there is not efficient enough. For us to test that hypothesis, so so I happen to had quite a bit of experience of, you know, parts of building gradient boosting trees. Uh, and their variants, so so effectively actually was was kind of like a a byproduct of that hypothesis studying, testing. At that time, I'm also competing a bit in data science challenges like I I worked on K D D cup. And then Kaggle come become bigger, right? So I kind of think maybe it's becoming useful to others. One of my friends convinced me to try to do a Python binding of it.

• 5:07Speaker 02

  That tends to be like a very good decision, right? To effectively, usually when I build it, we we feel like, you know, maybe a command line interface is okay. And then we had a Python binding, we have our bindings. And then it realized, you know, it start to getting interesting. People started contributing different perspective like visualization and so on. So we we start to, you know, push a bit more on to, you know, building distributed support. To make sure it works on any more platform and so on. And even at that time point, like when I talked to Carlos, my advisor later. He's like, he never anticipated that we'll get to that level of success. And actually, why why I pushed for gradient boosting trees? Interestingly, at that time, he also disagreed. He he thinks that maybe we should go for kernel machines then.

• 5:51Speaker 02

  And it turns out, you know, actually, we are both wrong in some sense and deep neural network was the king in the hill. But at least the gradient boosting direction getting to something fruitful.

• 6:01Alessio Fanelli?

  Interesting. I'm always curious when it comes to these improvements, like what's the design process in terms of like coming up with it? And how much of it is a collaborative with like other people that you're working with versus like trying to be, you know, obviously in academia is like very paper driven, kind of research driven.

• 6:19Speaker 02

  I would say the XGBoost improvement at that time point was more on like, you know, I'm trying to figure out, right? But but it's combining lessons. Before that, I did work on some of the other libraries on like on matrix vectorization. That was like my first open source experience. Nobody know about it. Because you will find likely if you go and try to search for the package for SVD feature, you'll find some like a SVN repo that's somewhere. It's actually being used for some of recommender system packages. So I'm trying to apply some of previous lessons there and trying to combine them. The later projects like MXNet and then TVM is much, much more collaborative, you know, in a sense that.

• 7:03Speaker 02

  But of course, XGBoost has become bigger, right? So we, when we started that project, it's myself and then we have, it's really amazing to see people coming in. Like we have Michael, who was a lawyer and now he's he works on the AI space as well, on contributing visualizations. And then we have people from our community contributing different things. So it's actually was even today, right? This community of committers driving the project. So it's definitely something collaborative. And moving forward on getting some of the things continuously improved for our community.

• 7:37Alessio Fanelli?

  Let's talk a bit about TVM too, because we got a lot of things to run through in this episode.

• 7:42Speaker 01

  Well, if I did at some point, I'd love to talk about this comparison between sort of XGBoost or sort of tree based type AI or machine learning compared to deep learning. Because I think there is some, a lot of interest around, I guess, merging the two disciplines, right? And we can talk, talk more about that. I don't know where to insert that, by the way.

• 8:01Speaker 02

  Yeah, no. Yeah, actually, why I said, you know, you know, when we test hypotheses, the hypothesis is kind of, I would say it's partially longer, because the hypothesis we want to test now is, you know, can you run tree based models on image classification task where deep learning is certainly the no brainer right now today, right? But if you try to run it on tabular data. Still, like you'll find that most people opt for tree-based models. And that's the reason for that, in a sense that, you know, when you are looking at tree-based models, the decision boundaries are naturally rules that you're looking at, right? And they also have nice properties like, you know, being able to be agnostic to scale of input and be able to automatically compose feature together. And I know there are attempts, there are attempts on your building

• 8:46Speaker 02

  neural network models that works for tabular data, and I also sometimes follow them. I do feel like it's good to have a bit of diversity in the modeling space. XGBoost, actually, when we're building TBM, we build cost models for the programs. Actually, we are using XGBoost for that as well. I still think tree-based model is going to be quite relevant because, first of all, it's really to get it work out of box. And also, you will be able to get a bit of interoperability and control like monotonicity and so on. So, yes, it's still going to be relevant. I also sometimes keep coming back to think about, are there possible improvements that we can build on top of these models?

• 9:28Speaker 02

  And definitely, I feel like it's a space that can have some of a potential in the future.

• 9:34Speaker 01

  Are there any current projects that you would call out as promising in terms of merging the two, I guess, directions?

• 9:41Speaker 02

  I think there are projects that tries to bring like transformer-type model for tabular data, right? I don't remember specifics of them, but I think even nowadays, if you look at the peoples, what people are using, like tree-based models do, one of their toolkit, right? So I think maybe eventually it's not even a replacement. It will be just an ensemble of models that you can call, yeah.

• 10:07Speaker 01

  Perfect.

• 10:07Alessio Fanelli?

  Next step, about three years after XGBoost, you built this thing called TBM, which is now a very popular compiler framework for models. Let's talk about, so this came out about at the same time as Onyx. So I think it would be great if you could maybe give a little bit of an overview of like, how do two things work together? Because it's kind of like the model then goes to ONNX, then goes to the TBM. But I think a lot of people don't understand that, then wants this.

• 10:31Speaker 02

  I can get a bit of backstory on that. So actually, that's kind of an ancient history. Before XGBoost, I worked on deep learning for like two years or three years. I get a master before I start my PhD. And during my master, my thesis focused on, you know, applying convolutional restricted Boltzmann machine for image net classification like that. That is the thing I'm working on. And that was before AlexNet moment. So effectively, I had to handcraft Nvidia CUDA kernels on, I think it's GTX 2070 card. I have a 22070 card. It took me about six months to get one model working.

• 11:14Speaker 02

  And eventually, that model is not so good. And we should have picked a better model. But that was like an ancient history that when I really got me into this deep learning field. And of course, eventually, we find, you know, it didn't work out. So I, in my master, I ended up working on recommend assistant, which got me a paper. And I applied and got a PhD. But I always want to come back to work on the deep learning field. So after XGBoost, I think I started to work with some of folks on this particular MXNet. That at that time, it was like the frameworks are Caffe, Sciano, Torch. PyTorch haven't yet come out.

• 11:54Speaker 02

  And we're really working hard to optimize for performance, right, on GPUs. At that time, I find, you know, it's really hard to, even for Nvidia GPU, right? It took me six months. And then it's amazing to see like, you know, on different hardware, how hard it is to go and optimize code for the platforms that are interesting. So that gets me thinking like, you know, can we build something more generic and automatic? So that, you know, I don't need an entire team of so many people to go and build those frameworks. So that's the motivation of starting working on TVM. Their risk really to lower the bar of machine learning engineering needed.

• 12:36Speaker 02

  To support deep learning models on the platforms that we're interested in. Yeah, so I think it started a bit earlier than on X, but why it's got announced? I think it's in a similar time period, right? At that time. So yeah, so overall, how it works is that, you know, TVM, you will be able to take a subset of machine learning program that are represented in what we call a computational graph. Nowadays, we can also represent a loop level program. We ingest from your machine learning models. Usually, you know, you have model formats on X, right? Or in PyTorch, they have like FX tracer that allows you to trace the FX graph. And then it goes through TVM. We also realized that, well, yes, it's need to be more customizable.

• 13:18Speaker 02

  So we will be able to perform some of the compilation optimizations, like fusion operator together between smart memory plannings and more importantly, generate low level code. So that, you know, works for Nvidia and also is portable to other GPU backends, even non GPU backends out there. Yeah, so that's a project actually has been my primary focus over the past few years. And it's great to see, you know, how it started from where I think we are the early, very early initiator of machine learning compilation. I remember I was visiting, one of the student asked me, are you still working on deep learning frameworks? I tell them that, you know, I'm working on MO compilation. And they said, okay, compilation, that sounds very ancient. You know, it sounds like a very old field.

• 14:04Speaker 02

  And why are you working on this? And now it's starting to getting more traction, like if you say Torch compile and other things. I'm really glad to see this field starting picking up. And also we have to continue innovating here.

• 14:17Alessio Fanelli?

  I think the other thing that I noticed is, you know, it's kind of like a big jump in terms of area of focus to go from like X-ray boost to like TVM. You know, it's kind of like a different part of the of the stack. Why did you decide to do that? And I think the other thing about compiling to like different GPUs and eventually CPUs too. Did you already see kind of like the some of the strain that models could have? Like just being focused on one runtime, you know, only being on cooldown like that. And yeah, how much of that went into it?

• 14:49Speaker 02

  I think it's less about trying to get impact, more about, you know, wanted to have fun. Like I like to hack code. I had great fun hacking CUDA code. And of course, being able to generate CUDA code is cool, right? But but you know, now, after being able to generate CUDA code, okay, by the way, you can do it on other platforms. Isn't that amazing? So it's more of that attitude to get me started on this. And also, I think, when we look at different researchers, myself is more like, I think, a problem solver type. So I like to look at a problem and say, you know, okay, what kind of tools we need to solve that problem? So regardless, you know, it could be building better models. For example, when we build action boost, we build certain regularizations into it so that it's more robust.

• 15:33Speaker 02

  It also means building system optimizations, writing low level code. Maybe, you know, trying to write assembly and build compilers and so on. So so as long as they solve the problem, definitely go and try to do them together. And I also see it's a common trend right now, like if you want to be able to solve a machine learning problems, it's no longer at Aggresson layer, right? You kind of need to solve it from both Aggresson data and systems angle. And this entire field of machine learning system, I think, is kind of emerging. And that's not a conference around it. And it's really good to see a lot more people are starting to look into this.

• 16:11Speaker 01

  Are you talking about ICML or something else?

• 16:13Speaker 02

  So machine learning and systems, right? So so not only machine learning, but machine learning and system. So that's a conference called MLsys. It's a definitely smaller community than CMO, but but it's I think it's also also emerging, growing community where, you know, people are talking about, you know, what's implications of building system for machine learning, right? And how do you go and optimize things around that and code design models and system together?

• 16:37Speaker 01

  Yeah, and you were area chair for ICML in Europe as well. So you've just had a lot of conference and community organization experience. Is that also like an important part of your work?

• 16:46Speaker 02

  Well, it's kind of expected for academic, if you have a academic job, right? You need to do services for the community.

• 16:52Speaker 01

  Okay, great. Your most recent venture in ML-CIS is going to the phone with MLCLM. You announced this in April. I have it on my phone. It's great. I'm running Llama 2, Vicunia. I don't know what other models you offer. But maybe just kind of describe your journey into MLC. And I don't know how this coincides with your work at CMU. Is that some kind of outgrowth?

• 17:18Speaker 02

  I think it's more like a focused effort that we want in the area of machine learning compilation. Right? So it's kind of related to what we built in TVM. So when we built TVM was five years ago, right? And a lot of things happened. We build end-to-end machine learning compiler that works, the first one that works. But then we capture a lot of lessons there. So then we are building a second iteration called TVM Unity that allows us to be able to, you know, allow MO engineers to be able to quickly capture the new model and how our demand and building optimizations for them. And then MLCLM is kind of like, and MLC is more like a vertical-driven organization

• 17:58Speaker 02

  that we go and build tutorials and go and build projects like LLM to solutions. So that to really show like, okay, you can take machine learning compilation technology and apply it and bring something fun forward. Yeah. So yes, it runs on phones, which is really cool. But the goal here is not only making it run on phones, right? The goal is making it deploy universally. So we do run on, you know, Apple M2 Max, the 7 billion, 17 billion models. Actually, on a single batch inference, more recently on CUDA, we get, I think, the most best performance we can get out there already on the four bit inference. And then actually, as I alluded earlier before the podcast, we just had a result on AMD.

• 18:43Speaker 02

  And on single batch, actually, we can get the latest AMD GPU. It is a consumer card. It can get to about eighty percent of the forty nineteen, so media's best consumer card out there. So it's not yet like on par, but thinking about how our diversity and what you can enable. And the previous things you can get on that card is really amazing that what you can do with this kind of technology.

• 19:10Speaker 01

  So one thing I'm a little bit confused by is that most of these models are in PyTorch, but you're running this inside of TVM. I don't know, was there any fundamental change that you needed to do?

• 19:21Speaker 02

  Yeah.

• 19:22Speaker 01

  Was this basically the fundamental design of TVM?

• 19:25Speaker 02

  So the idea is that, of course, it comes back to program representation. So effectively, TVM have this program representation called TVM script that contains more like computational graph and operational representation. So yes, initially, we do need to take a bit of effort of bringing those models onto the program representation that TVM support. Usually, there are mixed of ways, depending on kind of model you're looking at. For vision models and stable diffusion models, usually we can just do tracing that takes PyTorch model on the TVM. That part is still being robustified so that we can bring more model in. On language model tasks, actually, what we do is we directly build some of the model constructors

• 20:09Speaker 02

  and try to directly map from hugging face models. The goal is, you know, if you have a hugging face configuration, we will be able to bring that in and apply optimization on them. So one fun thing about model compilation is that, you know, your optimization don't happens only at a soft language, right? If you write PyTorch code, you just go and try to, you know, use better fuse operator at a source code level. Torch compile might help you do a bit of things in there. In most of model compilations, it not only happens at beginning stage, but will also apply generic transformations in between, also through a Python API. So you can tweak some of that. So that part of optimization helps a lot of uplifting in getting both performance and also portability on the environment.

• 20:54Speaker 02

  And another thing that we do have is what we call universal deployment. So if you get The M O program into this TVM script format where there are functions, right, that that takes in tensor and output tensor. We will be able to have a way to compile it. So they will be able to load the function in any of the language runtime that TVM support. So if I hope you could load it in JavaScript. And that's a JavaScript function that you can take in tensors and output tensors. If you're loading now Python, of course, and C++ and Java. So so the goal there is really bring the M O model to the language that people care about and be able to run them on the platforms they they like. Yeah.

• 21:38Speaker 01

  It strikes me that I so I've talked to a lot of compiler people, but you don't have a traditional compiler background. You're inventing your own discipline called machine learning compilation or MLC. Do you think that this will be a bigger field going forward?

• 21:52Speaker 02

  First of all, I I do work with the people working on compilation as well.

• 21:57Speaker 01

  Right, you're not the only.

• 21:58Speaker 02

  Yeah. So so they are, you know, with that, we are also taking inspirations from a lot of early innovations in in the field like for example, TVM. Initially, we take a lot of inspirations from Halide, which is this like image processing compiler. And of course, since then, we have evolved quite a bit so that, you know, to focus on the machine learning related compilations. If you look at some of a conference publications, you'll find the machine learning compilation is already kind of a subfield. So if you look at papers in both machine learning venues, the M O SIS conferences, of course, and also system venues, every year there will be papers around machine learning compilation. And in the compiler conference called CGO, there's a C4MO workshop

• 22:42Speaker 02

  that also kind of trying to focus on this area. So definitely it's already started getting traction and becoming a field. I wouldn't claim that I invented this field, but definitely, you know, I helped to work with a lot of folks there. And I try to bring a perspective, of course, you know, trying to learn a lot from the compiler, compiler optimizations as well as trying to bringing in knowledges in, you know, machine learning and assistance together.

• 23:08Alessio Fanelli?

  So we had a George Hots on the podcast a few episodes ago, and he had a lot to say about AMD and their their software. So when you think about TVM. Are you still restricted in a way by like the performance of the underlying kernel, so to speak? So like if you're like, you know, if your target is like a CUDA runtime, you still get better performance, no matter like TVM kind of helps you get there, but then that level you don't take care of, right?

• 23:34Speaker 02

  There are two parts in here, right? So first for that is the lower level runtime, like no CUDA runtime. And then actually for NVIDIA, a lot of the mood came from their libraries, like Cutler, CUD and right, those library optimizations. And also for specialized workloads, actually you can specialize them like, because a lot of cases you'll find that if you go and do benchmarks, it's very interesting, like in like a two years ago, if you try to benchmark ResNet, for example, usually the NVIDIA library will give you the best performance. It's really hard to beat them. But as soon as you start to change the model to something, you know, maybe a bit of a variation of ResNet,

• 24:16Speaker 02

  not for the ImageNet detections, but for lane detection, so on, there will be some room for optimization because people sometimes overfit to benchmarks. These are people go and optimize things, right? So the people overfit the benchmarks. So that's the largest barrier, like being able to get a low level kernel libraries, right? In that sense, the goal of TVM is actually we try to have a generic layer to both, of course, leverage libraries when available, but also being able to automatically generate libraries when possible. So in that sense, we are not restricted by, you know, the libraries that they have to offer. That's why, for example, we will be able to run Apple M2, a web GPU where there's no library available, because we are kind of like an automatic generating libraries.

• 25:00Speaker 02

  That's makes it easier to support less well supported hardware, right? Like FAMO WebGPU is one example from a runtime perspective. AMD, I think before their锁锁driver was not

• 25:13Speaker 02

  Very well supported. Recently, they are getting good. But even before that, we'll be able to support AMD through this GPU graphics backend called Vulcan, which is not as performant, but gives you a decent portability across those hour.

• 25:29Alessio Fanelli?

  And I know we got other MLC stuff to talk about like WebLLM, but I want to wrap up on the optimization that you're doing. So there's kind of four core things, right? Kernel fusion, which we talked a bit about in the flash attention episode in the TinyG1. Memory planning and loop optimization, I think those are like pretty, you know, self-explanatory. I think the one that people have the most questions about is like-

• 25:52Speaker 01

  Can you quickly explain those?

• 25:55Alessio Fanelli?

  Yeah, go for it.

• 25:56Speaker 02

  Yeah, so there are kind of different things, right? Kernel fusion means that, you know, if you have operator convolutions or in the case of transformer like MLP, you have other operators follow that, right? You don't want to launch two GPU kernels. You want to be able to put them together. Yeah, smart way, right? And as a memory planning, it's more about, you know, hey, if you run like Python code, right? Every time when you generate a new array, you are effectively allocating a new piece of memory, right? Of course, PyTorch, another framework, try to optimize for you. So there is a smart memory allocator behind the thing. But actually, in a lot of cases, it's much better to statically allocate and plan everything ahead of time. And that's where like a compiler can come in.

• 26:38Speaker 02

  We need to, first of all, actually for language model, it's much harder because dynamic shape. So you need to be able to what we call symbolic shape tracing. So we have like a symbolic variable that tells you like the shape of the first tensor is n by 12. And the shape of the third tensor is also n by 12, or maybe it's n times 2 by 12. Although you don't know what n is, right? But you will be able to know that relation and be able to use that to reason about fusion and other decisions. So besides this, I think loop transformation is quite important. And it's actually non-traditional. Like originally, if you simply write a code and you want to get a performance, it's very hard. If I'm, you know, if you write matrix smart plan, right? Simplest thing you can do is you do for I J K, C I.

• 27:23Speaker 02

  J plus equal, you know, A I K times B I K. But that code is a hundred times slower than the best available code that you can get. So we do a lot of transformation, like being able to take that original code, trying to put things into shared memory, and making use of tensor cores, making use of memory copies. And all these, actually, all these things, we also realized that, you know, we cannot do all of them. So we also make the M L compilation framework as a Python package. So that people will be able to continuously improve that part of engineering in a more transparent way. So we find that's very useful, actually, for us to be able to get good performance very quickly on some of the new models.

• 28:05Speaker 02

  Like when Llama to came out, we'll be able to go and look at the whole, here's the bottom back and we can go and optimize those.

• 28:10Alessio Fanelli?

  And then the third one being a weight quantization. So everybody wants to know about that. And just to give people like an idea of like the the memory saving, you know, if you're doing like I F P thirty two is like four bytes per parameter. And eight is like one byte per parameter. So you can like really shrink down the memory footprint. What are like some of the trade offs there? Like, how do you figure out what like the right target is? And like, what are like the precision trade offs too?

• 28:37Speaker 02

  Right now, a lot of people like, also, we also mostly use int four now for language models. So so that really shrinks things down a lot, right? And more recently, actually, we started to think that, at least in M L C, we don't want to have a strong opinion on what kind of quantization we want to bring. Because there are so many research in the field. So what we can do is we can allow developers to customize the quantization they want. But we still bring the optimum code for them. So so we're we're working on this item called bring your own quantization. In fact, to be able to, hopefully, you know, M L C will be able to support more quantization format. And definitely, I think there are, there's an open field that's being explored. Like, can you bring more spacities?

• 29:19Speaker 02

  Can you quantize activations as much as possible and so on? And it's it's going to be something that's going to be relevant for quite a while.

• 29:27Alessio Fanelli?

  You mentioned something I wanted to double back on, which is most people use in for for language models. This is actually not obvious to me. Are you talking about the G G M L type people or even the trade, the researchers who are training the models also using it for?

• 29:40Speaker 02

  Sorry, so I'm mainly talking about inference, not training, right? So when I'm doing like a training, of course, um, in for is harder, right? Maybe you could do some form of mixed type precision for for inference. What I think in for is is kind of like a, um, a lot of cases, you will be able to get away with in for in all cases. And actually that does bring a lot of savings in terms of like the memory overhead and so on.

• 30:09Alessio Fanelli?

  Yeah, that's great. Let's talk a bit about maybe the, yeah, G G M L, then there's like Mojo. How should people think about M L C? Like, how do all these things play together? You know, I think G G M L is focused on kind of like model level reimplementation and like improvements. Mojo is like a language, super sad, you're more at the compiler level. Do y'all work together? Like, do people choose between them?

• 30:32Speaker 02

  So I think in this case, right, both, I think it's great to say the ecosystem becomes so rich with so many different ways, right? So in our case, I would say G G M O is more like, you know, you're implementing something from scratch in C, right? So that gives you ability to go and customize each of a particular hardware backend. But then you will need to write from a CUDA kernels and you write optimized for AMD and so on. So so the kind of engineering effort is a bit more broadened in that sense. Mojo, I have not looked at specific details yet. I think it's good to start to say, you know, it's a language, right? There's also, I believe there will also be machine and compilation technologies behind it. So so it's good to say, you know, interesting place in there.

• 31:17Speaker 02

  In the case of M L C, our case is that we do not want to have an opinion on how, where, which language people want to develop, deploy things on. And we also realize that actually there are two phases. We want to be able to develop and optimize your model. By optimization, I mean, you know, really bring in best CUDA kernels and do some of the machine learning engineering in there. And then there's a phase where you want to deploy it as a part of app. So if you look at the space, you'll find that GMO is more like, you know, I'm going to develop optimize in the C language, right? And the most of a low level languages they have. And module is that you want to develop and optimize in module, right? And you deploy in module. In fact, that's, that's the philosophy they want, they want to push for. In the other case, we find that actually if I want to develop models,

• 32:01Speaker 02

  machine learning community likes Python. Python is a language that you should focus on. So in the case of M O C, we really want to be able to enable, not only be able to, you know, just define your model in Python. That's very common, right? But also do M O optimization, like engineering optimization, CUDA kind of optimization, memory planning, all those things in Python that makes you customizable and so on. But when you do deployment, we realize that people want a bit of universal flavor. If you are a web developer, you want JavaScript, right? If you, if you want, if are maybe embedded system person, maybe you would like prefer C++ or C or Rust. And people sometimes do like Python in a lot of cases. So in our, in the case of M O C, we really want to have this vision of,

• 32:45Speaker 02

  you know, you optimize, build a generic optimizations in Python, then you deploy that universally onto the environments that, that people like.

• 32:54Speaker 01

  That's a great perspective and comparison, I guess. Um, one thing I, I wanted to make sure that we cover is that I think you are one of these emerging set of academics that also very much focus on your artifacts of delivery. You know, something we talked about to three is that he, he was very focused on his GitHub. And obviously you treated XGBoost like a product, you know? And then now you, you're publishing an iPhone app.

• 33:20Speaker 02

  Okay, yeah, yeah.

• 33:21Speaker 01

  What is this thinking about academics getting involved in shipping products?

• 33:24Speaker 02

  I think there are different ways of making impact, right? Definitely, you know, there are academics that are writing papers and building, building insights for people so that people can build product on top of them. In my case, I think the particular few that I'm working on machine learning systems, I feel like. Really, we need to be able to get it to the hand of people so that really we see the problem, right? And we show that we can solve the problem. And it's a different way, it's a different way of making impact. And there are academics that are doing similar things, like, you know, if you look at some of the people from Berkeley, right, that a few years, they will come up with big open source projects. Certainly, you know, I think it's just a healthy ecosystem to have different ways of

• 34:09Speaker 02

  making impacts. And I feel like really be able to do open source and work with, work with open source community is really rewarding because we have a real problem to work on, or when we build our research, actually those research bring together and the people will be able to make use of them. And we also start to see interesting research challenges that we wouldn't otherwise say, right, if we're just trying to do a prototype and so on. So I feel like it's something that is one interesting way of making impact, make contributions.

• 34:40Speaker 01

  Yeah, you definitely have a lot of impact there. And having experienced publishing Mac stuff before, the Apple App Store is no joke. It is the hardest compilation, human compilation effort. So one thing that we definitely wanted to cover is running in the browser. You have a 70 billion parameter model running in the browser.

• 34:58Speaker 02

  That's right.

• 34:59Speaker 01

  Can you just talk about how?

• 35:01Speaker 02

  Yeah, of course. So I think that there are a few elements that need to come in. First of all, you know, we do need a MacBook, the latest one, like M2 Max, because you need the memory to be big enough to cover that. So for a 17 billion model, it takes you about, I think, 50 gigs of RAM. So the, you know, M2 Max, the upper version will be able to run it, right? And it also leverages machine learning compilation. Again, what we are doing is the same, like whether it's running on iPhone, on server class GPUs, on AMDs, or on MacBook, we all go through that same MLC pipeline. Of course, in certain cases, maybe we'll do a bit of customization iteration for either ones.

• 35:44Speaker 02

  And then it runs on the browser runtime, there's this package of Web I O M. So that will effectively. So what we do is we will take that original model and compile to what we call Web GPU. And then the Web I M will be to pick it up. And the Web GPU is this latest GPU technology that that major browsers are shipping right now. So you can get it in Chrome for for them already. It allows you to be able to access your native GPUs from a browser. And then effectively that language model is just invoking the Web GPU kernels through there. So we actually, when when the Lama two came out, initially, we asked a question about, you know, can you run seventeen billion on MacBook? That was the question we're asking.

• 36:25Speaker 02

  So first, we actually, Junlu, who is the engineer pushing this, he he he get seventeen billion on MacBook. We had a CLI version, so in M O C you will be able to that runs through Metal accelerator. So effectively, you use the Metal programming language to get the GPU extration. So we find, okay, it works for the MacBook. Then we ask, you know, we had a Web GPU back and why why not try it there, right? So so so we just tried out. And it's really amazing to see like, you know, everything up and running and and actually it runs smoothly in that case. So so I do think there are some kind of interesting use cases already in this, right? Because everybody have a browser, you don't need to install anything.

• 37:06Speaker 02

  I think this doesn't make sense yet to really run a seventeen billion model on a browser, because you kind of need to be able to download the weight and so on. But I I think we're we're getting there, like effectively, the most powerful models, you will be able to run on consumer devices, kind of really amazing. And also, in a lot of cases, there might be use cases, for example, if I'm going to build a chatbot that, you know, that I talk to it and answers questions, maybe some of a component, like, you know, the voice to text could run on the client side. And so there are a lot of possibilities of being able to have something hybrid, like that contains the edge component, something that runs on our server.

• 37:47Alessio Fanelli?

  Do these browser models have been a way for applications to hook into them? So if I'm using, say, you know, you can use OpenAI or like you can use the local local model.

• 37:55Speaker 02

  Of course, so right now, actually, we are building. So so there's a N P M package called Web I O M, right? So that, you know, you will be able to, if only embed it on on to your web app, you will be able to directly depend on Web I M and it's you will be able to use it. We are also having a REST API that's OpenAI compatible. So that REST API, I think, right now, it's actually running on native backend. So that if a CUDA service faster to run on native backend. But also, we have a web GPU version of it that you can go and run. So yeah, so we do want to be able to have easier interactions with existing applications. And open API is certainly one way to do that.

• 38:36Speaker 01

  Yeah, this is great. I actually did not know there's an NPM package that makes it very, very easy to try out and use. I want to actually, one thing I'm unclear about is the chronology. Because as far as I know, Chrome shipped WebGPU the same time that you shipped WebLM.

• 38:53Speaker 02

  Okay, yeah.

• 38:54Speaker 01

  So did you have like some kind of secret chat with Chrome?

• 38:57Speaker 02

  The good news is that Chrome is doing a very good job of trying to have early release. So although the official shipment of the Chrome WebGPU is the same time of WebLM, right? Actually, you will be able to trial WebGPU technology in Chrome. Their unstable version called Canary about, I think, as early as two years ago, that's a WebGPU version. Of course, it's getting better. So we had TVM-based WebGPU backend two years ago. Of course, at that time, there's no language models. It's running on less interesting, well, still quite interesting models. And then this year, we really started to say it's getting matured and performance keeping up.

• 39:37Speaker 02

  So we have a more serious push of bringing the language model compatible runtime onto the WebGPU.

• 39:45Speaker 01

  I think you agree that the hardest part is the model download. Has there been conversations about a one-time model download and sharing between all the apps that might use this API?

• 39:55Speaker 02

  That is a great point. I think it's already supported kind of already in some sense, you know, when we download the model, WebLM will cache it onto a special Chrome cache. So if a different web app uses the same WebLM JavaScript package, you don't need to re-download the model again. So there is already something there, right? But of course, you have to download the model once at least to be able to use it, yeah.

• 40:19Speaker 01

  Okay, one more thing, just in general, before we're about to zoom out to AutoAI. The last question is, you're not the only project working on, I guess, local models.

• 40:28Speaker 02

  That's right.

• 40:28Speaker 01

  Alternative models. There's GPT4all, there's Ollama that this recently came out, and there's a bunch of these.

• 40:34Speaker 02

  Yeah.

• 40:34Speaker 01

  What would be your advice to them on what's a valuable problem to work on? And what is just? Thin wrappers around G G M L like what is what is what are the interesting problems in this space basically?

• 40:45Speaker 02

  I think making API better is certainly something useful, right? And in general, one thing that we we do try to push very hard on is this idea of easier universal deployment. So we are also looking forward to actually have more integration with M O C. That's why we're trying to build API, Web I M, so we're also looking forward to collaborate with all those ecosystem working working support to bring in models more universally and be able to also, you know, keep up the best performance when possible here more push button way.

• 41:15Alessio Fanelli?

  So as we mentioned in the beginning, you are also the co-founder of Octomel. Recently, Octomel released Octo A I, which is a compute service, basically focuses on optimizing model runtimes and acceleration and compilation. What has been the evolution there? So Octo started as kind of like a traditional M L Ops tool where like people were building their own models and like you help them on that side. And then it seems like now most of the market is shifting with to starting from like pre-trained generative models. Yeah, what has been that that experience for you and like what you've seen the market evolve and how did you decide to to release Octo A I?

• 41:52Speaker 02

  One thing that we find out is that on one hand, it's really easy to go and get something up running, right? So if you start to consider there's so many possible availabilities and scalability issues and and even integration issues since becoming kind of interesting and complicated. So really want to make sure to help people to get that part easy, right? And now a lot of things, if we look at the customers we talked to and the market certainly generative A I is something that that is very interesting. So so that is something that we really hope to help elevate. And also building on top of technology we build to enable since, you know, like a portability across hardwares. And you will be able to not worry about this specific details, right?

• 42:37Speaker 02

  Just focus on getting the model out. We'll try to work on infrastructure and other things that that helps on the other end.

• 42:45Alessio Fanelli?

  And when it comes to Getting optimization on the runtime. I see when I we run like an early adopters community and like most enterprises issue is like how to actually run these these models, you know. Do you see that as like a one of the big bottlenecks now? I think like a few years ago was like, well, we don't have a lot of like machine learning talent, we cannot develop our own models, you know, versus now it's like there's this great models you can use, but like I don't know how to run them efficiently.

• 43:12Speaker 02

  That depends on how you define by run then, right? So one whole, you know, it's one hand, it's easy to download even MLC like you downloaded, you you run on a laptop, but then there's also different decisions, right? What if you are trying to serve a larger user request? What if that request changes, right? What if availability of hardware changes? Like right now, it's really hard to get the latest hardware on media, unfortunately, because everybody's trying to working on this things using the hardware that's out there. Right, so I think they are kind of like when the definition of run changes, the kind of like there are a lot more questions around things. And also in a lot of cases, it's not only about running models, it's also about being

• 43:54Speaker 02

  to solve problem around them, like how do you manage your model locations? And how do you make sure that you get your model close to your execution environment more efficiently? So definitely a lot of engineering challenges out there that we hope to elevate. Yeah. And also, you know, if you if you think about our future, definitely I feel like right now the technology, given the technology and kind of how availability we have today, we would need to make use of all the possible hardware available out there. You know, that will includes a mechanism for cutting down costs, bringing something to edge and cloud in a more natural way. So I feel like still this is a very early stage of where we are, but it's already good to see like a lot of interesting progress.

• 44:35Alessio Fanelli?

  Yeah, that's awesome. I would love, I don't know how much you can go in depth into it, but what does it take to actually? Abstract all of this from the end user, you know, like they don't need to know what GPUs you run, what cloud you're running them on, you you take all of that away. What was that like as a engineering challenge?

• 44:51Speaker 02

  So I think that there are engineering challenges on, in fact, first of all, you would need to be able to support all the kind of hardware backend you have, right? On one hand, if you look at the media library, you'll find very surprisingly, not too surprisingly, most of the latest libraries works well for on the latest GPU. But there are other GPUs out there in the cloud as well. So certainly being able to have know hows and being able to do model optimization is one thing, right? Also infrastructures on on being able to scale things up, locate models. And in a lot of cases, we do find that on typical models, it also requires kind of vertical iterations. So it's not about, you know, build a silver bullet and that silver bullet is going to solve all the problems.

• 45:31Speaker 02

  It's more about, you know, we're building a product, we'll work with the users and we find out there are interesting opportunities in in certain point. And then when our engineer will go and go and solve that and it will automatically reflect it in a in a service.

• 45:45Alessio Fanelli?

  Awesome. We can jump into the lightning round and tell, I don't know, Sean, if you have more questions or T Q, if you have more stuff you wanted to talk about that we didn't get a chance to touch on.

• 45:54Speaker 02

  Yeah, we have talked a lot, so yeah.

• 45:56Speaker 01

  We always would like to ask, like, you know, if you have a commentary on other parts of E I and M L that is interesting to you.

• 46:03Speaker 02

  So right now, I think one thing that we are really pushing hard for is this question about how can, how far can we bring in open source, right? I'm kind of like a hacker and I really like to put things together. So I think this unclear in the future of what the future of A I looks like. On one hand, it could be possible that, you know, you just have a few big players, you just try to talk to those those bigger language models and that's can do everything, right? On the other hand, one of the things that Welling Academic had really excited pushing for, that's one reason why I'm pushing for M L C. Is that can we build something where you have different models? You have personal models that knows the best movie you like, but you also have bigger models that maybe know no more.

• 46:48Speaker 02

  And you get those models intact with each other. Right and and be able to have a wide ecosystem, you know, AI agents that helps each person while still be able to do things like personalizations, some some of them can run locally, some of them of course running on a cloud and how do they interact with each other. So I think that is we are in a very exciting town where, you know, the future is yet undecided, but I I feel like there are something we can do to shape that future as well.

• 47:18Speaker 01

  Yeah. One more thing, which is something I'm also pursuing, which is and this kind of goes back into predictions, but also back in your history. Do you have any idea or are you looking out for anything post transformers as far as architecture is concerned?

• 47:33Speaker 02

  Uh, I I think, you know, in a lot of this these cases, you can find there are already problems in models for long context, right? There are space based models. We are like, you know, a lot of some of my colleagues from Albert who he worked on this Hippo uh models, right? And then there is the open source version called RWKV. It's like recurrent models that allows you to summarize things. Actually, we are bringing RWKV to MOC as well. So so maybe you'll be able to see one of the models.

• 48:00Speaker 01

  We actually recorded an episode with one of the RWKV corp members.

• 48:04Speaker 02

  Okay.

• 48:04Speaker 01

  Um, yeah. It's unclear because there's no academic backing. It's just open source people.

• 48:09Speaker 02

  Oh, I see.

• 48:10Speaker 01

  So you like the merging of recurrent networks and transformers?

• 48:13Speaker 02

  I do love to see this model space continue growing, right? And and I feel like in a lot of cases, it's just that attention mechanisms getting changed in some sense. So I feel like definitely there are still a lot of things should be explored here. And that is also one reason why we want to keep pushing machine and compilation because one of the things we are trying to push was productivity. So that for machine learning engineering, so that as soon as some of the model came out, we will be able to, you know, empower them on to those environments that's out there. Yeah.

• 48:44Speaker 01

  Yeah, it's a really good mission. Okay, very excited to see that RWKV and space model stuff. I'm hearing increasing chatter about that stuff. Okay, lightning rounds is is always fun. I'll take the first one. Acceleration, what has already happened in AI that you thought would take much longer?

• 48:59Speaker 02

  Emergence of more like a conversation chatbot ability is something that kind of surprised me before it came out. This is like one piece that I feel originally I thought it would take much longer, but yeah, it happens.

• 49:13Speaker 01

  It's funny because like the original like Eliza chatbot was something that goes all the way back in time, right? And then we just suddenly came back again.

• 49:20Speaker 02

  Yeah, it's always to interesting came back, but with a kind of a different technology in some sense.

• 49:26Alessio Fanelli?

  What about the most interesting unsolved question in AI?

• 49:31Speaker 02

  That's a hard one, right? So I can tell you like what kind of I'm excited about. So so I think that I have always been excited about this idea of continuous learning and lifelong learning in some sense. So so how AI continues to evolve with the uh knowledges that being there seems that we're much getting much closer with all those latest recent technologies. So being able to be able to develop systems, support and be able to think about, you know, how AI continues to evolve is something that I'm really excited about.

• 50:01Speaker 01

  So specifically, just to double click on this, are you talking about continuous training?

• 50:06Speaker 02

  That's like a training, I feel like, you know, training adaptation and it's all similar things, right? You want to think about entire lifecycle, right? The lifecycle of collecting data, training, fine tuning, and maybe have your local context that getting continuously curated and feed on to models. So I think all these things are interesting and relevant here, yeah.

• 50:29Speaker 01

  Yeah, I think this is something that people are really asking, you know, right now we have moved a lot into the sort of pre-training phase and off-the-shelf, you know, model downloads and stuff like that. Which seems very counterintuitive compared to the continuous training paradigm that people want. So I guess the last question would be for takeaways. Uh what's be basically one message that you want every listener, every person to remember today?

• 50:54Speaker 02

  I think it's getting more obvious now, but I think what I would say is I always want to mention in my talks is that when you think about AI applications, originally. Is people think about algorithms a lot more, right? Our algorithm models, they are still very important. But usually, when you build AI applications, it takes, you know, both algorithm side, the system optimizations and the data curations, right? So it takes like a connection of so many facets to be able to bring together AI system and be able to looking at from that hardest perspectives really useful. Always start to build modern AI applications. I think it's going to continue, going to be more important in the future. Yeah.

• 51:36Speaker 01

  Thank you for showing the way on this and honestly, just making things possible that I thought would take a lot longer. So thanks for everything you've done.

• 51:46Speaker 02

  Thank you for having me, yeah.

• 51:47Alessio Fanelli?

  Yeah, thanks for coming on TQ. Have a good one.