AI Inference: Good, Fast, and Cheap, with Lin Qiao & Dmytro Ivchenko of Fireworks AI

In this episode, we delve into the intricate world of AI inference with cofounders of Firework AI. Discover the strategies behind optimizing AI performance, the importance of balancing latency and throughput, and the nuances of different AI architectures from GPT-3 to Stable Diffusion. Learn about their partnership with Stability AI, their unique focus on reducing total cost of ownership, and their vision for a seamless developer experience. RECOMMENDED PODCAST: How Do You Use ChatGPT with Dan Shipper. Dan Shipper talks to programmers, writers, founders, academics, tech executives, and others to walk through all of their ChatGPT use cases (including Nathan!). They even use ChatGPT together, live on the show. Listen to How Do You Use ChatGPT? from Dan Shipper and the team at Every, wherever you get your podcasts: https://link.chtbl.com/hdyuchatgpt SPONSORS: Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off https://www.omneky.com/ The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://bit.ly/BraveTCR Plumb is a no-code AI app builder designed for product teams who care about quality and speed. What is taking you weeks to hand-code today can be done confidently in hours. Check out https://bit.ly/PlumbTCR for early access. Head to Squad to access global engineering without the headache and at a fraction of the cost: head to https://choosesquad.com/ and mention “Turpentine” to skip the waitlist. CHAPTERS : (00:00:00) Introduction (00:08:34) Compute Stack (00:19:23) Fireworks Product Philosophy (00:24:11) Sponsors : Brave / Omneky (00:25:40) Fine-tuning Strategy (00:38:40) Sponsors : Plumb / Squad (00:40:33) NVIDIA Stack Overview (00:47:14) TensorFlow Triton Service (00:55:25) Reduced Precision Advantages (01:03:57) Different Deployment Scenarios (01:08:27) Seeking Intuition on Sharding (01:19:28) Announcing Stability AI Partnership (01:32:00) Closing Remarks

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  That is a complexity or application product developers as they are doing things, fun stuff themselves or in enterprises, they're all facing this challenge. So that's where we come in and say, you don't worry about it. We handle it all for you, so you just focus on your product application development.

• 0:19Speaker 01

  Can I just directly apply the techniques we learned from text models and image models? Because it has quality indicators, so you need to do some extra work to make sure the qualities are progressing. So that is quite a bit different.

• 0:34Speaker 00

  Over time, all these database managers have become smarter and smarter because they all have a layer called optimizer. Those optimizer observe the workload and it start to create, oh, you're doing a lot of filter on this particular column. So I'm going to create index. I'm going to partition those columns based on your future criteria. So it's much better search, much faster search.

• 0:53Nathan Labenz?

  Hello and welcome to the cognitive revolution where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz, joined by my co-host Erik Torenberg. Hello and welcome back to the cognitive revolution. Today, my guests are Lin Chow and Dmytro Ivchenko, co-founders of Fireworks AI, a company that specializes in inference compute, partnering with the world's leading generative AI researchers to serve the best models at the fastest speeds.

• 1:36Nathan Labenz?

  Lin and Dmytro both previously worked on PyTorch at Meta, which is today the default go-to AI framework powering applications used by billions. There, they gained firsthand experience with the immense challenges of running large language models at a massive scale and the many trade-offs between latency, cost and scalability that are always involved.

• 1:56Nathan Labenz?

  With Fireworks, they're building an end-to-end platform to make it radically easier and more cost-effective for any company to put generative AI into production. This spans the full technology stack, including providing simple tools for executing parameter-efficient fine-tuning techniques like LoRa that help developers iterate quickly toward product-market fit. Also, developing highly optimized deployments, leveraging multiple layers of abstraction, including custom CUDA kernels, to deliver consistently low latency. And managing and scaling hardware across major cloud compute providers in a way that's seamless to their customers. This is a wide-ranging discussion. Lynn and Dmytro share their hard-earned expertise on the intricacies of AI inference,

• 2:37Nathan Labenz?

  and we dive deep into the weeds on topics including: the different priorities that their customers have, such as minimizing time to first token, which is particularly important for voice applications; how some inference compute providers today are using Uber-style subsidized pricing to win business; and why Lynn thinks developers should be cautious about building on these platforms; also why she considers OpenAI and Anthropic to be Fireworks' real long-term competition; why Fireworks is betting that all small models, whether open or closed source, will ultimately converge in capabilities; the main parallelization techniques, including tensor and pipeline parallelism that they're using to spread models across GPUs in different ways with different benefits; why software is struggling to keep

• 3:19Nathan Labenz?

  up with the pace of advances in hardware; and how Fireworks is working toward an automated optimizer that will eventually allow even non-technical customers to choose the best configurations for their use cases. Finally, at the end, we brought Dmytro back for a short bonus discussion to cover their recently announced partnership with Stability AI, which has been powering Stable Diffusion 3 Generation on an exclusive basis. We talked a bit about some of the subtle differences between the image and text generation use cases, and overall, I came away with the sense that this partnership makes a ton of sense and might become a new pattern in the industry, as research groups look to make their work widely and effectively available, while also finding ways to earn a return on their investment.

• 4:03Nathan Labenz?

  Whether you are an AI engineer wrestling with model deployment or an executive evaluating AI platforms, Lynn and Dmytro offer a rare peek behind the curtain of the infrastructure layer, which may not get as much hype as the latest state-of-the-art model, but is obviously critical to realizing the potential of this technology.

• 4:19Nathan Labenz?

  If you, like me, default to high-speed podcast listening, I might suggest slowing this one down a little bit. Lin is originally from China and Dmytro is originally from Ukraine. And this conversation does get quite technical at times. So when I listened back, I found that 1.5x speed was as fast as I could comfortably go. As always, if you find value in the show, we'd appreciate an online share or review. And we always welcome your feedback via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Finally, for today, if you're an AI engineer looking for a new opportunity, or a business owner thinking about investing in AI tools or custom applications, I especially encourage you to get in touch. A number of companies that I'm connected with, including the executive assistant company Athena, which I've mentioned many

• 5:04Nathan Labenz?

  times, and Elicit, where I'm a small-time investor, plus a number of previous guests, are looking for AI engineering talent. And I'm also helping to start a boutique AI advisory and custom application studio, which will serve small and medium-sized businesses. That company is still in stealth mode for now, but the founder is offering free one-on-one consultations to business owners who have a practical problem that AI might be able to help solve. If any of this sounds interesting, send me a DM and I can connect you with the right people behind the scenes. Now, here's my conversation on AI inference optimized for scale, speed, and efficiency with Lin Chao and Dmytro Ivchenko of Fireworks AI. Lin Chao and Dmytro Ivchenko, co-founders of Fireworks AI. Welcome to the Cognitive Revolution.

• 5:47Speaker 00

  Thanks for having us.

• 5:48Nathan Labenz?

  Yeah, I'm excited for this conversation. You guys are in a super interesting business, which I will confess to not knowing a ton about. You provide primarily inference compute. And people broadly are well aware of the fact that compute is one of the hottest commodities in the world today and don't need to look any farther than Nvidia's stock price to get a sense for how high the demand is for compute. I also hear speculation that it's a tough business to be in because commodity businesses, long-term, can be tough. And then I also know that there's a lot of low-level execution details that really matter in businesses like these. And I'm super interested to learn more about some of the close-to-the-metal

• 6:31Nathan Labenz?

  work that you're doing. So maybe to start off, just give us a quick intro to Fireworks AI, how you guys got the idea to start the business and what your big-picture vision is for it.

• 6:41Speaker 00

  Yeah, definitely. So I think this is a very interesting question about, hey, is inference support provider a low-margin commodity? So a different way to answer these questions. First of all, reselling hardware is a low-margin business. I still remember when we just started, we were brainstorming all kind of different ideas, and we did notice a demand that because of GPU shortage, GPU arbitration could be an interesting problem to solve. And we decided to stay away from that because indeed, as you said, it's low margin and it's not a sustainable business. I have also seen artificially manufactured pricing from highly competitive landscape, and there's no way those providers can build

• 7:23Speaker 00

  a sustainable long-term business. And at least at the minimum, people need to take caution building on top of such stack, as when these companies or startups run out of funding, they will disappear. One of the important thing we want to build fireworks on top of is like focus on our specialty based on our experience. And here, when we think about GNI, so it is going to empower a whole slew of application and product disruption, particularly being consumer, prosumer, and developer-facing. The fundamental reason is this is a new revolutionary technology that didn't exist before, that can generate content emulating what human

• 8:04Speaker 00

  can generate. And by definition, the receiving part of this content are by and large human. And for those B2C application, latency is very important part of product experience because it has to be hyper-interactive. Without that interactiveness, it's not a viable product. So many of our customers come to us for extremely low latency requirements. At the same time, the content generated from those providers has to be high-quality.

• 8:31Speaker 00

  And we provide high quality through automated loop across fine tuning and inference. And last but not least, we provide low TCO in a highly sustainable way. And here, low TCO is very important because it's a different business from traditional application development. We're built on top of commodity CPUs. The gross margin is very high because CPU is so cheap. And now shift towards GPUs and GPU, not just the kind, the hardware expensive. It is very power hungry. It consumes a lot of power and generate a lot of heat. And traditional air cooling doesn't work. People have to use liquid cooling or immersed immersion cooling.

• 9:13Speaker 00

  That means put the GPU inside oil and fully immersed. And those are all costs of operating GNI inference. So that very high cost make it very challenging to justify business impact. And we specialize in reducing total cost ownership. So if you have a viable product, it can turn that into a viable business. So that's the the value add we have been focused on delivering from Firework's side. In terms of product, a quick overview. We provide the general AI platform to do fast experiment iteration and inference production scaling. So here, there are two development loops we're optimizing for. Inner loop, product experimentation.

• 9:58Speaker 00

  We optimize for iteration speed. And outer loop, production. We optimize for hard system metrics, including latency, TCO, scalability, reliability, I just mentioned. The specific product feature on these two loops are fine tuning and inference with on-demand serving and automation. This is for inner loop, fast iteration of experimentation. And we provide the fastest inference at scale for the outer loop. When you have product market fit, you want to scale a business. And this outer loop of production will help you to get the best speed and best TCO. So that's the very high level summary of Fireworks' product.

• 10:37Nathan Labenz?

  Cool. Okay. Several threads there that I want to follow up on. One. Interestingly, it sounds like you're saying that a lot of products on the market today are in an Uber moment where you think they're essentially being offered below cost and in a fundamentally unsustainable way. And by contrast, I understand that you are operating your business without doing that. You're not radically subsidizing the customer. Do I have that correct?

• 11:07Speaker 00

  That's correct, but also I just want to call out our value add is not extremely low cost. Our value add is low latency, high quality and low TCO. So I'm almost saying low TCO is a byproduct of our high performance.

• 11:22Nathan Labenz?

  Yeah, this is a real, I've experienced this in my own application development where TCO is not the natural way that a lot of engineers think of things and they specifically don't tend to factor in the cost of their own salaries as they start to build out infrastructure. I am a big believer in certainly the specialization of dedicated infrastructure providers, just because I've seen how hard it is to create a reasonably reliable stack when you're doing it on your own. But where are we in this market development cycle and where are things headed? It's an interesting observation off the bat that some of these bargain, the cheapest options that are out there are subsidized in an unsustainable

• 12:05Nathan Labenz?

  way. I know that's happening a lot at the application layer because certainly people are offering free demos, only that's obviously subsidized, right? Because all the tokens are costing money and people are certainly giving away a lot of free accounts and a lot of free inference to their end users. But I hadn't really considered how much that might be happening at the inference compute layer. So it's really interesting to consider that is also a reality in today's world.

• 12:29Speaker 00

  Yeah, a lot of our customers, whether they are developers or enterprises, come to us not because we are the lowest. Pricing provider at all. They come to us because, again, they are building consumer, personal, developer-facing applications that requires very low latency and they cannot get it by themselves, they cannot get it from any other providers, even from OpenAI and Topic. They didn't get the right latency or latency is not stable and they are seeking solution from our side. And getting to low latency is actually not easy and Dimitri can speak a lot to it today. But the high-level challenging is GNI model is among the largest model sizing and complexity

• 13:14Speaker 00

  in the whole spectrum of machine learning. In early days of machine learning, the algorithm is extremely simple, it's tens of megabytes. And now we're talking about the tens of billions of parameters. But that complexity, it doesn't change the nature of all these B2C applications, whether built on traditional machine learning or built on top of GNI. It doesn't change the latency requirements. It has to be extremely interactive and that put a lot of back pressure on the inference serving tier to do even more aggressive optimization. And that's the challenge where we are really good at addressing and that's our biggest value add. In addition to pricing, I don't think pricing in the long run is a sustainable value add.

• 13:58Nathan Labenz?

  Yeah, makes sense. Okay, cool. So I want to get into then how you're doing it to the greatest degree that you can share and that I can comprehend because I do think this stuff is going to get very into the weeds as we get close to the metal. Do I understand correctly that you are managing your own servers on racks? You're talking about cooling and all this sort of stuff, so are you vertically integrated to that level?

• 14:22Speaker 00

  Yeah, so that's a good segue to talk about our compute stack. And we currently build on top of CSPs. We're running on top of AWS, GCP and Oracle OCI. The reason is when we grow bigger, there are various different way to be more efficient. And right now, we are optimizing for velocity of our product development and those CSPs has been battle-tested. So that's why we build on top of them. And we also, as a company, aspire to run on top of the best hardware across the whole entire industry. And of course, it's the easiest to build on top of NVIDIA GPUs, and that's how we started. But at the same time, we see a lot of emerging hardware coming to this hardware landscape for GNI,

• 15:07Speaker 00

  including AMD, including Intel, including custom ASICs. So there are various different interesting trade-offs. We'd love to pass on to Dimitri to talk about the trade-offs across those different hardware providers.

• 15:20Speaker 01

  Sure, yeah. So as Lynn mentioned, historically, we've been predominantly dominated by NVIDIA in the overall AI landscape. And the reason for NVIDIA domination is twofold. First is they have pretty good hardware. And if you look at the latest, H100 has two PATA flops. This is pretty high. And they are pushing with a new blackwell to double that. And then you go to 10 PATA flops for their superchips. And they're also doubling, quadrupling memory bandwidth along the way. They're also improving on their cross-host interconnect. And the most relevant development there is the actual latencies which will allow us to

• 16:04Speaker 01

  of interconnect, which will in turn will allow us to reduce the latencies of our generation speed. That's one. But the second thing is NVIDIA historically has had a very robust developer stack. And because of these two reasons, I think NVIDIA has been dominating. But lately, I would say NVIDIA's arrival, AMD is coming back. And then now with the new MI300, which has very hard to respect than NVIDIA. And they are also working, I'm sure day and night, on their software stack. And the new welcome open source development on the Arachno stack. They are catching up basically in all aspects to NVIDIA as well. We also see other newcomers that are development from Intel with their Gaudi 2 or company Gaudi 3 release,

• 16:49Speaker 01

  and which is actually even more powerful than MI300. So it's in between H100 and 100 P200 U's. So that's like on the more programmable side of things. And now if you look at the different spectrum, the less programmable, more specialized hardware, namely ASICs. So here I would probably just select one is Rock, which has been making a bit of waves lately on Reddit because of their nice demo, the high generation speeds. But the thing is that ASICs are always compromising something. So and here is that if you look at the Rock and Surprisingly, there is very little detail and sometimes you need to guess what these guys do because they're not as open as Nvidia.

• 17:31Speaker 01

  But what they're fundamentally doing, they're compromising on the fact that they wanna remove the shared memory of this HBM in GPUs and they wanna put it as close in units as possible, they wanna put it on a die. But then they are compromising on the processing units, so they have achievable flops all over. But they're also, this memory is very expensive because putting anything on die is way more expensive than having a memory shared across GPU's. And now, what the result is that they have much lower achievable flops. What they can do faster, they can run the generations faster because they don't need to copy memory from the shared memory to the memory which is on die.

• 18:15Speaker 01

  So this is like fundamental kind of compromise. They're also way more expensive to operate because if you wanna host a single model, you need to have a human guest setup which costs millions of dollars. So these are kind of compromises they're going. Although it's very interesting and crazy cool demos, but practically see limited application of this in the real world. And sort of limited disruption, although it does solve some interesting cases, but there seem to be more edge cases right now where it wins over traditional programmable GPU hardware. There is also GPUs, of course, there has always been a case, they've always been around and they are setting up from programmability in between ASICs and GPUs. Although technically PyTorch supports GPUs,

• 18:58Speaker 01

  but to get the best performance, you have to go to JAX and it rules world, it rules software stack. So it becomes quite different operational mode, quite different business to operate. So this in a nutshell.

• 19:09Speaker 00

  Yeah, so speaking about that, as you can see, it's a spectrum. It's a spectrum from most programmable to hyper-specialized. And I want to talk a little bit about our Meta experience because both of us and actually a large portion of the funding team came from Meta. And we all worked on PyTorch extensively over the past five years. So it's a very interesting experience there. Of course, as you can see, Meta like other hyperscalers running on top of CPU, GPU, ASICs, this is the public information, a variety of hardware. And recently, they mentioned they deployed more than 600,000 H100 in-house. And what does that mean? Put that in perspective, that's equal to the power needed for 600,000 average American household. That's roughly twice the size of San Francisco.

• 19:55Speaker 00

  A huge power consumption, but Meta didn't get there overnight. It took us more than five years to go from basic machine learning on CPU to complex deep learning on GPU and ASICs. So there are a lot of interesting lessons learned. Because we work on software stack, right? So the first lesson learned is a good product cannot compromise. It need to be opinionated about what's a part of philosophy and what's the target audience. So at Meta, when we start to work on diving to AI infrastructure, we had at least three different AI frameworks. That's in 2018. We have Coffee2 for mobile, Onyx for server production. So our charter is to unify them into one and cover both server, mobile, research and production. It's called PyTorch1. This feels like mission impossible at the moment

• 20:41Speaker 00

  because it has so many different goals. So we merged all these teams together, but there's no consensus how to build this one framework. So we came up with an idealistic zipper approach. And the project was actually literally called zipper. Taking the PyTorch front end, which is very easy, simple to use, and zip it with Coffee2 back end, which is highly performant. It failed miserably because these two frameworks was never designed to work together. The integration overhead was more than writing a new framework from scratch. So we end up ditched that plan and we kept the PyTorch front end and rewrote this backend into As Torch script and that's the interpreter to execute PyTorch efficiently.

• 21:20Speaker 00

  So notice that the the design decision, the product choice we made is to hold a line for ease of use, hold a line for ease of AI innovation for developers built on PyTorch. And then we take all the complexity of you being building the backend. So that is the design decision. However, Torch script for PyTorch one dot O, it requires the user, the developers to annotate which part of PyTorch code is scriptable. That's not a very good UX because people need to know how to annotate. So two years ago, we also started Torch compile to fully automate the process. People don't need to annotate at all. So that's called PyTorch two dot O.

• 22:05Speaker 00

  We left before the project fully shipped, but I'm thrilled to see that there's great progress made from the PyTorch team. So another interesting takeaway we had in that process is we thought it should be easy to bring PyTorch to production. We thought the process is just swap the libraries because both PyTorch and Coffee two and Honest, they are all, they're just library, they're not services. Swap those libraries in the service. However, that's a wrong assumption. A framework or library innovation requires a completely new infrastructure of service to build, to support it. We end up have to build new data loaders, new data transformer, transformation layer for training for PyTorch. We need to build a new training loop for PyTorch.

• 22:47Speaker 00

  We need to new, build new distributed inference step for PyTorch. So after five years, we wrote a whole entire PyTorch platform from one app. It was serving more than fifty trillion inference per day across fifty data centers. So as you can see, this journey took us five years, but we start fireworks. We want to reduce, we want to shrink this five years journey into five weeks or even just five days for current broader set of developers in the industry. We have built better tested large scale AI systems. So we're confident that we can all innovate the increments and significantly shorten time to market. For everyone out there, I want to build, build disruptive applications and products on top of GNN.

• 23:31Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 23:35Nathan Labenz?

  If I try to summarize that and make the translation from the meta experience to the fireworks product direction, it seems like the key principles are never compromise on the product, user experience above all. The users hate to wait. I certainly know that from personal experience. And there's an echo of that at the developer layer, too. Developers hate complexity and don't want to have to deal with all this mess. So internal, focus on the end user experience above all. Focus on the developer experience second. Simplify everything. Bring all the complexity in-house. Abstract away from all the different kinds of hardware. And it sounds like the place that you want to carve out for yourselves

• 24:20Nathan Labenz?

  with fireworks is a sort of layer that sits between the developers and all the different hardware providers, so that folks can easily develop their applications and not really have to worry about what hardware they're running on, the relative strengths and weaknesses of all that, what bottlenecks they're going to hit at all these different, depending on all the different stacks. Tell me if that's right, first of all. And then I'm very curious to get into a little bit more detail on maybe what some of those bottlenecks are. Well, I'm like pretty rudimentary in my GPU knowledge, but I'm learning. And the audience will be

• 25:01Nathan Labenz?

  pretty varied in terms of how much command they have of this stuff. First of all, do I have that general story and kind of market position and vision, right?

• 25:10Speaker 00

  Yeah. So it's mostly right. A little bit more nuanced is what is getting the way of the fast iteration loop? Here, we're talking about current set of application product developers. They haven't jumped onto AI yet, and they have a slew of AI tools they have to learn, and pick and choose different models for their use cases, and also try to optimize for latency, and try to justify business impacts through low TCO. All of those are on their plate right now. And our goal is to take away those concerns, let them focus on where they should be focused on. Application and product development and we give them proper tools and higher level abstraction.

• 25:53Speaker 00

  So they get low latency, low TCO, high quality extremely easily. Of course, it sounds really nice and we can dive a lot deeper into how we get there. But I, at the high level, I will break it into two buckets. One is low latency, low TCO, that's a bucket. We should dive deeper into that's driven by our high system optimization, performance optimization. The other bucket is quality and we have to talk about quality because at the end, we think our competition are not other inference provider. Our real competition is actually OpenAI and Anthropic. So solving and addressing people's quality issue is very high priority of our company goals.

• 26:33Nathan Labenz?

  Yeah, okay, cool. Those are both really interesting. Now, let's maybe start with the quality one because that's probably the more intuitive one. And I was noticing in trying the product and I always do go in and try products on the, in my preparation for these conversations. I would say the experience is super easy to get started. Just create an account. Next thing you know, you're in a playground. You can choose from all these models. You've got what you call serverless models, which is literally just a very similar experience to what you get with OpenAI and Anthropic where you're making an API call. You're purely paying for tokens. And that's all super simple. You can test stuff out, try all the different models in the playground and then hit the

• 27:13Nathan Labenz?

  give me the code button and it, you know, pops up the code and you can go copy that code and drop it in your application. So all that is probably reasonably familiar to folks in our audience. They've done that kind of thing at least with an OpenAI or an Anthropic, if not another inference provider. Quality though is a tough challenge, right? Because with some notable exceptions that are starting to crack into the top 10, certainly still the very best models are proprietary to the OpenAIs and Anthropics and Google Deep Minds of the world.

• 27:45Nathan Labenz?

  Notably, too, the prices also started to get pretty low from those guys. The Haiku from Anthropic is like a really interesting point of comparison. The price that you guys have for the small tier of models, up to 16 billion parameters. I'm interested to hear how that cutoff was selected, but up to that 16 billion threshold is 20 cents per million tokens, which is crazy to think not that long ago, it was six cents per thousand tokens from OpenAI with the original GPT-3. So we've gone from six cents per thousand to 20 per million for better models than the original GPT-3 in just two years.

• 28:25Nathan Labenz?

  I always like to stand back and just marvel at how fast that cost has come down. But the leaders are not too far behind, right? There with Haiku, Anthropic is at 25 cents per million, at least on input. They charge a little more for the output, but with the long context, presumably a lot of the time it is pretty input-heavy workload there. So yeah, let's talk about quality. How do you win in a world of Haiku? I would assume that's like the number one direct competitor for those smallest models. Curious to hear how you're thinking about the quality challenge.

• 28:57Speaker 00

  Yeah, definitely. So between closed source vendor and open source vendor, who is going to win? It's clear that we are currently in a very intense race between open source and closed source models. This week is an interesting week. Mr. Joe's team just dropped a MOE, a new MOE model with eight experts of 22 billion parameters. And we just enable it. A meta will open source Llama 3 in a few weeks. And Google has been continuous to open source newer gamma models. And Qwen and E-models from those Chinese institutions are getting better and better also. At the same time, OpenAI and Anthropic keep improving their model quality, including at a smaller model size scale.

• 29:40Speaker 00

  So I think all these models in the same size bucket will converge in quality eventually. The reason is for smaller models, because of the size, it has the upper bound of how much knowledge it can absorb, and it determines capability. And as we are all also converge on how much data we can train a model. So just inevitable over time, whether open source or closed source, the model quality will be similar to each other. If that's true, and I will argue, open source models have a much stronger ecosystem potential. Because it has a lot more active people engagement, developers are engaging in tuning open source models and then they open source them. And this keeps going, it has compounding effects.

• 30:24Speaker 00

  So more and more people can build on top of each other's work. This is how I'm thinking, so that's why uh fireworks fundamentally build on top of those open source ecosystem. We also just launched our fine tuning service, it's relatively new. So in terms of revenue, it's not dominating any of other product features, but it's one of our fast growing product feature. With that said, I will still say, right now, we are not there yet, it's uh all open source, closed source models, small models are converging. Right now, the open source model is still need fine tuning, and the challenge of fine tuning is it's a lot, much longer development process than prompt engineering, and we fully acknowledge that.

• 31:07Speaker 00

  And that's why we as a company, we want to solve those problems, to make fine tuning much faster, easier and actionable, because once you see the result, hey, what is not working, it's actionable. And also, we will by and large automate a lot of this pain in fine tuning away. So our developers will have much simpler experience, as close to prompt engineering as possible. So that's one aspect. The other aspect is eventually OpenAI and Anthropic, their goal, including meta automation, their goal is to get to AGI, right? What is AGI? It's basically building a system that's smarter than human.

• 31:49Speaker 00

  It it means this system can solve very complex problem, hundreds of them at the same time. But if we look at the problems developers and enterprises, they are solving. They don't solve hundreds of very complex logical reasoning problem at the same time. They probably have a handful of very specific problems. For example, they want to solve classification problem from intent routing, based on intent route to different agents, to categorize product catalog nicely, to re-ranking, retrieve result, get top K of the best answers, or to do structured data extraction from images, like extract information

• 32:32Speaker 00

  from receipts, from medical bills, from insurance policies, to paraphrasing emails for better sales follow-up or for better marketing lead generation. And the list can go very long, but as you can see, every single examples I mentioned here, they're very specific. So that's create an interesting opportunity for us, for especially for smaller open source models. Think about model training as a process of aligning the model to optimize for a set of objectives. So this is very important framing, because what the model is good at is being decided at the beginning of the training process. How you form the training data set, what is the proportion of what kind of training data

• 33:14Speaker 00

  set you mix together is basically a product opinion you put in what is the capability of this end model. So no model today is gonna be good at solving all kind of problem. You have to pick and choose, so by set the objective at the beginning. So that's a general framing, but you can also think about it's much easier to align model to solve a specific problem very well, instead of ask the model to solve hundreds of problem very well. Solving, narrow down the objective to one problem is a much easier alignment process. So that's why in practice, all the practical problem we're seeing today is amenable to smaller models, because the problems are very narrow and well-defined.

• 33:59Speaker 00

  And we have seen a lot of success using fine tuning to solve those problem. And second, almost every single enterprise we talk with, they have data. They have data to align a model better and sometimes not just on par, even better than G P T four. So based on those observations, we're pretty bullish on the direction of continue down to make the feedback loop of quality between fine tuning, inference, data collection, cleansing. This will be very efficient for our users. And again, we will try to automate as much as possible in this process.

• 34:34Nathan Labenz?

  Okay, cool. I have a number of follow ups there as well. So just speaking to my own experience briefly, the general notion that fine tuning is, that seems like a really good answer to how do we compete with Haiku? Because I would agree that, and in fact, at Waymark, my company, we use a fine tuned G P T three point five currently to power our core script writing task. You know, the number one most important A I function in the product, we could use G P T for it. It's not really like a budget thing for us. We have a pretty high value use case. So our strategy is use whatever gives us the best results. Not just the best overall user experience, I'd say, and the quality of the output is probably the number one factor there.

• 35:20Nathan Labenz?

  Latency also is important as we've discussed, people don't want to wait. And so at the moment, we're on a three point five fine tuned model. G P T four could probably do the task pretty well. And we wouldn't be scared off by the cost, but it is a little bit slow sometimes. And also it's a little bit unwieldy. You're trying to prompt engineer your way. You have all these different caveats and rules and whatever, and it's hard to represent all that stuff in the prompt. And you could also think, huh, maybe we could use Haiku and maybe we could even put like ten examples into Haiku. And that could, in theory, get it to do the in context learning. But now we've gone up an order of magnitude in price because we're doing ten examples at every prompt. Fine tuning, I do think, is a really good answer. And certainly, as you said, companies have data, but even if they don't.

• 36:05Nathan Labenz?

  A lot of companies would probably be very surprised with how little data it really takes to do a reasonable fine tuning. Our data set is in the three figures, hundreds, but not even thousands of data points. And that works quite well. Actually, we don't have to have a huge thing for a narrow task. If you're trying to maintain generality, then you have a much bigger challenge on your hands. But for this one task of write a script for this business, for this video, hundreds of data points, we have found pretty well suffices.

• 36:36Speaker 00

  That's excellent point. Yeah.

• 36:38Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 36:43Nathan Labenz?

  So tell me about that fine tuning product and also maybe get into some of the bottlenecks. I looked at the documentation. I didn't actually do a fine tuning myself, but maybe I'll put that on my to do list. But it is very similar to the OpenAI product where you can set up your sort of chat, you know, records and their JSONL lines and you upload a bunch of JSONL lines and run it. You build that by token. Is it accurate to say that it's basically a kind of at the spec level same thing as the OpenAI experience?

• 37:15Speaker 00

  Yeah, this is a great way to summarize it because we are, one of the product design philosophies is to be OpenAI compatible. Because OpenAI has a lot of draw for the initial set of developers to try out ideas. And the benefit of being OpenAI compatible from fine tuning to inference is it's really easy to migrate. So even if our developer don't start for us, but once they get to a stage, they need more interesting fine tuning that OpenAI doesn't provide and they need low latency and low TCO for, for production and they can move to us. So you're absolutely right and the API will feel very familiar. But on top of that, we're adding also new APIs that doesn't exist in OpenAI. So that's also our unique advantage.

• 38:00Speaker 00

  But coming back to fine tuning, yeah, so here we started offer a special fine tuning called PAF, performance efficient fine tuning, actually found a long time ago. The most popular technique in that bucket is called LoRa. LoRa stands for low rank adaptation. So the idea is you freeze the pre-trained model weights and inject trainable rank decomposition matrices into each layer of the transformer architecture. And the end result is greatly reduce the number of trainable parameters for downstream tasks. So. Usually fine tuning, you can think of it as short training, right? So you need to go forward, backwards and do all this stuff. With LoRa, then your base model is only forward.

• 38:43Speaker 00

  And then your additional adapter, you go through the full training steps. So that's how it saves compute and be very efficient. And also, you just mentioned, actually, you figure out, many people, this consistent feedback, you don't need a lot of training samples or tuning samples. Typically, like around 1,000. In your example, just hundreds of them will be sufficient. A compound with very efficient LoRa fine tuning. That means your feedback loop is going to be really fast. And that make the whole fine tuning process even more appealing alternative. So let me spend a little bit of time explaining what LoRa means. So under the hood, it leverages a concept again called rank decomposition.

• 39:26Speaker 00

  And ranking decomposition is basically allow us to represent a high dimensional matrix with a product of two lower dimensional matrices. So if the pre-trained weight matrix, let's say of dimension N by K, and then it can be represented with a product of two matrices. Let's call it N by R for the first one. And the second one is R by K. So if we do a multiplication across these two matrices, you will still get N by K, right? So that's the high level idea. And let's make the saving more concrete. N is 2,000, K is 5,000, and R is 1 for extreme case. Just make the point here.

• 40:07Speaker 00

  So the total of parameters in the original matrix is 2,000 by 5,000. That's 10 million. And then if we decompose into lower rank, then the first matrix is 2,000 by 1 is 2,000. And second is 1 by 5,000 is 5,000. So total of parameters across these two lower rank, lower dimensional matrix is 2,000 plus 5,000 is 7,000. So 7 is a significant lower number of parameters than 10 million. That's the original matrix. So this is more than 1,000 times reduction. That's the idea behind why using LoRa to do fine tuning is so much faster. It's not just for faster fine tuning.

• 40:51Speaker 00

  At inference, we can also do something very interesting. So we actually have many customer come to us. They need to deploy many LoRa adapters against the same base model. So the naive way to deploy those LoRa adapter is you merge it with the base model. And then you deploy this a hundred. If they have a hundred LoRa adapters, you have to deploy models. And all these a hundred models sit in memory. We are one of the first to deploy many LoRa's by sharing the same base model. You can imagine it looks like a tree. There's a trunk, that's a base model. And each LoRa adapters are hanging on the trunk.

• 41:33Speaker 00

  So by doing this sharing, we can save a lot of cost in production for inference. Because for a hundred LoRa adapters, basically originally without this saving, you have to deploy a hundred times. And then you just need to deploy the one base model that's dominating the cost. So by that, you save a hundred times of the cost. We kind of use LoRa pretty extensively across both fine tuning and inference. And that significantly increased the velocity of iteration. If you haven't used, try that. I strongly encourage you to try our fine tuning service.

• 42:03Nathan Labenz?

  So yeah, let's just flesh that out a little bit more for people. You tell me if I'm going wrong anywhere. The high level situation is you want to have a model deployed that you can get inference from quickly. If you have all your computers turned off, you have the cold start problem of, okay, I got to boot up a Docker container and load new stuff in. And then these like billions of parameters take time to move in. So typically, most services are going to try to have some mechanism of not making you wait for a full cold start.

• 42:38Nathan Labenz?

  So instead, what you have when something is deployed is you have a GPU sitting there with the model loaded into the high bandwidth memory, which is to say the second tier memory. I've gone into this a bit on a previous episode where I dug into the Mamba architecture. But basically on your GPUs, you've got your many computation cores. There is the SRAM that is the like highest or let's say the lowest latency, closest to the computation core RAM. But it's small, you can't fit the whole model in there. So you have to have this second tier RAM, the high bandwidth memory where the actual billions and billions of parameters of the model sit.

• 43:21Nathan Labenz?

  And then when you're actually doing inference, there you're paging in parameters from high bandwidth memory into the SRAM for the actual computation, paging them in and out to do stuff. And what you're saying with the LoRa, like the many LoRa's configuration is that you could, the naive approach as you described it would be to say, okay, I'm going to have 100 different servers. Each one will have its base model and its LoRa and the LoRa is whatever, 1 to 3% or something, as many parameters. Again, depending on the setup and exactly how you do it. So you could have 100 of those, each with the base model and the LoRa, or you could have your base model and a bunch of LoRa's all sitting in high bandwidth memory

• 44:05Nathan Labenz?

  on a single GPU. And then you just have to manage the paging in or keeping track of which LoRa we're using for which use case. That kind of stuff gets a little bit more complicated, but it allows you to save on having to set up all these servers and have them waiting there. What's the next, you know, level of sophistication in terms of the analysis there? That could be like, what are the bottlenecks? What are the trade offs that you're facing? Tell me, what do I need to know next to get smarter beyond that base level description?

• 44:37Speaker 00

  Yeah, so we did a lot of very complicated inference stack optimization to bring down latency. As we have discussed in this episode, our hyper focus on latency. But also while we hyper focus on very low latency, we also hold the latency bar, we can pump up throughput to very high, and that is a result of low TCO. I will pass it to Dimitri to talk about all the trade offs, nuances we have put into inference optimization stack.

• 45:06Speaker 01

  So I want to here and there, take a step back and try to go over some very basic steps of how to build your own GNI inference service and what are the key points you should pay attention to because we have right now a huge information overload, there's so much information. Oh, this attention inclination is the best, or this Mamba architecture is the next best thing. Or we need to run a chart model differently, and there's so many different techniques. But the question still remains, what are the actual techniques which really do matter, right? So that's what I want to focus on these important points here. So, okay, now let's say we want to build the GNI inference and right now, let's always focus maybe on one or two cases.

• 45:49Speaker 01

  One is text generation and then another is image generation. Luckily, the very welcome development is that the new model from stability for the image generation is based on a transformer architecture. So the old architecture based on the rest of it is gone. And with that, all the convolutions are not as important anymore. So all we have to worry about is a transformer now, which in some extent is great, but it has its own challenges. Now, what does matter for transformer? So if a transformer, I would say there are only two operations which matter. One is the the most ubiquitous matrix multiplication. And that's all the AI workloads, GNI or not, but Mule is the most important one. And the second most important one is is of course attention.

• 46:33Speaker 01

  And attention is a bit special kind of Mule. I would say kind of it's back to back batched Mule. And optimizing it is quite critical. Okay, so now let's say you optimize these two operations, but guess what? There are different flavors that you need of these operations. And then these flavors are coming from the text generation workload. Because text generation is a bit special, there is input tokens and output tokens. And processing them is vastly different. The processing of input tokens is mostly compute bound. And generating tokens is mostly memory bandwidth bound. Now you have to have flavors of these two operations. One is flat optimized, compute optimized.

• 47:14Speaker 01

  And the second is a memory bandwidth optimized to speed up this paging loading of model weights from this HPM to the SRAM and then to registers. So that's what you need to implement. Now, how to implement it? So let's take Nvidia stack for example.

• 47:34Nathan Labenz?

  Can I just interject for one second? I just want to make sure I understand and that it's clear to the audience also that the two, we're all within inference here, right? But the kind of two forms of latency that matter are time to first token and then speed of token generation from there. And these are essentially two different phases of the process. Because, and here's where I'm learning from you and putting two and two together in real time. You said that the input tokens, AKA the processing that happens for time to first token, is going to be compute bound.

• 48:13Nathan Labenz?

  And I understand that to be because in that process, we're doing all the attention, all the MLPs for all the tokens, but we can do that in such a way where we don't have to page in and out the parameters for each token. We can process all the tokens with those parameters, page new parameters in, process all the tokens with those parameters. And so that's why it's compute bound, right? Because we're not paging in and out as much on a per token basis versus then when that's done and we get into token by token generation, now we need all the parameters in and out for every single token. Is that right?

• 48:50Speaker 01

  So it's actually much simpler than that. So the thing that, if you look at typical use cases, is they have, I would say roughly 10 to 1 ratio of input to output. So now when you have your input, and the input also typically like 1,000 and more tokens goes to 10,000, practically it's very common to see this kind of workload. Now, that's for the in, for the generation when you run the generation. First of all, you generate one token at a time. Unless you do some speculating generation, which is separate, but it's mostly one, one token at a time. And of course, yes, you wanna also batch multiple generations together, but there is also a limit there.

• 49:34Speaker 01

  Because your big prompt length, you need to allocate a lot of memory to host your intermediate activations. Okay, so in fact, what we see is that the batch size for the generation is around, I would say it goes from like sixteen to two, and the batch size for pre-fill can go to tens of thousands. So that's why here you see this, what order, even more, two orders of magnitude difference. And that's why here the compute versus memory bound thing comes in. So for the pre-fill, you have so many tokens to process. So your bottleneck is actually, it moves, your bottleneck is flaps, and the computer on the GPU. For the generation, you don't have that many.

• 50:15Speaker 01

  And your bottleneck is actually, but in the board situation, you need to load the weights from H B to all the way to register. You do it in both situations, but just the bottleneck on the generation is this loading, it's not the flaps. So it's quite simple, actually.

• 50:30Nathan Labenz?

  Okay, I'm not fully sure I understood the distinction between what I was trying to say and what you said. The big idea, as I understood it, was that when you do the input tokens, like to create an attention matrix, for example, you have to do every token, right? And all the attention heads throughout the thing. And so there's a lot of compute there. And so just on a relative basis, there's more compute relative to the moving of parameters in and out. Whereas in the generation side, now you're generating one at a time. There's not really a way to get around the fact, you have to run the whole transformer, right? To generate one token.

• 51:06Speaker 01

  So you do have to run the full transformer, but the fundamental thing here is that you use fundamental transition, which is called KV caching. So in this case, nutshell, it is, as you mentioned, for the pre-fill initial stage, all tokens have to attend to all previous tokens. So it's basically quadratic divided by two. So that's the generation, you need to do the same, but. For the previous tokens, you can cache this attention. You don't have to run it again and again and again.

• 51:32Nathan Labenz?

  This is the KV cache.

• 51:34Speaker 01

  Yeah, exactly. Results into only the token you're trying to generate attending to the prior ones. So this is, as you see, is much cheaper, because it doesn't have this quadratic nature to the linear nature.

• 51:46Nathan Labenz?

  Yeah. How big does the KV cache get? And does it have to come out of the SRAM and go on to the high bandwidth memory? Or does it just stay in the SRAM the whole time?

• 51:58Speaker 01

  Yeah, so it gets a little bit, right? And it varies because of multiple factors. One is the ratio of the Q heads to KV heads. So one of the MQA paper popularized this, and then now all the big models, they have this. So they reduce amount of KV heads, usually order of around four to eight, sometimes even more. But eight has been very common there. So this one thing, another thing, you can shut the model, right? And now you're reducing as well. So what we see when game models, KV cache is not typically big of a problem. But in the end of the day, you have to keep it in HBM for fast access. In some case, you can actually put M CPU memory and it's going to be permissible. But that introduces a lot of complexities as well.

• 52:44Speaker 01

  Yeah, but most typical case, you keep it in HBM.

• 52:48Nathan Labenz?

  Okay, cool. So you were just about to say, when I interjected, okay, so let's do that on Nvidia.

• 52:53Speaker 01

  Ah, yeah. So going through the, how do you implement these met moles and attentions very efficiently? So Nvidia stack has a lot of APIs and they go from very low level to very high level. And let's start in the other direction. So from the high level, Nvidia has now this ready service TFC LLM. So it's based on trying to infer a service, it's already ready to go. But it's easy to get started, although it's not as easy as some open source offering. But compared to the rest, it's much easier. But it's the most rigid as always. So if you want to change anything, it's C++ code, basically good luck. And then when you go down the stack, you have enough flexibility from this setup.

• 53:34Speaker 01

  You can go and take a step down and you can try to write, you want to use your also customized met model using, for example, Kubernetes. These are pre-compiled versions, there's a config you can tweak, but can fundamentally change them. Then you want to take even step down, you want to have even more flexibility. Now with looking at the CUPUS library and with the new sub-library, which is called CUTE. It's pretty cute, I would say, much better than the older API. This is very programmable, way more programmable. It's basically C++ code library. You can perform a lot of customizations there. But even if that is not enough, you can program yourself in CUDA C++.

• 54:14Speaker 01

  Even sometimes that is not enough, then you can go all the way to the hardware instructions and you can program in PTS. So all these options are possible. It's interesting how GPU programming evolved, right? So it all started with the programming CUDA cores. And if you look at GPU, that what it used to be. So you have CPU core, which is very programmable, but not as parallelizable. Now we're going to the GPU and GPU has this CUDA cores. It's also programmable, not as CPU. They are fast. Now the time passes by and that is also not enough. Because as I mentioned, we are optimizing really for MITM rules and this attention, which is also sort of MITM. So how can we do that? Now we need to do, next best thing is embed ASIC into your GPU.

• 54:58Speaker 01

  And that's where the tensor cores come in. That's basically ASIC embedded in GPU. But guess what? It complicates the programming big time. And especially on the newest hardware on H, the tensor cores are asynchronously programmable from the CUDA cores. Now you have two levels of asynchronously. You go to CPU to launch CUDA kernels and then from CUDA, you also asynchronously launch tensor core kernel. So this becomes really complicated. And what we see as a result of it, I see practically very few good kernels which are geared towards H100. V100 and A100, they had a lot of good CUDA kernels geared towards them. People wrote it and enjoyed GPU programming.

• 55:40Speaker 01

  Comes Hopper with there is a sync programming nature. And the amount of those public cores just drops crazily. And there is a very good explanation to it because the program becomes even more complicated. So now the question is, how do we solve this problem? One good step on the welcome development is Triton from OpenAI. So actually, it's a Different compiler. I would say that C++ was a good choice probably back in the day when Nvidia chose it for CUDA. But now it's really getting in the way. Like in some cases, it's much easier to program in assembly for CUDA than CUDA C++ to be honest. Because of that, we see a lot of experimentation and this Triton is a very neat idea.

• 56:21Speaker 01

  And so there you literally program in Python and there is a lot of magic happening behind this. Of course, there is never free lunch, right? So we basically introduce some kind of structure to your GPU programming and it splits it in different levels. So first is like a very simple layer where you program just in Python. Then it translates in what's so-called MLIR, intermediate representation. And then even that one is a low-level MLIR, which is closer to hardware. And then gets translated to PTS, so they go to the low style right away. The nice thing about this is this structure. Because of this very complex GPU architecture, we talked about this structure actually makes it much easier

• 57:01Speaker 01

  to understand the existing new kernels and longer term maintenance is much better. Because if you compare it to the C++ implementations based on CUDA to Triton, those are really hard to understand. There are thousands, thousands lines of code. This provides a very nice clean separation. But again, no free lunch. And if you hope that, oh, I'm just going to write a CUDA kernel in Triton and just write Python and it's going to be super fast, I'm most likely, unless you're doing a point wise, probably not. You have to still look in what is getting generated, what instructions they do to GPUs. Are they pipeline, are they not? What tweaks I need to do and so forth. You have to still look in the output in there. There is no way around it to get to achieve a cutting edge performance.

• 57:44Speaker 01

  But the nice thing is that it has this compiler, because I don't really want to write an assembly, right? Because I need to like keep track of all the registers and there's like 255 registers and it's just that code is unreadable, right? So here, by introducing this structure, the hope is, and it does work in many cases, although not a silver bullet, this structure actually makes the code much, much more readable and maintainable. And you can still achieve state of the art performance. In fact, the Yamat Mullen attention implementations, although they are not as flexible as you can write yourself, they achieve close to state of the art performance, which is very nice proof of this.

• 58:21Nathan Labenz?

  Okay, so maybe try a basic summary and then ask a couple of basic questions. There's a lot of layers between electrons moving around at the very lowest level to Python level scripting, or now we could even say prompting GPT-4 to write me the Python to do the things that I want it to do. So I, and certainly I would expect most of the audience is pretty familiar with the fact that the higher level at which you're working, the more you sort of are at the mercy of all the other layers to determine what your ultimate performance is going to be. I have lived a pretty privileged life where most of the things I've done have been fine at the Python layer, and I've only rarely had to think too much about intensive optimization. So that much I think is fairly intuitive. Like

• 59:04Nathan Labenz?

  you go lower level to do more optimization. What is not super intuitive to me is like, what is happening that is causing the need for lots of ongoing optimization today? One might naively think, hey, if it's all transformers, there's presumably only so much optimization that would need to be done across these handful of operations within the transformer. And yet, it sounds like that's not really the case. So what are the things that are driving the need for continual optimization? Or why isn't just like a finite set of problems that have already all been solved by the community?

• 59:41Speaker 01

  Good question. Yeah. I would say number one reason is a new hardware, a new hardware trying to get more optimal. And it's not just getting better with the existing precision. So like, it used to be we do everything in a single precision float 32 bit. And now it's shifted to a few years ago, we're doing half precision, FP16 for inference, FP16 for training. I would say kind of a new standard, so new precision, go to new precision. But guess what? That is not also good enough. And Nvidia needs to have, and they need to have new stat functions. And they, what they're doing, they're lowering the precision. So

• 1:00:22Speaker 01

  now our go to precision inference and to some extent for e-training is FP8. It's only eight bits, right? We use for a single parameter. And we also do compute in eight bits. Then look at the B one hundred, guess what will it use? Of course, they go to four bits. And look at the recent papers and one of the best papers which I like is so-called one point five bit. It's when they call one bits, not one bit. But fundamentally, the theory is that to train L L M, you just need zero, one minus one. So basically, go left, go right or stay put. So this is like a fundamental property. And if you can have that, you can train a model and looks like this is enough.

• 1:01:06Speaker 01

  You can basically replace the larger parameters model with it is the smaller parameters. Of course, you have to retrain. This is a crazy reduction precision. But it looks like these models perform on par with the half of full precision models. So this is a very welcome development. You will see even more hard optimizations. Because if you do this zero, one minus one, guess what? You don't need much application, it's all addition now. You'll see even more optimizations from media and A and D in coming coming years. Once we train bigger models, because they only like one, three billion models trained for this. Once you train in bigger models, this optimization will become even more critical.

• 1:01:46Speaker 01

  So now, these new precisions, they require very different instructions. And because you see like here, the ratio of generation of G P U is really optimized for this specific ratio of computing and memory back. So here, when the ratio changes, they change the nature of the A P I's. And you need to basically code from scratch, to be honest, for the G P U generation. So all the tests, they works, but it's much slower than it could have been. A good example could be, I would say, a flash attention limitation. Flash attention two was called for A one for for sixteen precision. And then you run H one hundred, you have just half of what you can get. It's same kernel, but it just uses all the operations, all the instructions from in peer,

• 1:02:31Speaker 01

  while the hop instructions are different. So this is like the way fundamentally it's happening and we still have a lot of work ahead of us. For a few years ahead, until we reach three, once we reach these three bits, let's see what's next. But there is still a long way to get there.

• 1:02:47Nathan Labenz?

  Yeah, that three bit paper is, or the 1.58 bit paper is super interesting.

• 1:02:52Speaker 00

  I will add to that, right? Dimish just mentioned, hey, hardware keeps evolving, right? So not just Nvidia hardware keeps moving forward and the basic instruction set keeps changing. But also there's other lines of product, right? There's M D, there's Intel, there's customistics and so on, so they're very broad. And we want to simplify that for our user. But also if we look at upper stack, so we are looking down. If we look up at the application level, so we mentioned latency. You mentioned time to first token, right? So sometimes latency also means time to the first 30 tokens, because they are voice streaming out and they have to get the 30 tokens before they can streaming. Or they care about the end to end latency. So latency also means different things. And people always want to get a spectrum of latency cost curve.

• 1:03:38Speaker 00

  They want to see a curve and they want to pick a point in that curve best for their business. And then on top of that, the input output ratio is different per application. We see a lot of RAG usage where it pushed the input to output ratio to be a very high. It can be 10 to one or even much higher. Or sometimes it people just generate one token for classification. Or sometimes they are generating a lot, like during generating code. So the ratio keeps changing. And the best deployment to find depends on what kind of latency you care, which dot in this latency cost curve.

• 1:04:19Speaker 00

  What is your input output ratio, what are generating, what kind of is your input, your context window length? It all matters. It's very complicated to optimize upwards towards where your application looks like. Does it have repetitive prompts, for example, as another application vertical? Down to ever moving hardware landscape. That is a complexity or application product developers as they are doing things, fun stuff themselves or in enterprises. They're all facing this challenge. So that's where we come in and we say, you don't worry about it. We handle it all for you. So you just focus on your product application development.

• 1:05:03Speaker 00

  So I just want to double click back to what is our roles in this complex ecosystem.

• 1:05:10Nathan Labenz?

  Yeah, but it's funny that you mentioned the curve and picking the point on that curve. I'm putting together a little talk for business leaders and developing a top 10 or whatever things to know for business leaders. One of my tips for them is to learn to think in Pareto curves. And this is something that I see at literally every level of the stack, including like just false positives, false negatives. Truly, every level seems to have this. So do you envision a future of your product experience being literally just showing people these curves? And like, should I expect to see a number of Pareto curves where I can pick like, okay, for this application, I want absolute minimum time to first token.

• 1:05:55Nathan Labenz?

  I'll take the highest cost for this one. I'll take the happy medium for this one. I want the lowest cost. It's a background process or whatever. So it's okay to wait. Is that the kind of experience, the kind of choice that you want to expose ultimately to developers?

• 1:06:07Speaker 00

  So I will draw analogy here. So 20, 25 years ago, database is a new domain and database management systems or start to come into pictures. Actually, it's very interesting analogy. Now I'm thinking about it on the fly. So database queries has strict patterns. It's called SQL. You have your select clause, you have your where clause, goodbye, doing aggregation. So this is a very clear pattern. If you think about that, Jenny and I. The models has very clear pattern to. You have your operator layers. It's more or less stable right now. Although the pattern is simple, but depends on how many columns you have, which column you're gonna put a filter on, which column you're gonna do aggregation,

• 1:06:52Speaker 00

  which column you're gonna do group by. There are all different ways to optimize your query. And at early days, none of these database management systems are smart. That's why it created a whole new entire career. It's called DBA. So basically, all database has knobs for human to tune, to try to optimize things. And those people make a lot of money. Because once they optimize, it's a much better experience. It save a lot of money too. It's very similar to what we just talked about latency and TCO. But over time, all these database management systems become smarter and smarter because they all have a layer called optimizer. Those optimizer observe the workload and it start to create, oh, you're doing a lot of filter on this particular column. So I'm gonna create index.

• 1:07:37Speaker 00

  I'm gonna partition those columns based on your future criteria. So it's much better search, much faster search. You can skip a lot of things. You have to do sequential scanning and so on. So there, then the optimizer becomes smarter and you don't need to hire DBAs at all. It's fully automated. So as long as you observe the workload, you have the workload defined, it tuned, it's self-tuned towards the best outcome. So that's our vision. Today, we have multiple configurations based on what you tell us, your workload pattern, what you care about, we deploy for you, which is really good. But over time, we want to automate ourselves away from that process. Basically, we'll learn from what you are running and the system becomes smarter and smarter. It can become lower latency over time.

• 1:08:22Speaker 00

  It can be higher quality over time. So that's what we aspire to build.

• 1:08:26Nathan Labenz?

  So these options in the product line today would be in the dedicated deployments product, right? Can you run through some of the? I scan these documents, but I wouldn't say I conceived of it in the way that you're describing it now. So what are some of the choices that I get to make today, depending on what my application needs are? And how does that evolve in the short and medium term?

• 1:08:51Speaker 00

  Yeah, for example, if you look at our product offering, we have three tiers. The developer tier, the business tier, the enterprise tier. So the business tier is more like serverless, pay as you go. In enterprise tier, usually they have clear workload definition. And they know where the latency cost curve, where they want to be. Cause they have budget. They also have product requirements. And they know their workload, like in terms of input, how much rack they are using and how much generation and so on. And then we come in and pick a perfect spot for them. And then we deploy a specific configuration towards that workload. But over time, this can be more and more automated. And because their workload can evolve, they don't always have to get us in the loop.

• 1:09:36Speaker 00

  And our system are just self adjusting towards ever changing workload. So that's how the direction we're heading towards.

• 1:09:44Nathan Labenz?

  So could you give me a little bit more intuition for, let's say I'm an enterprise customer and I have three applications. I have one that's user facing and it's going to generate tokens for a voice app. And those first thirty tokens, as you said, are super critical to get fast because I want to have conversational fluency. So that's one. Then the other thing is like a background job where I'm just like, give it to me as cheap as I can. And then there's maybe a chatbot one that's in the middle. I don't want to pay top dollar for that, but I don't want it to be slow. How does that trickle down into what you are actually deploying for those three different scenarios?

• 1:10:18Speaker 00

  So usually in your enterprise, they have many applications. And usually the distribution of traffic is always skewed, right? They're heavy hitters, they're very high volume and they're long tails. They each doesn't have high volume, but they add up. So we recommend they basically think about long tail as one bucket of deployment. And we give them one configuration and we help them optimize for the heavy hitters. Each of the heavy hitter, as I said, they care about different kind of latency. And those heavy hitter probably are coming from their product team. Usually we also work with ML infra team coming from their product team and they have a certain product budget. So then we work together to figure out, hey, which thought you want to pick in this curve? When it comes to quality, then the prompt length coming into picture, right?

• 1:11:02Speaker 00

  How much instruction tuning you put there? Whether you fine tune, take away the instruction prompt? Or you put a lot of a rag information as the context to constrain the model to do less hallucination? So all these kind of product context start to come into picture. And then we have different kind of deployment to hyper optimize for very long context window processing versus they do a lot of generation. So those are also different configurations.

• 1:11:28Speaker 01

  Yeah, I can give a some concrete example where you have to have different deployments for specific use cases. So let's say you really want to get a very old time to first token, the problem is long. And say your model is small, smallish meaning that like say seven billion parameters. You don't have to shut shut it really to put in different GPUs. But the thing is that it's still not fast enough to run on a single GPU because the problem is so long. So now for this specific use case, you want to shut it, but there is a cost for shut. So there are like fundamentally three shutting techniques. One is to shut your model weights. Another one is to shut the activation of the input.

• 1:12:11Speaker 01

  The third one is to chop model in pieces that is called pipeline parallelism. So the last one doesn't really help you with the latencies. It just helps when the model is too big. The case when you shut model weights, also called model parallelism, aka tensor parallelism. This help also with the model size and does help with the latencies. Although there is a cost for shutting. So basically your throughput overall declines. And the first one, the input shutting, usually it doesn't help you with the model size. And it's pretty cheap, but there is a caveat attention. So you need to take care of attention because you are tempted to add a token. So now you have to have a bit more complexity in your setup.

• 1:12:54Speaker 01

  So the bottom line here is that for the pre fail, for example, most common shining right now is the tensor parallelism. If you choose it, you don't want to always shut. You really wanna don't shut as much as possible up to the point where you have to. Because there is a cost because your throughput will be will be tanking the more you shut. But so much you have to shut because you cannot meet the latency budget. So this is like fundamental thing and I don't think this problem goes away any any time soon.

• 1:13:19Nathan Labenz?

  Yeah, okay. So I think again, everybody will be conceptually familiar with the idea that. There is a latency throughput trade off. That seems to be a very recurring pattern. But if I understand what you're saying correctly here, it is that if you have a really long prompt, even if you have a small model that can fit onto one GPU, you may want to split across GPUs. I'm not quite clear on the nature of the splitting. Are you splitting like layer wise? Like you'd have early layers on one and then later layers on another?

• 1:13:51Speaker 01

  So you have you lay your split. Well, it's more complicated, but it's like very nutshell because it goes to Megatron sharding. When you try to make moves, even do one columnized another row wise and minimize the amount. Because in that, once you do this, you do reshuffle, you do all reduce. So you want to minimize that. And this Megatron style of sharding helps you. Fundamentally, yeah, you're splitting actually the weights across GPUs. And then you need to all reduce the activations.

• 1:14:19Nathan Labenz?

  Can you give us a little bit better intuition for that? Because that's something that is not super clear to me. And I suspect a lot of people are not going to be very clear on it either. I've seen examples of this in my study of the Mamba and Mamba related things recently. I've been trying to get a better handle on this. And one thing that's like surprising or counterintuitive is that there are ways to take advantage of basically the associative property, where you can do these quick counterintuitive algorithms, even for just like adding up numbers that kind of look pretty weird, but end up being faster, especially because it allows for this kind of parallelism. Am I on the right track with that?

• 1:14:57Speaker 01

  Yes, so here it applies not necessarily. This kind of sharding is like, I would say, model architecture independent. So these are the input data parallelism or tensor parallelism and pipeline parallelism. They've been there like before even GNI kicked in. But they're kind of fundamental to machine learning. But then which one you want to apply totally depends on the architecture because the one architecture is much better. For example, if you don't have a sequential nature, you're using pipeline parallelism is actually no goal. It's not going to say anything because you cannot even chop the model. And it used to be like that. All transformer models tend to have layers, so it's much better. But something like if you look back on the image, older image model based on the ResNet, here's state connections.

• 1:15:40Speaker 01

  It's become harder to do pipeline parallelism. Yeah, and then the tensor parallelism actually. Conceptually pretty is harder to understand and the data input parallel is easier because you just split data and data is independent. So you could just duplicate the models with the inputs, right? And so you make a process and then join them in there. That is easy. The model parallel, the test parallel, which actually do use right now, hardest to understand because once the way started interacting with each other, you need to make sure your math is correct because you cannot just shard across if you need to, for example, to values, if you need to add them in there. If you want to shard them, but you have to still do the addition. If you need to multiply, you still have to do the multiplication.

• 1:16:22Speaker 01

  So when you do that, you need to probably gather them on one rank or another rank or send them across them and then split these operations across the ranks. Of course, sending across GPUs is way more expensive than just sending from a single GPU to the registers. It might be a bit worse, even more. So it becomes quite complicated, but these are the kind of compromises you have to go through. So basically, the other is you need to make sure that your math is still correct once you do the shard. So this is the other of this and there are different techniques and basically not every operation can be easily split. Besides, you need to look at specific operations, specific matrix multiplication, how you can split.

• 1:17:03Speaker 01

  Yeah, so if you want to research about this, yes, Megatron is a good starting point. It has a ton of descriptions, how you do the mathematical shard.

• 1:17:12Nathan Labenz?

  Yeah, okay, cool. I'll check that out. That's a good pointer. Let me just try to summarize the three kinds and make sure I at least have the conceptual framework right. Simplest one is data parallelism. That is just make multiple instantiations of the model. You can split the data across them. That's essentially like a load balancing setup. The next level up, which maybe is the most complicated, is actually splitting the weights. This is the kind of thing that you would do because you have a super long prompt, for example, and you want to have a fast time to first token. So you need to have even more parallelism of the computation and it can be worth it.

• 1:17:53Nathan Labenz?

  But it comes at this cost of now you've got GPU to GPU communication, plus the complexity of what the hell am I doing across all these different, now that I'm splitting weights, like, okay, that's a lot to keep track of. And then the third one is the pipeline, and that is where you actually split early layers from middle layers, late layers, whatever, and you're feeding through multiple things in a pipeline in a sequential way. Okay, cool. That's helpful. Lots to learn. I've been doing this around the clock, I feel like, for a few years now, and still a target-rich environment for new things to learn.

• 1:18:27Speaker 01

  Yeah, and they can combine them in many different ways and go to really complicated setups. So it gets hairy.

• 1:18:34Nathan Labenz?

  Yeah, independent dimensions as well. Okay, so we've covered a lot of ground. One other question I wanted to ask about is, with all this complexity of everything we've talked about, the one thing that was surprising to me is that you are also training your own models. You have your own branded fireworks function calling models. And especially given your earlier comment that you think at least the small models are converging, how does that fit into the overall strategy? Like, why bother training your own models as opposed to just focusing on the insane complexity of the inference stack and letting that convergence happen independently of your efforts?

• 1:19:15Speaker 00

  That's a great question. And one of the biggest, like, the north star of the product web building is not just serving individual models. So we want to build the inference stack for knowledge extraction. And when we talk about knowledge, we can first look at the knowledge provided by large language models. It's very capable now. It can answer many questions. It surprises us. But at the end, all large language models has limited knowledge. Why? The knowledge is limited by its training data. Its training is limited in terms of time range. It doesn't have the most latest information. And it doesn't have the information. People cannot crawl from internet.

• 1:19:58Speaker 00

  Those public informations, so there are just limits, limit to the corpus of information people can crawl directly from. And many of those knowledge also live outside of large language models. Because there are foundation models, they are generate images, generate videos, generate audios. They also do information extraction from images. So I want to create a framework to think about the knowledge distribution is beyond just a large language model. And even for given a large language model, it's limited. And there are other modalities that carry a lot of information. Beyond that, there are a lot of knowledges hidden behind APIs that we cannot extract from.

• 1:20:41Speaker 00

  And we use API very extensively day to day. For example, we do search, right? So there's Google search, there's weather API, there's dog extraction API, there's map API. For personal productivity, um, there are APIs for docs, for spreadsheets, for calendars, and so on, so forth. So APIs are everywhere. Within enterprise, one enterprise internally can have hundreds of APIs, surface areas too. There are a lot of key value stores, document stores, production logs, pub sub systems, internal search. So API itself contain a large corpus of knowledge. So think about function calling. What is function calling? So our file function, the model we build ourselves is serving the function calling area.

• 1:21:24Speaker 00

  And function calling is the layer to tie knowledges together behind all different modalities of foundation models and APIs together. So to us strategically, this is a critical layer for us to build the best inference tier for knowledges. And with that, you can build a lot of an application. For example, you can build a personal admin to knock down and to everyone on the planet that can sort out tedious work. This application can learn what you like over time and can preemptively finish work before you even ask and put out the reminders or even suggest things ahead of time. I wish I could have a personal admin like that.

• 1:22:04Speaker 00

  But those kind of new application has to depend on tools and ways to extract knowledge across the board from foundation models and from APIs. That's the fundamental reason why we built our own function calling model. We have released two versions, open weights, so everyone can use it. You can download from Hugging Face. We are currently working on our third version that's gonna come out soon and will have a great quality bump. So stay tuned.

• 1:22:32Nathan Labenz?

  Cool. I can look forward to that. I know we're just about at the limit of our time together today, and you guys have been very generous with your time and knowledge. Is there anything that we haven't managed to touch on yet that you want to make sure you mention?

• 1:22:44Speaker 00

  I think we touch on a lot. Yeah, I think you have one question about SSM.

• 1:22:48Nathan Labenz?

  Yeah, that's a hobby course of mine. Is that on your customers radar? Is it on your radar? Is it on your roadmap?

• 1:22:54Speaker 00

  So actually, we haven't heard too much about the requirements for our customers. I think it's a very cool technology. So of course, when we talk about SSM, it's in the context of Mamba. The main barrier to practical adoption is mainly quality in my mind compared with transformer models. So we haven't seen a pure Mamba SSM-based model emerge as highly competitive in quality to Mistral, Claude, GPT 3.5 or 4 yet. I think AI 21 Labs just open source Jamba. It's a hybrid of SSM and transformer models, but the real quality benchmark is one thing. Real quality in practice, another thing. So real quality using practical use cases yet to be verified.

• 1:23:39Speaker 00

  So that's why I think we haven't seen a huge demand there, and I guess people are more cautious. It's hard to say what will be the technical challenge. I think Dimitri was pretty confident we could support it in no time. But again, coming back to quality, it's a little bit hard to assess. So those kind of architecture, Mamba and SSM is really good for long context by removing the chaotic nature of transformer. But for long context, it's hard to assess the quality. Needle in the haystack is the most common used benchmark, but it's not very comprehensive. I don't think as an industry, we have standardized the benchmark for measuring long context quality yet. But this is another area the whole industry needs to move forward.

• 1:24:22Speaker 01

  This type of benchmarks still show that there is room to go for Mamba architecture to meet the current transformer-based models.

• 1:24:34Nathan Labenz?

  Yeah, my money's on the hybrids, for what it's worth. There's been some really interesting kind of very fine-grained studies of the comparative strengths and weaknesses. And there will be, I'm sure, many more developments. And so we are doing a quick bonus recording, 10 minutes or whatever, on some exciting company developments. This goes to show how fast everything is moving in AI.

• 1:24:57Nathan Labenz?

  I think it's only been seven days since we recorded and I noticed that you mentioned the new image generation model from Stability on our first call, but I didn't know that out of that, Fireworks has been in partnership talks and now has announced a partnership with Stability AI to be the exclusive inference provider of Stable Diffusion 3. That's pretty awesome. I'm sure that is a position that a lot of inference providers would love to have. And I am just very curious to hear more about how that came to be and what the competition was like, what sort of work you had to do to make it happen, anything you can share about the nature of the partnership. But for starters, congratulations.

• 1:25:39Speaker 01

  Thank you. Yeah, we've been working on this for a while now and overall the partnership makes total sense because if you look at on the company level, Stability has a lot of very good resources who train very good models. If you look historically, they've been very successful monetizing that compared to OpenAI. And on another hand, Fireworks, we have very good production-minded engineers who are also very good at model performance, talked about at length during our previous conversation. We can bring these models, which are trained by those amazing researchers, to production in a long time and run them very cheaply.

• 1:26:22Speaker 01

  So that's the high-level, partly partnership makes sense. I can talk a little bit more about optimization strategies, what we implore for the image models and how they are different from the text models and also similar at the same time. So one exciting development about the image models is that with the new architecture of Stable Diffusion 3, people move to the Transformer base architecture and that changes a lot of things because if you look at the performance profile before, it's mostly like that and it has that some convolution. There's nothing common with the test models, but now it's all transformer based.

• 1:27:02Speaker 01

  So we were able to apply a lot of optimizations which we learned of running hosting in the uh the text models on these image models. So that actually provided was quite a bit of boost. For example, with the applying most optimal kernels, applying different sharing strategies. Yeah, in the same time, the image models are still different. Uh they don't have this generation phase. I would say they are running in the custom preview phase, so to speak. And there is no only one big batch sizes, but you know, which size one is a big enough to sustain GPUs. But there are still a lot of these commonalities because of the transformer architecture. So we're excited about that.

• 1:27:43Nathan Labenz?

  Yeah, that's cool. I think folks will probably be fairly familiar with multimodal language models that accept images as input. So in that context, there's a translation rate, depending on the size of your image to the number of tokens that sort of counts as. And broadly speaking, again, people will know that you tend to break the image down into patches and feed it in that way. And then in most of these multimodal models gets mapped to a language latent space and then was used to inform your generation. This is obviously quite a different situation. So can you give us a little bit more about how it works? Are we generating one image patch at a time?

• 1:28:26Nathan Labenz?

  And how should we think about familiar concepts from the transformer like the vocabulary size or the attention window? How do those concepts map onto this image generation use case?

• 1:28:36Speaker 01

  Yeah, for the image generation, the best analogy here is to map the prefill part of the text generation to the image generation. So those map pretty nicely. The biggest difference there that on the text generation, the input is variable, right? Whereas on the image generation, yes, you have an encoder. But then once you run through those tokenizers, right? And encoders, that input actually to the transformer layer, this is the heaviest by far, is fixed. So this side is fixed, which actually makes, I would say, optimizing a bit more straightforward comparatively in the text models. The new three architecture, so the whole of it is quite different, right?

• 1:29:21Speaker 01

  But in the same time, the heaviest part by far is the transformer layer now. And so that's what our focus has been. And only we're learning is from the text models about the shot. This still apply to this transformer. But it's still like a very long sequential model. And basically, all these techniques do apply.

• 1:29:44Nathan Labenz?

  So what is a token output? I assume it's like a square that consists of a certain number of pixels, but I haven't studied this in depth.

• 1:29:54Speaker 01

  Yeah, I would say that concept is still the same way across the diffusion models. It didn't change with the introduction of the transformer architecture. Yeah, so basically, in the image generation, you have to map these pixels essentially to default input values. And so there is basically two different modes of how you process the input. For this, for example, there is an X driven image generation. So there you have to organize the text, right? And then get transformed into a, and they all map the input to the image generation. But I would say the input is quite different because what you say, you generate this noise because it's diffusion process, right? So that's what the fundamental difference is. So you basically generate this noise. It's literally a randomized noise generation.

• 1:30:40Speaker 01

  And then you run into these diffusion steps one by one. And you try to reverse diffusion process and you enhancing this image with each step. So that's what's happening in that. Yeah, and in the end, you run and it convert this output of the transformer to the actual pixels.

• 1:30:59Nathan Labenz?

  Yeah, let me try to state it back to you. You start with noise, you're running then multiple forward passes?

• 1:31:07Nathan Labenz?

  Where each forward pass constitutes one denoising step? And you have how many denoising steps on the way to a full image?

• 1:31:16Speaker 01

  It depends. So for this stable division, three, you can choose, but the four one, we run on our fifty steps. But for the turbo model, it's quite a little bit different architecture. And there, actually, you're running around around four steps. So very few. In which body is worse, but it's smaller. The turbo model really or you're just speed. So speed is the number one factor. And for this table division, three, the quality is the number one factor.

• 1:31:43Nathan Labenz?

  Gotcha. And in each forward pass, unlike a language model where you're auto-regressing and you are holding all the current tokens fixed and adding one to the end. Here you are starting with a bunch of patches and all of those patches get denoised. So your input is N patches and your output is N patches all denoised. And you do that fifty times in a row until you get to the refined final image.

• 1:32:11Speaker 01

  Yeah, but this stays constant, right? So that's very different from the text models. So this stays constant.

• 1:32:17Nathan Labenz?

  Yeah. How many patches or is it patches and tokens basically equivalent, right?

• 1:32:22Speaker 01

  Yeah, you can say that. And they have different settings on the different model sizes. So again, this is not the constant thing that changes.

• 1:32:29Nathan Labenz?

  Gotcha. Okay, cool. So how did this? It sounds like something that was maybe driven by the stability leadership team saying, hey, we're good at one part of this, but we're not exactly set up to do the other part. Let's go looking for a partner. Did they come to you directly and say, hey, we think fireworks is the company for us? Whatever you can share is cool with me. If I'm inferring, reading between the lines correctly, it sounds like they identified you and this was not like a tournament that you participated in, but more of a they knew who they wanted to go with, it sounds like, which is obviously.

• 1:33:01Speaker 01

  Yeah, so there's many factors which were in play. And again, like from the company's show us this standpoint, our knowledge and leadership with the performance during our models plus the writing and in production reliably. So those were the most important factors. And yeah, so if you ask people who know fireworks, mostly people say number one is for this speed. And so I think those two. That close to play the hero.

• 1:33:27Nathan Labenz?

  Gotcha. Cool. I tried it this morning. It was very simple to get started because I just saw the the news of the partnership. So I went to fireworks directly to try to find the model in the playground, whatever. And then I realized, oh, it's actually still presented as part of the stability API. So I ended up on the stability docs and got the code there and made a little notebook and started making calls. Is there anything that you can tell about how it works behind the scenes? It seems like I'm still hitting their domain, right? So there's sort of a middle layer and then they're routing over to you. Um. Is there anything interesting in the way that part of the technical setup is constructed?

• 1:34:06Speaker 01

  The main reason for this kind of setup is that stability is the company they're quite worried about people misusing their models. And they want to sanitize the input. And that is also, I would say, quite relevance related work on the train models to do that. So it makes for the sense if they can control that process of their input sanitization. So that's one of the reasons we have need to route this way. Yeah. Plus for image generation, you can say, oh, it reduces some latency is and we are actually working on reducing them. But for image generation, typically, I would say quality is that is one of the main things.

• 1:34:47Speaker 01

  Plus, the cost of running models. So that's why this routing is actually works out pretty well in this situation.

• 1:34:54Nathan Labenz?

  Cool. Yeah, it's interesting. Any more detail or interesting aspects of the optimization strategy that you'd like to share?

• 1:35:02Speaker 01

  Yeah, basically to repeat what I just said, I emphasize that we're very excited about the proliferation of transformer architecture. And next thing we try to work on is a low precision quantization for image models. As we are, yeah, still running on in a P16 precision. But next thing, one of the next things to mostly to reduce cost of running these models will be looking in the or basically compared to the text models, which are the most performant ones in the P A. So we're not doing that for image models yet. So that's one of the areas of studying. A interesting line of intelligence directly.

• 1:35:42Speaker 01

  The techniques we learn from text models on image models because it has quality implications. So you need to do some extra work to make sure the qualities are progressing. So that is quite a bit different. So for image, if you think about it, I was different from that. As is for text to generate in token. So it's only choosing the right token, next token. You're good. It doesn't really matter what other tokens were really in play. For for text, you're producing the image. So any kind of glitches on the image that become invisible. So those are the kind of main differences. And if you go on a lower level is that if you look at the condensation techniques is that on the image versus text models, on the text models,

• 1:36:23Speaker 01

  you really wanna incorporate the high spikes on activations, for example, which is quite a long path of weight distribution. And you really wanna incorporate them because they do affect the next token choice, right? At the center, the token choice is the most important thing. Also with the image models, those high activation are not as important. So you can actually cut them and I think what is not, although image what is not as a fact. So it's more precision of every activation is as aggregate more important. So that's like a kind of higher reading intuition on the condensation techniques.

• 1:36:58Nathan Labenz?

  Yeah, that's quite interesting. Here's another way to say that, basically, sort of mental model that I have for the language models is that they have obviously very messy, noisy kind of overlapping, but nevertheless, some sort of internal circuits that a given forward pass kind of slots into, such that you have these, as you said, high activation points in the network that seem to be doing most of the work beyond a certain layer. Whereas here, it's like all to all, the whole, whole way through because you need every ultimately.

• 1:37:36Speaker 01

  Yeah, every next of course, exactly, yeah. We'll definitely be announcing more things going forward. So watch out for the, for newer allow spans. And we also be adjusting based on the impact we're receiving from our audience. So yeah, some more things to come.

• 1:37:51Nathan Labenz?

  Guys, I know you gotta go. This has been a great conversation. Lin Chow and Dmytro, Dmytro Ivchenko, co-founders of Fireworks AI. Thank you for being part of the cognitive revolution. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected], or you can DM me on the social media platform of your choice.