Why Compound AI + Open Source will beat Closed AI

We have a full slate of upcoming events https://www.latent.space/community: AI Engineer London, AWS Re:Invent in Las Vegas, and now Latent Space LIVE! at NeurIPS in Vancouver and online. Sign up to join https://lu.ma/LSLIVE and speak https://forms.gle/xHACUUeNBrYb7XZu9! We are still taking questions for our next big recap episode! Submit questions and messages on Speakpipe here https://www.speakpipe.com/LatentSpace for a chance to appear on the show! We try to stay close to the inference providers as part of our coverage, as our podcasts with Together AI https://www.latent.space/p/together?utm_source=publication-search and Replicate https://www.latent.space/p/replicate?utm_source=publication-search will attest: However one of the most notable pull quotes from our very well received Braintrust episode https://www.latent.space/p/braintrust was his opinion that open source model adoption has NOT gone very well and is actually declining in relative market share terms (it is of course increasing in absolute terms): Today’s guest, Lin Qiao, would wholly disagree. Her team of Pytorch/GPU experts are wholly dedicated toward helping you serve and finetune the full stack of open source models from Meta and others, across all modalities (Tex https://docs.fireworks.ai/guides/querying-text-modelst, Audio https://docs.fireworks.ai/api-reference/audio-transcriptions, Image https://docs.fireworks.ai/api-reference/generate-a-new-image-from-a-text-prompt, Embedding https://docs.fireworks.ai/guides/querying-embeddings-models, Vision-understanding https://docs.fireworks.ai/guides/querying-vision-language-models), helping customers like Cursor and Hubspot scale up open source model inference both rapidly and affordably. Fireworks has emerged after its successive funding rounds with top tier VCs as one of the leaders of the Compound AI movement https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/, a term first coined by the Databricks/Mosaic gang at Berkeley AI and adapted as “Composite AI https://www.gartner.com/en/newsroom/press-releases/2024-09-25-gartner-identifies-composite-ai-and-supply-chain-data-governance-as-highest-impact-supply-chain-planning-technology-trends” by Gartner: Replicating o1 We are the first podcast to discuss Fireworks’ f1 https://fireworks.ai/blog/fireworks-compound-ai-system-f1, their proprietary replication of OpenAI’s o1. This has become a surprisingly hot area of competition in the past week as both Nous Forge and Deepseek r1 have launched competitive models https://buttondown.com/ainews/archive/ainews-deepseek-r1-claims-to-beat-o1-preview-and/. Full Video Podcast Like and subscribe! https://youtu.be/hQ7rppq-eZc Timestamps * 00:00:00 Introductions * 00:02:08 Pre-history of Fireworks and PyTorch at Meta * 00:09:49 Product Strategy: From Framework to Model Library * 00:13:01 Compound AI Concept and Industry Dynamics * 00:20:07 Fireworks' Distributed Inference Engine * 00:22:58 OSS Model Support and Competitive Strategy * 00:29:46 Declarative System Approach in AI * 00:31:00 Can OSS replicate o1? * 00:36:51 Fireworks f1 * 00:41:03 Collaboration with Cursor and Speculative Decoding * 00:46:44 Fireworks quantization (and drama around it) * 00:49:38 Pricing Strategy * 00:51:51 Underrated Features of Fireworks Platform * 00:55:17 Hiring Transcript Alessio [00:00:00]: Hey everyone, welcome to the Latent Space Podcast. This is Alessio, partner at CTO at Danceable Partners, and I'm joined by my co-host, Swyx founder, Osmalayar. Swyx [00:00:11]: Hey, and today we're in a very special studio inside the Fireworks office with Lin Qiang, CEO of Fireworks. Welcome. Yeah. Lin [00:00:20]: Oh, you should welcome us. Swyx [00:00:21]: Yeah, welcome. Yeah, thanks for having us. It's unusual to be in the home of a startup, but it's also, I think our relationship is a bit unusual compared to all our normal guests. Definitely. Lin [00:00:34]: Yeah. I'm super excited to talk about very interesting topics in that space with both of you. Swyx [00:00:41]: You just celebrated your two-year anniversary yesterday. Lin [00:00:43]: Yeah, it's quite a crazy journey. We circle around and share all the crazy stories across these two years, and it has been super fun. All the way from we experienced Silicon Valley bank run to we delete some data that shouldn't be deleted operationally. We went through a massive scale where we actually are busy getting capacity to, yeah, we learned to kind of work with it as a team with a lot of brilliant people across different places to join a company. It has really been a fun journey. Alessio [00:01:24]: When you started, did you think the technical stuff will be harder or the bank run and then the people side? I think there's a lot of amazing researchers that want to do companies and it's like the hardest thing is going to be building the product and then you have all these different other things. So, were you surprised by what has been your experience the most? Lin [00:01:42]: Yeah, to be honest with you, my focus has always been on the product side and then after the product goes to market. And I didn't realize the rest has been so complicated, operating a company and so on. But because I don't think about it, I just kind of manage it. So it's done. I think I just somehow don't think about it too much and solve whatever problem coming our way and it worked. Swyx [00:02:08]: So let's, I guess, let's start at the pre-history, the initial history of Fireworks. You ran the PyTorch team at Meta for a number of years and we previously had Sumit Chintal on and I think we were just all very interested in the history of GenEI. Maybe not that many people know how deeply involved Faire and Meta were prior to the current GenEI revolution. Lin [00:02:35]: My background is deep in distributed system, database management system. And I joined Meta from the data side and I saw this tremendous amount of data growth, which cost a lot of money and we're analyzing what's going on. And it's clear that AI is driving all this data generation. So it's a very interesting time because when I joined Meta, Meta is going through ramping down mobile-first, finishing the mobile-first transition and then starting AI-first. And there's a fundamental reason about that sequence because mobile-first gave a full range of user engagement that has never existed before. And all this user engagement generated a lot of data and this data power AI. So then the whole entire industry is also going through, falling through this same transition. When I see, oh, okay, this AI is powering all this data generation and look at where's our AI stack. There's no software, there's no hardware, there's no people, there's no team. I want to dive up there and help this movement. So when I started, it's very interesting industry landscape. There are a lot of AI frameworks. It's a kind of proliferation of AI frameworks happening in the industry. But all the AI frameworks focus on production and they use a very certain way of defining the graph of neural network and then use that to drive the model iteration and productionization. And PyTorch is completely different. So they could also assume that he was the user of his product. And he basically says, researchers face so much pain using existing AI frameworks, this is really hard to use and I'm going to do something different for myself. And that's the origin story of PyTorch. PyTorch actually started as the framework for researchers. They don't care about production at all. And as they grow in terms of adoption, so the interesting part of AI is research is the top of our normal production. There are so many researchers across academic, across industry, they innovate and they put their results out there in open source and that power the downstream productionization. So it's brilliant for MATA to establish PyTorch as a strategy to drive massive adoption in open source because MATA internally is a PyTorch shop. So it creates a flying wheel effect. So that's kind of a strategy behind PyTorch. But when I took on PyTorch, it's kind of at Caspo, MATA established PyTorch as the framework for both research and production. So no one has done that before. And we have to kind of rethink how to architect PyTorch so we can really sustain production workload, the stability, reliability, low latency, all this production concern was never a concern before. Now it's a concern. And we actually have to adjust its design and make it work for both sides. And that took us five years because MATA has so many AI use cases, all the way from ranking recommendation as powering the business top line or as ranking newsfeed, video ranking to site integrity detect bad content automatically using AI to all kinds of effects, translation, image classification, object detection, all this. And also across AI running on the server side, on mobile phones, on AI VR devices, the wide spectrum. So by the time we actually basically managed to support AI across ubiquitous everywhere across MATA. But interestingly, through open source engagement, we work with a lot of companies. It is clear to us like this industry is starting to take on AI first transition. And of course, MATA's hyperscale always go ahead of industry. And it feels like when we start this AI journey at MATA, there's no software, no hardware, no team. For many companies we engage with through PyTorch, we feel the pain. That's the genesis why we feel like, hey, if we create fireworks and support industry going through this transition, it will be a huge amount of impact. Of course, the problem that the industry is facing will not be the same as MATA. MATA is so big, right? So it's kind of skewed towards extreme scale and extreme optimization in the industry will be different. But we feel like we have the technical chop and we've seen a lot. We'll look to kind of drive that. So yeah, so that's how we started. Swyx [00:06:58]: When you and I chatted about the origins of fireworks, it was originally envisioned more as a PyTorch platform, and then later became much more focused on generative AI. Is that fair to say? What was the customer discovery here? Lin [00:07:13]: Right. So I would say our initial blueprint is we should build a PyTorch cloud because a PyTorch library and there's no SaaS platform to enable AI workloads. Swyx [00:07:26]: Even in 2022, it's interesting. Lin [00:07:28]: I would not say absolutely no, but cloud providers have some of those, but it's not first class citizen, right? At 2022, there's still like TensorFlow is massively in production. And this is all pre-gen AI, and PyTorch is kind of getting more and more adoption. But there's no PyTorch-first SaaS platform existing. At the same time, we are also a very pragmatic set of people. We really want to make sure from the get-go, we get really, really close to customers. We understand their use case, we understand their pain points, we understand the value we deliver to them. So we want to take a different approach instead of building a horizontal PyTorch cloud. We want to build a verticalized platform first. And then we talk with many customers. And interestingly, we started the company in September 2022, and in October, November, the OpenAI announced ChatGPT. And then boom, when we talked with many customers, they were like, can you help us work on the JNS aspect? So of course, there are some open source models. It's not as good at that time, but people are already putting a lot of attention there. Then we decided that if we're going to pick a vertical, we're going to pick JNI. The other reason is all JNI models are PyTorch models. So that's another reason. We believe that because of the nature of JNI, it's going to generate a lot of human consumable content. It will drive a lot of consumer, customer-developer-facing application and product innovation. Guaranteed. We're just at the beginning of this. Our prediction is for those kind of applications, the inference is much more important than training because inference scale is proportional to the up-limit award population. And training scale is proportional to the number of researchers. Of course, each training round could be very expensive. Although PyTorch supports both inference and training, we decided to laser focus on inference. So yeah, so that's how we got started. And we launched our public platform August last year. When we launched, it was a single product. It's a distributed inference engine with a simple API, open AI compatible API with many models. We started with LM and then we added a lot of models. Fast forward to now, we are a full platform with multiple product lines. So we love to kind of dive deep into what we offer. But that's a very fun journey in the past two years. Alessio [00:09:49]: What was the transition from you start to focus on PyTorch and people want to understand the framework, get it live. And now say maybe most people that use you don't even really know much about PyTorch at all. You know, they're just trying to consume a model. From a product perspective, like what were some of the decisions early on? Like right in October, November, you were just like, hey, most people just care about the model, not about the framework. We're going to make it super easy or was it more a gradual transition to the model library Swyx [00:10:16]: you have today? Lin [00:10:17]: Yeah. So our product decision is all based on who is our ICP. And one thing I want to acknowledge here is the generic technology is disruptive. It's very different from AI before GNI. So it's a clear leap forward. Because before GNI, the companies that want to invest in AI, they have to train from scratch. There's no other way. There's no foundation model. It doesn't exist. So that means then to start a team, first hire a team who is capable of crunch data. There's a lot of data to crunch, right? Because training from scratch, you have to prepare a lot of data. And then they need to have GPUs to train, and then you start to manage GPUs. So then it becomes a very complex project. It takes a long time and not many companies can afford it, actually. And the GNI is a very different game right now, because it is a foundation model. So you don't have to train anymore. That makes AI much more accessible as a technology. As an app developer or product manager, even, not a developer, they can interact with GNI models directly. So our goal is to make AI accessible to all app developers and product engineers. That's our goal. So then getting them into the building model doesn't make any sense anymore with this new technology. And then building easy, accessible APIs is the most important. Early on, when we got started, we decided we're going to be open AI compatible. It's just kind of very easy for developers to adopt this new technology, and we will manage the underlying complexity of serving all these models. Swyx [00:11:56]: Yeah, open AI has become the standard. Even as we're recording today, Gemini announced that they have open AI compatible APIs. Interesting. So we just need to drop it all in line, and then we have everyone popping in line. Lin [00:12:09]: That's interesting, because we are working very closely with Meta as one of the partners. Meta, of course, is kind of very generous to donate many very, very strong open source models, expecting more to come. But also they have announced LamaStack, which is basically standardized, the upper level stack built on top of Lama models. So they don't just want to give out models and you figure out what the upper stack is. They instead want to build a community around the stack and build a new standard. I think there's an interesting dynamics in play in the industry right now, when it's more standardized across open AI, because they are kind of creating the top of the funnel, or standardized across Lama, because this is the most used open source model. So I think it's a lot of fun working at this time. Swyx [00:13:01]: I've been a little bit more doubtful on LamaStack, I think you've been more positive. Basically it's just like the meta version of whatever Hugging Face offers, you know, or TensorRT, or BLM, or whatever the open source opportunity is. But to me, it's not clear that just because Meta open sources Lama, that the rest of LamaStack will be adopted. And it's not clear why I should adopt it. So I don't know if you agree. Lin [00:13:27]: It's very early right now. That's why I kind of work very closely with them and give them feedback. The feedback to the meta team is very important. So then they can use that to continue to improve the model and also improve the higher level I think the success of LamaStack heavily depends on the community adoption. And there's no way around it. And I know the meta team would like to kind of work with a broader set of community. But it's very early. Swyx [00:13:52]: One thing that after your Series B, so you raced for Benchmark, and then Sequoia. I remember being close to you for at least your Series B announcements, you started betting heavily on this term of Compound AI. It's not a term that we've covered very much in the podcast, but I think it's definitely getting a lot of adoption from Databricks and Berkeley people and all that. What's your take on Compound AI? Why is it resonating with people? Lin [00:14:16]: Right. So let me give a little bit of context why we even consider that space. Swyx [00:14:22]: Because like pre-Series B, there was no message, and now it's like on your landing page. Lin [00:14:27]: So it's kind of very organic evolution from when we first launched our public platform, we are a single product. We are a distributed inference engine, where we do a lot of innovation, customized KUDA kernels, raw kernel kernels, running on different kinds of hardware, and build distributed disaggregated execution, inference execution, build all kinds of caching. So that is one. So that's kind of one product line, is the fast, most cost-efficient inference platform. Because we wrote PyTorch code, we know we basically have a special PyTorch build for that, together with a custom kernel we wrote. And then we worked with many more customers, we realized, oh, the distributed inference engine, our design is one size fits all. We want to have this inference endpoint, then everyone come in, and no matter what kind of form and shape or workload they have, it will just work for them. So that's great. But the reality is, we realized all customers have different kinds of use cases. The use cases come in all different forms and shapes. And the end result is the data distribution in their inference workload doesn't align with the data distribution in the training data for the model. It's a given, actually. If you think about it, because researchers have to guesstimate what is important, what's not important in preparing data for training. So because of that misalignment, then we leave a lot of quality, latency, cost improvement on the table. So then we're saying, OK, we want to heavily invest in a customization engine. And we actually announced it called FHIR Optimizer. So FHIR Optimizer basically helps users navigate a three-dimensional optimization space across quality, latency, and cost. So it's a three-dimensional curve. And even for one company, for different use cases, they want to land in different spots. So we automate that process for our customers. It's very simple. You have your inference workload. You inject into the optimizer along with the objective function. And then we spit out inference deployment config and the model setup. So it's your customized setup. So that is a completely different product. So that product thinking is one size fits all. And now on top of that, we provide a huge variety of state-of-the-art models, hundreds of them, varying from text to large state-of-the-art English models. That's where we started. And as we talk with many customers, we realize, oh, audio and text are very, very close. Many of our customers start to build assistants, all kinds of assistants using text. And they immediately want to add audio, audio in, audio out. So we support transcription, translation, speech synthesis, text, audio alignment, all different kinds of audio features. It's a big announcement. You should have heard by the time this is out. And the other areas of vision and text are very close with each other. Because a lot of information doesn't live in plain text. A lot of information lives in multimedia format, images, PDFs, screenshots, and many other different formats. So oftentimes to solve a problem, we need to put the vision model first to extract information and then use language model to process and then send out results. So vision is important. We also support vision model, various different kinds of vision models specialized in processing different kinds of source and extraction. And we're also going to have another announcement of a new API endpoint we'll support for people to upload various different kinds of multimedia content and then get the extract very accurate information out and feed that into LM. And of course, we support embedding because embedding is very important for semantic search, for RAG, and all this. And in addition to that, we also support text-to-image, image generation models, text-to-image, image-to-image, and we're adding text-to-video as well in our portfolio. So it's a very comprehensive set of model catalog that built on top of File Optimizer and Distributed Inference Engine. But then we talk with more customers, they solve business use case, and then we realize one model is not sufficient to solve their problem. And it's very clear because one is the model hallucinates. Many customers, when they onboard this JNI journey, they thought this is magical. JNI is going to solve all my problems magically. But then they realize, oh, this model hallucinates. It hallucinates because it's not deterministic, it's probabilistic. So it's designed to always give you an answer, but based on probabilities, so it hallucinates. And that's actually sometimes a feature for creative writing, for example. Sometimes it's a bug because, hey, you don't want to give misinformation. And different models also have different specialties. To solve a problem, you want to ask different special models to kind of decompose your task into multiple small tasks, narrow tasks, and then have an expert model solve that task really well. And of course, the model doesn't have all the information. It has limited knowledge because the training data is finite, not infinite. So the model oftentimes doesn't have real-time information. It doesn't know any proprietary information within the enterprise. It's clear that in order to really build a compiling application on top of JNI, we need a compound AI system. Compound AI system basically is going to have multiple models across modalities, along with APIs, whether it's public APIs, internal proprietary APIs, storage systems, database systems, knowledge to work together to deliver the best answer. Swyx [00:20:07]: Are you going to offer a vector database? Lin [00:20:09]: We actually heavily partner with several big vector database providers. Which is your favorite? They are all great in different ways. But it's public information, like MongoDB is our investor. And we have been working closely with them for a while. Alessio [00:20:26]: When you say distributed inference engine, what do you mean exactly? Because when I hear your explanation, it's almost like you're centralizing a lot of the decisions through the Fireworks platform on the quality and whatnot. What do you mean distributed? It's like you have GPUs in a lot of different clusters, so you're sharding the inference across the same model. Lin [00:20:45]: So first of all, we run across multiple GPUs. But the way we distribute across multiple GPUs is unique. We don't distribute the whole model monolithically across multiple GPUs. We chop them into pieces and scale them completely differently based on what's the bottleneck. We also are distributed across regions. We have been running in North America, EMEA, and Asia. We have regional affinity to applications because latency is extremely important. We are also doing global load balancing because a lot of applications there, they quickly scale to global population. And then at that scale, different content wakes up at a different time. And you want to kind of load balancing across. So all the way, and we also have, we manage various different kinds of hardware skew from different hardware vendors. And different hardware design is best for different types of workload, whether it's long context, short context, long generation. So all these different types of workload is best fitted for different kinds of hardware skew. And then we can even distribute across different hardware for a workload. So the distribution actually is all around in the full stack. Swyx [00:22:02]: At some point, we'll show on the YouTube, the image that Ray, I think, has been working on with all the different modalities that you offer. To me, it's basically you offer the open source version of everything that OpenAI typically offers. I don't think there is. Actually, if you do text to video, you will be a superset of what OpenAI offers because they don't have Sora. Is that Mochi, by the way? Mochi. Mochi, right? Lin [00:22:27]: Mochi. And there are a few others. I will say, the interesting thing is, I think we're betting on the open source community is going to proliferate. This is literally what we're seeing. And there's amazing video generation companies. There is amazing audio companies. Like cross-border, the innovation is off the chart, and we are building on top of that. I think that's the advantage we have compared with a closed source company. Swyx [00:22:58]: I think I want to restate the value proposition of Fireworks for people who are comparing you versus a raw GPU provider like a RunPod or Lambda or anything like those, which is like you create the developer experience layer and you also make it easily scalable or serverless or as an endpoint. And then, I think for some models, you have custom kernels, but not all models. Lin [00:23:25]: Almost for all models. For all large language models, all your models, and the VRMs. Almost for all models we serve. Swyx [00:23:35]: And so that is called Fire Attention. I don't remember the speed numbers, but apparently much better than VLM, especially on a concurrency basis. Lin [00:23:44]: So Fire Attention is specific mostly for language models, but for other modalities, we'll also have a customized kernel. Swyx [00:23:51]: And I think the typical challenge for people is understanding that has value, and then there are other people who are also offering open-source models. Your mode is your ability to offer a good experience for all these customers. But if your existence is entirely reliant on people releasing nice open-source models, other people can also do the same thing. Lin [00:24:14]: So I would say we build on top of open-source model foundation. So that's the kind of foundation we build on top of. But we look at the value prop from the lens of application developers and product engineers. So they want to create new UX. So what's happening in the industry right now is people are thinking about a completely new way of designing products. And I'm talking to so many founders, it's just mind-blowing. They help me understand existing way of doing PowerPoint, existing way of coding, existing way of managing customer service. It's actually putting a box in our head. For example, PowerPoint. So PowerPoint generation is we always need to think about how to fit into my storytelling into this format of slide one after another. And I'm going to juggle through design together with what story to tell. But the most important thing is what's our storytelling lines, right? And why don't we create a space that is not limited to any format? And those kind of new product UX design combined with automated content generation through Gen AI is the new thing that many founders are doing. What are the challenges they're facing? Let's go from there. One is, again, because a lot of products built on top of Gen AI, they are consumer-personal developer facing, and they require interactive experience. It's just a kind of product experience we all get used to. And our desire is to actually get faster and faster interaction. Otherwise, nobody wants to spend time, right? And then that requires low latency. And the other thing is the nature of consumer-personal developer facing is your audience is very big. You want to scale up to product market fit quickly. But if you lose money at a small scale, you're going to bankrupt quickly. So it's actually a big contrast. I actually have product market fit, but when I scale, I scale out of my business. So that's kind of a very funny way to think about it. So then having low latency and low cost is essential for those new applications and products to survive and really become a generation company. So that's the design point for our distributed inference engine and the file optimizer. File optimizer, you can think about that as a feedback loop. The more you feed your inference workload to our inference engine, the more we help you improve quality, lower latency further, lower your cost. It basically becomes better. And we automate that because we don't want you as an app developer or product engineer to think about how to figure out all these low-level details. It's impossible because you're not trained to do that at all. You should kind of keep your focus on the product innovation. And then the compound AI, we actually feel a lot of pain as the app developers, engineers, there are so many models. Every week, there's at least a new model coming out. Swyx [00:27:09]: Tencent had a giant model this week. Yeah, yeah. Lin [00:27:13]: I saw that. I saw that. Swyx [00:27:15]: It's like $500 billion. Lin [00:27:18]: So they're like, should I keep chasing this or should I forget about it? And which model should I pick to solve what kind of sub-problem? How do I even decompose my problem into those smaller problems and fit the model into it? I have no idea. And then there are two ways to think about this design. I think I talked about that in the past. One is imperative, as in you figure out how to do it. You give developer tools to dictate how to do it. Or you build a declarative system where a developer tells what they want to do, not how. So these are completely two different designs. So the analogy I want to draw is, in the data world, the database management system is a declarative system because people use database, use SQL. SQL is a way you say, what do you want to extract out of a database? What kind of result do you want? But you don't figure out which node is going to, how many nodes you're going to run on top of, how you redefine your disk, which index you use, which project. You don't need to worry about any of those. And database management system will figure out, generate a new best plan, and execute on that. So database is declarative. And it makes it super easy. You just learn SQL, which is learn a semantic meaning of SQL, and you can use it. Imperative side is there are a lot of ETL pipelines. And people design this DAG system with triggers, with actions, and you dictate exactly what to do. And if it fails, then how to recover. So that's an imperative system. We have seen a range of systems in the ecosystem go different ways. I think there's value of both. There's value of both. I don't think one is going to subsume the other. But we are leaning more into the philosophy of the declarative system. Because from the lens of app developer and product engineer, that would be easiest for them to integrate. Swyx [00:29:07]: I understand that's also why PyTorch won as well, right? This is one of the reasons. Ease of use. Lin [00:29:14]: Focus on ease of use, and then let the system take on the hard challenges and complexities. So we follow, we extend that thinking into current system design. So another announcement is we will also announce our next declarative system is going to appear as a model that has extremely high quality. And this model is inspired by Owen's announcement for OpenAI. You should see that by the time we announce this or soon. Alessio [00:29:46]: Trained by you. Lin [00:29:47]: Yes. Alessio [00:29:48]: Is this the first model that you trained? It's not the first. Lin [00:29:52]: We actually have trained a model called FireFunction. It's a function calling model. It's our first step into compound AI system. Because function calling model can dispatch a request into multiple APIs. We have pre-baked set of APIs the model learned. You can also add additional APIs through the configuration to let model dispatch accordingly. So we have a very high quality function calling model that's already released. We have actually three versions. The latest version is very high quality. But now we take a further step that you don't even need to use function calling model. You use our new model we're going to release. It will solve a lot of problems approaching very high OpenAI quality. So I'm very excited about that. Swyx [00:30:41]: Do you have any benchmarks yet? Lin [00:30:43]: We have a benchmark. We're going to release it hopefully next week. We just put our model to LMSYS and people are guessing. Is this the next Gemini model or a MADIS model? People are guessing. That's very interesting. We're watching the Reddit discussion right now. Swyx [00:31:00]: I have to ask more questions about this. When OpenAI released o1, a lot of people asked about whether or not it's a single model or whether it's a chain of models. Noam and basically everyone on the Strawberry team was very insistent that what they did for reinforcement learning, chain of thought, cannot be replicated by a whole bunch of open source model calls. Do you think that that is wrong? Have you done the same amount of work on RL as they have or was it a different direction? Lin [00:31:29]: I think they take a very specific approach where the caliber of team is very high. So I do think they are the domain expert in doing the things they are doing. I don't think there's only one way to achieve the same goal. We're on the same direction in the sense that the quality scaling law is shifting from training to inference. For that, I fully agree with them. But we're taking a completely different approach to the problem. All of that is because, of course, we didn't train the model from scratch. All of that is because we built on the show of giants. The current model available we have access to is getting better and better. The future trend is the gap between the open source model and the co-source model. It's just going to shrink to the point there's not much difference. And then we're on the same level field. That's why I think our early investment in inference and all the work we do around balancing across quality, latency, and cost pay off because we have accumulated a lot of experience and that empowers us to release this new model that is approaching open-ended quality. Alessio [00:32:39]: I guess the question is, what do you think the gap to catch up will be? Because I think everybody agrees with open source models eventually will catch up. And I think with 4, then with Lama 3.2, 3.1, 4.5b, we close the gap. And then 0.1 just reopened the gap so much and it's unclear. Obviously, you're saying your model will have... Swyx [00:32:57]: We're closing that gap. Alessio [00:32:58]: But you think in the future, it's going to be months? Lin [00:33:02]: So here's the thing that's happened. There's public benchmark. It is what it is. But in reality, open source models in certain dimensions are already on par or beat closed source models. So for example, in the coding space, open source models are really, really good. And in function calling, file function is also really, really good. So it's all a matter of whether you build one model to solve all the problems and you want to be the best of solving all the problems, or in the open source domain, it's going to specialize. All these different model builders specialize in certain narrow area. And it's logical that they can be really, really good in that very narrow area. And that's our prediction is with specialization, there will be a lot of expert models really, really good and even better than one-size-fits-all closed source models. Swyx [00:33:55]: I think this is the core debate that I am still not 100% either way on in terms of compound AI versus normal AI. Because you're basically fighting the bitter lesson. Lin [00:34:09]: Look at the human society, right? We specialize. And you feel really good about someone specializing doing something really well, right? And that's how our way evolved from ancient times. We're all journalists. We do everything. Now we heavily specialize in different domains. So my prediction is in the AI model space, it will happen also. Except for the bitter lesson. Swyx [00:34:30]: You get short-term gains by having specialists, domain specialists, and then someone just needs to train like a 10x bigger model on 10x more inference, 10x more data, 10x more model perhaps, whatever the current scaling law is. And then it supersedes all the individual models because of some generalized intelligence slash world knowledge. I think that is the core insight of the GPTs, the GPT-123 networks. Right. Lin [00:34:56]: But the training scaling law is because you have an increasing amount of data to train from. And you can do a lot of compute. So I think on the data side, we're approaching the limit. And the only data to increase that is synthetic generated data. And then there's like what is the secret sauce there, right? Because if you have a very good large model, you can generate very good synthetic data and then continue to improve quality. So that's why I think in OpenAI, they are shifting from the training scaling law into Swyx [00:35:25]: inference scaling law. Lin [00:35:25]: And it's the test time and all this. So I definitely believe that's the future direction. And that's where we are really good at, doing inference. Swyx [00:35:34]: A couple of questions on that. Are you planning to share your reasoning choices? Lin [00:35:39]: That's a very good question. We are still debating. Swyx [00:35:43]: Yeah. Lin [00:35:45]: We're still debating. Swyx [00:35:46]: I would say, for example, it's interesting that, for example, SweetBench. If you want to be considered for ranking, you have to submit your reasoning choices. And that has actually disqualified some of our past guests. Cosign was doing well on SweetBench, but they didn't want to leak those results. So that's why you don't see O1 preview on SweetBench, because they don't submit their reasoning choices. And obviously, it's IP. But also, if you're going to be more open, then that's one way to be more open. So your model is not going to be open source, right? It's going to be an endpoint that you provide. Okay, cool. And then pricing, also the same as OpenAI, just kind of based on... Lin [00:36:25]: Yeah, this is... I don't have, actually, information. Everything is going so fast, we haven't even thought about that yet. Yeah, I should be more prepared. Swyx [00:36:33]: I mean, this is live. You know, it's nice to just talk about it as it goes live. Any other things that you want feedback on or you're thinking through? It's kind of nice to just talk about something when it's not decided yet. About this new model. It's going to be exciting. It's going to generate a lot of buzz. Right. Lin [00:36:51]: I'm very excited to see how people are going to use this model. So there's already a Reddit discussion about it. And people are asking very deep, mathematical questions. And since the model got it right, surprising. And internally, we're also asking the model to generate what is AGI. And it generates a very complicated DAG thinking process. So we're having a lot of fun testing this internally. But I'm more curious, how will people use it? What kind of application they're going to try and test on it? And that's where we really like to hear feedback from the community. And also feedback to us. What works out well? What doesn't work out well? What works out well, but surprising them? And what kind of thing they think we should improve on? And those kind of feedback will be tremendously helpful. Swyx [00:37:44]: Yeah. So I've been a production user of Preview and Mini since launch. I would say they're very, very obvious jobs in quality. So much so that they made clods on it. And they made the previous state-of-the-art look bad. It's really that stark, that difference. The number one thing, just feedback or feature requests, is people want control on the budget. Because right now, in 0.1, it kind of decides its own thinking budget. But sometimes you know how hard the problem is. And you want to actually tell the model, spend two minutes on this. Or spend some dollar amount. Maybe it's time you miss dollars. I don't know what the budget is. That makes a lot of sense. Lin [00:38:27]: So we actually thought about that requirement. And it should be, at some point, we need to support that. Not initially. But that makes a lot of sense. Swyx [00:38:38]: Okay. So that was a fascinating overview of just the things that you're working on. First of all, I realized that... I don't know if I've ever given you this feedback. But I think you guys are one of the reasons I agreed to advise you. Because I think when you first met me, I was kind of dubious. I was like... Who are you? There's Replicate. There's Together. There's Laptop. There's a whole bunch of other players. You're in very, very competitive fields. Like, why will you win? And the reason I actually changed my mind was I saw you guys shipping. I think your surface area is very big. The team is not that big. No. We're only 40 people. Yeah. And now here you are trying to compete with OpenAI and everyone else. What is the secret? Lin [00:39:21]: I think the team. The team is the secret. Swyx [00:39:23]: Oh boy. So there's no thing I can just copy. You just... No. Lin [00:39:30]: I think we all come from a very aligned culture. Because most of our team came from meta. Swyx [00:39:38]: Yeah. Lin [00:39:38]: And many startups. So we really believe in results. One is result. And second is customer. We're very customer obsessed. And we don't want to drive adoption for the sake of adoption. We really want to make sure we understand we are delivering a lot of business values to the customer. And we really value their feedback. So we would wake up midnight and deploy some model for them. Shuffle some capacity for them. And yeah, over the weekend, no brainer. Swyx [00:40:15]: So yeah. Lin [00:40:15]: So that's just how we work as a team. And the caliber of the team is really, really high as well. So as plug-in, we're hiring. We're expanding very, very fast. So if we are passionate about working on the most cutting-edge technology in the general space, come talk with us. Yeah. Swyx [00:40:38]: Let's talk a little bit about that customer journey. I think one of your more famous customers is Cursor. We were the first podcast to have Cursor on. And then obviously since then, they have blown up. Cause and effect are not related. But you guys especially worked on a fast supply model where you were one of the first people to work on speculative decoding in a production setting. Maybe just talk about what was the behind the scenes of working with Cursor? Lin [00:41:03]: I will say Cursor is a very, very unique team. I think the unique part is the team has very high technical caliber. There's no question about it. But they have decided, although many companies building coding co-pilot, they will say, I'm going to build a whole entire stack because I can. And they are unique in the sense they seek partnership. Not because they cannot. They're fully capable, but they know where to focus. That to me is amazing. And of course, they want to find a bypass partner. So we spent some time working together. They are pushing us very aggressively because for them to deliver high caliber product experience, they need the latency. They need the interactive, but also high quality at the same time. So actually, we expanded our product feature quite a lot as we support Cursor. And they are growing so fast. And we massively scaled quickly across multiple regions. And we developed a pretty high intense inference stack, almost like similar to what we do for Meta. I think that's a very, very interesting engagement. And through that, there's a lot of trust being built. They realize, hey, this is a team they can really partner with. And they can go big with. That comes back to, hey, we're really customer obsessed. And all the engineers working with them, there's just enormous amount of time syncing together with them and discussing. And we're not big on meetings, but we are like stack channel always on. Yeah, so you almost feel like working as one team. So I think that's really highlighted. Swyx [00:42:38]: Yeah. For those who don't know, so basically Cursor is a VS Code fork. But most of the time, people will be using closed models. Like I actually use a lot of SONET. So you're not involved there, right? It's not like you host SONET or you have any partnership with it. You're involved where Cursor is small, or like their house brand models are concerned, right? Lin [00:42:58]: I don't know what I can say, but the things they haven't said. Swyx [00:43:04]: Very obviously, the drop down is 4.0, but in Cursor, right? So I assume that the Cursor side is the Fireworks side. And then the other side, they're calling out the other. Just kind of curious. And then, do you see any more opportunity on the... You know, I think you made a big splash with 1,000 tokens per second. That was because of speculative decoding. Is there more to push there? Lin [00:43:25]: We push a lot. Actually, when I mentioned Fire Optimizer, right? So as in, we have a unique automation stack that is one size fits one. We actually deployed to Cursor earlier on. Basically optimized for their specific workload. And that's a lot of juice to extract out of there. And we see success in that product. It actually can be widely adopted. So that's why we started a separate product line called Fire Optimizer. So speculative decoding is just one approach. And speculative decoding here is not static. We actually wrote a blog post about it. There's so many different ways to do speculative decoding. You can pair a small model with a large model in the same model family. Or you can have equal pads and so on. There are different trade-offs which approach you take. It really depends on your workload. And then with your workload, we can align the Eagle heads or Medusa heads or a small big model pair much better to extract the best latency reduction. So all of that is part of the Fire Optimizer offering. Alessio [00:44:23]: I know you mentioned some of the other inference providers. I think the other question that people always have is around benchmarks. So you get different performance on different platforms. How should people think about... People are like, hey, Lama 3.2 is X on MMLU. But maybe using speculative decoding, you go down a different path. Maybe some providers run a quantized model. How should people think about how much they should care about how you're actually running the model? What's the delta between all the magic that you do and what a raw model... Lin [00:44:57]: Okay, so there are two big development cycles. One is experimentation, where they need fast iteration. They don't want to think about quality, and they just want to experiment with product experience and so on. So that's one. And then it looks good, and they want to post-product market with scaling. And the quality is really important. And latency and all the other things are becoming important. During the experimentation phase, it's just pick a good model. Don't worry about anything else. Make sure you even generate the right solution to your product. And that's the focus. And then post-product market fit, then that's kind of the three-dimensional optimization curve start to kick in across quality, latency, cost, where you should land. And to me, it's purely a product decision. To many products, if you choose a lower quality, but better speed and lower cost, but it doesn't make a difference to the product experience, then you should do it. So that's why I think inference is part of the validation. The validation doesn't stop at offline eval. The validation will go through A-B testing, through inference. And that's where we offer various different configurations for you to test which is the best setting. So this is the traditional product evaluation. So product evaluation should also include your new model versions and different model setup into the consideration. Swyx [00:46:22]: I want to specifically talk about what happens a few months ago with some of your major competitors. I mean, all of this is public. What is your take on what happens? And maybe you want to set the record straight on how Fireworks does quantization because I think a lot of people may have outdated perceptions or they didn't read the clarification post on your approach to quantization. Lin [00:46:44]: First of all, it's always a surprise to us that without any notice, we got called out. Swyx [00:46:51]: Specifically by name, which is normally not what... Lin [00:46:54]: Yeah, in a public post. And have certain interpretation of our quality. So I was really surprised. And it's not a good way to compete, right? We want to compete fairly. And oftentimes when one vendor gives out results, the interpretation of another vendor is always extremely biased. So we actually refrain ourselves to do any of those. And we happily partner with third parties to do the most fair evaluation. So we're very surprised. And we don't think that's a good way to figure out the competition landscape. So then we react. I think when it comes to quantization, the interpretation, we wrote actually a very thorough blog post. Because again, no one says it's all. We have various different quantization schemes. We can quantize very different parts of the model from ways to activation to cross-TPU communication. They can use different quantization schemes or consistent across the board. And again, it's a trade-off. It's a trade-off across this three-dimensional quality, latency, and cost. And for our customer, we actually let them find the best optimized point. And we have a very thorough evaluation process to pick that point. But for self-serve, there's only one point to pick. There's no customization available. So of course, it depends on what we talk with many customers. We have to pick one point. And I think the end result, like AA published, later on AA published a quality measure. And we actually looked really good. So that's why what I mean is, I will leave the evaluation of quality or performance to third party and work with them to find the most fair benchmark. And I think that's a good approach, a methodology. But I'm not a part of an approach of calling out specific names Swyx [00:48:55]: and critique other competitors in a very biased way. Databases happens as well. I think you're the more politically correct one. And then Dima is the more... Something like this. It's you on Twitter. Lin [00:49:11]: It's like the Russian... We partner. We play different roles. Swyx [00:49:20]: Another one that I wanted to... I'm just the last one on the competition side. There's a perception of price wars in hosting open source models. And we talked about the competitiveness in the market. Do you aim to make margin on open source models? Oh, absolutely, yes. Lin [00:49:38]: So, but I think it really... When we think about pricing, it's really need to coordinate with the value we're delivering. If the value is limited, or there are a lot of people delivering the same value, there's no differentiation. There's only one way to go. It's going down. So through competition. If I take a big step back, there is pricing from... We're more compared with close model providers, APIs, right? The close model provider, their cost structure is even more interesting because we don't bear any training costs. And we focus on inference optimization, and that's kind of where we continue to add a lot of product value. So that's how we think about product. But for the close source API provider, model provider, they bear a lot of training costs. And they need to amortize the training costs into the inference. So that created very interesting dynamics of, yeah, if we match pricing there, and I think how they are going to make money is very, very interesting. Swyx [00:50:37]: So for listeners, opening eyes 2024, $4 billion in revenue, $3 billion in compute training, $2 billion in compute inference, $1 billion in research compute amortization, and $700 million in salaries. So that is like... Swyx [00:50:59]: I mean, a lot of R&D. Lin [00:51:01]: Yeah, so I think matter is basically like, make it zero. So that's a very, very interesting dynamics we're operating within. But coming back to inference, so we are, again, as I mentioned, our product is, we are a platform. We're not just a single model as a service provider as many other inference providers, like they're providing a single model. We have our optimizer to highly customize towards your inference workload. We have a compound AI system where significantly simplify your interaction to high quality and low latency, low cost. So those are all very different from other providers. Alessio [00:51:38]: What do people not know about the work that you do? I guess like people are like, okay, Fireworks, you run model very quickly. You have the function model. Is there any kind of like underrated part of Fireworks that more people should try? Lin [00:51:51]: Yeah, actually, one user post on x.com, he mentioned, oh, actually, Fireworks can allow me to upload the LoRa adapter to the service model at the same cost and use it at same cost. Nobody has provided that. That's because we have a very special, like we rolled out multi-LoRa last year, actually. And we actually have this function for a long time. And many people has been using it, but it's not well known that, oh, if you find your model, you don't need to use on demand. If you find your model is LoRa, you can upload your LoRa adapter and we deploy it as if it's a new model. And then you use, you get your endpoint and you can use that directly, but at the same cost as the base model. So I'm happy that user is marketing it for us. He discovered that feature, but we have that for last year. So I think to feedback to me is, we have a lot of very, very good features, as Sean just mentioned. I'm the advisor to the company, Swyx [00:52:57]: and I didn't know that you had speculative decoding released. Lin [00:53:02]: We have prompt catching way back last year also. We have many, yeah. So I think that is one of the underrated feature. And if they're developers, you are using our self-serve platform, please try it out. Swyx [00:53:16]: The LoRa thing is interesting because I think you also, the reason people add additional costs to it, it's not because they feel like charging people. Normally in normal LoRa serving setups, there is a cost to dedicating, loading those weights and dedicating a machine to that inference. How come you can't avoid it? Lin [00:53:36]: Yeah, so this is kind of our technique called multi-LoRa. So we basically have many LoRa adapters share the same base model. And basically we significantly reduce the memory footprint of serving. And the one base model can sustain a hundred to a thousand LoRa adapters. And then basically all these different LoRa adapters can share the same, like direct the same traffic to the same base model where base model is dominating the cost. So that's how we advertise that way. And that's how we can manage the tokens per dollar, million token pricing, the same as base model. Swyx [00:54:13]: Awesome. Is there anything that you think you want to request from the community or you're looking for model-wise or tooling-wise that you think like someone should be working on in this? Lin [00:54:23]: Yeah, so we really want to get a lot of feedback from the application developers who are starting to build on JNN or on the already adopted or starting about thinking about new use cases and so on to try out Fireworks first. And let us know what works out really well for you and what is your wishlist and what sucks, right? So what is not working out for you and we would like to continue to improve. And for our new product launches, typically we want to launch to a small group of people. Usually we launch on our Discord first to have a set of people use that first. So please join our Discord channel. We have a lot of communication going on there. Again, you can also give us feedback. We'll have a starting office hour for you to directly talk with our DevRel and engineers to exchange more long notes. Alessio [00:55:17]: And you're hiring across the board? Lin [00:55:18]: We're hiring across the board. We're hiring front-end engineers, infrastructure cloud, infrastructure engineers, back-end system optimization engineers, applied researchers, like researchers who have done post-training, who have done a lot of fine-tuning and so on. Swyx [00:55:34]: That's it. Thank you. Thanks for having us. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We are so back. This is Charlie, your AI co-host. We're still between studios, so today we're filming on location at Fireworks AI HQ in Redwood City. We're here for a deep dive into the massive Compound AI wave coined by Databricks co-founders Mathai Zahariah, Ali Godsey, three-time Latent Space guest star John Frankel, and others, with Lin Qiao, co-founder and CEO of Fireworks AI, which is the leading Compound AI platform after two red-hot funding rounds with Benchmark and Sequoia Capital, and an incredible customer list from Superhuman to Cursor to Quora to HubSpot. We normally strive to have great offline relationships with the guests we bring you,

• 0:45Speaker 00

  but this case is more special than most. After Lin chatted with us briefly for the NURIPS 2023 podcast, Swyx has had the privilege of advising the company for a number of its launches from Fire Attention, which has 15x higher throughput than VLLM, to Fire Function and Fire Optimizer and Real-Time Audio Launch, to its most recent launch of F1, one of a new batch of long-inference reasoning models built atop open models to compete with OpenAI's O1 alongside of Newsforge API and DeepSeek R1 as well, which were released after this recording. In Latent Space news, we now have a stacked meetup calendar,

• 1:26Speaker 00

  from the third AI engineer meetup in London, to the AWS Reinvent listener meetup in Las Vegas, to the new Latent Space Live micro-conference at NURIPS 2024. The NURIPS event in particular will be hosted both online and in person, and for the first time ever, we are confirming three prize categories for our speakers. One, too hot for NURIPS, for papers that are too new or wrongly rejected for NURIPS. Two, Best Papers of 2024 for SurveyTalks nominating best papers of the year in a given domain. Three, Oxford-style debates for hot topics in the AI research and engineering

• 2:10Speaker 00

  community. Head to https://ludotma/lslive to sign up and apply to speak or sponsor. Limited in-person tickets available now. Lastly, we are still taking listener questions for our end-of-year recap. Head to speakpipe.com/laten space to submit questions and messages for a chance to appear on the show. Watch out and take care.

• 2:40Alessio Fanelli?

  Hey everyone, welcome to the Laten Space Podcast. This is Alessio, Partner and CEO of Decibel Partners, and I'm joined by my co-host Swix, founder Osmiley Ayer.

• 2:48Speaker 02

  Hey, and today we're in a very special studio inside the Fireworks office with Lin Chiao, CEO of Fireworks. Welcome. Oh, you should welcome us.

• 2:56Speaker 03

  Yeah, welcome.

• 3:00Speaker 02

  Yeah, thanks for having us. It's unusual to be in the home of a startup, but like, it's also, I think our relationship is a bit unusual with, compared to all our normal guests.

• 3:10Speaker 03

  Definitely. Yeah, I'm super excited to talk about very interesting topics in that space with both of you.

• 3:17Speaker 02

  You just celebrated your two-year anniversary yesterday.

• 3:19Speaker 03

  Yeah, it's quite a crazy journey. We circle around and share all the crazy stories across these two years, and it has been super fun. All the way from we experienced Silicon Valley bank run.

• 3:32Speaker 02

  Right.

• 3:34Speaker 03

  To we delete some data that shouldn't be deleted operationally. We went through massive scale where we actually are busy getting capacity to...

• 3:49Speaker 03

  Yeah, we we learn to kind of work with it as a team with a lot of brilliant people across different places joining a company. It has really been a fun journey.

• 4:00Alessio Fanelli?

  When you started, did you think the technical stuff will be harder? Or the bank run and then the people side? I think there's a lot of like amazing researchers that want to do companies. And it's like the hardest thing is going to be building the product. And then you have all these different other things. So were you surprised by what has been the your experience?

• 4:19Speaker 03

  Yeah, to be honest with you, like my focus has always been on the product side. And then after product, go to market. And I didn't realize the rest has been so complicated, operating a company and so on. But because I don't think about it, I just kind of manage it. So it's done. So I think I just somehow like don't think about it too much and you know, solve whatever problem coming our way and it worked.

• 4:44Speaker 02

  So let's, I guess, let's start at the pre-history, like the pre, the sort of initial history of Fireworks. You ran the PyTorch team at Meta for a number of years. And we previously had Sumit Chintala on. And I think we were just all very interested in like the the history of GNI. Maybe not that many people know like how deeply involved FAIR and Meta were in like prior to the current GNI revolution.

• 5:10Speaker 03

  Hmm. My background is a D P in distributed system, data mix management system. And I joined Meta from the data side. And I saw this tremendous amount of data growth, which costs a lot of money. And we're analyzing what's going on. And it's clear that AI is driving all this data generation. So it it's a very interesting time because when I joined Meta, Meta is going through ramping down mobile first, finishing the mobile first transition and then starting AI first. And there's a fundamental reason about that sequence because mobile first gave a full range of user engagement that was has never existed before. And all this user engineering generate a lot of data. And this data power AI.

• 5:55Speaker 03

  So then the whole entire industry is also going through, following through this same transition. When I see, oh, hey, this AI powering all this generation and look at where's our AI stack? There's no software, there's no hardware, there's no people, there's no team. I like, I want to dive it up there and help this movement. So when I started, it's very interesting industry landscape. There are a lot of AI frameworks. It's a kind of proliferation of AI frameworks happening in the industry. But all the AI frameworks focus on like production. And they use a very certain way of defining the graph of new network. And they use that to drive the model iteration and productionization.

• 6:41Speaker 03

  And PyTorch is completely different. So Big Kuro Sozumiy, then he was the user of his product. He, as a researcher, faced so much pain using existing AI frameworks. This, this is really hard to use and I'm gonna build something different for myself. And that's the origin story of PyTorch. PyTorch actually started as the framework for researchers. Don't care about production at all. And as it grow in terms of adoption. So the interesting part of AI is research is the top of funnel for production. There are so many researchers across academic, across industry. And they innovate and they put their results out there in open source. And that powered the downstream production addition.

• 7:22Speaker 03

  So it's brilliant for Meta to establish PyTorch as a strategy to drive massive adoption in open source. Because Meta internally is a PyTorch shop. So it's created a flying wheel effect. So that's kind of a strategy behind PyTorch. But when I took on PyTorch, it's kind of a castle. Meta established PyTorch as the framework for both research and production. So no one has done that before. And we have to kind of rethink how to architect PyTorch so it can really sustain production workload. The stability, reliability, low latency, all this production concern was never concerned before. Now is a concern and we actually have to adjust its design and make it work for both sides. And that took us five years because Meta has so many AI use cases.

• 8:07Speaker 03

  All the way from ranking recommendation that's powering the business top line. For ads ranking, newsfeed, video ranking, to site integrity, detect bad content automatically using AI, to all kind of effects, translation, image classification, object detection, all this. And also across AI running on the server side, on mobile phones, on AI VR devices, wide spectrum. So by the time, we actually basically managed to support AI across ubiquitous everywhere, across Meta. But interestingly, through open source engagement, we work with a lot of companies. It is clear to us, like, this industry is start to take on AI first transition. And of course, Meta's hyper scale always go be ahead of the industry.

• 8:52Speaker 03

  And we feel like, it feels like when we start this AI journey at Meta, there's no, no software, no hardware, no team. For many companies we, we engage with through PyTorch, we feel the pain. That's the genesis why we feel like, hey, if we create fireworks and supporting industry going through this transition, it will be huge amount of impact. Of course, the problem that the industry facing will not be the same as Meta. Meta is so big, right? So it's kind of skewed towards extreme scale and extreme optimization, the industry will be different. But we feel like we have the technical on chopping, we have seen a lot, would like, love to kind of drive that. So yeah, so that's the, that's how we started.

• 9:35Speaker 02

  When you and I chatted about like the origins of fireworks, it was originally envisioned more as a PyTorch platform. And then later it became much more focused on generative AI. Is that, is that fair to say like that?

• 9:47Speaker 03

  Right.

• 9:48Speaker 02

  What was the customer discovery here?

• 9:49Speaker 03

  Right, so I would say our initial blueprint is, hey, we should do the PyTorch cloud because PyTorch is a library and there's no SaaS platform to enable AI workloads.

• 10:02Speaker 02

  Even in 2022?

• 10:04Speaker 03

  I would not say absolutely no, but like cloud providers have some of those, but it's not first class, it is different, right? Because at 2022, there is still like Tesla flow is massively in production. And this is all pre-generative AI. And the PyTorch is kind of getting more and more adoption. But there's no PyTorch first SaaS platform existing. At the same time, we are also a very pragmatic set of people. We really want to make sure from the get-go, we get really, really close to customers. We understand their use case, we understand their pain points, we understand the value we deliver to them. So we want to take a different approach. Instead of building horizontal PyTorch cloud, we want to build a verticalized platform first. And then we talk with many customer.

• 10:49Speaker 03

  And interesting, we started company September two thousand twenty two and October, November, then OpenAI announced ChatGPT. And then boom, then when we talk with many customer, they are like, can you help us work on the GNI aspect, right? So of course, there are some open source models. It's not as good at that time, but people are already like putting a lot of attention there. Then we decide that if we're gonna pick a vertical, we're gonna pick GNI. The other reason is all GNI model are PyTorch models. So that's another reason. We believe that because of nature of GNI, it's gonna generate a lot of human consumable content. It will drive a lot of consumer, consumer developer facing application and product innovation guaranteed. Right? We're just at the beginning of this.

• 11:35Speaker 03

  Our prediction is for those kind of application, the inference is much more important than training. Because inference scale is proportional to the up limit of world population. And training, training scale is proportional to the number of researchers. Of course, each training round could be very expensive. Although PyTorch support both inference and training, we decide to later focus on training, uh, inference. So yeah, so that's how we got started. And we launched our public platform August last, last year. And when we launched, it's a single product. It's a distributed inference engine with simple API, open and compatible API with the many models. We started with LLM and they don't add a lot of model. Fast forward to now, we are full platform with multiple product lines.

• 12:19Speaker 03

  So we love to kind of dive deep into what we offer. Uh, so but that's a very fun journey in the, in the past two years.

• 12:26Alessio Fanelli?

  What was the transition from you start focus on PyTorch? And like, people want to understand the framework, get it live. And now I would say maybe most people that use you don't even really know much about PyTorch at all. You know, they're just stronger to my model. From a product perspective, like over some of the decisions early on, like right in October, November, you were just like, hey, most people just care about the model, not about the framework. We're gonna make it super easy. Or was it more gradual transition to the model library you have today?

• 12:53Speaker 03

  Yeah, so our product decision all based on who is our ICP. And one thing we want to acknowledge here is the GNI technology is disruptive. It's very different from A I before G N I. So it's a clear leap forward. Because before G N I, the companies that want to invest in A I, they have trained from scratch. There's no other way. There's no foundation model. Doesn't exist. So that means then to start a team, first hire a team who is capable of crunch data. There's a lot of data to crunch, right? Because training from scratch, you you have to prepare a lot of data. And then they need to have, uh, they need to have G P U's to train. Uh, and then you start to manage G P U. So then it becomes a very complex project.

• 13:37Speaker 03

  Uh, it takes a long time and not, not many company can afford it, actually. And G N I is a very different game right now because it is a foundation model, so you don't have to train anymore. That make A I much more accessible as a technology. As an app developer or product manager, even not a developer, they can interact with G N I models directly. So, and our goal is to make A I accessible to all app developers and product engineers. That's our goal. So then getting them into the building model doesn't make any sense anymore with this new technology. And then building easy accessible A P I is the most important.

• 14:18Speaker 03

  Our, uh, early on, when we got started, we decided we're gonna be OpenAI compatible. It's just kind of very easy for developers to adopt this new technology. And we will manage the underlying complexity of serving all these models.

• 14:32Speaker 02

  Yeah, OpenAI has become

• 14:34Speaker 03

  The standard.

• 14:34Speaker 02

  Standard, uh, even to, as we're recording today, Gem and I announced that they have OpenAI compatible A P I's.

• 14:41Speaker 03

  Interesting.

• 14:42Speaker 02

  Then we just need a topic to hold in mind, and then we have everyone.

• 14:44Speaker 03

  Yeah, that's interesting because, um, we are working very closely with Meta as one of the partners. And Meta announced, Meta, of course, is kind of very generous to donate many very, very strong open source models, expecting more to come. But also they have announced Lama Stack.

• 15:01Speaker 02

  Yeah.

• 15:01Speaker 03

  Uh, which is, uh, basically standardized the upper level stack built on top of Lama models. So they don't just want to give out models and you figure out what the upper stack. They instead want to build a community around this stack and then build, build a kind of new standard. I think this is interesting dynamics in, playing in, in the industry right now, when there's more like Standardize across OpenAI because they are kind of creating the top of Inferno. Or standardize across Lama because this is the like most used open source model. So I I think do a lot of fun working at this time.

• 15:37Speaker 02

  I've been a little bit more doubtful on Llama stack. I think you've been more positive. Basically, it's just like the meta version of whatever Hugging Face offers. You know, or TensorRT, or BLM, or whatever the open source opportunity is. But like, uh, it it to me, it's not clear that just because Meta opensizes Llama. That the rest of Llama stack will be adopted and it's not clear why I should adopt it. So I I don't know if you agree.

• 16:03Speaker 03

  Yeah, it's very early right now. That's why I kind of will will work very closely with them and give them feedback. The feedback to the Meta team is very important, so then they can use that to continue to improve the model and also improve the higher level stack. I think the success of Llama stack heavily depend on the community adoption and there's no way around it. And I I know Meta team will like to kind of work with the broader side of community, but it's very early.

• 16:28Speaker 02

  One thing that uh after your Series B, so you you raised from a benchmark and then Sequoia. I remember being close to you for at least your your Series B announcements. You started betting heavily on this term of Compound AI. It's not a term that we've covered very much in the podcast, but I think it's definitely getting a lot of adoption from like Databricks and Berkeley people and all that. What's your take on Compound AI? Why is it resonating with people?

• 16:52Speaker 03

  Right, so uh let me give you a bit of context why why we even consider that space.

• 16:58Speaker 02

  Yeah, because like pre-Series B, you were not, there was no lesson. Yeah, now it's like on your landing page.

• 17:04Speaker 03

  So it's kind of very organic evolution from when we first launched our product platform. We are single product, we are distributed inference engine, where we do a lot of uh innovation on customized CUDA kernels, Rockon kernels, on running a different kind of hardware. And build distributed disaggregated execution, inference execution, build all kind of caching. And so that is one. So that's kind of one product line is the fast, most cost efficient inference platform. Because we wrote Python code, we know, we basically have a special Python build for that. Uh, together with a custom kernel we wrote. And then we work with many more customer, we realize, oh, the distribution inference engine is, our design is one size fits all.

• 17:49Speaker 03

  Right? We want to have this inference endpoint, then everyone come in. And they, no matter what kind of form and shape or workload they have, it will just work for them, right? So that's great. But the reality is, they, we realize all customer have different kind of use cases. The use case come in all different form and shape. And the end result is the data distribution in their inference workload doesn't align with the data distribution in the training data for the model, right? It's a given, actually, if you think about this. Because researcher has to estimate what is important, what's not important during, like, in prepared data for training. So because of that misalignment, then we leave a lot of quality, latency, cost improvement on the table.

• 18:33Speaker 03

  So then we are saying, okay, we want to have it invest in a customization engine. And we actually announced it called Fire Optimizer. So Fire Optimizer basically help user navigate a three dimensional optimization space across quality, latency, and cost. So it's a three dimensional curve. And even for one company, for different use case, they want land in different spot. So we automate that process for our customer. It's very simple, you have your inference workload, and you inject um into the optimizer along with our objective function. And then we spit out a inference deployment config and the model setup. So like it's your customized setup.

• 19:17Speaker 03

  So that is a completely different product. So that product thinking is one size fits one, different from one size fits all. And now on top of that, we provide a huge variety of state of art models, uh, hundreds of them, varying from text, the large. State our large English models. That's where we started. And as we talk with many customer, we realized, oh, audio and text are very, very close. Many who are customers start to build assistant, all kind of assistant using text. And they immediately want to add audio, audio eating audio out. So we support transcription, translation, speech synthesis, text, audio alignment, all different kind of audio features. It's a big announcement we're gonna, you should have heard.

• 20:01Speaker 04

  By the time this.

• 20:01Speaker 03

  By the time this is out.

• 20:02Speaker 04

  Yeah.

• 20:03Speaker 03

  And the other areas of vision and the text are very close with each other. Because a lot of information doesn't live in plain text. A lot of information live in multimedia format, live in images, PDFs, screenshots, and many other different formats. So oftentimes solve a problem, uh, we need to put the vision model first to extract information and then use language model to process and then send out results. So vision is important, we also support vision model. Very different kind of vision models specialize in processing different kind of source and extraction. And we're also gonna have another announcement of a new API endpoint we'll support for people to upload various different kind of multimedia content

• 20:44Speaker 03

  and then get the, uh, extract very accurate information out and feed that into L M. And, and then of course we support embedding because embedding is very important for semantic search, for RAG and all this. And in addition to that, we also support text to image, image generation models, text to image, image to image, and we're adding text to video as well in our portfolio. So it's very comprehensive set of model catalog that built on, run on top of file optimizer and distributed inference engine. But then we talk with more customer, they solve business use case and they realize one model is not sufficient to solve their problem. And it's, it's very clear because one is the model who listens. And many customer, they, when they onboard this G N I journey, they thought this is magical.

• 21:28Speaker 03

  G N I is gonna solve all my problems magically, but then they realize, oh, this model who listens. It listens because it's not deterministic, it's probabilistic. Uh, so it's designed to always give a answer, but, uh, based on probability, so it listens. And that's actually sometimes the feature for creative writing, for example. Sometimes it's a bug because, hey, you don't want to give misinformation. And different model also have different specialties. To solve a problem, you want to ask different special model to kind of decompose your task into multiple small task, narrow task, and then have expert model solve that task really well. And of course, the model doesn't have all the information. It has to make knowledge because the training data is finite, not infinite. So model oftentimes doesn't have real time information.

• 22:13Speaker 03

  It doesn't know any proprietary information within enterprise. It's clear, you know, had in order to really build a compiling application on top of GNI, we need a compound AI system. Compound AI system basically is gonna have multiple models across modalities along with APIs, whether it's public APIs, internal proprietary APIs, storage systems, database system, knowledge systems to work together to deliver the best answer.

• 22:44Speaker 02

  Are you gonna offer vector database?

• 22:45Speaker 03

  We actually heavily partner with several big vector database providers.

• 22:51Speaker 02

  Which is your favorite.

• 22:54Speaker 03

  They are all great in different ways, but it's public information, like MongoDB is our investor, and we have been working closely with them for a while.

• 23:03Alessio Fanelli?

  When you say distributed inference engine, what do you mean exactly? Because when I hear your explanation, it's almost like you're centralizing a lot of the decisions through the fireworks platform or like the quality and whatnot. What do you mean distributed is like you have GPUs and like a lot of different clusters? So like you're sharding the inference across same model.

• 23:21Speaker 03

  Right, right, right. So first of all, we run across multiple GPUs. But the way we distribute across multiple GPUs is is unique. We don't distribute the whole model monolithically across multiple GPUs. We chop them into pieces and scale them completely differently based on what's the bottleneck. We also are distributed across regions. We have been running in North America, EMEA and Asia. We have regional affinity.

• 23:48Speaker 03

  To applications because latency is extremely important. We are also like doing global load balancing because a lot of application there. They quickly scale to global population. And then at that scale, like different content wakes up at a different time. And you want to kind of load balancing across. So all the way in a week, we also have, we manage various different kind of hardware skill. From different hardware vendors and different hardware design is best for different type of workload. Whether it's long context, short context, long generation. So all these different type of workload is best fitted for different kind of hardware skill. And we can even distribute it across different hardware for a workload.

• 24:33Speaker 03

  So yeah, so the distribution actually is is all around in the full stack.

• 24:39Speaker 02

  At some point, we'll show on the YouTube. The image that Ray, I think, has been working on with like all the different modalities that you offer. Like to me, it's basically you offer the open source version of everything that OpenAI typically offers, right? I don't think there is, actually, if you do text the video, you will be a subset of what OpenAI offers because they don't have Sora. Is that Mochi, by the way?

• 25:01Speaker 03

  Mochi.

• 25:02Speaker 02

  Mochi, right?

• 25:02Speaker 03

  Yes. Mochi, and there are a few others, like there. I would say the interesting thing is, I think we are betting on the open source community is gonna grow, like proliferate. This is literally what I'm seeing.

• 25:16Speaker 02

  Yeah.

• 25:17Speaker 03

  And there's amazing video generation companies.

• 25:20Speaker 02

  Yeah.

• 25:21Speaker 03

  There is amazing audio companies, like cross board, the innovation is off the chart. And we are building on top of that. I think that's the advantage we have compared with a closed source company.

• 25:34Speaker 02

  I think I want to restate the value proposition of Fireworks for people who are comparing you versus like a raw GPU provider like a RunPod. Or Lambda, you know, anything like those, which is like you create the developer experience layer. And you also like make it easily scalable or serverless or, you know, as as an endpoint. And then I think for some models, you have custom kernels, but not all models.

• 26:00Speaker 03

  For almost for all model, for all large language models, audio models.

• 26:05Speaker 02

  All day long?

• 26:05Speaker 03

  And the V R S.

• 26:08Speaker 02

  Yeah, yeah.

• 26:09Speaker 03

  Yeah, almost for all models we serve, we have.

• 26:11Speaker 02

  And so that is called fire attention?

• 26:13Speaker 03

  That's called fire.

• 26:13Speaker 02

  I don't remember the the speed numbers, but apparently much better than BLM, especially on a concurrency basis.

• 26:20Speaker 03

  Right, so fire attention is specific for um mostly for language model, but for other modalities will also have a customized kernel.

• 26:28Speaker 02

  Yeah, I think the the the typical challenge for people is understanding like that has value. Uh and then like there are other people who are also offering open source models, right? Like your mode is is your ability to offer like a a good experience for all these customers. But if your existence is entirely reliant on people releasing nice open source models, other people can also do the same thing.

• 26:49Speaker 03

  Right, yeah, so I would say we build on top of open source model foundation. So that that's the kind of foundation we build on top of. But we look at our the value prop from the lens of application developers and product engineers. So they want to create new UX. So what's happening in the industry right now is people are thinking about completely new way of designing products. And I'm talking to so many founders, it's just mind blowing. They help me understand existing way of doing PowerPoint, existing way of coding, existing way of managing customer service. It's actually putting a box in our head. For example, PowerPoint, right?

• 27:30Speaker 03

  So PowerPoint generation is we always need to think about how to fit into my storytelling into this format of slide one after another. And I I'm gonna juggle through like design together with you know what story to tell. But the most important thing is what's your storytelling lines, right? And why don't we create a space that is not limited to any format? And those kind of new product UX design combined with automated content generation for GNI.

• 28:00Speaker 03

  Is the new thing that many founders are doing. What are the challenges they're facing? And let's go from there. One is, again, because a lot of product built on top of Genai, they are consumer personal development facing. And they request interactive experience. It's just a kind of product experience we we all get used to. And our desires actually get faster and faster interaction. Otherwise, nobody want to spend time, right? So again, and then that requires low latency. And the other thing is the nature of consumer personal development facing is the your audience is very big, you you want to scale after product market fit quickly. But if you lose money at a small scale, you're going to bankrupt quickly. So it's actually a big contrast is I actually have product market fit.

• 28:44Speaker 03

  But when I scale, I scale out of my business. So that's kind of very, very funny uh to think about this. So then have low latency and low cost is essential for those new application and product to survive and really become a generation company. So that's the design point for our distributed inference engine and the file optimizer. File optimizer, you can think about that as a feedback loop. The more you feed your inference workload to our inference engine, the more we help you improve quality, lower latency, further lower your cost. It basically becomes better. And and we automate that because we don't want you as app developer or product engineer to think about how to figure out all these low level details.

• 29:29Speaker 03

  It's impossible because you are not trained for to do that at all. You should kind of keep your focus on the product innovation. And then the compound AI, we actually feel a lot of pain as the app developers, engineer, they there are so many models. Every week, there's at least a new model coming out.

• 29:46Speaker 02

  Tencent had a giant model this week.

• 29:49Speaker 03

  Yeah, yeah, I saw that. I saw that.

• 29:52Speaker 02

  Like five hundred billion.

• 29:53Speaker 03

  Yeah. Um, so they're like. Should I keep chasing this or should I forget about it, right? So and which model should I pick to solve? What kind of subproblem? How do I even decompose my problem into those smaller problems and fit the model into it? I have no idea. And then there are two ways to think about this design, right? I think I talked about that in the past. One is imperative, as in you tell, you figure out how to do it, right? You give developer tools to how to detect how to do it. Or you build a declarative system where developer tells what they want to do, not how. So these are completely two different designs, right? So like analogy I want to draw is in the data world.

• 30:36Speaker 03

  The database management system is a declarative system because people use database use SQL. SQL is a way you say what do you want to extract out of database? What kind of result you want? But you don't figure out which node is gonna, how many nodes you're gonna run on top of? How you read file in your disk? Which index you use, which project? You don't need to worry about any of those. And database management system will figure out, generate the new best plan and execute on that. Right, so database is declarative and it makes it super easy. You just learn SQL, which is learn a semantic minimal SQL and you can use it. Imperative side is there are a lot of ETL pipelines. And people design this stack system of with triggers, with actions and

• 31:19Speaker 03

  and you detect exactly what to do and it fails and then how to recover. So that's a imperative system. And we have seen a range of system in the ecosystem like go different ways. I think their value of both, their value both. I don't think one is gonna subsume the other. But we are leaning more into the philosophy of build a declarative system. Because from the lens of app developer and product engineer, that would easiest for them to integrate.

• 31:44Speaker 02

  I understand that's also why PyTorch won't. Well, this is the one of the

• 31:49Speaker 03

  Ease of use. So yeah, focus on ease of use and then let the, let the system take on the hard challenges and complexities. So we follow, we extend that thinking into current system design. So another announcement is we will also announce a, our next declarative system is gonna be, appear as a model that has extremely high quality. And this model is inspired by Owen, announcement for OpenAI. You should see that by the time we announce this or, you know, soon.

• 32:22Speaker 02

  Trained by you?

• 32:23Speaker 03

  Yes.

• 32:24Speaker 02

  Is this the first model that you trained at like this?

• 32:27Speaker 03

  It's not the first. Uh, we actually have, uh, trained a model called Fire Function. It's a function calling model. It's our first step into company AI system because function calling model can dispatch a request into multiple APIs. We have pre-baked set of APIs, the model learned. You can also add additional APIs to, through the configuration to let model dispatch accordingly. So we have a very high quality function calling model that already released. We have actually three versions. The latest version is very high quality. But now we take a further step, and you don't even need to use function calling model.

• 33:06Speaker 03

  You use our, uh, new model we're gonna release, uh, it will solve a lot of problem, uh, approaching very high like OpenAI's quality. So I'm very excited about that.

• 33:17Speaker 02

  Do you have any benchmarks yet or?

• 33:19Speaker 03

  We have benchmark. We're gonna release it. Uh, hopefully next week. We just put our model to L M S S and people are guessing, is this the next Gemini model? Or a Mada's model? People are guessing. That's very interesting. We're like watching the Reddit, uh, discussion right now.

• 33:36Speaker 02

  I mean, I have to ask some more questions about this. When OpenAI released the one, a lot of people asked about whether or not it's a single model or whether it's like a, a chain of models and no, and basically everyone on, on the strawberry team was very insisted that What they did for reinforcement learning, training of thought cannot be replicated by a whole bunch of open source model calls. Do you think that that is they are wrong? Have you done the same amount of work on RL as they have, or was it a different direction?

• 34:06Speaker 03

  I think they take a very specific approach where I do, the caliber of team is very high, right? So I I do think they are the domain expert in doing the things they are doing. But I I don't think there's only one way to achieve the same goal. We are on the same direction in the sense that the quality scaling law is shifting from training to inference. We are definitely on the, for that, I fully agree with them. But we are taking a completely different approach to the problem. All of that is because, of course, we didn't train the model from scratch. All of that is because we built on the shoulders of giants, right? So the current model available, we have access to, is getting better and better.

• 34:45Speaker 03

  The future trend is the gap between the open source model, closed source model, it's just gonna shrink to the point there's not much difference. And then we are on the same level field. That's why kind of I think our early investment in in inference and all the work we do around balancing across quality, latency and cost pay off because we have accumulated a lot of experience there. And that empower us to to build and release this new model.

• 35:12Speaker 02

  Yeah.

• 35:12Speaker 03

  That is um approaching open source quality.

• 35:15Alessio Fanelli?

  I guess like the question is, what do you think the gap to catch up will be? Because I think everybody agrees with open source models eventually will catch up. And I think with four, then with Llama three two, three one, four or five B, we close the gap. And then O one just reopen the gap so much and it's unclear. Obviously, you're saying your model will have.

• 35:33Speaker 03

  We're closing that gap.

• 35:34Speaker 02

  Yeah, but you think you think like in the future, it's gonna, oh, it's gonna be like months?

• 35:38Speaker 03

  So here's the thing that's happened, right? There's public benchmark, it is what it is. But in reality, open source model in certain direct dimension already on par or beat closed source model, right? So for example, for in in the coding space, open source models are really, really good. And in function calling, like fire function is also really good. So it's all a matter of whether you build one model to solve all the problem and you want to be the best of solving all the problems. Or in the open source domain, it's gonna specialize, right? All these different model builders specialize in certain narrow area. And it's logical that they can be really, really good in that very narrow area.

• 36:20Speaker 03

  And that's our prediction is with specialization, there will be a lot of expert models, really, really good and even better than like one set of closed source models.

• 36:32Speaker 02

  I think this is the the core debates that I am still not one hundred percent either way on in in terms of compound AI versus normal AI. Cause you're basically fighting the bitter lesson.

• 36:45Speaker 03

  Look at the human society, right? We specialize. And you feel really good about someone specializing doing something really well, right? And that's how our like when evolve from ancient time, we're all journalists, we do everything.

• 36:58Speaker 02

  Yes.

• 36:58Speaker 03

  Try but to now we having specializing different domain. So I my prediction is in the AI model space, it will happen also.

• 37:05Speaker 02

  Except for the bitter lesson. You all you get short term gains by having specialists, domain specialists. And then someone just needs to train like a ten X bigger model on ten X more inference, ten X more data, ten X more model perhaps, whatever the the current scaling law is. And then it supersedes the all the individual models because of some generalized intelligence slash world knowledge. You know, like I think that that is the the core insight of the GPTs, that GPT one two three networks.

• 37:32Speaker 03

  Right, but the scaling law again, right? The the training scaling law is because you have increasing amount of data to train from. And you can do a lot of compute, right? So I think on the data side, we're approaching the limit. And the only data to increase that is synthetic generated data. And then there's like, what is the secret sauce there, right? Because you have a if you have a very good large model, you can generate very good synthetic data and and then continue to improve quality. So that's why I think even OpenAI, they are shifting from the training scaling law into inference scaling law and it's the test time and all this. So I definitely believe that's the future direction. And that's where we are really looking at and doing inference.

• 38:10Speaker 02

  Couple of questions on on that. Are you planning to share your reasoning traces?

• 38:15Speaker 03

  That's a very good question. We are still debating.

• 38:19Speaker 02

  Yeah.

• 38:21Speaker 03

  What's your debating?

• 38:23Speaker 02

  I would say if you, for example, it's interesting that like, for example, Sweet Bench. If you want to be considered for ranking, you have to submit your reasoning traces. And that has actually disqualified some of our past guests, like Cosine was doing well on Sweet Bench, but they didn't want to leak those results. So that's why you don't see O1, O1 preview on Sweet Bench, because they don't submit their reasoning traces. And obviously it's IP, but also if you're gonna be more open, then that's one way to be more open. So your model is not gonna be open source, right? Like, it's gonna be a endpoint that you provide.

• 38:54Speaker 03

  Yes.

• 38:55Speaker 02

  That's okay, cool. And then pricing, also same as OpenAI, just kind of baseline.

• 39:01Speaker 03

  Yeah, this is, I don't have actually information. Everything is going so fast, we haven't even think about that yet. Yeah, I should be more prepared, so.

• 39:09Speaker 02

  I mean, this is live, you know, it's, it's, it's nice to just talk about it as it goes live. Any other things that you're like, you want feedback on, or you're thinking through it? It's, it's kind of nice to just talk about something when it's not decided yet, about this new model. Like, I mean, it's, it's gonna be exciting, it's, it's gonna generate a lot of buzz.

• 39:26Speaker 03

  Right. I'm very excited about to see how people are gonna use this model. So there's already Reddit discussion about it, and the people are asking very deep medical questions. And since the model got it right, surprising. And internally, we're also asking the model to generate what is AGI? And it generates a very complicated DAG thinking process. So it's, we're having a lot of fun testing this internally. But I'm more curious, like, how will people use it? How, what kind of application they're gonna try and test on it? And that's where we really like to hear feedback from the community. And also feedback to us, like, what works out well, what doesn't work out well?

• 40:11Speaker 03

  What works out well but surprising them? And what kind of thing they they think we should improve on? And those kind of feedback will be tremendously helpful.

• 40:20Speaker 02

  Yeah, I mean, so I've been a production user of Preview Mini since March. I would say they're very, very obvious jumps in quality, so much so that they they made flaws on it and for all just. Like they they made the previous day of the art look bad, like it's really, it's really that stark that difference. The number one thing I I actually, you know, just feedback or request, feature request is people want control on the budget. Because right now in O one, it kind of decides its own thinking budget.

• 40:51Speaker 03

  Yeah.

• 40:52Speaker 02

  But sometimes you know how hard the problem is and you want to actually tell the model like spend two minutes on this. Or spend some dollar amount, maybe it's time, it means dollars, I don't know what the budget means, but they do.

• 41:03Speaker 03

  That that makes a lot of sense. We uh we actually thought about that requirement and it should be at some point we need to support that. Uh not initially, but that makes a lot of sense.

• 41:14Speaker 02

  Okay, so that was a fascinating overview of just like the things that you're working on. First of all, I realized that I don't know if I've ever given you this feedback, but I think you're guys are one of like one of the reasons I agreed to advise you. Because like, you know, I think when when you first met me, I was kind of dubious. I was like.

• 41:30Speaker 03

  Why you?

• 41:32Speaker 02

  Replicate this together, there is like a laptop, there's like a whole bunch of other players, you're in very, very competitive fields. Like, why will you win? And the reason I actually changed my mind was I saw you guys shipping. You know, I think your surface area is very big. The team is not that big.

• 41:48Speaker 03

  No, we're only forty people.

• 41:50Speaker 02

  Yeah, and now here you are trying to compete with OpenAI and you know, everyone else like how, what is the secret?

• 41:57Speaker 03

  I think the team, the team is the secret.

• 41:59Speaker 02

  Oh, boy. So there's no, there's nothing I can just copy you just.

• 42:05Speaker 03

  No.

• 42:06Speaker 02

  Yeah.

• 42:06Speaker 03

  I think we all come from very aligned on the culture. Uh because most of our team came from Meta.

• 42:14Speaker 02

  Yeah.

• 42:14Speaker 03

  And many startups, so we really believe in results. One is result and second is customer. We're very customer obsessed. Um, and we don't want to drive adoption for the sake of adoption. We really want to uh make sure we understand we are deliver a lot of business values to the customer and we are, we really value their feedback. So we would woke up mid of night and deploy some model for them, shuffle some capacity for them. Um, and yeah, over the weekend, no brainer. So yeah, so that's just how we work as a team.

• 42:55Speaker 03

  And the caliber of the team is really, really high as well. So like as plugin we're hiring, we're expanding very, very fast. So if we are passionate about uh working on the most cutting edge technology in the general space, uh come talk with us.

• 43:13Speaker 02

  Yep. Let's talk a little bit about those that customer journey. I think one of your more famous customers is Cursor. We were the first podcast to have Cursor on and then obviously since then they have blown up. Cause and effect are not related. But, but you guys especially worked on a fast apply model where you were one of the first people to to work on speculative decoding in in a production setting. Uh maybe just talk about like what was the behind the scenes of working with Cursor?

• 43:39Speaker 03

  Right. I will say Cursor is a very, very unique team. I think the unique part is the team has very high technical caliber. Right? There's no question about it. But they have decided, although like they, uh many companies building coding Copilot, they will say, I'm gonna build a whole entire stack because I can. And they are unique in the sense they seek partnership, not because they cannot. They are fully capable, but they know where to focus. That to me is amazing. And of course they want to find a best partner. So we spent some time working together. They are pushing us very aggressively uh because for them to deliver high caliber product experience, they need the latency, they need the interactive, but also high quality at the same time.

• 44:23Speaker 03

  So actually we extended our product feature quite a lot as we supporting Cursor. And they are growing so fast and we massively scaled quickly across multiple regions. And we developed. Pretty high intense inference stack, almost like similar to what we do for Meta. That, I think that's a very, very uh interesting engagement and through that, there are a lot of trust being built. As in, they realize, hey, this is a team they can really partner with and they can go big with. That comes back to, hey, we're really customer obsessed and all the engineers working with them. There's just enormous amount of time syncing together with them and discussing and we're not big on meetings, but we are like stack channel always on.

• 45:09Speaker 03

  Uh, yeah, so you almost feel like working as one team. So that, I, I think that's really highlight of.

• 45:15Speaker 02

  Yeah, for those who don't know, like, so basically Cursor is a vehicle for, but most of the time people will be using closed models, like I, I actually use a lot of Sonnet, so you're not involved there, right? It's not like you host Sonnet or you have any partnership with it, though, like you're involved where Cursor is small or like they're, they're their, you know, house brand models are concerned, right?

• 45:35Speaker 03

  I don't know what I can say, but the things they haven't said.

• 45:39Speaker 02

  It's very obviously the dropdown is for O, but and then Cursor, right? So like I assume that the Cursor side is the fireworks side and then the other side is they're calling out to the other, you know? Just kind of curious. And, and then like, do you see any more opportunity on like the, you know, I think you made a big splash with like one thousand tokens per second. That was because of speculative decoding. Is there more to push there?

• 46:01Speaker 03

  We push a lot. Actually, when I mentioned a file optimizer, right? So I think we have a unique automation stack that is one size fits one. Uh, we actually deployed to Cursor early on, basically optimized for their specific workload. And that's a lot of juice to extract out of there. And we see success in, in that product is actually can be widely adopted. So that's why kind of we, we started a separate product line called File Optimizer. So a speculative decoding is just one approach. And speculative decoding here is not static. We actually wrote a blog post about it. There's a zooming different way to do speculative decoding. You can pair a small model with a large model in the same model family.

• 46:40Speaker 03

  Or you can have eagle hats and so on. So there are different trade-offs of which approach you take, really depends on your workload. And then you, with your workload, we can align. The eagle heads or Medusa heads or, you know, small big model pair much better to extract the facility to reduction. So all of that is part of the far Optimizer offering.

• 47:00Alessio Fanelli?

  I know you mentioned some of the other inference providers. I I think the other question that people always have is around benchmarks. So you get different performance on like different platforms. How should people think about? You know, people are like, hey, Lama three point two is X on M M L U. But maybe, you know, using speculative decoding, you go down a different path. Maybe some providers from a quantized model. How should people think about how much they should care about how you're actually running the model? You know, like, what's the delta between all the magic that you do and like what a raw model?

• 47:34Speaker 03

  Okay, so there are two big development cycle. One is experimentation. Where they need fast iteration, they don't want to think about quality and they just kind of want to experiment with product experience and so on, right? So that's one. And then it looks good and they want to kind of post-part on market for scaling. And the quality is really important and latency and all the other things are becoming important. During the experimentation phase is just pick a good model. Don't worry about anything else. Make sure even like Jenny is the right solution to your product and that's the focus. And then post-part market fit, then that's kind of the three dimensional optimization curve start to kicking across quality, latency, cost, where you should land. And to me, it's a purely a product decision.

• 48:19Speaker 03

  To many product, if you choose a lower quality but better speed and lower cost, but doesn't make a difference to the product experience, then you should do it. So that's why I think inference is part of the validation. The validation doesn't stop at offline about. The validation is kind of will go through A B testing through inference. And that's why we can offer various different configurations for test, which is the best setting. So this is the like traditional product evaluation. So product evaluation should also include your new model versions um and different model setup into the consideration.

• 48:58Speaker 02

  I want to specifically talk about what happens a few months ago with uh with some of your major competitors. I mean, you know, all of this is public. What is your take on what happens? And maybe you want to set the record straight on how fireworks does quantization? Because I think a lot of people may have outdated perceptions or they didn't read the the clarification post on your post quantization.

• 49:21Speaker 03

  First of all, it's always surprised to us that uh without any notice, we got caught out.

• 49:27Speaker 02

  Specifically by name, which is normally not what.

• 49:30Speaker 03

  Yeah, in in a public post and have certain interpretation of our quality. So I I was really surprised and uh it it's not a good way to compete, right? We want to compete fairly. And oftentimes, when one vendor give out results from interpretation, another vendor is always extremely biased. So we actually regret ourselves to do any of those. And we happily happily partner with third party to do the most fair evaluation. So we are very surprised and we don't think that's a good way to figure out the competition landscape. So then we react, I think when it comes to quantization, the interpretation, we wrote out actually a very thorough broad post.

• 50:16Speaker 03

  Because again, uh no one says it's all. We have various different quantization schemes. We can quantify very different parts of the model, uh find ways to activation to cross D Q communication to like consider. They can use different quantization scheme or consistent across the board. Again, it's a trade off, it's trade off across this three dimensional quality, latency and cost. And for our customer, we actually let them like find the best optimized point. And that's kind of how, and we have very thorough evaluation uh process to to pick that point. But for self-serve, there's no only one point to pick, there's no like customization available. So of course, we depends on like what we we talk with many customer, we we have to pick one point and.

• 51:00Speaker 03

  And I I think the end result like AA published, uh later on AA pu- published a quality measure. And we actually, we looked really good. So I I don't, I wouldn't, that's what I wanna mean is. I will leave the evaluation of quality or performance to third party. And work with them to find the most fair benchmark approach methodology. But I'm not applaud of approach of calling out specific names and critique other competitors uh in a very biased way.

• 51:33Speaker 02

  Databases uh happens as well. I think you're the more politically correct one and then Dima is the more uh.

• 51:42Speaker 02

  It's you on Twitter.

• 51:44Speaker 03

  We uh.

• 51:45Speaker 02

  It's like the Russian uh.

• 51:47Speaker 03

  We we partner. No, actually, all these reactions we build together.

• 51:53Speaker 03

  We play different roles.

• 51:56Speaker 02

  Another one that I want is uh uh on just the last one on the competition side. There's a perception of price wars in ho- hosting open source models. You are you're and we talked about uh the competitiveness in the market. Um do you aim to make margin on open source models?

• 52:13Speaker 03

  Oh, absolutely, yes. So but I I think it it really, when we think about pricing, it's really. Need to coordinate with the value we are delivering. If the value is limited or. There are a lot of people delivering same value, there's no differentiation, there's only one way to go is going down. I so through competition. If I take a big step back, there is pricing from, we are more compared with like closed model providers, APIs, right? The closed model provider, their cost structure is even more interesting because we don't have any, we don't bear any training costs. And we focus on inference optimization and that's kind of where we continue to add a lot of product value.

• 52:53Speaker 03

  So that's how we think about product. But for the closed source API provider, uh model provider, they bear a lot of training costs. And they need to amortize the training costs into the inference. So that created very interesting dynamics of. Yeah, if we match pricing there, and I think how they are gonna make money is is very, very interesting.

• 53:14Speaker 02

  So for listeners, OpenAI is twenty twenty four, four billion in revenue, three billion in compute training. Uh two billion in compute inference. Um.

• 53:25Speaker 02

  One billion in research, compute amortization, and seven hundred million in salaries. So that is like.

• 53:36Speaker 02

  I mean, a lot of R and D.

• 53:37Speaker 03

  Yeah, so I think matter is basically like make it zero.

• 53:42Speaker 02

  Yeah.

• 53:43Speaker 03

  So that's a very, very interesting dynamics we're operating within. But coming back to influence, right, so we are, again, as I mentioned, our product is, we are a platform, we're not just a single model as a service provider, as many other influence providers, like they are providing single model. We have our optimizer to highly customize towards your influence workload. We have a component system where significantly simplify your interaction to high quality and low latency, low cost. So those are all very different for other providers.

• 54:14Speaker 02

  What do people not know about the work that you do? I guess like people are like, okay, if fireworks, you run model very quickly. You have the function model, is there any kind of underrated part of fireworks that more people should try?

• 54:27Speaker 03

  Yeah, actually, one user posted on X dot com. He mentioned, oh, actually fireworks can allow me to upload the Laura adapter to the service model with, at the same cost and use it at same cost. Nobody has provide that. But that's because we have a very special, like, we, we rolled multi Laura last year, actually. And we actually have this function for a long time. And many people has been using, but it's not well known that, oh, if you find your model, you don't need to use on demand. If you find your model is Laura, you can upload your Laura adapter. And we deploy it as if it's a new model.

• 55:10Speaker 03

  And then you use, you get your endpoint and you can use it directly, but at the same cost as the base model. So I'm happy that users marketing it for us. He discovered that feature, but we have that for last year. Uh, so I think we, I think to, um, feedback to me is, uh, we have a lot of very, very good feature as, as, um, shout out to the people.

• 55:33Speaker 02

  To the company and I didn't know that you had speculative decoding released, you know?

• 55:38Speaker 03

  We have Pom catching way back last year also. We have many, yeah. So yeah, so I I think that is one of the underrated feature. And if if they're developers, um you are using our self serve platform, please try it out.

• 55:52Speaker 02

  Yeah, yeah, yeah. The Laura thing is interesting because I think you you also like. The reason people add additional cost to it is not because they feel like charging people. Like normally in normal Laura serving setups, there there is a cost to dedicating loading those weights and dedicating a a machine to that inference. Uh how come you can't avoid it?

• 56:12Speaker 03

  Yeah, so so this is kind of uh our technique called multi Laura. So we basically have many Laura adapters share the same base model.

• 56:21Speaker 02

  Yeah.

• 56:21Speaker 03

  And basically we significantly reduce the memory footprint of serving. And one base model can sustain a hundred to a thousand Laura adapters. And then basically all these different Laura adapter can share the same like that direct the same traffic to the same base model where base model is dominating the cost. So that's how we advertise uh that way. And that's by how we can manage the tokens per uh dollar, uh million token, uh pricing, the same as this model.

• 56:50Speaker 02

  Is there anything that you think you want to request from the community or you're looking for model wise or tooling wise that you think like someone should be working on in this?

• 56:59Speaker 03

  Yeah, so we really want to get a lot of feedback from the application developers who are starting to build on Genai or on the already adopted or starting about thinking about new use cases and so on to try out our fireworks first. And let us know works out really well for you and what is your wishlist? And what is what sucks, right? So uh what is not working out for you? And we we like to continue to improve. And for our new product launches, typically we want to launch to a small group of people. Usually we launch on our Discord first to have a set of people use that first. So please join our Discord channel. We have a lot of communication going on there.

• 57:44Speaker 03

  Again, you can also give us feedback. We'll we'll have started office hour for you to directly talk with our DevRel and engineers to exchange your long notes.

• 57:53Speaker 02

  And you're hiring across the board?

• 57:55Speaker 03

  We're hiring across the board. We're hiring front engineers, infrastructure, cloud infrastructure engineers, backend system optimization engineers, applied researchers. And like researchers who has done post-training. Uh, who has a lot of fine tuning as well.

• 58:10Speaker 02

  Yeah, that's it. Thank you.

• 58:12Speaker 03

  Awesome.

• 58:13Speaker 02

  Thanks for having us.