Ep 50: Fireworks CEO Lin Qiao on Why There Won’t be a Single Model, Will Hyperscalers Win Inference & AI Use-cases with PMF

Lin Qiao, the co-founder of Fireworks.ai, sits down for a deep dive into the future of AI. Lin ran the PyTorch team at Meta, which developed some of the most fundamental open-source AI software in use today. She’s got a riveting perspective on the AI landscape that is a must-listen.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  Lin Xiao is the co-founder and CEO of Fireworks, a company building fast and efficient inference for compound AI systems. She was also one of the creators of PyTorch over at Meta and has raised over $75 million from investors as a Koyan benchmark. On unsupervised learning, we had a fascinating, wide-ranging conversation. We hit on Lin's thoughts on the evolving open source ecosystem, what infrastructure is required to make agents work in 2025. And we also hit on the challenges of building an AI infrastructure company when so many changes are happening at the model layer. I think folks are really going to enjoy this conversation. Without further ado, here's Lin.

• 0:31Jacob Effron?

  Well, Lin, thanks so much for coming on the podcast. Really appreciate it.

• 0:34Speaker 01

  Thank you for having me.

• 0:35Jacob Effron?

  Yeah, I've been looking forward to this one for a while. And I feel like it's probably a really interesting time to get to chat with you. Obviously, you guys have built just an incredible platform for developers and folks that are building in the AI space. And there's always, it's never a dull moment in the AI world, and I feel like there's always things changing. And, you know, one that I'd be really curious to get your take on is obviously, I think, everyone seems to be talking about test time compute right now. And one, and, you know, many models that seem that they will shortly follow. And I'm wondering, like, how does that change your kind of strategy or product that you offer at Fireworks, like these types of models?

• 1:06Speaker 01

  Yeah, I think from the inception of Fireworks, we always had that in mind. I think the, like, what is Fireworks? Fireworks is a gen AI platform with a focus on inference. And here, the top level goal is deliver the best quality, lowest latency, and lowest cost. Right? So in the inference stack, and we can, I'd love to dive into details there, because inference is not a single model as a service inference. It's not that simple. So it just kind of allude to test time inference, test time quality scaling law. The inference system we envision in the future is a complex inference system with logical reasoning, with access to a lot of hundreds of small expert models and so on.

• 1:50Speaker 01

  So the level of problems we're trying to solve is not, hey, you know, this is just API call, in, out, done. It's never this simple.

• 1:59Jacob Effron?

  And so it sounds like you kind of envision a world where, you know, the user puts in a query and like at Fireworks, you're doing the routing and figuring out, like, what is the perfect, like, you know, the best performance model for whatever that query is as a response?

• 2:11Speaker 01

  Right, so I think Let's take a big step back of what the problem we're trying to solve here, right? So there are a lot of limitation of models. And its limitation comes from the models are not deterministic. It's probabilistic, right? By nature, it's not desirable when you want to deliver factual results, always truthful result to your end user. So controlling that, hallucination is extremely important. And the second is a complex business problem. We have many customers using us. It has to assemble across multiple models across multiple modalities to solve those problems. As we're talking right now, right?

• 2:52Speaker 01

  So as kind of a lot of application are to consumer, to personal developers, we are not communicating, texting each other. I'm processing audio information and visual information to kind of deliver a good, like, interactive experience, a conversation we're having right now, similar to a lot of GNI-based native applications, consumer personal facing. They also need to process multiple, across multiple modalities. And even within same modality, let's take L M's. There are many different expert L M models specializing doing classifications, summarization, multi-turn chats, tool calling, and they're all slightly different from each other.

• 3:37Speaker 01

  So that's kind of single models very limited if you want to solve a real-world problem. And the last but not least, single model is very limited in knowledge. Its knowledge is limited by its training data. Training data is finite, not infinite. So a lot of like real information, real-world information lives behind APIs. Yeah. They are public APIs or even proprietary, private APIs within enterprise that there's no way you can access to it without like working directly with enterprise. So across the board, we envision that like the next barrier is how to

• 4:11Speaker 01

  Go beyond single model as a service, right? So the world needs a notion called compound AI system. As in multiple models across different modalities along with various different APIs. Holding knowledges, including databases, storage systems, knowledge base. Need to work together to deliver the best AI results.

• 4:32Jacob Effron?

  Yeah, but as you think about building these more complex compound AI systems, like what are the tools you think developers will need to, you know, be able to effectively build these?

• 4:39Speaker 01

  Maybe you can first start talking with like a polar opposite design points, right? So one design is is called imperative. Where you have full control of what is the workflow and what are the inputs, outputs? Everything you want to make it deterministic, right? So so that's kind of one level of design. The other design, so you basically design the the house.

• 5:06Jacob Effron?

  Yeah.

• 5:07Speaker 01

  Right, dictate that. The other design is called declarative. You define the what, the what problem this system should solve for you. But then you let the system figure out the house. So in general, in the industry, it's not unique to AI. In general, we have these two school of thoughts in approaching the system design. For example, in the database world, it's kind of they're full of those kind of examples. SQL is the example of declarative.

• 5:35Jacob Effron?

  Yeah.

• 5:36Speaker 01

  Where you data engineers or analysts, they define very like exactly what we want to retrieve out of the database. And database managers figure out the execution plan, the best, most efficient execution plan, right? And then the ETL process is declarative. And you basically define, oh, here's the source and what are the processing logic? How you trigger the next step and how do you do backfield? Very imperative. And those are nothing wrong with either of those approaches. Those are different, just different approaches. And the file was based on our. Pi Torch experience, we want to deliver, ultimate our design principles. We want to deliver the simplest the user experience.

• 6:21Speaker 01

  And hide all the kind of nitty gritty details on complexity in the backend as much as possible without hurting the speed of iteration. And it's a very fine balance to strike there. So we are leaning more towards a more declarative system with full debuggability and mental abilities.

• 6:39Jacob Effron?

  What are some of the, like, some examples maybe of where you're on the line and you're like, we could do it this way, that would be more declarative or a different way? Like, what are some of those trade offs you have to, or you're thinking about making?

• 6:49Speaker 01

  Yeah, so for example, where we started, we started with the lowest level abstraction, which is single model as a service. Where today, we provide hundreds of models across large engine model, across audio models, transcription, translations, speech synthesis and alignment, across vision models, where the source can be paid for images, screenshots and so on. And embedding models, image generation models, we're adding video models. So those are kind of the lowest, lowest level of building blocks. And developers can just assemble what they want on top of that, but there's a lot to assemble. A lot of pieces to assemble and the quality control is difficult because they, every week,

• 7:32Speaker 01

  there are multiple models get released. And they constantly ask us, we want to try this model, what's your suggestion? Should we even try or like, then what is our production stability, right? And how to do version control and all this becomes a problem for them. So, but we start there, so we, we know like, like how, we can understand how the industry is evolving. But then we quickly realized there's a huge gap of usability. Especially when it comes to enterprise and, and there's a huge gap there and we want to fill in the gap.

• 8:04Jacob Effron?

  When you're talking to folks, like, why, why aren't there more Gen AI applications out there?

• 8:08Speaker 01

  So I think there are a lot of barriers actually. One is, there's no like one model fits all, that's our observation.

• 8:19Speaker 01

  It's kind of the nature of the training process. The training process is a very opinionated process that you have to pick which subset of problem across like thousands of problem in the world you care the most about and you devote the biggest amount of resource and money to acquire data and make sure your data quality and a variety of data in that space is the best. And which subset of problem you don't care and we're in the middle, right? So because of that, the end result of the model will be really, really, really good at doing things and then really bad at doing certain things, right? So that's just the nature of model training. We believe the future are the hundreds of small expert models.

• 9:02Speaker 01

  Because when you shrink the problem down to a narrower space, it's inevitably much easier for small model to thrive in pushing the quality. And as a matter of fact, in the open source world, this is really great for the open source community because with open source based models, it gives a lot of control for you to customize your model. There are a lot of model providers, they focus on post-training or fine-tuning and deliver specialized model, giving back to open source community that's really, really good at solving certain problems very well. So we believe the future are hundreds of small models.

• 9:45Speaker 01

  I think that's kind of where we see the enterprise world is moving towards, they want to have more control. They want to have stability.

• 9:53Jacob Effron?

  In this world of like hundreds of small expert models, I can imagine one world in which, you know, there's so many people taking open source models and tweaking them in some ways and so fireworks just has a big library of them and I'm in enterprise and I have a problem and, you know, you're like, hey, here's a dozen really great models that will orchestrate together. I could also imagine there's probably some cases where you tell an enterprise, hey, you should really fine tune for this use case or you might actually, maybe in some cases, even pre-train a model to some extent. How do you think about where that plays long term? Like, how do you think in five years, like, will enterprises be fine tuning a lot of models or pre-training or how does that evolve?

• 10:26Speaker 01

  We deeply believe in customization, again, as I mentioned, but the process is it's kind of, it's not, I would say it's not straightforward, like, it's not like, oh, like, one minute done, right? So it's longer than that. So we see a fine balance between

• 10:42Speaker 01

  Like fine tuning and problem engineering. Right, it's, it's, there's a strong trade off there and that's where we are working on. Because we're believing in customization, we're working on how to make customization extremely easy. So the balance is falling, right, so um it's much immediate. You can see the result immediately, it's much more responsive, interactive using problem engineering. This is kind of, that's the nature of it. And we see a lot of enterprise or developers start with problem engineering, right? So hey, is this model steerable towards the direction I want to do? And I can test it out quickly. Um but then we see thousands of lines of system problems. Yeah. At some point, if you want to continue to steer it, it got stuck. Yeah. Because you don't know where to change and you know, it's gonna all get lost.

• 11:27Speaker 01

  Uh so, so then um. So solving a very complex system problem management is a problem myself. Yeah. Uh huh, so actually that eventually is the key to how to make customization easy. Is to solve that problem. And, and we are gonna launch a product to solve that problem. Interesting. Yeah, so um and then we see, um then at that phase. Leave that aside, at that phase, you have thousands of lines of system problem, then what do you do? Um and usually that's the time, it's the prime time for you to do, use fine tuning to absorb that system problem into the model itself. Because by that time, you should already proven the model is steerable to solve your problem.

• 12:10Speaker 01

  You, you are able to let the model follow your instruction. Uh your instruction is pretty much the data you provide to the model. Uh and then, you know, it would be good candidate to do the fine tuning. Uh and that already mean you're probably evolving from pre-part of market fitting to post-part of market fitting, product scaling. And if, once you absorb the long lengthy system prompt into the model, the model can run faster also. And the cheaper and also with high quality. So we see that movement, it's very organic movement. But still, um getting stage of managing, like even kind of uh before you absorbing to, to fine tuning.

• 12:47Speaker 01

  Managing these long process and problem is still a very challenging thing. So, yeah.

• 12:52Jacob Effron?

  Do you think pre-training ever makes sense for enterprises?

• 12:54Speaker 01

  Pre-training is kind of the whole entire industry is going to hey, pre-training is being consolidated, right? So into hyperscaler, that's pretty much the reality. We do feel like there are, we do see enterprise, some enterprise do pre-training models because they have, this is kind of core to their business or for other reasons.

• 13:19Speaker 01

  But, yeah, so, but the differentiation, the question we see a lot of enterprise asking themselves is, or even startup asking, what's the differentiation? Because pre-training is very expensive and RI has to justify you threw a lot of money and the human resources into, into doing this. And if you can just do post-training on top of very strong base model, then RI is much stronger and you are much more agile to test different ideas.

• 13:43Jacob Effron?

  You said, you said an interesting vantage point because you see so many people building on top of fireworks. Like, what use cases for Gen AI do you feel like have product-market fit today?

• 13:52Speaker 01

  I would say there are, I would classify those use cases in probably two different buckets.

• 13:59Speaker 01

  But in summary, I think most of use case we've seen successful getting adoption is kind of human in the loop, automation.

• 14:10Speaker 01

  Not human, other loop automation yet. So my hypothesis is a Gen AI system has to be, it has to be human, debuggable, understandable, maintainable, operable. If a human cannot evaluate it, cannot maintain it or put influence over it and operate that in production, it will be really, really hard to get adoption. So because of that nature, we see a lot of kind of vibrant kind of product offering from all different kind of assistant to human in different categories.

• 14:52Speaker 01

  We have seen people building assistant to doctors to make their scriving much easier. I've seen assistant to teachers and students and people want to learn foreign languages, educational. We have seen assistant to coding. Of course, that's a highly competitive space, but we have worked very closely, for example, with Cursor, Resource Web and many other great companies. We have seen assistant to, like medical assistant. Apparently, there's shortage of more than million nurses in North America. And there are a lot of patient need being like paying attention to. So that is a whole slew of like to consumer, presumer, like developer space. So that is one.

• 15:38Speaker 01

  The other is kind of more B two B, as in there are automation, like call center automation. But more like call center automation is a great example. It has, you can build assistant to human agent to make them more productive and answer the questions better or replace human. We have seen a lot of success of building this assistant to help human agents, right? So that's another example. A lot of process of optimizing business logic, business workflow, efficiency and so on.

• 16:10Jacob Effron?

  And then what about on the model side? What have you noticed in terms of the models that are actually getting used by companies?

• 16:15Speaker 01

  We see a lot of convergence of variation of Llama models. I think that's a testimony to the quality of the model. And it's a very strong base model, very, very good instruction following and very good for tuning. So and of course, meta backing it is very important and see a lot of enterprise adoption there.

• 16:38Jacob Effron?

  Obviously, I'm sure something your enterprise customers think a lot about or, you know, is just like evals. And I'm sure, you know, you obviously have thought a lot about this, like, what's kind of the current state of evals? And, you know, how do you think about the right building blocks to give, you know, enterprises that might have very different use cases and as a result, very different ways of evaluating success?

• 16:59Speaker 01

  Many enterprise do at the stage of eval is vibe based.

• 17:05Speaker 01

  Which is not surprising, right? So yeah, yeah, yeah, so they are, but they quickly realize. I mean, for the early stage, product development is kind of more vibe based. I just want to get a sense of how this product is like feel like with, with, with different models and so on, right? And they quickly evolve to the stage that they are consciously building eval. And they know this is kind of an investment area. But for them to stay on top of the state of art, they have to be able to evaluate. And also they have to be able to evaluate once they get into kind of more deeper problem engineering or fine tuning. Like how to just evaluate quality is important. They cannot always go to A B testing.

• 17:48Jacob Effron?

  Right.

• 17:49Speaker 01

  To, A B testing is ultimate process to, to determine the product impact. But it's longer cycle, so.

• 17:58Jacob Effron?

  It is funny that we talked to some, I think Source Graph's customer of yours said they don't even bother with evals. They just release it out, A B test it, and they'll know pretty quickly from their developers which, which model is better.

• 18:08Speaker 01

  Yeah, so but the, the evals are very important because.

• 18:14Speaker 01

  And, and they, for our customers' point of view, they realize investing in generating good eval data set and, and adding more and more to it can give them a very clear view.

• 18:29Speaker 01

  What matters, what doesn't matter, because like in this process of catching up with the constant moving train of those small expert model. Their, the model is not just every week, there's a better model like beating leaderboard. The model is also getting more and more specialized. And they, they will also, they are, many of our customer, they start from the mod, their product is open ended. They don't have a strong opinion what kind of question they should expect from their user to their they start to get more and more clear opinion. And they're hardened their product design into open-ended product to like specialized product in different product features. And then from those product features, they want to have specialized model to solve this problem. So it's a natural evolution, evolution of product development.

• 19:14Jacob Effron?

  And I guess like, you know, maybe just talking about some of the stuff you guys have built. Obviously, you're taking the great models that exist in the open source world. And then you also build your own models, right? You built F1. Maybe talk a little bit about why, you know, why you decided to do that?

• 19:28Speaker 01

  So we release F1 under API and that's a model API. Basically, how you use F1 is as if you are using a new model. But F1 is really a complex logical reasoning inference system.

• 19:43Speaker 01

  So which underneath, we haven't talked too much about what's underneath F1. But it's kind of here is a snippet of what's underneath. So underneath there are actually multiple models. And also there is a logical reasoning steps that we implement in our system. So then actually building a system is very complex. It's not, it's not like just regular single model as a service inference. And also we have to solve a lot of, a lot of quality related problem. Because now you let the model talk with itself, let the model talk with each other.

• 20:23Speaker 01

  And how, what kind of information they communicate to drive quality and they are sensitive to that, right? And how to do quality control in this complex system? I think the complexity is actually higher than building a database management system.

• 20:41Speaker 01

  In that sense. And yeah, and then of course, because there are multiple steps involved. And then your overall influence latency and then cost becomes more interesting. So that's our bottom bar, because we're a team like really specializing optimized for that. And then we view the our like when we start a company, we build a lower level of single models of service. That's a building block towards this complex logic reasoning stack.

• 21:09Jacob Effron?

  Interesting. And then obviously, I think a big part of it is function calling as well, right? And like, what are the barriers, I guess, to getting function calling even better today? It feels like we're still kind of in the nascent stages of that in many ways.

• 21:18Speaker 01

  So function calling, like many, many people recognize that actually, we see a lot of after we launched one. We have, we have a waiting list for people to get in. And we talked with many of them. Apparently, their use case, the majority of use case is actually building agents. And they need function calling. So what is function calling, right? Function calling in basics is an extension point for, for this model to call into other tools to enhance the quality of, of the answer. But function calling is not just about calling into one tool. It's actually very complex if you think about it. Usually, people use those function calling model in the multi-turn chat context, right?

• 22:00Speaker 01

  The model is, it should be, it need to be able to hold a long context of what has, what the commerce. And then use that context to influence which is as tools call. And oftentimes, they also need to call into multiple tools, up to like, could be up to hundreds of tools, tool selections. And also, they need to, it's not just call one tool at a time. Oftentimes, they also need to call like multiple tools in parallel. And it's sequential, so it's a complex coordination plan in one shot. So our function calling is able to do parallel sequential complex planning. And then orchestrate and execute on that plan.

• 22:44Speaker 01

  Of course, then, the precision is important, like if I ask the model to like do a general chart of a stock price of top three cloud providers. Underneath, it will return the answer quickly and then show you a chart. But underneath, there's a search to find out top three. There's three parallel function calls to get the stock price, and then one call into the chart to get the chart back. So this is a very simple example, right? But it demonstrates the complexity of orchestration. And the model need to have very strong capability of understanding, if you're plugging your tool, when to call this tool, and how to drive that position is very important. So that makes the tuning process very complicated.

• 23:25Speaker 01

  But we have been investing this space for now about a year.

• 23:30Jacob Effron?

  Yeah.

• 23:31Speaker 01

  So yeah, so like, we start to see, like, when we start to release the first function calling model, I think we are ahead of the adoption curve.

• 23:38Jacob Effron?

  Yeah.

• 23:39Speaker 01

  The use case hasn't kind of fully taken off.

• 23:41Jacob Effron?

  And how do you think about like what, you know, why decide to build that versus, I'm sure if you waited at some point, there'd be, you know, a good open source model with some of those capabilities. Like, how do you kind of figure out when to build things in-house on the model side?

• 23:52Speaker 01

  Yeah, so we heavily build on top of open source community, right? That's absolutely, it's the case. And we are betting on that because we believe the hundreds of small expert model will come from the open source community.

• 24:05Jacob Effron?

  Yeah.

• 24:06Speaker 01

  So that is directly aligned with our vision. But at the same time, we're also heavily investing in the compound AI system, where it's all about composing those hundreds of small expert model to solve a complex business task in a very easy way. And the critical part of this composability lies in this layer of orchestration. So the model was intelligent enough to be able to call into different tools. And here, you can think about the general way of thinking about tool is each individual small expert model is also a tool.

• 24:41Jacob Effron?

  Right.

• 24:42Speaker 01

  In addition to this particular large language model. So then it becomes a critical ingredient of tying everything together. And we will invest in this strategically critical area that we cannot just wait to see what happens.

• 24:59Jacob Effron?

  Where do like the reasoning models end up fitting in? I mean, obviously, at least the ones we've seen today are, you know, I think of them as bulky and generally good at reasoning of all things. Do we end up with like small expert, you know, like test on compute intensive models as well? Or like, do you imagine we'll only kind of route to those in very specific instances? Or like, as I think about these compound systems, I'm just curious like where these O1-like models will fit into that.

• 25:23Speaker 01

  I think even for reasoning, there are different paths to solve reasoning problems. And there will be different models specializing in different paths. One path is to have a very strong base model itself to do self-inspection. And a lot of technique already being discussed with chain of thought, truth of thoughts, and backtracking and all these different technologies. So that's this one. And there will be a new set of models. They can do logical reasoning, not in the prompt space. And this is something I'm very excited about, but in latent space.

• 26:06Speaker 01

  Because you can think about, when we think, right? We don't have to talk in words, like our thinking process. We can, but we don't have to, right? So I'm pretty sure when we process a problem in our head, we're probably thinking in a different space.

• 26:21Jacob Effron?

  Right.

• 26:22Speaker 01

  Right? So similar to the model as well. So there are a lot of active research happening, like to how to make the thinking process much more efficient and much more native to the process. So I'm very excited about that kind of research. There will be other flavor of logical reasoning. So we are not, we don't want to be very opinionated about which one is going to win. Instead, we're going to integrate all these different flavor into our logical reasoning system.

• 26:51Jacob Effron?

  I love that. I guess outside of what you're doing at Fireworks, what do you feel like are some of the other big unsolved problems in AI infrastructure today?

• 26:59Speaker 01

  I think we have seen a lot of movement into building agentic workflows.

• 27:06Jacob Effron?

  Yeah.

• 27:08Speaker 01

  And again, I think we're still at the early stage of figuring out what is a right user experience.

• 27:16Speaker 01

  What is the abstraction, the whole industry, right? What is the right abstraction? What we should hide behind the system? We should expose to developer. We're still kind of in very early stage and there it's, I think it's very experimental right now. But where the abstraction lies will determine the complexity of the infrastructure hiding behind the abstraction. But we, I think we now start to form opinion around that.

• 27:48Speaker 01

  And in a month or two, we're gonna kind of GA F1. And so, okay, I don't think in process is building F1 is one is kind of, F1 is our own exercise to understand the system abstraction and the complexity of building logical reasoning engine.

• 28:11Speaker 01

  And from there, when we GA, we want to expose developer facing, plugins and how developer can build their own F1s. So yeah, so I'm very excited about that direction.

• 28:24Jacob Effron?

  That's super cool because obviously you learn the tools and the abstractions that are necessary by building it yourself. And then, you know, you basically allow developers to recreate that. I love that as an approach. I guess you know, a lot of, it feels like on the hardware side, I mean, obviously everyone's using Nvidia, but there's a lot of competitors have been popping up. And I'm curious like what you make of that space and the viability of some of those efforts. I think you guys support AMD chips for inference on the platform. Like, you know, when does that make sense for developers? And how do you see that kind of evolving?

• 28:52Speaker 01

  In general, we see a kind of scarcity in the developer space that they understand hardware, low-level hardware optimization. And we also see a very

• 29:07Speaker 01

  Big change in the kind of hardware development cadence. Before, it's kind of every three years, there's a new hardware skill. Not every one year, there's a new hardware skill for each hardware vendors. And then the whole hardware space is moving very fast. And how to kind of access to the best hardware? And by best, I really mean, there's, again, there's no one size fits all. So it really depends on your workload pattern. Even for the accessing the same model, there's no best hardware for one model even. Because your workload distribution is gonna determine where is your biggest bottleneck. And different hardware skill is gonna be the best for solving, kind of removing certain bottleneck the best.

• 29:49Speaker 01

  So we even at the fireworks layer, we absorb the burden of integrating and determining which hardware is best for what kind of workload. Even for one workload, when you have mixed access pattern, we can route to different hardware. So yeah, so we want to really kind of alleviate the concerns and burden of developer, they should focus on building product. And we will, we will kind of take the complexity of managing and optimizing for the hardware.

• 30:20Jacob Effron?

  Yeah, it seems like you're obviously super focused on these compound systems. You know, there's obviously always gonna be some set of folks that like, you know, just wanna call a Lava model as it is and run it. In that world, you know, it feels like for a while, there was really a lot of competition on the inference side. Like a new model, an open source model would come out and everyone would be offering their prices and it would get lower. You know, in that game, do you think that like, eventually that's the hyperscalers are best suited to win? Just like kind of straight inference on an open source model? Or like, how does that kind of develop?

• 30:49Speaker 01

  That's a very complex question to answer. I think, I think all hyperscalers, they want to be the Apple, right? So they want to build iPhone. They want to kind of build a vertically integrated stack because they can. So, but again, the direction we're heading towards is we feel like the future lies in hundreds of small expert models. And we want to harvest that energy and build a compound system that leverage the best models. To solve a complex problem. So, in my view, I feel like the hyperscalers, what's their biggest benefit?

• 31:30Speaker 01

  Their biggest benefit is that to solve the problem requires a lot of resources in terms of money and the human. Those examples are like building data centers, acquiring powers.

• 31:43Speaker 01

  And install tons and tons of machines, deploy them, light them up. Yeah, and turn up large-scale storage and compute. Cloud provider, please solve those problems, right? So this kind of massive problem at scale.

• 32:01Speaker 01

  And we specialize in solving problems that requires a lot of the combination of engineering craftsmanship and deep research. And then we can deploy at scale. So I think inference is like that. I work at Meta, it's not like Meta has a thousand people inference team, right? Because it, once you have a, a system that can be horizontally scalable with high production quality. Then the rest is just let it scale. That's the beauty of designing a highly scalable system.

• 32:41Speaker 01

  So, so in that sense, I think this is squarely in our, you know, our strength and expertise. And also, again, the inference system is not simple. It is actually gonna, we're evolving to, to, to get into this compound logical reasoning inference system. Where there's a lot of complexity in building that. And that complexity is not by throwing people and money to solve, so, yeah.

• 33:08Jacob Effron?

  Yeah, I guess with these smaller expert models, one, you know, one thing I wonder is, you know, as these models get smaller. It feels like there's a big trend to running them locally too. And I'm curious like how you see that playing out over, over the next few years? And then, you know, if that, you know, becomes part of the fireworks platform as well.

• 33:23Speaker 01

  Yeah. I've seen a lot of argument for running models locally for two reasons. And also, we should talk about what does local even mean, right? One reason is cost-saving, because yeah, you need to pay GPU on cloud and, you know, running on desktop, you don't need to pay. And yeah, so the other is privacy, right? So I have different thoughts on that. I think that makes sense to kind of offload compute from cloud to desktop. I think many application like Zoom, for example, that save a lot of cost for them, right? But offload, offload compute from cloud to mobile is a different story.

• 34:06Speaker 01

  Because mobile phone has a very, like, it's getting like beefy and beefier, but it has a very limited power. Right, so at Meta, for the application, a lot of application metrics are closely monitored, including cost of time, your power, like, consumption. All this affect application rating and user adoption and experiences. So because of that, the model that can practically deploy to mobile is tiny. Like, 1B, 10B, and very, very small. And those model has limited capability, right? So I think like, between mobile and cloud, this kind of the difference is extremely big.

• 34:51Speaker 01

  And offload to desktop, I think for many like consumer personal facing application makes sense. But also, like, there's another argument of, hey, you know, it's more private. And privacy is interesting, right? Because we have a lot of personal data is already on the cloud.

• 35:07Jacob Effron?

  Right.

• 35:07Speaker 01

  Most of our personal data is not on like local desk anymore. So then, kind of, hey, does that even make sense for privacy? Yeah, so that, I think that's a separate concern.

• 35:19Jacob Effron?

  Yeah, super interesting. I mean, I guess on the, on the open source side, you mentioned, obviously, you used to work at Meta. It feels like, you know, obviously, they've been providing a huge service to ecosystem by training these larger and larger models. Like, how far do you think they go on continuing to scale spend on, on open source models before, like, at some point? I mean, I guess everyone's asking this for all the players, not just the open source players. But how do you think about like the, the extent to which it makes sense to keep pushing on, on pre-training spend?

• 35:43Speaker 01

  I think that's a good question to Zach.

• 35:49Speaker 01

  No, I'm just excited. We work very closely with Meta team, of course, we came from Meta. And we actually co-design, they're not just open source Llama models, they're also building a standard called Llama Stack.

• 36:04Jacob Effron?

  Yeah.

• 36:04Speaker 01

  And Llama Stack, the intention is kind of standardize the

• 36:09Speaker 01

  the tools stack around Llama models. And we co-design with them because we are kind of, we have a lot of information coming from our customer that can help them like make design choices. So, so I think that's kind of ambition from Meta is to build this Android world where everything is nicely standardized and you plug into different components and make it very easy to adopt. And from my vintage point of view, my understanding is Llama 4 is coming soon, right? So there, there, there will be continuous investment from Meta that train. I haven't seen that stopped.

• 36:55Speaker 01

  Of course, like if it's a general question, when would model providers stop or invest less on pre-training is when there's not much ROI. So why there's not much ROI is because we're hitting a wall of data. Where we're now data, everyone call the same internet data. There's just, we have exhausted synthetic generated data. We have exhausted a combination of multimedia and text data. Then kind of there's less return.

• 37:25Speaker 01

  Before we hit that wall, I think there will be continuous investment.

• 37:29Jacob Effron?

  When do you think we'll hit that wall?

• 37:31Speaker 01

  I think we're, it's definitely getting longer and longer. Like from like OpenAI's release point of view, right? So.

• 37:40Speaker 01

  And I definitely see a lot of investments start to shift from pre-training to post-training, from post-training to inference.

• 37:47Jacob Effron?

  Right.

• 37:48Speaker 01

  So I think we're already hitting a soft war, not a hard war. So yeah, so I think in general, as an industry, we're the ROI is kind of transitioning from pre-training to post-training to inference.

• 38:04Jacob Effron?

  As you think about like all the sets of tools that are required to make like an agent successful, I mean, there's one world where as fireworks, you can be a one-stop-shop and like build all of them. There's others where you could partner and you go, look, we're not going to build, you know, this part of the thing. Like, how do you think about that?

• 38:16Speaker 01

  Yeah, so we always are very compatible with imperative agentic tools, right? I think most notably is LanQian.

• 38:28Jacob Effron?

  Yeah.

• 38:28Speaker 01

  And we have been a very strong partner with them because we are not gonna like, LanQian is doing great work in what they are building. Huge community following and adoption, really awesome. And we just want to simplify the next level above single model as a service by like composing wherever it makes sense, multiple models to solve the problem much better. So and we are fully kind of, we can, we will, we will not change our position, we'll plug into LanQian. To, for example, as one example, there's also, of course, many other examples in the community.

• 39:10Jacob Effron?

  Yeah, makes a ton of sense. As you think about kind of like the competitive landscape today, I mean, I imagine when people think of fireworks, they also probably lump you in with like Together AI and, you know, I know Databricks talks a lot about these like compound AI systems too, like, how do you think about what like differentiates fireworks today? And then, you know, five years from now, like what will ultimately determine who kind of like captures this really interesting space?

• 39:30Speaker 01

  Yeah, we are probably thinking, so compound AI actually is the term, we didn't coin that term, right? So it's kind of coined by Berkeley. And we know that Databricks is also like looking into that space. So I think it's great, it's a new category that we're defining, right? So definitely multiple players will kind of be in the same space and we are, we're very happy that Databricks also think about that problem. So, so I think that that space is very meaningful because it's a very complex space to like for. Application developer, product developer engineers to get into.

• 40:11Speaker 01

  And there will be a rich set of tools that emerge in that space to make development much more efficient, and today we are not there yet.

• 40:20Jacob Effron?

  Yeah.

• 40:20Speaker 01

  So we are determined to kind of be a key player in that space. Other than that, I, I, we are not, we are not in the space of being a GPU cloud, for example, and provide access to cheap GPUs, that's not our play at all. So we actually, we build on top of GPU cloud to, to offer a complex inference stack.

• 40:46Jacob Effron?

  I guess one thing that's, I think's really interesting about the company is, I guess you started in Fireworks like, what, a few months before ChatGPT? And so I'm curious, like, you know, what, like, as you think about the original vision for the company, and then, you know, where you've come through today with this like Gen AI craziness, like, how much of it was the same, you know, sitting there in September twenty-two? How much of it had you had to like change as the world has just changed so much in the last few years?

• 41:09Speaker 01

  Yeah, um, it's actually, it's, yeah, this is a very interesting question. When I started Fireworks, there was a actually active debate, is AI here or not?

• 41:21Jacob Effron?

  Yeah.

• 41:23Speaker 01

  Because this pre kind of awareness of foundation models, and it's kind of, oh, there's so many like fragmented AI application, and the data is not there, you know, AI is not there, right? So, but we clearly see that AI is coming, right? So usually hyperscalers are like three to five years ahead of the whole entire industry, and Meta is massive AI powered. And through, clearly the industry is kind of through PyTorch adoption, it's clear, they see that, you know, the wave is coming. So that's kind of the timing about Fireworks. And the Gen AI did skew, ChatGPT heavily skewed the adoption curve of AI.

• 42:07Speaker 01

  And that's a special kind of AI. So it's very interesting, Gen AI is a different beast, not because it's Gen AI, it's magical, because it's fundamentally changed the

• 42:18Speaker 01

  The accessibility. So pre GNI, when it's like traditional machine learning or early deep learning models, any company investing AI particularly have to first hire a machine learning team. Because they have to train model from scratch. There's no other alternative. There's no other model that's pre-trained you can build on top of. And that means this team has to curate, spend a lot of time curate data. And then the training process, because training from scratch is kind of take a long time. So they have to have the resource to hire those very scarce talent and through money into training.

• 42:59Jacob Effron?

  And was the original fireworks product like built for those people? Or like, cause you know, that was originally what it felt like was going to be needed to get adoption.

• 43:05Speaker 01

  Yes, yes. That's what I, when we think about, when we started, we think, we think about that cohort. And then with GNI, it makes it much, it's kind of change the landscape, right? You don't have to hire a big team to carry data to, because it has a, the beauty of GNI is it creates a foundation model that absorb a majority of the knowledge. And then you just build on top of it as is or you tune, right? You have thousands of samples you tune. So then you either don't need to have any machine learning team. You just need to have that application product team to build on top of that directly. Or you have a small machine learning team, right? So this is a massive, massive unblocking of accessibility of, of that technology.

• 43:49Speaker 01

  And that, that's why we see kind of the adoption curve take off like crazy. Because of that accessibility is fundamentally different before after. So because of that, we, these are focused on it because the pull is much stronger in that direction. And the other side effect is all these models, all GNI models are PyTorch models.

• 44:07Jacob Effron?

  Yeah.

• 44:08Speaker 01

  And we wrote PyTorch code for years. And we operate PyTorch, large, complex PyTorch model in production with high volume traffic for years. And that's what we're really without doing.

• 44:19Jacob Effron?

  Yeah, very well situated for, for that. You mentioned when you, you know, you, like when you're at a place like Meta, you get kind of a window into the future. Like before, it's obvious. Like, was there something you saw when you were at Meta where you're like, God, this is just going to be clearly so big? You know, shortly thereafter?

• 44:33Speaker 01

  I think it's more the kind of AI is going to be very big wave. That's clear. So I worked at Meta for seven years, right? It's interesting time. When they joined Meta, they were going through the mobile first transition and the kind of tail end. And then mobile first, and then the industry is also going to mobile first, right? So from desktop to mobile. And mobile first enable consumers to be able to access application everywhere. And that drove up adoption and engagement. And because of that, it produced a lot of data. And a lot of data is powering AI, right? So that's, then the whole entire industry is also following the mobile first, drive a lot of data, and they're power AI. So that trend is very clear.

• 45:18Jacob Effron?

  What AI research do you pay attention to? Maybe even outside of like the core day-to-day of fireworks?

• 45:23Speaker 01

  I usually pay attention to two kind of research. One is like model system co-design kind of research. Because there is a interesting organizational division, as in there's researchers, they focus on quality only.

• 45:41Jacob Effron?

  Right.

• 45:42Speaker 01

  There's a system builder like us, we focus on like deliver the best quality with low latency and low cost, right? So like we solve a system problems. But oftentimes, the best ROI is think about that together.

• 45:56Speaker 01

  And that's how we operate at Meta is kind of the research team and kind of info teams sit very closely to discuss trade-offs, right? And the co-design. So there are a lot of research in the co-design space. I think it's gonna be very appealing to make, to kind of find the best design point across quality, latency, and cost. As a three-dimensional optimization. The other kind of research is kind of fundamentally different disruptive kind of research. As in transformer as a technology is overdue for disruption.

• 46:32Speaker 01

  Right? So where are the next generation of transformers, right? That's gonna change how we train model, change how we do inference, and so on. So that's very interesting. And also, in the agentic world. How different agents is gonna like communicate with each other? As I mentioned, thinking the latest space, that kind of very new kind of thinking is very, very interesting.

• 46:58Jacob Effron?

  I mean, one challenge that I imagine in building an infrastructure company is that obviously the pace of change is so fast, both like the improvement in models, plus just like how enterprises are thinking about and actually using these things. And I can imagine like one school of thought would be it's changing so fast, like, you know, we'll just kind of quickly update what we're, you know, what our kind of core tools are because of how fast it's changing. And at some point, it will settle down and then we can kind of build the standard set of tooling. How do you think about that tension of like, you know, building for however folks are doing things today? With an, you know, without knowing exactly, maybe in two, three years, like models have all these other different capabilities and we're designing systems in all these other different ways.

• 47:34Speaker 01

  Yeah, that's a really good question. That's all we think about all the time, right? So we don't want to keep chasing what's coming because like chasing is exhausting. We'll always want to stay on top of the curve.

• 47:49Speaker 01

  So I think the model capability is definitely evolving, has evolved from two years ago.

• 47:58Speaker 01

  Especially like if we just look at OpenAI's model capability from G P three to now today. It's there and I differences.

• 48:08Speaker 01

  And but fundamentally, like, like a few trend that doesn't change, right? So one is, again, anchor back to our vision is we believe the direction is specialization and customization. That doesn't change how the model core capability evolves. Because we just don't believe that one size fits all different workloads with proprietary data and so on. And we believe there's a much better solution if you can customize, if you can steer, if you have control the destiny here. So that's why we built our stack to to heavily kind of enable that in an easy way.

• 48:52Speaker 01

  Specifically, we offer on top of our inference engine, which is one size fits all. We have an optimizer, file optimizer, that's one size fits one. We take inference workload as input and also your customization objective as input and spit out inference deployment configuration along with potentially a model, adjusted model for you to have the control whether you want to deploy that or not. So this is just a process of we kind of close the loop and make it super easy to customize. So I don't think that's going to change in the future.

• 49:30Jacob Effron?

  Totally. Just what you're customizing may change, but like still the way you will go about it. I think that makes a tremendous amount of sense. Well, we always like to end our conversations with a quick fire round where we get your take on a standard set of questions. And so maybe to start, would love just one thing you think is overhyped in the AI world right now and one thing you think is underhyped.

• 49:48Speaker 01

  I think overhyped is the perception GNI is magical. It's a recipe of our problems. We should just ask it. Any questions, it's kind of, we'll have the right answer. I think, I think it's going through the correction time right now. And again, that's where kind of we believe like there's no magical one model that solve all the problems the best way or in the correct way.

• 50:11Jacob Effron?

  What's one thing you've like changed your mind on in the AI world in the last year?

• 50:14Speaker 01

  I think my hypothesis in building this company and how this GNI technology can adapt, adopted. And our go-to-market strategy in my mind is always like in a sequential way. Startups, startups are like GNI native, so they will be on the frontier of adoption. And then we will work with digital native. They have very strong engineering resource and always kind of be more tech-forward. And then traditional enterprise because they have more like other kind of problem to solve. But now we're working with all of them at the same time.

• 50:50Speaker 01

  And to me, it's a little bit crazy.

• 50:54Speaker 01

  Of course, the use case are not all the same, but the adoption curve is happening all at the same time. It's very different from what I was imagining.

• 51:05Jacob Effron?

  Totally. I mean, on the application side, it feels like the normal rules of how long it takes to land one big enterprise and then land a few after. It's all out the window, right? Some of these application companies are scaling so fast in the large enterprise. I think there's just a tremendous appetite for it.

• 51:17Speaker 01

  That's right. That's right. So that's why I feel like we're in this revolutionary way. So a lot of things will be done differently. Not just the application we build will be different. Technology adoption curve will be different. But also, even how to think about go-to-market will be different. The sales cycle is much shorter, and people are open to think the procurement process differently. So it's very interesting, a ripple effect from this massive transformation.

• 51:48Jacob Effron?

  Do you feel like, is there any difference in what you have to build for enterprise versus the startup folks? Or is actually the needs pretty similar?

• 51:55Speaker 01

  I think, my observation is startups, they are typically, like, they want to get access to, and when we talk about abstraction, right? They want to get access to the low-level abstraction.

• 52:12Jacob Effron?

  Yeah.

• 52:13Speaker 01

  Because they want to tinker a lot, right? So assemble a lot of things. And for traditional enterprise, I'm not saying this in an exhaustive way, but typically, they want to hire a higher-level abstraction. If they don't need to solve for low-level details they don't need to pay attention to, rather not to pay attention to. So even at the matter, we have those different abstraction layer for different kind of team, because they want to pick and choose, right? So usually, two-level abstraction will suffice. Kind of like they were happy with, at least have two choices. And we see a little bit of those.

• 52:47Jacob Effron?

  Totally. Must be interesting to build both at the same time, as you're rapidly scaling.

• 52:51Speaker 01

  Right, right. So we need a low-level abstraction for ourselves anyway. So to us, it's not additional overhead, but we see the adoption will happen at different layers.

• 53:02Jacob Effron?

  Yeah. I know it's probably, I won't ask you to pick a favorite, but what is one favorite application that's going to work today?

• 53:10Speaker 01

  This is a tricky question.

• 53:14Speaker 01

  I'll say Cursor.

• 53:17Speaker 01

  Which.

• 53:19Speaker 01

  But we see a lot of similar company in in kind of productivity space, right? Not just Cursor, but also Source Graph, Z, Cognition, Factory. They're all kind of very forward-thinking company. And I think in that space, it's still early. But you know, that's the space we're very excited about, the whole chat space.

• 53:40Jacob Effron?

  Do you think someone will spend a hundred billion dollars on a on training a model in the next few years?

• 53:45Speaker 01

  Ooh, maybe. I think it only makes sense there if they are training using a fundamentally different model architecture.

• 53:54Jacob Effron?

  Yeah.

• 53:55Speaker 01

  It's so disruptive, it's worth that investment.

• 53:58Jacob Effron?

  Yeah. You know, you kind of already mentioned the coding space and then some of those startups. But any other like AI startup or space that you're really excited about outside of fireworks right now?

• 54:07Speaker 01

  Again, I think the the agentic.

• 54:13Speaker 01

  The agentic word hasn't been fully figured out yet.

• 54:16Jacob Effron?

  Have you seen any applications like do that well?

• 54:18Speaker 01

  We have seen like, we have seen early applications. For example, the digital SDRs, the, you know, the digital marketing. And yeah, so we have seen kind of good startups getting great adoption there. But I do, I do think in the future, there will be a lot more complexity in that space. And it was too early.

• 54:50Jacob Effron?

  Well, this has been a fascinating conversation. I'm sure folks will want to pull on all sorts of threads. So I want to leave the last word to you. Where can folks go to learn more about you, about what you're building at Fireworks? The floor is yours.

• 55:00Speaker 01

  Yeah, so yeah, definitely. So we offer for the benefit of our developer community, we offer a self-service platform. Very simple, just go to fireworks.ai and you can access to our playground and hundreds of model capabilities we have. Yeah, and feel free to reach out to me also. Connect me on LinkedIn and love to hear your use case, your challenges and pain points.

• 55:25Jacob Effron?

  Amazing. Well, thanks so much, Theresa. This was a fascinating conversation.