Open-Weight AI Models

Open-weight models are AI systems whose trained parameters are publicly released, which allows developers to run, fine-tune, and deploy them independently rather than accessing them only through a hosted API. While closed-weight models from companies like OpenAI or Anthropic are delivered as managed services, open-weight models give organizations direct control over how the models are deployed and used. Importantly, the performance of these models is steadily improving and they’ve become credible alternatives for production workloads, with advantages in customization and data privacy. ​ Fireworks https://fireworks.ai/ AI is building a platform focused on serving and customizing open-weight models at scale. The platform includes optimized inference infrastructure, multi-hardware support across NVIDIA and AMD, and reinforcement fine-tuning capabilities. ​ Benny Chen https://x.com/the_bunny_chen is a Co-Founder of Fireworks AI. In this episode, he joins Gregor Vand to discuss his path from Meta’s ML infrastructure teams to co-founding Fireworks AI, why open-weight models are becoming increasingly competitive, how custom kernels and speculative decoding improve performance, reinforcement fine-tuning, and much more. https://i0.wp.com/softwareengineeringdaily.com/wp-content/uploads/2023/11/Gregor-Vand.jpegGregor Vand is a security-focused technologist, having previously been a CTO across cybersecurity, cyber insurance and general software engineering companies. He is based in Singapore and can be found via his profile at vand.hk or on LinkedIn. Please click here to see the transcript of this episode. http://softwareengineeringdaily.com/wp-content/uploads/2026/03/SED1919-FireworksAI.txt Sponsorship inquiries: [email protected] https://cms.megaphone.fm/organizations/619b680e-d813-11ea-9750-e7ecac406436/podcasts/e60108fe-e328-11ea-b2b6-1348249f900a/episodes/3913370a-bc44-11ed-ace6-8b1f96d82bba/[email protected] The post Open-Weight AI Models https://softwareengineeringdaily.com/podcasts/open-weight-ai-models/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

• 0:00Speaker 00

  Open weight models are AI systems whose trained parameters are publicly released, which allows developers to run, fine tune and deploy them independently, rather than accessing them only through a hosted API. While closed weight models from companies like OpenAI or Anthropic are delivered as managed services, open weight models give organizations direct control over how the models are deployed and used. Importantly, the performance of these models is steadily improving, and they've become credible alternatives for production workloads, with advantages in customization and data privacy. Fireworks AI is building a platform focused on serving and customizing open weight models at scale.

• 0:43Speaker 00

  The platform includes optimized inference infrastructure, multi hardware support across Nvidia and AMD, and reinforcement fine tuning capabilities. Benny Chen is a co-founder of Fireworks AI. In this episode, he joins Gregor Van to discuss his path from Meta's ML infrastructure teams to co-founding Fireworks AI, why open weight models are becoming increasingly competitive, how custom kernels and speculative decoding improve performance, reinforcement fine tuning, and much more. Gregor Van is a security-focused technologist, having previously been a CTO across cybersecurity, cyber insurance, and general software engineering companies.

• 1:25Speaker 00

  He is based in Singapore and can be found via his profile at van.hk or on LinkedIn.

• 1:44Speaker 01

  Hello and welcome to Software Engineering Daily. My guest today is Benny Chen. Welcome Benny.

• 1:51Speaker 02

  Thanks for having me.

• 1:52Speaker 01

  Yeah, great to have you here. So we're going to be talking all about Fireworks AI, which is a company that I believe you co-founded, I think, three, three and a half years ago, is that right?

• 2:04Speaker 02

  Yeah. It's been a while.

• 2:06Speaker 01

  Yeah. So before we dive into FireX AI, and I think especially today, this is quite pertinent to sort of where maybe FireX AI came from. You spent a lot of time at Meta and, you know, in their ML team, which means you were doing things with ML probably way before many of us really had it on our radar. But what was kind of your path? Like, what's been your path to like through software engineering? And especially the meta phase as well?

• 2:35Speaker 02

  Yeah, that's definitely, uh, it's been a while. So let me what the journey was like in the beginning. So in the very beginning, I joined as a software engineer in 2014 on the integrity team, where most of, I would say, the sort of non recommendation system experiments started. Early on, it was sort of like decision trees for different fraud behaviors. And then also on the team, we started doing like image classifiers for different rules for advertising. And then come 2016, switched over to the ads infrastructure.

• 3:18Speaker 02

  So worked on supporting the recommendation system models. And then in 2017, I think the Facebook leadership back then started thinking about having a ASIC in-house that's more like Google's TPU. So we started collaborating with Intel back then on a ASIC for recommendation systems. So is there anything about like doing something that's not cool before everyone remembers? Definitely doing ASICs in 2017 isn't necessarily cool. But yeah, it was an interesting project for sure. And I always tell people that the A six I worked on supporting back then was seventeen watts.

• 4:02Speaker 01

  Oh, wow.

• 4:03Speaker 02

  If you look at the new Nvidia GPUs, that's like a thousand watts. Some of the peripherals on that chip is more than seventeen watts. But yeah, it was a very small chip. It's like, how many years ago? Almost nine years ago, nine years ago at this point. So it's like time really flies and things really change up quickly in the area, I guess. But yeah, I worked on supporting that chip for about two years. And then fast forward, Nvidia started shipping A one hundreds. That's when everyone realizes, hey, I guess A six are not gonna be as good as A one hundreds. And then worked on supporting the PyTorch enablement for all the ads models in twenty nineteen, twenty twenty, like around the pandemic time.

• 4:49Speaker 02

  But yeah, like Nvidia also wasn't a huge company back then, so I probably should have just loaded up on Nvidia stock rather than doing anything else. And after that, it was sort of like all the Nvidia GPUs for another two years until we decided that it was time to start something new and then started working on fireworks.

• 5:11Speaker 01

  Yeah, nice, because I, yeah, I was gonna sort of ask, I guess, yeah. Why leave Meta, I guess, but I mean, everyone's always got their own reasons for, you know, doing their tenure at like one of the big, big tech and then and then starting their own thing, but like, was there a sort of defining moment for you? Was it just like the time is now or, you know?

• 5:29Speaker 02

  Yeah, to be frank, like I probably could have premeditated more. I probably could have think it through more analytically. At the same time, I do think like AI infrastructure will take off. We in fact started before ChatGPT came out, so we couldn't time it better, like we started maybe like five, six months before ChatGPT was shipped. But yeah, it's not so much that there's any like particular trigger, but definitely being a Meta for about eight years. I mean, like all good show must end, right? So.

• 6:03Speaker 01

  Yeah, absolutely.

• 6:04Speaker 02

  And for me, I think I also maybe is a little bit more risk prone than some of my friends. I do think taking on more risk is a good thing, so I probably didn't think it through as hard as I should have in retrospect.

• 6:19Speaker 01

  Yeah, but I mean, that's where some of the best things come from. So yeah, and that's probably why we're sitting here today, speaking about Fireworks AI, which I think, yeah, let's get on to what you did next. So let's just sort of start a super high level. What is Fireworks AI?

• 6:35Speaker 02

  Fireworks is a platform that serves and train open source models. And we mostly stuck to that mission since ChatGPT shipped till today. And to be frank, our play hasn't been super sophisticated for our customers. At the same time, the work itself is very complicated. And a lot of our customer appreciate the support we're able to give them. Specifically, a lot of our customers are very, they're either AI native or they are very AI leaning enterprises. Who are looking to either offer products that's based on language models or looking for automating large part of their organization through large language models.

• 7:19Speaker 02

  We're here to help them customize their open source models and then scale them up. A lot of startups we work with also start with the frontier models. And as they scale, they needed to make sure that their unit economics are good. And we're here to help customize open source models so they can reduce their total cost of ownership for their customers and start making money. In general, I really appreciate a lot of our customer support for having trust in such a young company versus all the big clouds and other offerings out there. And yeah, we're here to help.

• 7:54Speaker 01

  Nice. And I think usually we kind of later in episodes go into sort of customer stories, I guess. But I think one that's maybe useful to pull out now to help set the scene for for the listeners is Cursor. So I believe that you guys do work with Cursor. I'm taking a guess that this is like, for example, when, well, I think code completion is like one area that that you guys supply to Cursor. It's like, could you just talk a little bit about that? Because I think that's going to help kind of set the scene for the rest of the episode, just in terms of what does Fireworks actually? Do and for whom and yeah.

• 8:29Speaker 02

  Yeah, honestly, the Cursor people are amazing and we're mostly here to help support them. I think early on, a lot of work around supporting Cursor was their custom models like tapped and editing models. And we helped design solutions that would work very fast with their model in a cost-effective way. I think one thing that we publicized early on was the fast supply model we serve for Cursor, which require a lot of like special support for speculative decoding in order to support the model properly with fast decoding. So in those settings, inside the editor, you want to edit a very large file in one go.

• 9:14Speaker 02

  A lot of times, there's a lot of nuances on how to serve those kind of models. And we work with them to set up a sort of like a dedicated algorithm so that we can serve those models very, very cost-effectively. Yeah, and I think like recently Cursor also published a blog around how to do like online learning with their tab models. Yeah, they are like very sophisticated team.

• 9:44Speaker 01

  Yeah, no, I mean, I was, I've been a fan of Cursor for a long time, I think, since it almost came about. Unfortunately, I don't do as much hands-on coding today as I did even eighteen months ago. But that's just a function of where I've gone in, in life. But, uh, but yeah, Cursor, still love Cursor as a, a product, still use it if I do code. So let's kind of walk through some of, I guess, like the features, if you like, of like how FireX's inference infra actually operates. I mean, just to kind of give some scale idea here, probably should have mentioned it this, at the start, but you guys process something like thirteen trillion tokens a day, is that, is that right?

• 10:26Speaker 02

  Yeah, yeah. And I think I'm probably not here to share the latest numbers, but at the same time, like, I think the thirteen trillion number was larger than the Gemini and OpenAI number they shared for the APIs. I think Gemini was like ten, eleven trillion, give or take. And we've been growing quite a bit since we shared a number last time as well. But yeah, open source models are very strong. Definitely, it was a leap of faith from us to focus on open source models. At the same time, I'm like very pleasantly surprised on like how strong the open source model has been. Getting to a point where it is like price competitive against closed source models. This year, just in the beginning of this year is amazing.

• 11:09Speaker 02

  I do think a lot of open source models last year. Shows promising benchmark scores, but are not competitive end to end. I think what really helped set the scene this year is one. Open Claw came out, which one people realized, wow, these models are amazing. Two, wow, these models are expensive. Like I had a friend who like set up Open Claw and he's like, yeah, this is so good. At the same time, I don't know why one message cost me like a million tokens. Just like send one message on Telegram and then a million token gone. And he doesn't even know where it went, just sees the money going away. And two, yeah, like a lot of the open source models that are offered.

• 11:52Speaker 02

  Are very competitive in like a real open core setting. Where you don't necessarily want the most expensive model just to book you a restaurant. You, you probably just want something normal, can book you a restaurant, but with one twentieth of the price. So yeah, I think there's a lot of competition heating up. And like Fireworks here is here to support the open source model thing as much as we can.

• 12:17Speaker 01

  And I mean, would it be fair to say that because you came from Meta ultimately and they had been working? Well, the thing is, I'm trying to look at like line up the timelines here, which is Lama. And was that really a thing internally when you were still at Meta? Or did that only really come out? Or like what I've been getting at is, were you already working on something Lama related? Before you left and that got your kind of brain jogging on these open weight models or not really?

• 12:48Speaker 02

  Maybe the closest answer is kind of. So when I was working at Meta, the Llama or like the large language model program wasn't really that well funded. So it was different from where OpenAI was doing the Yolo run for like G B four and with like, I forgot, like thousands of A one hundreds. There was no nowhere near as much of a commitment in Meta back then as of now. But I worked a lot on recommendation system models that had transformers. And there was a clear sign where you just throw more compute at the problem and the model just gets better. The ROI for those increased compute may or may not be worth it. At the same time, there was a clear trend. So while people were talking about scaling laws for

• 13:32Speaker 02

  language models, I guess we were sort of like seeing the scaling law for recommendation system models in real time as well. So the migration between the tiny seventeen watt A six into the two hundred, three hundred watts A one hundred, there was like a clear payoff for that. And then keep on migrating to H one hundred. There was also a clear payoff for that as well. So yeah, I guess I was working in related fields and seeing the scaling law working, but in a different way. At the same time, I think the commitment or like the belief in open source models right now seems obvious, but I think it was definitely contrarian three years ago. People forget the models we had back then were OPT Lama one and Falcon.

• 14:18Speaker 02

  And if those models could hold a three turn conversation, that was already amazing. And people also forget, three years ago, there was no function calling. It's like we also worked on open sourcing function calling models, which wasn't really a straightforward thing back then. So yeah, I would say it wasn't straightforward at all to say, hey, like, Lama is here and it's here to stay. In fact, it's not here to stay anymore. But sort of like the belief we had because we worked on open source software for so long and sort of like the belief that the models are more like software rather than some kind of hardware kind of setup.

• 15:00Speaker 02

  I think that sort of leap of faith was important for us and I think carried a lot of weight even in today.

• 15:06Speaker 01

  Yeah, and I realize we're sticking a lot on on models generally, but I think this is quite interesting just for a second. I mean, I think just to be clear, a lot of these open, well, first of all, just to clarify, you mentioned open source models. I mean, is there a distinction here, open weight? Like, are you using interchangeably with open weight or yeah?

• 15:27Speaker 02

  I am using it interchangeably and for all the people out there who really understand the distinction, I think they will hate me for using these terms interchangeably. I'm more of an attitude where like, I don't really make those distinction when the goodies are still flowing, to be honest. As long as the final artifact is produced, I think it's good. But I have huge respect to both like the OMO team and the NEMOTRON team. The OMO team is like LNAI, they have funding from NFS as well as I think other sources and they really publish all the results as well as the all the intermediate artifacts. I think that's like a really good foundational work for everyone working in the field.

• 16:10Speaker 02

  And then I think Nvidia also is really committing to keep pushing on open source models. And they also publish all their training code or their recipes to benefit the community.

• 16:21Speaker 01

  Yeah, absolutely. So yeah, I guess to keep some of our listener base happy when we talk about open weight models, which is a lot of, I guess, what FireX is working with. If I look at most of those names, or especially on the FireX platform site, but I think just known generally, they're mostly coming, would it be fair to say from Chinese companies and Chinese tech is that is like good. Yeah, yeah. So I mean, I'm someone who's based in Asia. I fully appreciate just the amazing talent and innovation that comes out of this part of the world. And I think it is helpful to maybe just talk through, have you hit any bumps with the fact that you are a US company?

• 17:05Speaker 01

  And then pushing models like this from that part of the world? Or I'm just curious like how that looks because it's something that obviously gets talked a lot about, maybe in the news even. You know, especially when DeepSeek landed, if you want to call it that. And, and I think all that's kind of settled down to some respect. But what was your thinking around all of that when putting these models into Fireworks?

• 17:25Speaker 02

  Yeah, I think that's a good question. So I would say 90% of our customers don't really care about the origin of the model, especially if they are doing fine tuning. Because they will be sort of like running these models in very specific environments. For example, for coding environment, often you don't really care about the political preference for these models. Because it's just, you're just writing React, Rust, you know, like it doesn't really matter as much. And then definitely for certain customers who are more like consumer facing, they are much more aware of the origin of the model. And we are more constrained to serve things like Nimtron, GPT OSS, these kind of American models.

• 18:06Speaker 02

  I do think Fireworks here is here to help support our customers and we don't really judge why are they making these distinctions? We're just here to help support them as much as we can. Mistral people also do a great job and they also publish very strong models and sometimes also serve Mistral models as well. But yeah, like I will say, like one thing I keep chatting with my colleagues on for like, it's like a year on. And GPT OSS is surprisingly competitive today amongst all the American models. So huge respect for OpenAI as well, for open sourcing the model back then. But yeah, there are new models coming out every day from Nvidia, from OMO. Those are also getting more competitive.

• 18:47Speaker 02

  And I do think at the end of the day, given like the same data set and the same model size, the results will converge. And then the data set should converge. As people start sharing more and more of those intermediate artifacts and recipes. So I I bet the American models will catch up this year very quickly. I honestly feel like there's so little secret out there today. I feel like there's so many conversations I mean where like people think they have alpha and then they sort of exchange notes and be like, oh, I didn't realize some other people are doing the same thing. But yeah, I I do think like a lot of things will converge and the American models will be, it will be very competitive this year.

• 19:26Speaker 01

  Yeah, and I guess like Firex has a platform which we're gonna get into in more detail in a second, but I guess part of your offering is to help your customers understand which model would suit their case best, because I think, at least to me, when I was doing a bit more hands-on, call it twelve months ago. I think trying to pick through like which model, why even the, okay, this is kind of the same model, but one's like, you know, four B and one's sixteen B or whatever, you know, like this is, I'm sure where you're actually able to give more direction and input to your customers, given the size of them and like, this isn't just a you know, you can't kind of make the quote wrong decision on these things.

• 20:09Speaker 02

  Yeah, that's a good question. How we talk with our customer on when to use which model depends on the use case quite a bit. I would say for probably a third of the use case, our customer is more sophisticated than us. They have ran the evaluations in-house. They explicitly come to us and be like, this is the model you need to serve. Don't argue with me. I will pay you for this. Just serve this model. And then one third is more somewhere in the middle where they kind of know they're gonna pick between two or three models. They mostly need some judgment call on the cost of serving on like whether it scales on our, on our platform. So they run the evaluation locally and then they just need to know the cost or the scalability of the setup.

• 20:53Speaker 02

  And then the last one third is where the customer are really looking for us for advice on which model to use and which use case. And in those cases, we share our evaluation results internally and try to paint a full picture. Like, hey, this model is much better coding. This model is much more malleable and a good starting point for reinforcement learning. And then show them a data point and give them the judgment call. To answer that question more concretely, for example, a lot of choices are more nuanced depending on what you want to do. For example, like Kimi is a very big model. So it's very easy to do reinforcement learning or fine tuning. Easy as in like infrastructure, it's difficult, but it's easy to push results out of big model.

• 21:35Speaker 02

  Rather than trying to R L like a small model and try to get the small model to be smarter. At the same time, the evaluation initially may be worse. So then it is not going to be obvious to our customers. Hey, like, hey, Kimi is a much better starting point than a lot of other models. But for example, when some of our customers are just looking to serve open source models and not customize. Then like, for example, if it's like a coding use case, then G R M and Minimax both are great. And then just depends on the cost effectiveness depending on their use case. So we help try to paint a full picture to our customers and still rely on them to decide what's the best for them.

• 22:15Speaker 01

  Yeah, no, it was very interesting. I maybe hadn't appreciated, yeah, what you said. So towards the beginning, a good chunk of people come to say, this is the model, already figured that one out. Just run it for me. So let's get on to the just run it for me bit. So you know, you have quite advanced infra as it has to be running, running all of this. I believe you have things called or like a piece of the puzzle called fire attention. For example, like, maybe could you walk us through what is that? Let's just kind of start there and we can go through some of these interesting bits of the fireworks stack.

• 22:50Speaker 02

  Yeah, we set up our own kernel in-house for a few reasons. One, a lot of kernels in the open source world are not numerically pristine. In the sense that it will work, it may not work in the way that you expected. Or it may not be working because of the reason you think it's working. So this always new kernels coming out every which way. We try to make sure our customer, when they use fireworks to serve the model, don't think about all the complexity for like all these weird kernels. They just understand that we provide the best trade-off between quality and speed.

• 23:31Speaker 02

  We'll push as much as we can on speed, but not compromise on quality. And a lot of our customer appreciate that. That's sort of like the first reason why we set the kernels in-house. The second reason is we have a very, I would say, expensive but important commitment to do multi-hardware. So we work with AMD team quite a bit to make sure we can properly support their hardware. And not everyone is willing to sink the time and effort into supporting different hardware. So yeah, we set up those AMD kernels in-house as well, just to make sure that we can serve on different hardwares and try to find the best price for our customers. And also, maybe the last reason I can go through on these internal fire attention kernels is

• 24:16Speaker 02

  because we do a lot of reinforcement learning workload. Making sure that the training inference consistency is minimized is very, very important. You see a lot of fancy algorithm getting published every day. There's like so many variants of GRPO at this point. I bet if you put two random letters in front of it, there's probably a paper for it. I would not know where it came from, but there's probably a paper on it. For all three letters, for certain people. But from our observation, those algorithms are important, but not as important as controlling the numerics and making sure the numerics are aligned across training inference. So you see a lot of headlines on, hey, like, there's like a reinforcement learning stack

• 24:58Speaker 02

  where we use like trainer from one stack and an inference from another stack. And then like it should just work for a big model. From our experience, it's far from it. We really take it seriously and make sure that we align the numerics across training inference for those kernels. That's why we have to have those in-house and that's why we have to spend all the sweat and tears. So then our customer don't have to think twice and be like, hey, like, did these people do their job? And is this RL run not going up because their numeric sucks? We try to really make sure our customer trust us to make those happen.

• 25:32Speaker 01

  Yeah. And then something called speculative decoding, which again, maybe context-wise, I believe Cursor built their fast apply feature on this API. Could you maybe just like speak to that?

• 25:46Speaker 02

  Yeah, speculative decoding, I would say at this point is a pretty well-known concept for many serving stack. It's a setup where you have a small model that try to guess which token the big model would like. And just ask the big model, do you like these tokens? If yes, let's spit them out all at the same time. There's a lot of interesting research in this area. I think recently there's like using different model architectures to do the speculation. We also spend a lot of time doing those research in-house. Yeah, speculative decoding is definitely also workload dependent. And making sure that we can train different form of speculative decoding models. While the data is coming in to make sure that we keep up with the change in distribution in our customers' data.

• 26:30Speaker 02

  Those are also very important as well. I think one other thing that people often don't appreciate enough is that training an ego model is often like training a large language model. Or like a training a small language model. In the sense that data quality is important, training infrastructure is important, dev efficiency on the stack is important. There's a lot of nuances on how we can support our customer with good speculative decoding models. And we spend a lot of time and effort on that, making sure that when they bring a open source model that they find tune. We can quickly train a really good speculator model for them.

• 27:06Speaker 01

  Yeah, because I was gonna sort of just ask this, yeah, the idea of, you know, you've got effectively a sort of draft model and a target model. And I guess you're hinting that the target model is often actually the customer's fine-tuned model. And Firex is needing to pair that speculative draft model, yeah.

• 27:25Speaker 02

  Absolutely. A lot of our customers are very, very sophisticated. They may have most of the things like seventy, eighty percent of way there. And we are here to, let's say, like deploy the model, scale the model. And often part of scaling the model is to train the speculator for them as well.

• 27:40Speaker 01

  Yeah, nice. And then you have something called 3D fire optimizer. So yeah, I'll just let you take that one.

• 27:48Speaker 02

  Yeah, I think it's an interesting name when we came up with the concept.

• 27:53Speaker 01

  Sounds like something I find in gaming. So yeah, that's quite interesting.

• 27:57Speaker 02

  I think conceptually it's very straightforward, though. It's a database of all the previous performance optimization results we have, as well as predicted results of what the customer workload will be. Because there's so many dimensions to do trade offs. For example, like workload patterns, hardware types, cache hit rate, many, many different variants to deploy the same workload. And it's important for us to be able to scale the engagement for our customers in an automated way. So we have a database for these performance optimization techniques that our automation can make use of and make sure that we can get back to our customer and answer very quickly.

• 28:41Speaker 02

  Yeah, oftentimes, how to set up the workload correctly is like 50% of the work. And 3D Optimizer is an in-house stack that helps us get there.

• 28:50Speaker 01

  Nice. And then, I mean, you have mentioned hardware, I think, especially in relation to the Fire Attention kernel. You are a sort of hardware agnostic, if you like, shop and buy agnostic. I mean, we're talking basically two providers because that's kind of what it boils down to. But yeah, I mean, what is your thought process apart? Like, I mean, let's assume that cost comes into it somewhere. But cost plus what else sort of is what you're looking at when you're deciding Nvidia and AMD, basically, which I believe you run both.

• 29:23Speaker 02

  And just to be clear, both are our investors, so I don't want to.

• 29:27Speaker 01

  Gotcha.

• 29:29Speaker 02

  We do love Nvidia and work a lot with Nvidia, just to be clear. I do think when we're running a business, we need to make sure we maximize our customer value. And in the world where we maximize customer value, it's definitely a little bit weird to be locked into one hardware vendor. And I really think this year is honestly not so much about loyalty to Nvidia and whatnot, because Nvidia is already producing chips as fast as again. It's just that everyone else is. Buying them up. So practically speaking as well for multi hardware vendor, oftentimes it's all about like supply chain reliability, making sure that you can actually buy the cards when you have the money.

• 30:12Speaker 02

  And at any point in time, there may not be Nvidia cards available and you have to buy AMD cards. Available as in like available at reasonable price because of course, like if you are willing to pay, like there's always someone willing to sell you. It just said the premium will be very, very high. And honestly, like while I was working at Meta, like because I worked on A6, I was also in conversation with all the like procurement process. Of course, I'm not the one negotiating the contract, but because I was on as team, I was providing all the inputs on like how many CPUs and GPUs we need. And often in those, like even very early on in those conversations, it was AMD and Intel for CPUs back then.

• 30:53Speaker 02

  And it was important back then, even just to have like a dual supplier strategy that works surprisingly well. So in case where people think like two suppliers may or may not be enough, oftentimes it is enough. It would be great when you have three suppliers. That's really when you get to benefit. But even having like having more than one is very, very helpful. And that sort of is ingrained into how I think about this process. And that's why I multi hardware is so important for us. And I do want to be honest about it, that like it is a lot of work. And there's always a trade off between working on AMD for fire attention versus working on other stuff that we can serve our customer better.

• 31:36Speaker 02

  At the same time, we really believe that the AMD investment will pay off.

• 31:40Speaker 01

  Yeah, I think it's really, really helpful, I'm sure for our listeners to hear just sort of actually what are the things that you have to consider when running something like this? And obviously investors come into it as well. You know, I think that's what people forget about as well is most companies in this space have investors and that will often have a bearing on some direction at some stage as well. So I think that's really helpful. Let's maybe sort of move on to evals. I mean, most of our listener base should be, I think, somewhat. Familiar with the concept of evals and why it's important to anything, especially running your own or wanting to fine tune your own models, but even just picking between models as well. And I think at least as a company, you've kind of made the sort of statement,

• 32:25Speaker 01

  which a lot of people agree with, I think, which is one of the biggest barriers to this whole AI ROI problem or mystery to some people for those just ROI return on investment. So like, is the money being put into your AI within a company? Are you getting something that like effectively is larger than the investment you put in? It's very simple. And Firex is saying the biggest barrier to that isn't cost as such, but it is like define good. Like, what is it that you think good is? So, yeah, I mean, could you just sort of walk us through what does Firex do in this area? Like, do you help your customers do evals? Or yeah, just walk us through kind of what that looks like.

• 33:08Speaker 02

  Yeah, Fireworks investment on the eval is, I would say, like 70% on the infrastructure side and 30% on the consulting side. So the 70% on infrastructure, we have a project, open source project called Eval Protocol, where we help people author evals for reinforcement learning. Oftentimes, when I ask customers whether they have evals and then recommend them to write evals and whatnot, my pitch often is even if this engagement falls through, the worst outcome is that you have better evals,

• 33:41Speaker 02

  which hopefully isn't a bad outcome. Because there's so much innovation coming out of these frontier labs, closed source models even. How do you decide like which one to use? It's like every day there's something new. Okay, every day is an overstatement, but like every week there's something new. And this is to say like even like the most straightforward case, let's say you love Elon and you want to use XAI's model. How do you know when it's appropriate, right? Like, you can't even pay Elon unless you have the evals. So it's very important, I think, for a lot of our customers to realize that these are assets. And Fireworks is here to help you build up those assets.

• 34:22Speaker 02

  And as soon as people have evals, the gap between using that eval to evaluate model to the gap to use those evals to train in reinforcement learning, to train your own model, is very, very small. So it's very helpful for our customer to first be able to pick different models. And then second, once they are comfortable, start training new models on fireworks with reinforcement learning. Through eval protocol is something that we've seen like repeatedly happen over and over again. At the end of the day, if you know how to evaluate your model, you have all the power. You get to decide which supplier to use in what setting and when. And that power is very, very important to a lot of our customers.

• 35:05Speaker 02

  It's just that some of them haven't realized how important that power is. The other part is also we work with some of the large customers to help them set up the eval as well. Because in certain settings that it's not practical to sort of like leave our customer hanging and ask them to author the eval themselves. We have a lot of know how's in-house as well after all these engagements and we help our customers really use our know how's and author the evals themselves as well.

• 35:35Speaker 01

  And you have a eval framework that is actually being open source, is that right?

• 35:41Speaker 02

  Yeah, it's called eval protocol. It is focused on helping people author evaluations for reinforcement learning settings. So you focus on writing the evaluation itself and then fireworks can help take care of the rest. For doing rollouts on inference, passing those rollouts to the trainer and making sure that the reinforcement learning graph goes up and to the right. And also helps with observability for like which traces went wrong and why and what did the model trip on. It is often understated on how much people focus on sort of like the fancy algorithmic side of RL. But in practice, a lot of time, the observability is very, very important.

• 36:25Speaker 02

  Making sure that you have a open source SDK that help people hook into any part of the infrastructure however they want. And they need that assurance because unless they can own the code themselves, like they don't want to touch it. Making sure that all the observability stuff can be set up so they can see the environment's behavior at any point in time. Because oftentimes the RL problems are coming from the environment itself, like something broke in the environment. And you want to fix the environment to continue on training. So those things are all very, very important and eval protocol helps you with all of that.

• 37:01Speaker 01

  Yeah, and I encourage anyone interested in any of these topics. You guys have a really good blog and something was published on there. Traces are all you need to, brackets, to rank L L Ms. So this is in reference to production trace data and being sort of, you know, a better signal, say, than benchmark suites. And like, there's probably a catch somewhere. So I think just maybe if you could briefly unpack, what is that about?

• 37:26Speaker 02

  Yeah, I think when we talk with our customers, a lot of them feel that it's intimidating to author evaluations. A lot of them didn't start out as machine learning engineers. But that's the beauty of this wave of innovation. A lot of people coming in from product background, who has the ability to clearly articulate what is good and what is bad for their customers. And oftentimes, I feel like that's underappreciated in a lot of settings. Because honestly, if you can clearly articulate what is good, what is bad, you are ninety percent of the way there. The only delta is then now you need to take traces from your production workload and run a language model through those traces with your articulation.

• 38:11Speaker 02

  And oftentimes, I think in a lot of settings, like our customer, tend to have more elaborate setup for the evaluations and whatnot. But what we find is honestly, if you can't articulate it, you are ninety percent of the way there. There are definitely special cases where, for example, we have an example for like S V G agent, where you still need to render the output through a Chrome server to be able to get the screenshot to even use your articulation of like, what is a good S V G. At the same time, honestly, like Opus four point six, you say the word, it comes out the other end. I don't even know what happened in between, but like, it works. Then the only important bit is can you pull your traces out of your production system?

• 38:55Speaker 02

  And then can you articulate what is good or bad?

• 38:57Speaker 01

  Yep, so we're going to move on. Probably sort of the fairly, well, final fairly meaty topic is just on reinforcement fine tuning, something that I believe, again, FireWax helps people figure out. So, you know, supervised fine tuning, it's been around a while. So what does reinforcement fine tuning? Like, why does that change things like fundamentally? Which is sort of my understanding of it's a huge shift. It's not just a sort of small incremental change. It's kind of a step change in this area.

• 39:27Speaker 02

  Absolutely, absolutely, yeah. Maybe I can start with how it changes the industry and like then specifically how it changes fireworks. I would say reinforcement learning is a new lever that the industry found. While the pre-training sort of like a free riding slowed down. You definitely see a lot of articulation from like Dario on saying like, hey, like Anthropic does not see the improvement slowing down and whatnot. Pre-training is still giving a lot of gains. At the same time, my current understanding is that you need sort of like exponential amount of compute to get the straight line going. And at some point, the money aspect will kick in, like reality will have to kick in.

• 40:12Speaker 02

  Even if part of the reality doesn't kick in, the electricity part of reality will kick in. So there are limits to how far you can push these ideas. And reinforcement learning was a new paradigm that the industry found to keep the scaling going. Because even for relatively small models, if you can push RL, you can get really good results out of the model. And I think the other thing that people don't tend to talk about for reinforcement learning is that it really consolidates your evaluation as an asset. Meaning that the same evaluation on the same environment can be used across different generations of model without significant changes.

• 40:55Speaker 02

  This is different from SFT data sets where depending on what's needed for this model, you may need like more supervised fine-tuning data to sort of like push the model to a certain direction. And that is definitely important work that a lot of frontier labs, people are pushing in every day to make sure they curate better and better supervised fine-tuning data set. At the same time, when the models are really good, you have to throw certain things out, just to even like not confuse the model. Whereas for like asking the model to do an Excel spreadsheet is always gonna be valuable, no matter where you start. Because you always want to make sure that it knows how to read a spreadsheet and knows how to Manipulator spreadsheet, right?

• 41:39Speaker 02

  So those assets are sort of more enduring for these settings. And that's why I think so many frontier labs are investing so much money into these environments. Specifically for fireworks, it is very important because it now finally unlocks the customization loop from a software engineer directly to a tuned model. Previously to do supervised fine tuning, the conversation often goes like, oh, do you have a team of MLEs in-house who knows how to work with data labelers? Because these MLEs need to have some experience managing labelers and making sure that they can sort of clearly communicate with another set of human

• 42:21Speaker 02

  that they never worked with before on what is good, what is bad, and what I'm looking for. And then it also takes a few iterations because oftentimes these data labeling companies will have to assign you certain set of people repeatedly just so that they don't lose the context. But also you need to have quality control on the other end, making sure that the supervised dataset, the supervised fine-tuning dataset are consistent. And it will continue to be what you're looking for at the a thousandth label. That is a very tedious process, a very difficult process for many, many people. And the process also breaks down when it's long context. Because honestly, at certain context length, I really don't think I even understand what's going on.

• 43:03Speaker 02

  Like, it probably takes me like an hour just to read through all the conversations to figure out where we were wrong for me to like edit the conversation to make it work correctly. Whereas for reinforcement learning, as long as you have a product manager who can articulate what is good or bad, they will be able to author a language model as a judge snippet and then send it to fireworks and be like, hey, teach my model this, right? Everyone else is out of the loop and then we can sort of bootstrap this very, very quickly. And as the coding model gets better and better, I really think that there is a lot more we can push out of these coding models to get reinforcement learning more automated.

• 43:45Speaker 02

  I think that would be like an interesting topic to dig into more this year.

• 43:49Speaker 01

  Yep, and I believe it was Versal used, has used RFT with you guys. And that's sort of come out with, you know, putting numbers on it, something along the lines of 40x faster code fixing and with better outputs. So that's like sort of interesting use case for RFT.

• 44:09Speaker 02

  Yeah, absolutely, absolutely. Yeah, and we have a lot of smaller customers in general who benefit from RFT as well. I would say, for example, Versal has like a very, very strong engineering and product team. Those teams are exceptionally well fit to do reinforcement learning because we just with like two or three people, they can totally align on what's good and what's bad and they can just go.

• 44:31Speaker 01

  Nice. Yeah, that's a really kind of powerful image there. So we are sort of starting to wrap up, but something I did want to touch on is this competitor as a vendor problem that we've seen, you know, play out over the last, I guess, 18 months especially. So, you know, some examples on SED News, which is our monthly podcast that myself and Sean Falconer do, we've, we've covered actually quite a few of these. So this is why it's quite interesting. So the fact that Anthropic effectively cut out Windsurf and then, you know, OpenAI launching Codex, which competes with Cursor. You've then got things like Google Search, you know, versus perplexity in some respects.

• 45:10Speaker 01

  So if you're the vendor, like, how are you thinking about where this risk for you potentially lies or doesn't even so, yeah?

• 45:20Speaker 02

  I honestly think we are such a small player in this market and the market is so early still that I don't know if the competitive pressure matters yet. Yeah, I keep telling my colleagues, like, if we don't do single digit percent of what Nvidia is doing, I feel like we're doing something really wrong. And we're not there yet. So what Nvidia's, I forgot what the exact number is, like 400 billion this year, give or take.

• 45:51Speaker 01

  Something like that, yeah.

• 45:52Speaker 02

  Yeah, I would say like probably before, like we do a few percent of what Nvidia is doing, I really don't. Think that we will run into those hard constraints on like competitive vendors and whatnot. But I do think I can be more helpful for this answer in the sense that like when our customers are thinking about fireworks, what they're looking for specifically. And honestly, I think at the end of the day, it's mostly about trust. Trusting that we get the numerics right, so they don't have to figure out all the details. Trusting that we got all the serving details right, so that function calls can happen properly. The constraint generation happens properly.

• 46:31Speaker 02

  The trust on that we set up our reinforcement learning correctly, so then I have to like do the numerical debugging themselves. And I don't think a lot of those are competitor related in the sense that a lot of our customers are making honestly a lot of money. And they just want to make sure that we handle those complexities for them. I do think in the fast evolving field, maybe I should think more about the competitors landscape and whatnot. But I really don't think it matters as much yet. And it's more about helping our customers make as much money as they can at this stage.

• 47:09Speaker 01

  Yeah, exactly. And you know, you work with some very big names. So it's sort of anyone potentially smaller coming along and that's that trust piece. I think that's where you've got some of the bigger names to kind of back you up at this stage in terms of if you're able to work with the cursors and the Vercels, that surely says something. So. Yeah, I mean, we are, we are coming to time, but I think for anyone out there, developer or anyone, uh, maybe slightly, I want to say higher level because I see versus business is kind of the same thing. But someone who's not hands on keyboard, but I mean, if someone wants to get sort of quote started with fireworks, like what's kind of the best path there?

• 47:47Speaker 02

  Install OpenClaw, hook up Kimi on fireworks with OpenClaw. And honestly, Peter is amazing. I watch some of his podcasts, like the author for OpenClaw.

• 47:58Speaker 01

  Yeah. He's just joined OpenAI, at least.

• 48:01Speaker 02

  Yeah.

• 48:02Speaker 01

  Yeah, a couple of days ago before this recording, yeah.

• 48:04Speaker 02

  Like, I don't know how much he's paid, but he definitely deserved it. It is surprising, honestly, how fast everything is moving still. Just as soon as you expect things to slow down maybe a little bit, like, all the models come out in the last two weeks. Right? And then, like, OpenClaw gets their fame to acquisition in, like, a few weeks. It's, uh, the things are not slowing down.

• 48:29Speaker 01

  No.

• 48:30Speaker 02

  And I do think that for anyone who's listening, any amount of effort in this area, I think, will pay off. In anything, it's like, if it's not OpenClaw and, like, I don't know, Vibe Code, something. Because I really think a lot of the difference that I'm seeing for some of our customers is just that they are two months early. And the fact that they are two months early makes the world of difference.

• 48:54Speaker 01

  Yeah. Yeah, amazing. So, yeah, I mean, thank you so much for coming on today, Benny. Is there anywhere that you personally are on, like, that people can follow you? Are you on, I don't know, X or anything like that, or not really?

• 49:09Speaker 02

  I guess I'm old, I still say Twitter.

• 49:11Speaker 01

  Yeah, nice Twitter. Yeah, I would still say Twitter. I'm just, uh, I, the number of times I hear X formerly known as Twitter, like, well, we've got to pick one, don't we?

• 49:20Speaker 01

  Cool. So you're on Twitter. What's your, what's your handle on Twitter?

• 49:23Speaker 02

  Benny Chen.

• 49:24Speaker 01

  Oh, cool. Okay. Benny Chen. Nice.

• 49:26Speaker 02

  Yeah.

• 49:27Speaker 01

  Awesome. Well, yeah, thank you so much for coming on. I've learned a lot as well as, yeah, it's just kind of really awesome to see a company in this space. And looking back to the beginning, when you were saying, like, maybe you weren't analytical enough about when you left Meta and started this. But, I mean, this sounds amazing. And you've obviously done a huge amount of work in this space that someone else hasn't done. So I think that's, that says it all, really. So, yeah.

• 49:53Speaker 02

  Thank you for having me, Greg. Thank you.

• 49:55Speaker 01

  Thanks a lot. Okay. Well, I hope we get to catch up again in the future.

• 49:59Speaker 02

  Thank you.