How Cursor Trained Composer on Fireworks: Distributed Infrastructure for High-Performance RL

Cursor's Federico Cassano and Fireworks' Dmytro Dzhulgakov explain how they collaborated to build Composer as a specialized foundation model. The core insight: models have finite capacity in their weights, and allocating all those bits to the singular task of software engineering in Cursor frees the model to be both better at the task and far more efficient at inference. Rather than start from pre-training and work up, they took an unconventional top-down approach — mid-training and RL on top of an open-source base to get a useful model into users' hands fast, then specializing the model around real Cursor usage. With Fireworks providing distributed infrastructure, Composer delivers frontier-class coding performance with the speed of a much smaller model. Hosted by Sonya Huang, Sequoia Capital

• 0:00Speaker 00

  You need all the infrastructure to run these environments that have to mimic as closely as possible what a user's computer would look like. And it's very important, it's close as possible, because sometimes the model can actually figure out when it's being run in like a fake environment or a real one, and it has like different behaviors during RL than in production.

• 0:20Speaker 01

  Are you seeing it being conscious that it's in a fake environment and starts behaving differently?

• 0:26Speaker 00

  Yes.

• 0:27Speaker 01

  Interesting.

• 0:28Speaker 00

  Like, it's like, oh, I'm in a fake environment. I've learned a few tricks to like get better reward in this environment, and let me try them out. Models love to cheat. RL is really good at encouraging cheating.

• 0:53Speaker 01

  I'm delighted to welcome Federico from Cursor and Dima from Fireworks to the podcast today. Federico, you are the research lead on Composer 2 at Cursor, Cursor's new agentic coding model. And Dima, you spent how many of the last few months moonlighting at Cursor in order to support a lot of the infrastructure required to make this gargantuan training task happen? And so I'm excited to talk to both of you today about how the training of Composer 2 came together, what hard problems you solved together, and what do you think it means for the future of AI and foundation model companies?

• 1:26Speaker 02

  Exciting. Yeah. Thank you.

• 1:29Speaker 01

  Thanks for joining. Okay, let's dive right in. For those who haven't been following us closely, Cursor recently announced Composer 2, which is an agentic coding model meant for long horizon coding tasks. Federico, up till now, Cursor was mostly enabling other people's coding agents. What was the impetus for Cursor to lean so heavily into Composer 2? And how existential is it for you to become not just an application company, but also a foundation model company yourselves?

• 1:57Speaker 00

  The reason why we started looking into training our own models is you can sort of think about the model as sort of like like a storage drive. It has certain amount of bits that it can store in its weights. And the idea is very simple, you know, like we care about only one task. We don't even care about coding or programming necessarily. We care about software engineering inside Cursor and inside Cursor only. And so what if we were to allocate all of the bits of information that can be stored inside a model weights to that one particular task? Also, as people may have noticed, Composer is of order of magnitude the less expensive than Nopus and other like coding models.

• 2:41Speaker 00

  Because we can just simply specialize all of the model weights to that particular task. And so we can serve like a smaller model or something of that sort, yeah.

• 2:51Speaker 01

  So it's about, let's make sure every single bit of weight or information we have is dedicated toward the specific problem that we have at hand.

• 2:57Speaker 00

  Exactly.

• 2:57Speaker 01

  Got it. That seems like it's an almost generalizable problem. Dima, I'm curious your perspective. Do you think that every application company should be looking at Cursor as a harbinger of what's to come? Like, should they all be looking to do the same thing?

• 3:09Speaker 02

  Yeah, absolutely. I mean, we actually generally see it as a pattern of kind of evolution of the applications. You maybe start prototyping, you might be using kind of off-the-shelf model to get something running, maybe do some prompt engineering, figure out how your harness works. But the most kind of leveraged attribute of your application is actual usage of user data or particular specific aspects of how the application works, maybe some aspects of your harness. Which tools do you provide? How the application works? Kind of really important bits which are important for your application. And the right way to capture that, you can do a little bit of this through prompting. But really, the right way to do this is craft your model to act in your environment.

• 3:45Speaker 00

  Yeah, absolutely. Like, there are certain tools, the agent calls, that it's very hard to succinctly describe exactly the behavior of that tool to the model. And, you know, with just like post-training, we can bake in the optimal way to use those tools. Like Composer, we do serve a prompt to Composer, but I think the way we are training it, it would work even without a prompt and it would know what to do, just because like we are intrinsically pushing the model to like the right direction of how it should act throughout our training.

• 4:15Speaker 02

  Basically, there's kind of like upper bound of like how far you can get this prompting during and if you want to craft really great AI products, you have to go through kind of fine tuning and influence a model behavior. That's kind of one reason. I mean, reason number two is what the recommendation is kind of cost trade off or like speed trade off. Like the way we kind of viewed the fireworks is that that you're trying to do optimization, you had this like three dimensional trade off between quality, speed and cost. And you can go quite far and we are doing it with all the customers. Initially, you can go quite far with just optimizing infrastructure. But when you start getting to model trading, you can really push this trade off much further and you can get better model at fraction of the cost running much faster. And you know, composer is a great example

• 4:57Speaker 01

  of Can I push on this a little bit? I want to ask you if this approach is bitter lesson pills. And we were, we were actually all talking about tab nine on the walk in. I'm remembering before the LLM era, there were these like small specialized coding models. And one of the things that was, I think, surprising to, to a lot of people was as you've scaled up, you know, you scaled up just training on the internet and a lot of bunch of English texts and other languages. Actually, the models themselves got inherently better at coding as well. And so at least the trend line I've seen so far is just like bigger models perform better on everything, including on coding. Is what you guys are saying, does that go against the grain of the bitter lesson?

• 5:33Speaker 00

  I think no, but one, one sort of like thing to point out is that the big models trained by the labs train on a lot of code as well. Like code is one of the main tasks the labs are interested in pushing. And so they don't just generalize to it. They're a bit specialized as well. I think for our case, actually, you know, if we believe about the bitter lesson, we are just pushing very hard on the data dimension. And we know that the models inherently have finite capacity. And so if we want to saturate all that capacity, we need to scale data. And in order to ingest more data, we, we need to like free up the weights from distractions the model may have.

• 6:13Speaker 01

  Hmm. Okay, got it. Super interesting. Okay, let's dig into the training of Composer 2. You launched a couple of weeks ago, immediately grabbed attention. Strong benchmark number, is much lower cost to run inference on. What's the short version of how Composer 2 works? And what you guys did to make it so performant?

• 6:30Speaker 00

  We started from a very strong base, which is KIMI 2.5. It's like a 1 trillion parameter MOE that's 30B active. So very, very sparse, actually. We sort of like looked at the stack and realized there are like two axes. So mainly Composer 1 was just pushing on one of these axes, which is reinforcement learning. But Composer 2 pushes in two different axes. One is continual pre-training and the other is reinforcement learning. So the thing that made Composer 2 very good is pushing in both of these directions. So we started off the training run by doing lots of mid-training on code tokens, almost sort of pre-training scale, actually.

• 7:13Speaker 00

  And then coming out of that mid-training run, we took the checkpoints and we did very large-scale RL on lots of, lots of tasks.

• 7:23Speaker 01

  Okay, and then the premise here would be because Cursor sits in the middle of so many interesting coding tokens, you actually pretty uniquely have access to data to be able to train at almost pre-training scale.

• 7:33Speaker 00

  Yeah.

• 7:33Speaker 01

  Why not pre-train your own model, then?

• 7:35Speaker 00

  We just think about our approach from top-down instead of bottom-up. So like, how do we get a model that's useful to users in the least time possible? If we were to start from the bottom, sort of figure out how we do pre-training and then scale it up to mid-training and then, okay, now we figured out mid-training, I would do reinforcement learning. That would take a very long time to get a model out to our users. By doing it the other way around, we were able to give a useful model to our users in very little time. So hopefully, you know, like, next Composer versions are gonna be our own model instead of basing it off an open-source base.

• 8:15Speaker 01

  And what is the model roughly learning in the kind of mid-training step? And what is the model learning in the post-training step for you?

• 8:22Speaker 00

  Yeah, so in mid-training, it's sort of just kind of learning about libraries of code and learning about specific code patterns that are very common, like just world knowledge as well. There is like web data there as well. And this is sort of just creating a wider distribution that then reinforcement learning can sharpen on. And so during reinforcement learning, you know, the model gets to play directly with the cursor harness. And so it gets to learn about the world the model is going to live in for the rest of its life, right? In some way. And so then during reinforcement learning, that's where it learns how to call tools properly, how to navigate its environment, how to write correct code. Because during mid-training, it learns how to write code.

• 9:07Speaker 00

  That doesn't necessarily mean it learns how to write correct code. We try to train on code that is largely only correct, but the model doesn't actually know how to differentiate between the two. While in RL, one of the key things that we are doing is we're kind of tuning the feature of the model saying, hey, now you get to write correct code all the time. Exactly.

• 9:27Speaker 01

  Interesting. And is the is the model after mid-training, is that similar to the model that you guys have on Tab autocomplete? Or is that a different core competency?

• 9:36Speaker 00

  Yeah, I mean, it's, yeah, I think I would put it like that because like during mid-training, we are just doing next token prediction, you know, like how well you predict the next token and then the token after that. So, yeah.

• 9:47Speaker 01

  So why not just post-train on your Tab autocomplete model then? Why mid-train the different models?

• 9:50Speaker 00

  Yeah, I mean, Tab is a very small model because it's like a super low latency model. We want it to be very fast. So like the core two distinctions about the base models here is that Tab is like small and composer is quite large.

• 10:06Speaker 01

  I see, I see. Okay, so it seems like a lot of the focus of what you guys did for Composer 2 was this large-scale reinforcement learning run. Can you break that down for us? Like, what goes into that? And what are the various hard problems you solved along the way?

• 10:20Speaker 02

  When you do a R, it's quite different from like pre-training and mid-training, because you're not just trying to predict next token. You're actually running the entire harness, like the entire experiment. You're letting the model act in the environment, see how it performs for a given rollout. That's the terminology which is called rollout. And kind of assign the reward, whether it did something correctly or not, which might be some using LM as a judge, or maybe something verifiable, like does this code compile? Or something like this, which actually means that compared to this regular training, you need a bunch of other components. Like you still need large-scale training. You still need to orchestrate tens of thousands of GPUs to the forward, backward propagation, though all this stuff you do in mid-training and pre-training. But now you also need to orchestrate a bunch of environments.

• 11:03Speaker 02

  You need to run model inference, because when you do this rollout, you're effectively running like real Cursor session in some sense, right?

• 11:11Speaker 01

  So a rollout is like a forward pass?

• 11:13Speaker 02

  No, rollout is basically your entire like agent session from Cursor, right? So it basically means it might take something like 50 turns. Model will take your initial prompt, then decides to call some tools. You want to execute those tools, then model generates a bunch of other code. Kind of entire session which you, when you interact with agent in Cursor, right? You kind of simulate this entire session as a part of your training run. You get to final reward and use the signal to now go back to trainer and incorporate it in the model weights. So you have this kind of very big loop, update loop, which is very heterogeneous, right? Because you have all this like different components working together. And now you're trying to orchestrate all of this to work efficiently, work with high throughput, because GPUs are expensive

• 11:57Speaker 02

  and you want to get your model trained quickly in economic fashion. So that by itself is like very interesting kind of problem and intersection of algorithms and infrastructure, because there are a lot of trade-offs how you can kind of co-optimize and co-design the system. One aspect is kind of people call about like this async R of pipeline R. The idea is basically, okay, you're trying to update this model in steps, right? So you have your current model version and you're trying to do a bunch of rollouts with it. What does your trainer do while you're doing this rollout, right? Like, maybe approach would say that, okay, now I'm gonna stop my trainer. I'm gonna do a bunch of sessions and those sessions might run for like five, 10 minutes or even longer if it's like longer horizon tasks. I'm gonna get those outcomes and now I'm gonna pause my inference and I go back to training, trying to do updates.

• 12:43Speaker 02

  That's like very theoretically algorithmically robust because you are not precisely simulating everything. But it's very system inefficient because half of your capacity is sitting idle all the time. So you can do all the clever like algorithmic tricks. What can you do instead? Yeah, you can like kind of pipeline all of this. So imagine this as a gigantic like factory, right? You have this like trainer building and you have rollouts building. They're always turning, right? There's a rollouts always take like latest model version and try to do new sessions and kind of simulate new agent sessions. And trainer always takes new outcomes as they come and try to compute updates. So everything is moving along all the time. The trade off is that why I'm saying that algorithmically it's different because now by the time you finish some test rollouts in your kind of simulated environment.

• 13:28Speaker 02

  Maybe model weights will already updated on some other data. So you have this kind of staleness like delay between how quickly model can learn updates. Because by the time you kind of process or some interaction session with a simulated environment, your model base changed. And that introduces interesting training dynamics and there are clever ways how you can address this. But the flip side of that is that your all your GPUs, all your computers kind of loaded and chiming all the time. Which actually you're using more more flops and to your bitter lesson example, yeah, you you have a higher compute efficiency. You can get to a better model in a smaller amount of time. Yeah. Maybe you're losing a few percent from being asynchronous and not doing like perfect mathematical updates.

• 14:11Speaker 02

  But your way compensate for that by effectively not leaving half your capacity on the table. And there are a lot of kind of depths and interesting interaction in that part.

• 14:20Speaker 00

  And we're very serious about performance at Cursor because unlike the big labs, you know, we have tens of thousands of GPUs, not millions. And so, yeah, we do all sorts of tricks to make, get the most out of GPU. Like, we train in production with FP4 even. We work with Fireworks to like push on inference as well. Because the thing about RL infrastructure is just like it's just inherently more complex than pre-training because you need all the pre-training infrastructure. That's just like one of the requirements. Then you need all the infrastructure to run these environments that have to mimic as closely as possible what a user's computer would look like. And it's very important to do as close as possible because sometimes the model can actually figure out when it's being run

• 15:04Speaker 00

  in like a fake environment or a real one. And it has like different behaviors during RL than in production.

• 15:11Speaker 01

  Are you seeing it being conscious that it's being, it's in a fake environment, it starts behaving differently?

• 15:16Speaker 00

  Yes, yes. Like, it's like, oh, I'm in a fake environment. I've learned a few tricks to like get the better reward in this environment and let me try them out.

• 15:24Speaker 02

  Models love to cheat. RL is really good at encouraging cheating.

• 15:27Speaker 01

  Yeah.

• 15:28Speaker 00

  Yeah, and then we need a really efficient inference. So this is really important. So there is like actually this kind of myth that during RL you spend more, way more inference flops than training flops. This is sort of like just because the open source inference engines are very unoptimized. Instead of actually being a property of RL, roughly the same ratio is kind of the same. In theory, if you push the GPUs to the maximum, you should have one third of your training GPUs allocated to inference, right? Because training is effectively three forward passes. You have the forward pass, you have the data gradient, the weight gradient. While if you really hit the critical batch size on inference, you should only have a single forward password of flops.

• 16:12Speaker 01

  So that's why you guys use Fireworks instead of using an open inference engine?

• 16:15Speaker 00

  Yeah, I mean, the other alternative is we would build one in-house. But, you know, we have finite engineers like everybody else. We would like prefer to have engineers make training more efficient and more precise rather than like spin up like I inference F4L.

• 16:31Speaker 01

  Okay, that's super hardcore. What about, I think you mentioned in your technical paper that you were doing this in a kind of globally distributed way? Why globally distributed? And then what makes that hard? Yeah.

• 16:41Speaker 00

  Yeah, well, there are various reasons. One, you know, like this very large contiguous clusters are hard to find in the market. And so what we can do instead is we have one cluster that's going to run all of training. You know, we can't do global training cluster. But then the inference component of reinforcement learning, we can globally distribute that across small clusters all over the world. So I think for the composer to run, we used the four clusters in total that were all over the world, very far away from each other. And we even used some of our production traffic when it was least used. So like we had the composer 1.5, the previous model served.

• 17:23Speaker 00

  And when it was least used by people, we just grabbed some inference GPUs and we put them to speed up training. And so we can do this sort of things and sort of easily scale up our training ground without having one large contiguous cluster. And the thing that enables it, maybe Dima can talk more about it.

• 17:41Speaker 02

  Kind of like to reiterate what Federico said is basically RL training is like very heterogeneous, right? And by leveraging heterogeneity, how different components, like what infrastructures they need, you can actually drive efficiency and you see this pattern kind of across the board everywhere. Specifically for for training, you have all this like highly interconnected clusters. You need high speed network, kind of need to work in lockstep. So those clusters are expensive, right? And actually, it's really hard to find big ones, right? Basically, as a scale with which composer was trained, finding like 2X larger clusters is like significantly harder than finding the current size one. And that's why if you can disaggregate these components and put them on different places, one, you don't need to find such a big cluster. Two, you can actually find like different trade-offs of hardware.

• 18:25Speaker 02

  Because for inference, you don't need that kind of wide interconnect. You can have smaller groups of GPUs interconnected together. You can have heterogeneous types of GPUs. You can have different generations of GPUs. You can kind of play all these games of optimization. And finally, like inference, it's much easier to scale up and down as you go. And yeah, it's very conventional, like when you have OFP covers, you can view all your. Kind of inference pool as one set of GPUs, serving production traffic for real users or serving simulated environments for RL purposes and kind of balance between those. Of course, it's a very interesting systems problem. So you mentioned like the KIMI model is like 1 terabyte. Training step takes somewhere between like 5 to 15 minutes.

• 19:08Speaker 02

  So it basically means like every like every 5 to 10 minutes, you are producing like 1 terabyte new snapshot of weights. So the question is like, how are you going to ship it to a different cluster on the other side of the world very efficiently, right? And you want to like do it quickly because remember, you don't want to get this staleness to get out of hand. So I think that was probably one.

• 19:27Speaker 00

  Yeah.

• 19:27Speaker 02

  The kind of the most fun part which we figured out together is that despite, you know, full model being like 1 terabyte, not all the weights change every step, right? Because RL does a lot of very precise adjustments, especially the training going on. So actually, there are very kind of regular patterns in like which subset of weights gets changed. Maybe not all of them change every time. So if you were to look at like how my model changes within one training step, like after 10 minutes, there is relatively small delta between those. You can write a compression algorithm, which basically leverages this property. And now you end up with kind of like database systems problem, which is, okay, I have my delta and I just want to like ship it across across the world. My delta might be like 20 times smaller than what's shipping

• 20:11Speaker 02

  the full model with, and that and that makes it practical. But of course, now you need to build all this kind of machinery from storage systems of full snapshots and deltas and recovery and reconciliation, etc. We were able to build it kind of in lossless fashion. Basically means that you always end up with bit equivalent model on the other side. So you don't need to worry about any math aspects of this. And you can do it really fast. You can you can do it under, you know, under a few minutes, even in the worst conditions, usually it's under a minute. And most importantly, you like pause only for like maybe 30 seconds to swap the weights in your actual inference server.

• 20:44Speaker 00

  We also like fully like saturated the band, the egress of the cluster by like sharding the upload and the download as well.

• 20:51Speaker 02

  So you can do all these like system tricks to bring this stand down. It is quite a few complexity, but you can kind of abstract it out and just make it work great. Like it doesn't interfere with your training algorithm. And on the flip side, you have this kind of power to disaggregate, to leverage other clusters to do that. And that kind of goes against kind of conventional wisdom of how you should do RL infrastructure. Because conventional wisdom is like, okay, you're going to have this really huge one cluster connected with our DMA. And it's going to be very expensive and you're going to probably spend, you know, maybe you're going to allocate like one sort to training and two sorts to inference. And sure, if you have a very expensive network, it's much easier to copy this one terabyte quickly. But now we have like three times larger cluster.

• 21:32Speaker 02

  Now, if your inference engine is more optimized, then maybe you're going to save one sort of that cluster in terms of GPUs anyway, because you're just more efficient. And you can take, you know, half of this cluster somewhere else in a maybe cheaper hardware in a different region. So your cost comes down quite a bit.

• 21:47Speaker 01

  I love that you guys are just grinning as you describe this because it's like, it's so hard. And this is like a systems engineer's dream, right? And so it's just like a, it's an amazing, amazing system you guys have built.

• 21:57Speaker 00

  We spent a bunch of nights working on this.

• 21:59Speaker 01

  You look like you spent a long time, a lot of time together. What about, I mean, you mentioned at the beginning that Kimi is a very large sparse MOE model. Does that make the RL run tricky in any way?

• 22:11Speaker 00

  Yeah.

• 22:11Speaker 01

  How so?

• 22:12Speaker 00

  Well, when you do inference, you're essentially doing like a forward pass is just kind of like autoregressive. And in this forward pass, it produces like log probabilities of like the tokens it has sampled. When we ship back the like generations of the model to the trainer, we have to rerun that forward pass. Because as we mentioned, we are doing asynchronous training. So the model that has produced the pass may have been like actually a few steps behind what the trainer is at. And so we have to rerun that forward pass and reproduce log probabilities now. The problem is, in theory, this log probability should be exactly the same if it's the same model version.

• 22:54Speaker 00

  But even with the same model version, you get slightly or sometimes very different log probability values for the same tokens. So this is often called like a numerical mismatch for inference. You hear this about all the time these days.

• 23:09Speaker 01

  Why is that? Why does that happen?

• 23:12Speaker 02

  I mean, primarily because like fundamentally floating-point arithmetic, which is doing this is non-deterministic.

• 23:18Speaker 00

  Yeah.

• 23:18Speaker 02

  So if you-

• 23:19Speaker 01

  Sorry, floating-point arithmetic is non-deterministic?

• 23:21Speaker 02

  So you know, we learned this called like if you take a plus b plus c, right? And like c plus b plus a, it's gonna be the same result. If you're doing this with integers, with whole numbers on the computer, that's gonna be always true.

• 23:32Speaker 01

  Yep.

• 23:32Speaker 02

  If you're going to do it with floating-point numbers, which are actually like approximation numbers, you have this like mantis and exponent, etc. A plus b plus c and c plus b plus a is gonna give you like different results, or even like a plus b and b. So basically, fundamentally, it's accumulation order of like all these operations, which models do, is basically like multiplication and additions. And like addition order matters to your final result. It's all like small differences, but they get amplified through like millions and billions of operations. So when you do inference of models, usually it doesn't matter that much. Because you pre-train your model, you're actually pretty robust. If you like flip some bits, it's still gonna produce you like good results, your benchmarks are not gonna change. But a real in particular, because you're using this very, very like weak signal to teach the model.

• 24:17Speaker 02

  The noise from these numerical differences can make or break your training, and that's like particularly important. And again, it's an interesting intersection between like algorithmic and systems part, because you know, you can write a beautiful math and it just doesn't work in practice. There are ways how you can drive this difference to pretty much zero. There are all this like batch invariant ways, where you can be very, very careful and write all your GPU kernels, so they always add numbers in the same order. So you always do like a plus b plus c and not a different order. It's possible, but it always has like trade-offs, right? Basically, your like your system becomes maybe like 2X or 3X slower. Again, it becomes an interesting trade-off, like, okay, what is there 10% Of slowdown which we can take, or in fact, it's actually a few percent of slowdown we can take to address 90% of this difference.

• 25:02Speaker 02

  That's, you know, the right trade off which, you know, we find together through iteration. And you mentioned that particularly for MOEs and sparsity is hard. The reason for that is that like the way MOEs work is that you would take your activations at every layer. And you would run it through gating layer, which basically decides, okay, for this token, I'm going to run out of 384 experts. I'm going to run this eight, right? So it's going to do like some mess and like top eight scores. Those eight experts are going to be activated, other ones will not be activated for this token. This operation amplifies your small numerical differences quite a bit. Because maybe your hidden states were like difference by like fifth digit after dot, doesn't really matter.

• 25:42Speaker 02

  But this difference made it so you picked expert number seven versus expert number nine as a kind of as a cutoff. And suddenly you went and like activated totally different part of the model and your difference got amplified quite a bit. And my models by definition like were more sensitive to this mismatch. Again, when you do inference or when you do kind of regular rollout, it usually doesn't matter as an average out. But now if you're trying to base this model learn, this difference is huge because your inference activated expert number seven. Now in your training, you're trying to like update expert number nine, which didn't even contribute to that during inference.

• 26:18Speaker 01

  So were you guys handwriting GPU kernels then to help get around this problem?

• 26:21Speaker 02

  Yes, so you can, again, you can address a lot of this for GPU kernels and there's always trade off. Specifically for ME, you can do this interesting trick, which people call router replay. But basically you can have your inference just pass extra information to training and say that, hey, I activated expert seven for this token. This very small bit piece of information is just one integer saying that like, okay, this is the expert which you activated. So trainer can be aligned with that. And a lot of this numerical alignment is basically doing tricks like that, matching quantization levels, matching kernels, etc. To drive the divergence between training inference implementation down. And that makes huge difference in between, you know, your own maybe diverging completely or being, you know, multiple X less compute efficient. Because you'll need much more data to address to this mismatch.

• 27:06Speaker 01

  I'd love to maybe chat a little bit more about the RL kind of recipe. Can you say what about the reward signal you're using is like, or you can't, okay, can't say, got it. Top secret stuff, top secret stuff. Okay, that makes sense. Like it seems like this is a, almost like the equivalent of learning in sim. This is simulated rollouts versus like, you have so much actual user data that you could be learning on. Why not just do RL on your, your actual user data and your actual user harness versus doing this in sim?

• 27:33Speaker 00

  Yeah, we're also doing that. So that's what we call a real time RL.

• 27:37Speaker 01

  Okay.

• 27:37Speaker 00

  And we use the same technology to do like the inference with sync with like fireworks to do this. We find like user signals where the user was happy or sad about a particular model generation. And we're able to update that model live. And so then ship a new version of the model continuously every few hours. We're working on decreasing that time. Actually, at some point, we'll have to increase that time. Because as the horizon of the model gets longer and longer, we'll have to re-extend that time. It's like an interesting play. Like right now, we are trying to decrease the time for stability. Because we were figuring out the right hyper parameters.

• 28:17Speaker 00

  And then after we figured it out, we have to re-extend it again, just because we want to lengthen the horizon of these models. Yeah.

• 28:24Speaker 01

  Do you need to do any of the kind of like pre-training simulated RL? You have so much actual user data. I imagine that's just like much more valuable to, to train and tune on. Like, why not just go straight to the online RL step? Why, why do you have to do the, the offline RL?

• 28:37Speaker 00

  The online RL currently is pretty inefficient. We suffer from this problem that the GPUs are offline for a long time, essentially.

• 28:45Speaker 02

  And besides that, there are also like different trade-offs.

• 28:48Speaker 00

  Yeah.

• 28:48Speaker 02

  Also in terms of efficiency and user experience.

• 28:50Speaker 00

  Yeah.

• 28:50Speaker 02

  If you do simulation, you actually do multiple rollouts from the same prompt, right? You effectively take a task and you ask a model to do 16 tries at a task, like 128 tries on task, like different rollouts from the same prompt. Some of them are gonna go, go well, some of them are not going to go well. And by doing it multiple rollouts in parallel, you are able to get much more precise signal. Maybe like, you know, maybe model is very good and it's, does it well 90% of the time. Maybe it's not very good. Losses like GRPO, like group, group policy gradient, like kind of work by doing multiple rollouts at the same time. If you're doing online, you have only one rollout hanging back. And that, so, so trade-offs of like how you do it algorithmically. Different and most importantly, if your simulated rollout goes wrong, it's not it's not too bad.

• 29:34Speaker 02

  Right? I mean, you just, you know, maybe you spend some time on it if you if it's an actual user, you have much higher like minimum bar on this. Because effectively, you're doing A B test, right? So if the model produces something weird, like that's a bad user experience.

• 29:47Speaker 01

  Okay, so you can go off policy more often when it's not a real user. Because you can like you can experiment with like crazy things and without affecting the user experience. You can do a lot more rollouts, you do GRPO. And then you can basically like bootstrap some level of performance that's good enough to even put in front of users.

• 30:05Speaker 00

  Yeah, like we teach reasoning through like the offline RL, which is actually like called online RL. Offline RL is more like DPO kind of technique, the sort of reinforced kind of RL is online. And then we there we like teach the reasoning to the model. We give it some kind of input of the behavior it should have. We try to give it new information about the world and we teach it tool calling. And then we put it live to users. Because you could imagine like if the model is bad, users don't want to use it, they're not going to give us any feedback, right? So the model has to meet some kind of bar to even like be put into online RL. Like we want to be really happy with the model and this is the model we ship. That's kind of the paradox of online RL or how we like to call it real time.

• 30:50Speaker 00

  Is that, you know, we can't use this to really like create the model from scratch. Because users need to be using the model and so it has to be good already and we can only make it better.

• 31:02Speaker 01

  Yeah.

• 31:02Speaker 02

  It's kind of like cherry on top to really get this super delightful experience.

• 31:06Speaker 01

  Yeah, totally.

• 31:07Speaker 00

  Hopefully, one day it will be like big, big cherry, you know?

• 31:12Speaker 01

  Yeah, Dan Roberts presented at our conference last year. I think you were there. It's like traditionally is the big cake and the little cherry.

• 31:19Speaker 02

  The Yann Le Cun's cherry, yeah.

• 31:20Speaker 01

  Little cake, big cherry. I'm curious, the Andre Karpathy line of like right now RL is, you know, still super inefficient. You do a big, big, long rollout and then you kind of get like, you know, a little bit of information at the end. And it's still like, I think slurping bits from a straw. What do you think? And have you have you been able to figure out how to get more bits out of that path?

• 31:40Speaker 00

  I can't talk about that.

• 31:42Speaker 01

  Okay, okay, got it. We're back on the secret stuff. Good. That's how I know I'm asking the right questions.

• 31:49Speaker 01

  You mentioned the rollouts are a few minutes at a time. It seems like the whole field is pushing towards making like long horizon agents, agents that can work for a long period of time uninterrupted and generally not failing. I love that meter scaling chart. What goes into the RL process to try to get the agent to run for longer?

• 32:07Speaker 00

  Several things. So one problem about sort of like reinforcement learning is that the longer the trajectory is, the harder it is to do credit assignment. So you can imagine like we are giving thumbs up, thumbs down at the bundle right at the end of its work. And sort of like to simplify the problem is like the model asks itself, okay, where did I do right and where did I do wrong? That's basically the problem called credit assignment. It gets harder as this gets longer, so you have to do a bunch of tricks there. The other problem is just like you run out of space, right? Like these models have a finite context window and at some point they're gonna reach that.

• 32:47Speaker 00

  So actually, the way we solve this at Cursor is we put compaction inside the RL loop. So we call this self summarization. So during reinforcement learning, the agent actually learns how to continue and go on forever. So in practice, our model is like a 200,000 context window model. But in reality, it can go on for millions of tokens. And just because of this ability that it can summarize its work and then take that summary to restart its context window while still trying to accomplish the task. And through RL, because RL pushes the model to do things correctly towards the goal. At the same time jointly, we are training the model to produce a good summary.

• 33:31Speaker 00

  And then we're training the model to listen to that summary very well at the same time. And so this is kind of like a continuation to reasoning almost, I feel like.

• 33:40Speaker 02

  I find it fascinating because usually context management can see like part of the harness, right? In this case, you're effectively co-optimizing like how part of the harness and like model itself work together and throwing all of that into optimization loop. And we've seen this again and again in AI. They're like, the more you throw computers a problem, the more you can solve the problem end to end. The magic of computing bitter lesson works and you get much better system which can work together.

• 34:06Speaker 01

  Totally, totally. Do you think every company is gonna be RLing their own harnesses? Like, do you think that every company has the same shape of problem as Cursor?

• 34:15Speaker 00

  If they are using AI and they're like producing lots of tokens and they have a product to optimize against, I think it's it's like the right move and the right direction to train models.

• 34:26Speaker 01

  Yeah, yeah, interesting, interesting. And so it seems like most of the reinforcement learning you guys did then was on the kind of like the harness slash tool use part rather than on the get good at, you know, completing the next token for code. Is that roughly the pattern that other founders should have in mind when they're trying to think about? Where should I use reinforcement learning? So like if you're trying to get an agent to perform tasks with tools over a long horizon, you need RL. If you're trying to create a model that's good at summarization or a next token or whatever, you probably don't need RL. Is that a good framework for when you need RL?

• 34:58Speaker 00

  I think RL fits everywhere. So even for TAB, we use RL. Personally, this is just my theory and it's not backed up by anything. When you pre-train a model, they're just, the models are just ingesting the totality of human knowledge. Let's say you're training a model for math. The model sort of like learns all the math on Stack Exchange. The model, when it's presented with a math problem, and this is a model that hasn't gone through RL, the model is, needs to wonder what kind of person it is. Is it the expert or is it the student that's trying to learn? And so one of the things that I think happens during RL is that we are tuning this knob, letting the model know, hey, you are the expert, you need to do things correctly.

• 35:43Speaker 00

  So that's like one thing that happens is we are sharpening this distribution. Sort of like RL has a few phases. So like there is the very first phase where the model learns and becomes very good, very quickly. And then there is like a second phase where like it takes a lot of compute to continuously improve the model. And like you see the model starts reasoning and have this pattern. So in the very first phase of the curve, I think that's where we're just tuning the knob, telling the model, hey, you should do things correctly here. And so RL in the small compute case is also very useful just to let the model know that it has to do things correctly. That's sort of like my case to this.

• 36:25Speaker 02

  Yeah, I mean, second that, I mean, you, we see this pattern because many of these cases, we helped RL fine-tuning generally for many customers. And we see this usually you kind of continuous pre-training, mid-training, like regular supervised fine-tuning is simplifying. You can say this transfer of new knowledge, kind of in abstract way. And RL is kind of sharpening the behavior or like particular qualities you would, you would want from, from the model. And usually you end up needing both. And even to your example of summarization, it's actually like RL may be very useful for this because sometimes it's, if you want particular style out of summarization, right? It's really hard to like come up with examples of like good and bad summarization, et cetera, like really describing this precisely. But if you use, for example, LLM as a judge, right, you can actually say very precise rubrics.

• 37:10Speaker 02

  You can kind of prompt eval saying like, okay, this is a criteria, how I'm going to evaluate whether summarization good or not. Throw that into RL loop and let the model kind of experiment with different summarization styles, figure out what you actually want from it. While maybe another LLM kind of evaluates whether it's matched in particular rubric or not. And that's kind of type of pattern which you see a lot, not just in coding.

• 37:33Speaker 01

  I see. Okay, I'm going to ask this question to Dima because Federico is going to plead the fifth.

• 37:38Speaker 01

  You've mentioned LLM as judge a couple of times. Do you think that ultimately companies will be more successful having like experts hand examining RL rollouts? And, you know, hand coaching the model behavior in some way? Or do you think LLM as judge, other automated rubrics are likely to get us there?

• 37:53Speaker 02

  You don't really like put experts directly in judging RL rollouts. I mean, that would be some kind of like, I mean, real-time RL if it's actually users or like some form of, I don't know, like RLHF or DPO. I mean, generally, the more verifiable your reward is, the better, because it allows you to like scale the compute and just get better outcome. In some case, and by verifiable, I mean, basically means like, hey, how, can you automatically produce it without a human? Of course, if it's like mass or coding and you can craft something like very deterministic, that's the best. The reason why LLM as a judge works is that it's actually, it's kind of like generator discriminator distinction. Like it's much easier to judge, I mean, they're central for humans, right? It's easier to judge than, than to create.

• 38:35Speaker 01

  There's LMA VC.

• 38:37Speaker 02

  No, no implication there. But yeah, it's much easier to judge and you can craft precisely like different criterias you want to rank some answer. And you see this pattern where you might have like very complicated eval from multiple aspects, right? Because if you dump multiple aspects to a single LM, it might be get confused how to judge, right? Like you, you might break it down. Okay, you're gonna judge rubric based, based on style, based on like some different aspects, like based on factuality. Kind of really craft these rewards. Some of them will be the GNC, some of them will be LM based. And that's what guides your model behavior. Then you just turn on, turn on more computes and see the graph go up.

• 39:14Speaker 01

  Do you think that we're gonna see RL be more effective in the harder to verify domains? Like, do you think LLM as judges sufficient?

• 39:20Speaker 02

  That's one of the techniques you would, you would, you would start, right? Ideally, you want to figure out what is their actual outcome, what is their actual metric you want to get, right? So kind of trying to approximate this realm is one way. Trying to get bigger simulated environments is another, right? Like if you can simulate more of your product, if you can simulate more of your environment. Usually you have like final metric which you care about. It's just harder to capture. If you can figure out how to capture this, that's great. And to your point about, you know, experts, I mean, experts are still, still needed, right? Because crafting this task and actually encoding the product experience you want, that's, that's what matters, right? We went through software one point or two point or three point, right? Instead of crafting software directly, we went to crafting training data, right? Now you're effectively crafting the evaluation rules.

• 40:04Speaker 02

  But that's still very important. You need to look at examples, you need to look at the data, you need to look at like where your product fails and how to nudge the model in the right, in the right behavior.

• 40:13Speaker 01

  I want to ask about RL environments, which is maybe related to what you were talking about. It seems like there's been a huge explosion and just the revenue, the scale that some of these RL environments companies are reaching. What do they provide that's actually useful? Because I think cursor, for example, you have so much data on like how your customers are actually using your environments. What do the RL environment vendors offer you on top of what you already have?

• 40:38Speaker 00

  Yeah, we don't actually use any of the environment vendors. I think so it's very difficult to construct working environments. It's a valuable product for people that do not like have access to this. However, for coding particularly, there is like a very large amount of working coding environments available to everybody. That's GitHub, right? You can go in and maybe like you can have a model like just install all of the dependencies for a repository. And that's like a working environment. I think a lot of the difficulty comes from the infrastructure as well. So you can imagine that a environment that's that works well for a particular task may need like services up.

• 41:20Speaker 00

  You're like making a change that, um, let's say like a database migration to test that is actually working, you need the database up, right? And so those kind of things are very tricky. I think like these environment companies are like quite helpful for that that kind of stuff.

• 41:35Speaker 02

  There are kind of two aspects to this, right? First, if you look at like Frontier Labs, right? They're trying to build generic model which is good at everything, right? So they need to cover all these different tasks underneath, package up in one model and kind of encourage it to generalize. Right? Uh, so that's that's kind of one part and that's that's very helpful, right? In cases like Composer, right? You have you have your actual product, right? And I think that's what also kind of video with fireworks, like, yeah, if you have your actual product, you should you should do a R L against it, right?

• 42:03Speaker 01

  The most powerful environment is your own product.

• 42:05Speaker 02

  Exactly, because like that's where your model actually will be used. And and of course, uh, if you're Frontier Lab, you're not gonna do it across all the products, right? But if you're if you're trying to build the best model for your product, specialize and tailor it, you should just use your production environment. Of course, you want to isolate it properly, right? You don't want to model havoc havoc on your production database. You want to flaunt it, etc. And there are some, you know, tools from environment companies, just like from general infrastructure, which makes it easier. But generally, you want your R L environment to be as close to real production as possible. And that's what, you know, as an example, we see it is if you look at kind of toy R L examples, the toy R L frameworks, they always start like, oh, there's this like. Toy environment, I'm gonna spin up a Docker container and run everything in it, which is great for like toy examples.

• 42:50Speaker 02

  If you're trying to teach model how to play Atari or whatever, right? But if you're actually transitioned to like production cases, you can't just put your real, real production application in a Docker container. And we found it pretty early ourselves, like working as many folks, like in case of cursor trainer on their side. Some other customers, we run trainer on our training platform. But for environments, we actually default to running them on the customer side because that's where the actual implementation is. And you effectively have the same setup of trainer, even if it's part of fireworks platform or on the customer side, calling the actual production environment, not trying to kind of wrap it and componentize it.

• 43:27Speaker 00

  Yeah.

• 43:28Speaker 02

  On the, on the hosted platform, because that's really hard and that introduces differences.

• 43:32Speaker 00

  Yeah, like, I mean, what we call RL environments is really three components. One is the harness. So the harness is like where the model can submit tools and the tools get executed. And the second thing is, let's call it the like kind of operating system, right? So like, what is the actual like world, the, the state where the model is like interacting with? And then there is like the reward component, we need, which needs to check at the end that the work is done correctly. And generally, the harness is pretty portable. You can take the harness and put in many different environments. The thing that's key is the operating system. And to replicate this, just normal containers don't really work very well.

• 44:14Speaker 00

  So at cursor, we actually built like a whole virtual machine stack. And so we can spin up like virtual machines really quickly. And it has to be super bursty because you can imagine like we are asking this system, please give me a hundred thousand virtual machines now. And it has to come, all come up and um, um, yeah.

• 44:33Speaker 01

  Awesome. I really enjoyed this conversation today. I think cursor is such an inspiration in what you all are doing as a company towards going from application company to really a frontier model lab. And I think the work you did with composer too, really leads that charge. So really special to hear about it. And then Dima. Really cool to hear about the hardcore infrastructure problems, actually, that the two of you solved together in the trenches over many, many late nights to make it all possible. So, thank you, thank you guys for joining today.

• 45:00Speaker 00

  Thank you so much for having us. Thank you.