⚡️Mercury: Ultra-Fast Diffusion LLMs — Estefano Ermon, CEO Inception Labs

Watch the full show on YouTube: https://www.youtube.com/watch?v=2fDBeMu6xjk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Alessio Fanelli?

  Hey everyone, welcome back to another Light in Space Lightning Pod. This is Alessio, Partner and CTO at Decibel, and I'm joined by swyx founder, Smiley AI.

• 0:11Speaker 01

  Hey, today we are doing another Lightning Pod, but this time, I think, our first diffusion language model pod with Stefano Ermon from Inception. Welcome.

• 0:22Speaker 02

  Thank you. Yeah, I'm excited to be here.

• 0:23Speaker 01

  So Inception, you've been making a little bit of noise with diffusion models over the last few months, but I think that the big one is now that you are actually competitive on coding tasks, and obviously, the tokens per second is really impressive. So basically, could you tell us the overall story of Inception and maybe like, why you're betting on diffusion language models?

• 0:47Speaker 02

  Sure, yeah. So the story started back in 2019, actually. At that time, you know, I've been doing research in generative models since basically 2014, 2015, when I started my faculty position at Stanford. Been working on a variety of different kinds of generative models. But yeah, in 2019, I was working on image generation. And back then, the best generative models were GANs, generative adversarial networks. And we were pretty unhappy with GANs and how hard they were to train. And so we started pushing this idea of building a different kind of generative model with my student, Yang Song. And we kind of like showed that there was a better way of building

• 1:32Speaker 02

  image generative models, and that you could train this diffusion kind of model that instead of generating images in one shot, it kind of like generates them through this iterative refinement procedure. And so we kind of like pushed out some of the first papers on diffusion models for image generation using deep neural networks. We used to call them score-based generative models. And that kind of like was picked up very quickly by industry, by folks at Stability, by Midjourney, and then kind of like became the state of the art for image generation, video generation, basically, any kind of continuous modality, you know, diffusion models and variants of diffusion models are now the kind of like the best approach that exists.

• 2:16Speaker 02

  And since basically two thousand and twenty, two thousand and twenty one, I've been trying to get diffusion models to work on the discrete data. So tax generation, code generation, DNA generation. Um, and it's been really hard. The, after many years of trying, we were able to show that diffusion models. We figured out a way to train basically some kind of discrete diffusion model that was competitive with auto-regressive models at small scales. And so we published a paper at ICML last year that won the best paper award, showing that for the first time, basically, discrete diffusion models were competitive on language generation without the regressives models. Up to the G P T two kind of scale.

• 2:59Speaker 02

  And I got very excited about those results and I really wanted to see what happens if you train bigger models on more data and you put all the kind of engineering efforts needed to really get to state of the art results on tasks that matter. And so that's how Inception came along. And so I recruited some of my best former PhD students, a lot of Stanford folks. And then we started a company last year, last summer. And yeah, we've been working on basically improving those research ideas that we had. And we've been successful in training the first commercial scale diffusion language models. Uh, we call this model Mercury. We released the first one, Mercury Quarter, back in February.

• 3:37Speaker 02

  And we just announced a couple of weeks ago, the first diffusion language models for more like chat, kind of like text use cases, summarization, translation, reply prediction. And yeah, these models are really exciting. They are much faster than other aggressive models. They have some very interesting new capabilities and we're excited about the future of this technology. I can see a world where in the future, all language models will be diffusion based. I think it's a superior kind of like approach. And so I'm very excited to be at Inception working on this.

• 4:04Alessio Fanelli?

  Can you maybe do a quick intro on diffusion models for discrete text? Because I think people have an understanding of image generation, which is you add the noise, you remove the noise. But it's kind of hard to grasp how you actually train a diffusion text model.

• 4:19Speaker 02

  Yeah, it is actually a very similar kind of idea. Typical language models, LLMs, are auto-aggressive, meaning that they generate objects left to right, typically one token at a time, one word at a time. And you can do the same thing for image generation, like you can generate images one pixel at a time. It's just not a very good idea to do that. Diffusion models, on the other hand, they work by generating objects, kind of like in a coarse-to-find way. You start with a rough guess what the answer should be, and then you refine it, again, using a neural network typically. And diffusion language models work the same way. You kind of like start with a guess of what the answer should be, and then you refine the answer using a neural network by modifying essentially multiple tokens effectively in parallel.

• 5:04Speaker 02

  That's kind of like the reason diffusion language models are much faster compared to auto-aggressive models, is that each neural network evaluation doesn't just give you one token, like in the typical auto-aggressive world, but it's able to output essentially or modify multiple tokens. And so that's where you get the speed ups, the fact that you're able to modify many things basically in parallel.

• 5:25Alessio Fanelli?

  And so the... When you're creating a training dataset for a diffusion model, what does it look like? Is it just a normal pre-training corpus? And then instead of masking a token, you're like, I don't know, what are you?

• 5:37Speaker 02

  Yeah, so there is a lot of secret sauce involved, but roughly, yeah, the approach is similar. So you can start with the usual training data that you would use for training an auto-aggressive model. You have to specify a neural network, which could be the same kind of architecture that you would use for an auto-aggressive LLM, so it could be a transformer if you want. The difference is that instead of training it to predict the next token, you train it to de-noise a piece of text or a piece of code. And there is different ways of adding noise or removing noise. In the case of images, you would typically add Gaussian noise, and you would train the model to remove the noise, although other ways are possible. In the case of text, again, there is different mechanisms of adding noise. You could imagine masking out some parts of the sentence,

• 6:20Speaker 02

  and then training the model to infill, to guess basically what the missing tokens were. Or you could imagine different kind of noise models where maybe you flip tokens. So it's just more similar to the Gaussian kind of case where you you kind of like randomly flip some of the tokens. And then you train your neural network to basically fix these mistakes and and correct the the this noisy data point and map it back to to the clean data that you started from.

• 6:46Speaker 01

  Yeah, um, I noticed that you uh so that you did publish a paper recently, I think yesterday, the or something. Where you you confirm that you're actually using transformers for at least the Mercury models, yeah? Um, and basically, does it have to be trained end to end? Or can you just take any existing open model backbone and uh uh and you know, do this process? Like, do you have to initialize from zero, basically? Or can you just, can you lean, can you sort of leverage off of existing uh other other language models?

• 7:17Speaker 02

  There have been attempts in the literature um where people have tried basically using um existing open source models and then kind of fine tune them. The challenge is that, yeah, the training objective is quite different because you are training uh based on the noise as opposed to next token prediction. Diffusion models are not causal and that is also kind of problematic. Uh, I mean, it's a big advantage of diffusion models that you don't have to be causal. You can actually look at the whole context to the left and to the right as you decide kind of like the edits that you want to do. Which is giving, you know, it's a very powerful thing if you think about how you generate objects. Uh, but that makes it quite different, quite difficult to adapt

• 7:59Speaker 02

  kind of like models that have been trained with with causal masking to to this to this to this to this new task. And so there have been attempts in the literature annealing the the the masks, uh, the attention mask. Like, there have been various tricks and attempts at doing that, but it's not straightforward.

• 8:19Speaker 01

  Yeah, and then on the sort of the equivalent of contrastive loss or guidance or anything like that. Does that require any innovations or is it kind of, does it transfer over nicely from the image side?

• 8:31Speaker 02

  Transfer over pretty nicely. You can do, yeah, you can do basically classified.

• 8:37Speaker 01

  Yeah, basically, I'm just trying to see where you can shortcut, you know, leverage off the existing open model ecosystem. But it sounds like you can't, you have to do everything from scratch. Like, there's nothing to reuse.

• 8:48Speaker 02

  Well, you can use architectures. I think that at least the shapes that can be leveraged. So you don't have to reinvent necessarily completely different neural network architectures. And a lot of the data can be used. Like, I think perhaps there are optimizations and you can think about what kind of data sets work best for diffusion models. But it's pretty much compatible in terms of like the types of data that you would need to do pre-training, mid-training, post-training. It's all, it's all quite compatible, but you have to change. You have to innovate on the, on the training losses, on the objectives, which is actually exciting because you can do things that are not possible without the regressives models and suddenly they become possible with the diffusion.

• 9:27Speaker 01

  Yeah, okay. So yeah, no, broadly stepping back a little bit. Why do you think diffusion models, diffusion language models are working now? Um, you know, I think it's been talked about for, for a while, for the last maybe three, four years. But all the examples I've seen were always like very kind of, uh, yeah, not, not like, you know, is there a why now? Obviously you have some secret sauce, but like, I'm just trying to figure out like, is there anything fundamental?

• 9:51Speaker 02

  I mean, depends how you define fundamental. I think there was some math that developed in terms of like thinking about, I like to think of diffusion models in terms of like learning score functions and, and kind of like gradients, basically of probability density functions, which are not really well defined in the discrete world, but there are kind of like mathematical analog goes objects that you can kind of train using denoising like objective, like score matching like objective. So there was some pretty fundamental math that had to be developed.

• 10:21Speaker 02

  Then there's also a lot of engineering and, and tricks that to, to build on top. And then I would also add the fact that, you know, once somebody shows that it's possible, you know, suddenly other people are much more incentivized to, to get their own thing to work and, and I think everybody progresses more rapidly once there is kind of like a proof of concept that, that something is possible. And so, yeah, we've seen, you know, the, the Google guys also have a, have a model. So it's exciting to see that. The field is really starting to to pick up speed.

• 10:51Speaker 01

  Yeah, amazing. Yeah. Any thoughts on the Google side? You know, obviously you were working on this for a while. Was it a surprise?

• 10:57Speaker 02

  I mean, I know they've been trying for for a while, too. It was great to see that. My understanding was that, and I don't know exactly how they do it. I thought they were, um. You know, I don't know exactly what's their recipe, what's their architecture, what's their model, but it was, it seems to perform pretty well. It was it was exciting to see that at least some other player in the space was able to come up with a independently come up with a solution that actually works, because I'm pretty sure it's quite different from what we do at inception. So, yeah, it's it's exciting, I think.

• 11:27Speaker 01

  Yeah, like to to my knowledge, Gemini diffusion hasn't been widely released yet, but like, I'm curious how to test it. Like, if you got Gemini diffusion today, would you run anything? Is there is there any use case that you'd be very interested in? Any prompts? Or I don't know how to evaluate?

• 11:43Speaker 02

  So there is a playground, um, I think you.

• 11:46Speaker 01

  For Gemini diffusion? I didn't know that.

• 11:49Speaker 02

  Yeah, yeah, there's a playground. Yeah, yeah. So you can play with it. It's pretty fun. It does a lot of in feeling tasks pretty well, which is also kind of like what we've we are seeing. Diffusion models, not surprisingly, they work pretty well at the infilling, where you really need to be able to use context to the left and to the right. All are completed. One of the main use cases is also for for Mercury Coder. And it seems like their model is also doing pretty well, like kind of editing, in painting kind of tasks. To me, they've released some benchmark numbers. They seem to be pretty close to to the numbers that we were getting with the Mercury Coder back in some, you know, back in early this year. So,

• 12:30Speaker 02

  yeah, it's it's it's exciting that to see that they've also been able to get this technology to work.

• 12:35Speaker 01

  And to wrap on maybe the pre-training phase, is there also post-training in the same way that people do with normal L L M's? Well, not normal, but you know what I mean? Or like instruction following and things like that? Does that work the same as those models? Or is it different in diffusion?

• 12:50Speaker 02

  So our pipelines, yeah, we have something similar in the sense that we also use, we can do basically S F T and we can also do. We do reinforcement learning at LHF, we do reinforcement learning from human feedback. Again, some of the algorithms have to change. For example, we have a DPO algorithm specialized for diffusion language models. I was one of the original authors of the DPO paper that was originally designed for LLMs, auto-regressive LLMs. I worked on a paper extending DPO for diffusion models, more continuous diffusion models like image generation, video generation. And that's also been quite impactful. Like if you look at some of the papers that have been released on some of the leading open source image generative models based on diffusion.

• 13:34Speaker 02

  You can see that the DPO piece using human feedback to really align the models to human preferences is really important if you want to improve quality. And we've extended a variant of DPO to diffusion language models. And so we're actually also able to use preference data to to fine tune the models. And but yeah, basically, we have pipelines that are compatible with the existing SFT and RHF data sets. And we we do that with customers, like we often work with customers that have their own internal data sets. They want us to fine tune the models on their proprietary data and we can easily do that. It's all pretty easy once they share the data with us. They can read it.

• 14:12Speaker 01

  They can read it.

• 14:13Alessio Fanelli?

  Maybe give people also an idea of on the compute side when it comes to pre-training, how different the skill is. And then as you move to inference, also what that looks like, but in terms of throughput, compute requirements and also like context size. I don't know if there's much to say there, but if there's anything.

• 14:29Speaker 02

  Yeah, so unfortunately, I'm not able to share much on the training side in terms of like the amount of compute or the number of tokens that we're using. On the inference side, what we're seeing is that diffusion models are much more efficient. We're actually able to Pareto dominate auto-regressive models. If you think about the typical trade off between throughput versus latency, which you kind of like cannot can can can get by changing the inference parameters, like the batch size and things like that. What we're seeing is that we're actually able to Pareto dominate auto-regressive models. For the same throughput, we can get better latency, or for the same latency, we can get better throughput. And so throughput basically means cost effectively.

• 15:11Speaker 02

  So for the same cost, we can make the models go fast or for the other way around.

• 15:19Alessio Fanelli?

  Is that the killer use case of diffusion? Or are the model also just better?

• 15:26Speaker 02

  Right now, that's one of the real differentiating factors. Where we're seeing a lot of traction, like a lot of our users care about latency. So we're seeing a lot of use cases where that are latency sensitive. And so people really care about being able to give answers to their users quickly. Lots of applications where there is a real time, kind of like component, voice agents, where you're interacting with a person in real time, or IDEs, or like vibe coding kind of things, where you really want to be able to see results quickly. Like maybe you're vibe coding a website or a UI. So lots of these use cases where speed matters. We're seeing a lot of traction. And it can mean two things. It can mean either just giving even faster results to the users to improve the experience. Or sometimes the value proposition can be,

• 16:11Speaker 02

  people are currently using small auto-regressive models to meet the latency constraints. What you can do is you can put in a bigger diffusion language model. That's going to be faster because it's a diffusion model. And it's going to be higher quality because it's a bigger model, has just more parameters, more data. And so that's also been a very valuable kind of like use case of diffusion language.

• 16:34Alessio Fanelli?

  How should people think about the parallel frontier today? As far as capability of model? I know you had, you released some numbers on the coder model, which is roughly the same as 40 many, Mistral small three. Anything else on non-coding? Or maybe I missed it in all the announcements?

• 16:52Speaker 02

  Yeah, it's actually, it's been released recently. We had our generalist model evaluated by artificial analysis. And the intelligence score from AA, artificial analysis, around 40. So it's comparable to GPT 4.1 nano, Claude High Q, a kind of like closed source, speed optimized models. It's roughly comparable in terms of intelligence, but significantly faster, 5, 10x faster compared to this kind of speed optimized auto-regressive. We're working on next generation models that are going to be even more intelligent. But that's not currently publicly available.

• 17:25Alessio Fanelli?

  Is there something, so if you think about use cases, is there something that people should not be using diffusion models for? Or do this is just kind of, in your mind, it's like a superior, just swap everything over to these and then as we scaled them. It's funny because you're seeing image generation going towards auto-aggressive models and then you're seeing the text models go to diffusion. How do you kind of see the the model use case match?

• 17:47Speaker 02

  So the API is the same, it's like text in, text out. So in principle, any use cases that are supported by an auto-aggressive model is also supported by a diffusion language model. In terms of capabilities, yeah, as I mentioned, like we're not frontier level. And so there's certainly use cases where you probably wouldn't want to use a diffusion language model, or at least not the current existing diffusion language models. And so, yeah, I think at the moment it's more a function of kind of like, what's the intelligence level, how much do you care about latency or cost? So there's definitely a bunch of use cases where people, where diffusion language models are already competitive, they're already the best solution for those use cases. There are others that are still yet to come. And I think that will require scaling up and making further improvements

• 18:31Speaker 02

  to the to the training and the data and various kinds of things. I'm pretty optimistic about a future where diffusion models can become the dominant solution. I've seen it happen before with GANs a few years ago. So I wouldn't be surprised if that's the case also here. I really like this idea of using context to the left and to the right, the fact that you have error correction that is built in. So you think about an auto-aggressive model, once you output something, you can never take it back. And so if you want to do, you know, if you want to fix mistakes, maybe you can do a reasoning chain, maybe you can do it that way. But it's kind of unnatural, it's kind of clunky. Diffusion model has this nice thing of kind of fixing mistakes as you go. So I think it's going to be pretty exciting

• 19:15Speaker 02

  once we have next generation models, maybe with reasoning built in. I think we're going to see some some very interesting capabilities emerge there.

• 19:23Speaker 01

  Yeah, yeah. So in your mind, it's just a scaling question left, not a, you know, missing capabilities from the architecture, like limitations of of it.

• 19:32Speaker 02

  Right, yeah. No, I think there is gonna be potentially new improvements that can be, I mean, I'm sure that we're still pretty far from a local optimum. Like, if you think about autoregressive L L M's, people have been trying all kinds of things. And so maybe there's not that many low hanging fruits. What I'm, what is exciting to me as a researcher and as an engineer, if I think about the future language models, it's all very new. We have something that already works pretty well, but I'm sure that there is a lot of design choices that were suboptimal. Uh, there is still a lot of ideas to be discovered. And so there could be big leaps, big improvements possible just by doing R and D on the way we train the models, the way we do inference. You have a lot more knobs, it's a different design space. So there is a lot of exciting R and D to be to be done.

• 20:16Speaker 02

  And we're seeing it even at the company, like the improvements that we're seeing week over week are pretty substantial. And so it's, uh, it's, I'm personally very excited about this.

• 20:25Speaker 01

  Yeah, awesome. Um, one more thing about on the inference side. I think a lot of people are very cognizant of the serving requirements between autoregressive and diffusion. A lot of people rely on the batching characteristics in in production. You would not have that, right? Are you aiming to like, uh, you know, are you aiming to be to abstract that problem away for people? Or are you just gonna release the models open in open ways and then people will deal with it however they want?

• 20:51Speaker 02

  So we don't have a plan at the moment to to release models, to open source any model. And in fact, it's a little bit tricky even to, even if you wanted to release, let's say, a small, maybe less capable version of the models. We'd have to release the inference code, which is also proprietary. Unlike an autoregressive model, where the inference code is kind of like straightforward. There is not that many ways you can do inference in an autoregressive model. You can tweak a little bit simply, but it's it's pretty straightforward. In a diffusion language model, there is a lot more knobs and a lot more innovation that can happen on the inference side. And so it's very hard for us to open source a model because we have to release the inference code. And by the way, we have our own inference engine.

• 21:33Speaker 02

  So just like you would normally serve an LLM using VLLM or SGLang or Tensor or TLLM, we have built our own inference engine. And so we are supporting production traffic already with our own inference engine. We support continuous patch and quantization. We support a lot of the features that are supported in these serving engines, caching, like prefix caching. A lot of that stuff is already handled by our own serving engine. But yeah, it gets a little bit tricky if we wanted to open source because a lot of the IP is around on the inference side as well.

• 22:03Speaker 01

  You know, we have a bunch of inference nerds listening to us and who might be interested in joining you. What's like, you know, one underappreciated problem or something interesting that has come out of this production work?

• 22:14Speaker 02

  There is a lot of, just like the workloads look different and there is different opportunities for optimizing aspects of the serving, the way we serve models. So yeah, we have a bunch of smart people that are kind of like working, like NLCs, I would call them. I think, you know, I've been working in this space for a while with students at Stanford. I was on the flash attention paper. So I've been focusing on this kind of issues for a long time. And the things you see once you start using additional language models are quite different. So there is new opportunities for optimizing the models, the inference algorithms, the way we implement them and we write kernels to really make them go fast.

• 22:56Speaker 02

  So there is a lot of exciting opportunities for doing work. And yeah, if you're interested, reach out to us. We're hiring at all levels. So I'll use this opportunity to do this plug for applying to our jobs.

• 23:08Speaker 01

  Yeah, absolutely. I mean, this is, you know, if we can get people a job, if we can get you an engineer, I think that's a win for us doing this, right? Okay, so I want to move on a little bit in terms of like, we, let's say we have production and high quality LLMs. You made an interesting controversial statement earlier that you said most frontier models or most language models will use diffusion in the future. I don't think that's consensus. Could you just elaborate why?

• 23:33Speaker 02

  I think that there is a chance. I mean, I don't know if that's gonna happen, but I feel that is, there is a world where that could happen. And I think it could be driven, if it happens, it's gonna be driven by efficiency. Like, we're all constrained by essentially power. And if you have, I mean, at the end of the day, it's all an inference game, right? Okay, training is expensive, but then the thing that matters is being able to serve these models cheaply and without consuming more energy that we have access to. And you only have so many gigawatts of power and so many data centers. And there is a growing demand for tokens that you need to be able to serve to users and all kinds of apps that are being built. And so there is a real need to find a solution that can give us maybe a 10x improvement on the efficiency side. And there's a really good chance that diffusion models are the thing.

• 24:19Speaker 02

  So if indeed it's possible to match the quality of frontier models in terms of quality, if you have a win on the inference side, that is gonna be the solution that will be, that will be deployed, right?

• 24:32Speaker 01

  Yeah.

• 24:33Speaker 02

  It's, at the end of the day, it matters. There is a real resource constraint. And so the better solution will be deployed and will win and will be scaled up.

• 24:43Speaker 01

  Yeah. I would just note that it's interesting that you guys on your paper, you noted that your context length goes up to 128K tokens and you use normal training recipes. That is the one thing, if I think about autoregressive versus diffusion, that you don't have a cache that you can pre-fill. But I'm sure you figured it out somehow, right? Like, you know, but I think that's kind of cool that, like, that just translates over.

• 25:05Speaker 02

  Yeah, there are interesting caching strategies that can be implemented, including some that have been in papers. There is some growing literature on

• 25:14Speaker 02

  KV cache-like mechanisms for diffusion language models. So you can find something in the literature as well, from NVIDIA, from academic groups. So there is definitely a lot of optimizations that can be done.

• 25:26Speaker 01

  Yeah. Yeah, yeah. So I would say like quality, it's anyone's game. It's really gonna be amount of compute, data, all that. Not necessarily driven by just the algorithm itself. But like, I think you unlock a lot of different use cases, which is very cool. And it's interesting that you released the ChatTune model because actually, maybe the focus should be everything but chat. Like chat, fine, auto-regressive, right? Back and forth, like sequential, okay. But like, everything else, like, nobody's releasing any base models. And actually, you can, you can offer something very different there. That, that'll be my, my pitch to you in terms of like, you don't have to fight with auto-regressive language models head to head on quality. But actually, you could just do everything else because they just don't, they cannot do it.

• 26:12Speaker 02

  Yeah, yeah, for sure. Yeah, I think that we're definitely, you know, exploring the market, working with customers, figuring out where we have an edge. Are there use cases where we're particularly good compared to auto-regressive models? Capabilities that are just not, like, for example, around controllability, like, or thinking about hallucinations, I don't control, like, there is all kinds of interesting things that are in theory possible with, with fusion language models. And so we're also excited to explore that. It's a lot easier to provide an open eye compatible endpoint that anybody can build on top of.

• 26:43Speaker 01

  Yeah, yeah, yeah. That's the, that's the industry standard. I mean, I think there's a demand for new form factors and new capabilities. And that would be your relative strength, you know?

• 26:52Speaker 02

  Sure.

• 26:52Speaker 01

  So I think you mentioned customers. I think that the, the last thing I wanted to touch on was just, what kind of customers are you looking for and that you're ideal for? Like, if someone says, I have this problem, they should come to you. What, what is that problem?

• 27:06Speaker 02

  Yeah, whenever they have applications where they are not able to get the level of quality they need from existing models, typically because they, they need to use one of these closed source speed optimized models or they're fine tuning a small auto-regressive model. So any kind of applications that is latency constrained, where customers are not able to get the quality that they need, diffusion language models can be a great alternative. We can increase the quality of the answers by plugging in a bigger model. Or we can make the existing kind of form factors go much faster and increase the user experience and improve the user experience. So any kind of like latency sensitive applications of other LAMs, diffusion models right now have a, have a real edge.

• 27:46Speaker 01

  Yeah, amazing. Stefano, thanks for coming on.

• 27:49Speaker 02

  Thanks for having me, yeah. It was really good to meet you.

• 28:02Speaker 02

  The