#310 Stefano Ermon: Why Diffusion Language Models Will Define the Next Generation of LLMs

This episode is sponsored by AGNTCY. Unlock agents at scale with an open Internet of Agents. Visit https://agntcy.org/ and add your support. Most large language models today generate text one token at a time. That design choice creates a hard limit on speed, cost, and scalability. In this episode of Eye on AI, Stefano Ermon breaks down diffusion language models and why a parallel, inference-first approach could define the next generation of LLMs. We explore how diffusion models differ from autoregressive systems, why inference efficiency matters more than training scale, and what this shift means for real-time AI applications like code generation, agents, and voice systems. This conversation goes deep into AI architecture, model controllability, latency, cost trade-offs, and the future of generative intelligence as AI moves from demos to production-scale systems. Stay Updated: Craig Smith on X: https://x.com/craigssEye on A. I. on X: https://x.com/EyeOn_AI (00:00) Autoregressive vs Diffusion LLMs (02:12) Why Build Diffusion LLMs (05:51) Context Window Limits (08:39) How Diffusion Works (11:58) Global vs Token Prediction (17:19) Model Control and Safety (19:48) Training and RLHF (22:35) Evaluating Diffusion Models (24:18) Diffusion LLM Competition (30:09) Why Start With Code (32:04) Enterprise Fine-Tuning (33:16) Speed vs Accuracy Tradeoffs (35:34) Diffusion vs Autoregressive Future (38:18) Coding Workflows in Practice (43:07) Voice and Real-Time Agents (44:59) Reasoning Diffusion Models (46:39) Multimodal AI Direction (50:10) Handling Hallucinations

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  When you think about most existing LLMs, ChatGPT, the Gemini, they're also called autoregressive. And what that means is that when you ask a question, they will give you an answer and it will generate the answer. One word or one token, as we call it, at a time. And that generation is sequential and that's kind of like a structural bottleneck that cannot be overcome because it's kind of like built in into the autoregressive formulation. A diffusion language model is also using a neural network under the hood. And in our case, it's also a transformer neural network. But the task that the model is trained on is quite different. The model is not trained to predict the next token. Instead, it's trained to remove noise.

• 0:40Craig S. Smith?

  Build the future of multi-agent software with AGENCY. That's A G N T C Y. Now an open source Linux Foundation project. AGENCY is building the Internet of Agents. A collaborative layer where AI agents can discover, connect and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on AGENCY. Including robust identity and access management that ensures every agent is authenticated and trusted before interacting.

• 1:24Craig S. Smith?

  AGENCY also provides open standardized tools for agent discovery, seamless protocols for agent-to-agent communication and modular components for scalable workflows. Collaborate with developers from Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat and more than 75 other supporting companies to build next-generation AI infrastructure together. AGENCY is dropping code, specs and services, no strings attached. Visit agntcy.org to contribute. That's A G N T C Y.

• 2:10Craig S. Smith?

  I dot O R G.

• 2:11Speaker 00

  Hi everyone, uh, my name is Stefano Ermon. Uh, I'm one of the founders and the CEO of Inception. Before uh, starting Inception, uh, I was a professor of computer science at Stanford for about a decade. Um, I've been working on uh, generative models, what's now called generative AI. Uh, yeah, for about ten years, uh, I did very early work on uh, diffusion models, the technology that is now powering a lot of the state of the art image generation systems, video generation systems. Um, I've worked on a variety of techniques like flash attention, DPO,

• 2:50Speaker 00

  that are now widely used in industry to build large scale at the labs and other kinds of generative AI solutions. Uh, I've been uh, working on uh, diffusion language models for a while in my lab at Stanford. So kind of trying to figure out how to get diffusion models to work, not just on images and videos, but also on text and code generation. Uh, had some breakthroughs in my lab uh, last year where we showed that for like for the first time, this technology was able to compete with the leading uh, auto-regressive models. Uh, and then that prompted me to start the company to scale up the tech, the, the models. And uh, that's where I've been for the last year or so. And uh, yeah, we've been pretty successful at

• 3:34Speaker 00

  training the first commercially scaled diffusion language models that we call Mercury.

• 3:38Craig S. Smith?

  Yeah, and, and uh, I'm curious, I mean, it's fascinating that diffusion language models work. Uh, and maybe we'll talk about that first, but, but, but, but I want to ask why? Is there an advantage to a diffusion language model over an auto-regressive transformer-based L L M?

• 4:03Speaker 00

  Yes, for sure. And the reason is that when you think about most existing LLMs, like the ChatGPT, the Gemini, the Claude, they are also called autoregressive. And what that means is that when you ask a question, they will give you an answer and it will generate the answer, one word or one token, as we call it, at a time. And that generation is sequential. And that's kind of like a structure on bottleneck that cannot be overcome because it's kind of like built in into the autoregressive formulation. The neural network is trained to predict the next token and that's how you use it at the inference time when you ask a question to it. And sequential computations are generally very hard to accelerate

• 4:48Speaker 00

  because there is not so many, there is not many things you can do in parallel if you have to go through the tokens by the time. On the other hand, a diffusion language model is generating the answer all at once. And so it's, the neural network is trained to refine its answers to kind of like fix mistakes. And then at the inference time, you kind of like use this procedure where each neural network evaluation is able to essentially modify multiple tokens at the same time instead of just one. And so what we see is that these models can be much, much faster compared to autoregressive models of similar quality. And that's kind of like the key benefit.

• 5:30Speaker 00

  We're able to drastically speed up the generation, speed, the time it takes basically to generate answers with an LLM and reduce costs. That's the other big benefit. The computation is more parallel. We can make use of parallel compute of GPUs more effectively and we can effectively reduce costs for our users.

• 5:51Craig S. Smith?

  Yeah, the other, I don't know that this is an advantage, but one of the problems with LLMs, and I've been talking to people about sort of post transformer architectures. But one of the problems with LLMs is

• 6:07Craig S. Smith?

  Context, the compute required for for context, for the context window scales quadratically, so it very quickly becomes too expensive. And then there are all kinds of problems with with context windows that are too large. The L L M gets confused or lost. Does is is that something that diffusion L L Ms would address?

• 6:42Speaker 00

  Yeah, that's a great question. And yeah, I'm very familiar with these issues. I was one of the original authors of the flash attention paper that is widely used to yeah, accelerate the the kind of like very expensive quadratic computation that is required to to do attention in a transformer.

• 7:01Speaker 00

  I we have shared in the past, and I'm happy to to confirm that our diffusion language models are also based on transformers under their hood. So the architecture that we use is still a transformer with with self attention. And what this means is that we got the same pros and cons with respect to the context length that you would expect from an auto-regressive L L M. And so the current models that we serve in production have about 130 K tokens of context. We think we can expand it using the techniques that people have used for auto-regressive

• 7:41Speaker 00

  L L Ms, but in terms of pros and cons, there is not really any benefit or any downside compared to to auto-regressive models. We've also used different architectures. So just like people in the auto-regressive L L M world that they've been experimenting with state space models and MAMBS and alternative architectures, that's also possible in the context of diffusion language models. You can use any neural network architecture you want inside a diffusion language model. And we have some prototypes where we've replaced transformers with this alternative.

• 8:17Craig S. Smith?

  Architectures and you get the benefits and the downsides of those architectures.

• 8:22Speaker 00

  So

• 8:23Craig S. Smith?

  Yeah.

• 8:23Speaker 00

  Diffusion versus architecture, those are kind of like two orthogonal axes that you can use to accelerate the computation or reduce the memory footprint or kind of like trade off of different kinds of performance characteristics you might care about.

• 8:39Craig S. Smith?

  Yeah, well, for listeners and for myself, frankly, can you explain how diffusion works? And how transformers play a role?

• 8:54Craig S. Smith?

  Yeah, just, I mean, I know that there's, you inject noise during training and then the, the, the, it sort of reverses the noise down to the output. But yeah, if you could explain it for a layman, that would be terrific.

• 9:15Speaker 00

  Sure. So both autoregressive L L M's and diffusion L L M's rely on neural networks under the hood. So there is a neural network that takes as input a sentence and then outputs something. And in the context of a autoregressive L L M, this neural network, which is often a transformer, takes as input a sentence and then it predicts what is the next token, what is the next word that I should append to that sentence to, to make sense. And you can train this neural network to do a pretty good job at it by just feeding a lot of text or a lot of code to these models. And then eventually, after lots of training iterations, they become pretty good and understanding

• 9:58Speaker 00

  what kind of tokens make sense and how you should complete sentences. A diffusion language model is also using a neural network under the hood. And in our case, it's also transformer neural network. But the task that the model is trained on is quite different. The model is not trained to predict the next token. Instead, it's trained to predict, to remove noise. So again, you start with sentences of code that can be maybe, you know, taken from the internet. And then what you do is you artificially add noise. Perhaps you mask some tokens or you change the value of some of the tokens. And then you train the neural network to.

• 10:40Speaker 00

  Take as input this corrupted sentence and try to fix the mistakes. Try to give you back the original sentence. And so it's a it's a different training objective. It's not next token prediction, but it's more like a global transformation of the sentence, where you try to fix the mistakes. And at the inference time, these two models are used in a very different way. The traditional autoregressive model is just going to predict the next token. Then you fit it in and then you predict the next token and then you fit it in and you predict the next token and so forth. A diffusion language model instead, it starts with a guess of what the answer should be. And in theory, it can be anything. And then you pass this whole guess through the neural network. The neural network will fix some of the mistakes.

• 11:26Speaker 00

  And crucially, it will fix the mistakes across all tokens, across all words that form the sentence. So it's much more parallel. And then you repeat this denoising process a bunch of times. And then at the end, you get an answer that you that we serve to to our users. And as long as you don't need too many denoising steps, you don't need too many diffusion steps, this can be way more efficient. Because again, each neural network evaluation is able to modify multiple tokens as opposed to just one.

• 11:58Craig S. Smith?

  Yeah, in in the autoregressive model, I understand

• 12:06Craig S. Smith?

  the the token prediction and and the attention mechanism. So it's it's really looking for the most probable next token.

• 12:20Craig S. Smith?

  In diffusion, I I don't understand, because it's not predicting the next token. So how does the model know the answer? I mean, no, is is is it looking across a probability distribution of of the input?

• 12:42Craig S. Smith?

  Yeah, I just, I don't, I don't understand that at all.

• 12:47Speaker 00

  Yeah, so it's a, you know, like, at different time, yeah, the model never knows what's the correct answer, but during training, you have ground truth data, you have text and code that you've collected on the, on the internet, for example. And, and so you do know what is the, what is the correct answer. So you do know what, what was the right token that came before, that came after a given, a given context, a given sentence. Similarly, in the context of a diffusion language model, we start with ground truth text. We artificially add noise, we introduce some mistakes, we mask out some of the tokens. And then we feed this corrupted sentence to the neural network and we ask, we train

• 13:32Speaker 00

  the model to reconstruct the original signal. And so during training, we do have access to ground truth, just like in the context of hypergressive models. It's just a different kind of task that the neural network is trained on. It's trained to predict and modify multiple tokens at the same time. And under the hood, it does this by using attention. And so what it does is it takes as input this corrupted sentence. It does attention and so it kind of like tries to compare every token with every other token. It tries to figure out what is the meaning. It tries to figure out what should have been the right words that go at every, in every spot.

• 14:13Speaker 00

  And then it outputs its best guess of what the, the fix should be. And yeah, they're not perfect, but if you train them for a sufficiently long amount of time on a sufficiently large amount of data, they actually become pretty good at correcting mistakes, at fixing mistakes. And that's why they actually work quite well during, during inference time.

• 14:36Craig S. Smith?

  Yeah, and, and I'm gonna break this down and ask, ask, drill down a little bit to clear up my understanding. So in the training, you're, you're taking existing language.

• 14:59Craig S. Smith?

  You know, diffusing it or, you know, whatever the term is so that it's full of noise and de-noising it back down to the original sentence.

• 15:10Speaker 00

  That's right.

• 15:11Craig S. Smith?

  But where does the the prediction come from for to answer the question, is that purely from the attention mechanism? Or is there some next token prediction happening in the background before,

• 15:33Craig S. Smith?

  you know, the de-noising?

• 15:35Speaker 00

  Yeah, so it's not just the max token in the sense that it's that that's the other kind of like advantage of the model is that it's not necessarily just, you know, sequentially going through one token at a time. It's actually able to use context to the left and to the right to fix mistakes. So say like if the first token is missing and everything else is known in the sentence, then the diffusion language model can try to predict what should the first token be, given that I know everything else. And so it's kind of like a generalization of the task that you would normally do in an auto aggressive model, where you always just predict one token left to right.

• 16:16Speaker 00

  In a diffusion language model, you have a richer kind of set of tasks that you need to be able to solve during training, which could be just predicting the next token, but maybe it's predicting the first token, given that you know everything to the to the right of it. And so it's more like right to left prediction. And it's actually even more complicated because there is all kinds of mistakes that could be there in the sentences. And the model needs to figure out how to fix all of them at the same time. But in some sense, they are they're kind of like similar in the sense that in both cases, you you try to reconstruct part of the input. And in the context of auto aggressive language models, you always reconstruct left to right, one token at a time.

• 16:59Speaker 00

  In the context, in the context of a diffusion language model, you reconstruct the sentence all at once globally. And so it's a it's a it's a more complex kind of like reconstruction, de-noising task that but we can still train neural networks to to solve it.

• 17:19Craig S. Smith?

  Yeah, on image diffusion models.

• 17:23Craig S. Smith?

  Uh. They're sort of notoriously hard to control. Uh. They'll certainly uh give you an output, but it it it may not be what you intended. Uh. Even if it fits the prompt. Uh. How how does that? I mean, and there's been a lot of uh

• 17:49Craig S. Smith?

  progress on that, but but how does that problem relate to language?

• 17:54Speaker 00

  Yeah, in my opinion, a diffusion model is actually much more controllable than a typical autoregressive model. And the reason is that in a diffusion model, uh, you kind of like have access to the whole object uh from the very beginning. And so you kind of like know from the very beginning whether or not it's consistent with the prompt, whether or not it satisfies some safety constraints or whatever, whatever is the the control signal that you want to use to steer the generation. Uh. Throughout the generative process, you always know uh whether or not you're going in the right direction, whether or not the constraints are satisfied. And and that's why actually diffusion diffusion models are relatively easy to control and there is a a lot of techniques

• 18:38Speaker 00

  that people have come up with to steer the generation process in a certain direction. Um. Because again, they can kind of like check whether or not you're satisfying the constraints and you can kind of push the generation in the direction that you want. And in fact, uh and in instead, if you think about an autoregressive model, you kind of like need to wait until you get to the very end, to the last pixel to know whether or not uh this is satisfying the constraints. And so that's why uh we we, in my opinion, diffusion models are more controllable than autoregressive models. And we're seeing uh or we're starting to explore some of these benefits also in the context of diffusion language models, where uh yeah, there is there is still quite a bit of uh.

• 19:21Speaker 00

  Yeah, it's it's a problem that that comes up sometimes that you you want the model to behave in a certain way. Of course, you want the model to be safe. Uh. But maybe you want the model to be on brand, like there are there are more complex ways of defining what you want uh the model to do to to and we are starting to explore ways to use our diffusion language technology to uh allow our customers to control the the the language models in a more fine-grained way.

• 19:48Craig S. Smith?

  Yeah, uh, the um.

• 19:53Craig S. Smith?

  Okay, let me gather my thoughts here.

• 19:57Craig S. Smith?

  The, the, uh, so.

• 20:01Craig S. Smith?

  Well, I've got a bunch of different questions. One is about training. How do you train these? The other is about.

• 20:11Craig S. Smith?

  Guiding them to, uh, or, or, uh. You know, the, in, in autoregressive models, they use, uh.

• 20:23Craig S. Smith?

  Reinforcement learning with human feedback is, are they the same kinds? Can you, because I'm going to cut my stumbling around here out. What, how do you train these? Is it the same process as training, uh, an autoregressive model?

• 20:42Speaker 00

  Yeah. So, uh, at a high level, it's kind of like what I discussed, like it's a, we're training this neural networks to, uh, you know, is, uh, and, uh, the, the inputs to remove noise from sentences. We haven't disclosed a lot of the details in terms of like the exact training objective or the exact size of the models or the amount of training data that we use. There are certain papers in the, that are, you know, publicly available are in the open where people have studied. You know, the, the scaling laws and the performance that you see if you train the fusion language models versus autoregressive models.

• 21:24Speaker 00

  And there is some evidence in the, in the academic literature that the fusion language models are much more data efficient. In the sense that kind of the intuition is that because you, you're training the model to solve a many different kinds of tasks, not just predict the next token, but you know, let's say predict, uh, in any order, uh, then they, they tend to be more data efficient. So you need less training data to achieve a certain level of quality. And of course, everything depends on the devil is in the detail and there is all kinds of things that you can do, but we have our own recipe and we've seen that it does really, really well. On top of that, it's also possible to do reinforcement learning from human feedback.

• 22:08Speaker 00

  It's also possible to do reinforcement learning using verifiable words if you are in the context of, let's say, coding or places or math, where there is a verifiable word, there is a uniquely correct or wrong answer that you can use to check the quality of the results. And internally, we've been able to use all of these techniques and adapt them to the, to the different kinds of generative models that we are, that we're dealing at this action.

• 22:35Craig S. Smith?

  Yeah. And, and, and so for evaluation, do you, do you use other diffusion models for evaluation? Or do you use autoregressive L L M's in, in your evaluation?

• 22:49Speaker 00

  So it's, it's, I mean, evaluating L L M's is notoriously difficult. And the way we track performance internally is by using benchmarks. So there are a variety of benchmarks that the, the community has developed that are kind of like trying to measure how good the model, the models are at different tasks that people care about, like how good they are at math, how good they are at following instructions, how good they are at doing question answering, how good are they at writing code in different kinds of languages? And so internally, what we do is we monitor the performance of our models across all

• 23:31Speaker 00

  the variety of benchmarks. And then we keep adjusting our training methods and the architectures and the inference schemes to try to get as good a performance as we can, while keeping the computation costs down, trying to also make the models as efficient as possible. So on the evaluation side, it's actually, it's exactly the same. It's all very, very similar to what you would do with a traditional L L M.

• 23:59Craig S. Smith?

  Yeah. And what are the sizes, parameter sizes of the models that you have today?

• 24:07Speaker 00

  Yeah, unfortunately, we cannot disclose the the exact size of the models. Yeah, we are in a very competitive kind of space and that's information that we're not able to disclose at this time.

• 24:18Craig S. Smith?

  Well, that's interesting, a competitive space. Are there other people working on diffusion models? A diffusion L L M's?

• 24:27Speaker 00

  Diffusion L L M's, yes. So there is a fairly active community in academia that is training diffusion language models, relatively small scale models, including from my lab at Stanford, I mean, early work at the G P T two kind of size, where we trained models with less than a billion parameters, let's say. And that, you know, it's very valuable kind of research because you, it allows you to compare apples to apples, like a small autoregressive model, a small diffusion language model, and see which one scales better.

• 25:04Speaker 00

  But these models that are sort of like either open source or that have been developed in academia, they are typically not the, they don't perform particularly well because the scale is relatively small and the the kind of data that was used to train them is not exactly the the best that is available out there. There have been attempts at building diffusion language models in other, in the industry labs. There's been a Gemini diffusion paper and release, blog post at least, where researchers from from Google, from DeepMind have talked about their internal efforts around building a diffusion language model.

• 25:48Speaker 00

  So far, it's not been deployed in production, like it's a, it's a demo, but it's not available, as far as I know, to customers. It's not serving production traffic. So as far as I know, Inception is the only player right now with a working model that is used to sort of production traffic, who were the first ones to release a commercial scale model. And we're still way ahead of the competition in terms of like how mature our models are and our cervix engine and our kind of like inference stack compared to anybody else. But there is certainly research attempts in different places and people are actively exploring this this direction.

• 26:30Craig S. Smith?

  Yeah, and and the the scaling of

• 26:35Craig S. Smith?

  The transformer-based models, that's part of what's made them attractive. It's linear scaling, not in the context window. But is it the same? Do you benefit from the same scaling laws in these D L L M's?

• 26:54Speaker 00

  Yeah, there's certainly certain we're observing is predictable sort of like behaviors that we can use to guide our experiments and decide what kind of architecture and design choices to prioritize. And again, there have been papers in the literature published showing that diffusion language models can be can scale better than ultra-aggressive L M's. And that's part of the reason we're so excited about this. It's not only extremely efficient at inference time, so the scaling at inference time is quickly better than what's possible with an ultra-aggressive model, which in my opinion is the thing that matters because ultimately it's all going to be an inference game.

• 27:40Speaker 00

  People are building these massive data centers. It's not for training. It's because they expect everyone will be using A I, generative A Is, L L M's. They're going to be everywhere. There's going to be a massive need to serve these models efficiently, quickly, cheaply. And the technology that scales better is the one that is going to win. And that's why we're so excited about this. But also on the training side, we've been able to quickly come up with models that are competitive with, you know, state-of-the-art, speed-optimized models from Frontier Labs. And so we're making rapid progress in terms of like increasing the capabilities.

• 28:17Craig S. Smith?

  Build the future of multi-agent software with AGNTCY.

• 28:25Craig S. Smith?

  Now an open source Linux Foundation project, AGENCY is building the Internet of Agents, a collaborative layer where AI agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on AGENCY, including robust identity and access management that ensures every agent is authenticated and trusted before interacting. AGENCY also provides open standardized tools for agent discovery, seamless protocols

• 29:08Craig S. Smith?

  for agent-to-agent communication, and modular components for scalable workflows. Collaborate with developers from Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat, and more than 75 other supporting companies to build next-generation AI infrastructure together. AGENCY is dropping code, specs and services, no strings attached. Visit agntcy.org to contribute. That's A-G-N-T-C-Y dot O-R-G.

• 29:49Craig S. Smith?

  Yeah, and you're talking about Mercury is the publicly released model, right? You're talking about it as a code assistant, is that right primarily? Is there a reason why you're focused on writing code?

• 30:09Speaker 00

  Yeah, so we started with code because it felt like, well, first of all, we're all computer scientists, and so code is pretty close to our heart, and we understand if the model is doing well or not. It's also a pretty interesting application for diffusion language models because it's a little bit less auto-regressive, in the sense that this bias of, you know...

• 30:33Speaker 00

  Going left to right, kind of like makes sense if you're thinking about writing a document like it's kind of like how we write, although I would argue that maybe there is still a little bit of course to find generation where we start with the with an outline and then we fill in the details, but at least I would say if you think about code, there is a lot less left to right bias, like if you think about different files that are not necessarily in a particular order. There are many more tasks where you really want to be able to look at a whole code base and infill a particular function or functionality that you're trying to develop. And that's where the future models can shine because they are not constrained to be to go, they're not being,

• 31:16Speaker 00

  you know, this idea of going left to right is not baked in. And so that's why we kind of like focused on code initially. Because yeah, it's an area where we were expecting to see big improvements. And also it's very commercially and it's super important from a commercial point of view. That's where we're seeing a lot of the usage of L L M's where people are getting most of the value, I would say today. From using L L M's to write code faster and find the coding and all those kinds of things that people are so excited about. And so it was a pretty natural kind of like first step for us.

• 31:56Craig S. Smith?

  Yeah. And and can these models be fine tuned in the same way that auto regress?

• 32:04Speaker 00

  Absolutely. Yes, we we can fine tune them for specific tasks. And in fact, that's one of the modes of when we engage with enterprise customers. We sometimes are able to, you know, if they're able to provide

• 32:20Speaker 00

  internal data or or specific data that they care about. We're actually able to fine tune our models on their data and deliver even higher performance on the on the task they care about.

• 32:33Craig S. Smith?

  Yeah. And and is it the same as with, I mean, you you said you can't disclose parameter counts, but is it the same sort of progression that the larger the the number of parameters, the more accurate the model gets?

• 32:52Speaker 00

  Yes, typically, yes. The bigger the model, the higher, I mean, as long as you trained with

• 33:02Speaker 00

  proportionally larger amount of training data. So there are some caveats, but typically you can expect larger models to perform better, but they also become more expensive and slower. So that's kind of like the trick that you're facing.

• 33:16Craig S. Smith?

  Yeah. And from your point of view, the advantage is is speed and cost or power efficiency or not accuracy or or yeah, how do you measure accuracy against the retro? I mean, the autoregressive?

• 33:35Speaker 00

  Yeah, so it depends on you. If you think about a different L L M's that exist today, there is always a trade off between quality, like accuracy, cost and and speed. And, you know, if you want the the fastest models, they are typically not the most accurate because they are smaller, because they are, people need to use all kinds of, you know, tricks to make them as efficient as possible. And so what we can achieve is we are actually for a certain speed profile, we are the most accurate models, right? We are not the most accurate models overall. So there are larger

• 34:14Speaker 00

  models that have been trained by, you know, OpenAI and Atropic and Gemini that are more accurate in an absolute sense compared to where we are today. But in a certain speed class, we are the most accurate model. And so the way we're thinking about it is that there is a parade of frontier between quality and speed. We've been able to shift that parade of frontier, or at least for a certain quality level. And the next stage of the company is to kind of like keep improving the quality of diffusion language models until we actually catch up with frontier level quality intelligence that right now it's only available in in a few models that that are only autoregressive.

• 34:59Craig S. Smith?

  Yeah. And I mean, do you see diffusion language models as an alternative to autoregressive L L M's? Or or do you do you see?

• 35:17Craig S. Smith?

  Diffusion L L Ms as as you know, competing with uh autoregressive L L Ms and with the hope that someday they'll surpass

• 35:32Craig S. Smith?

  autoregressive L L Ms.

• 35:34Speaker 00

  Yeah, we certainly uh like they're they're functionally out of placement of each other. Like whenever you can use an autoregressive L L M, you can also use a diffusion language model. So from that perspective, uh they are certainly competing solutions for the same problem.

• 35:51Speaker 00

  Even right now, um we are surpassing autoregressive L L Ms and our customers are switching from autoregressive L L Ms to diffusion language models, uh because there are a number of low latency kind of applications where you wanna you need to be able to give an answer quickly to a developer or to a customer because you're pulling a voice agent. There are a variety of applications where latency matters and our models for that latency budget are actually more accurate than than the autoregressive models that people would use uh in order to be sufficiently uh sufficiently quick at outputting an answer. And so the question, of course, is, you know, what will happen as we scale our models? As diffusion language models get better?

• 36:37Speaker 00

  We don't know. Um, I I I can see a future where uh all models will be diffusion based. Usually, if I look at the back at the history of computer science, the more parallel solution is the one that wins. And the diffusion language models are built to be parallel from the grounds up. And so that's why we're betting on this. But we don't know. It's also possible that multiple technologies will exist. Then maybe one would be better for certain use cases. Another one would be better for other use cases. It's entirely possible that they're gonna make, you know, maybe neither of them will be perfect. They're gonna make mistakes, but as long as the mistakes are different and they are not correlated with each other,

• 37:19Speaker 00

  uh then there is gonna be value in having agents that are maybe autoregressive, based on autoregressive models, based on diffusion models, they can interact with each other, they can learn from each other. And you're gonna get value by having um a completely different intelligence stack to build intelligent machines effectively.

• 37:38Craig S. Smith?

  Yeah, uh actually, that's an interesting idea. You you could have a system where the The autoregressive model is checking the work of a diffusion model and back and forth to refine the answer. Is that something you've experimented with?

• 37:56Speaker 00

  Not yet, but I think, yes, it's something that in the future, I can see how that could work. There's going to be humans, there's going to be agents, and the agents are going to be based on different technologies. As long as they complement each other in interesting ways, and they're going to be useful in combination. And, you know, the sum is going to be greater than the parts.

• 38:18Craig S. Smith?

  Yeah. And for coding, how do you see people using it?

• 38:26Craig S. Smith?

  Is it really as a coding assistant? Is it, how accurate is it compared to some of the best coding models? Are people writing complete code bases using a D L L M? Yeah.

• 38:44Speaker 00

  Yeah. So if you think about code assistance, there is different types of use cases of L L Ms. The simplest one and the first one that was actually deployed is autocomplete. It's kind of like, you know, you're writing code and the L L M will suggest how you should complete that line or those lines of code. That's where we started. That's an application that is very latency sensitive. Like a developer is not going to wait a minute to get a suggestion. Like the suggestion has to be instant. It has to be very quick for the developer to be in the flow and for the experience to be good.

• 39:23Speaker 00

  And what we have right now, our Mercury models, are state of the art in terms of like code completions that they provide. There is this. Benchmark called CouPilot Arena. It's basically, it's run by researchers at TMU. And it's a, it's a benchmark that they use to evaluate different kinds of L L M's specifically for autocomplete. And the way it works is that they have an IDE. And they, there are developers around the world that are using this IDE. And as they write code, they get to see not just one completion, but they get two completions. And these two completions, they're coming from two different models.

• 40:07Speaker 00

  The developers, they don't know which models they are being, they're using. And then they vote which one is better. And then they come up with an ELO score, an ELO ranking, same thing that people use for, for chess players. To kind of like compare different L L M's in terms of like their quality. As measured by how often developers prefer one L L M versus the other. And our Mercury models are currently number one, tied with a few other L L M's. But we are number one in terms of quality. And we are number one by a, by a fair margin in terms of speed. So it's, it's already quite good as evaluated by external benchmarks.

• 40:47Speaker 00

  And even if you look at adoption, you know, where the default in quite a few coding IDEs. And in a few other places where developers can choose the model, they, they bring your own key, where developers can plug and play different models. There is a lot of developers that are using Mercury models to, you know, generate better autocomplete suggestions and in their, in their workflows.

• 41:12Craig S. Smith?

  Yeah.

• 41:13Speaker 00

  Yeah. Space where we're seeing a lot of adoption is NextEdit. Just kind of like autocomplete, a more general version of autocomplete. Where you're not just suggesting how to complete a sentence, but you're suggesting more general edits. Maybe you want to delete some code. Maybe you want to rename some variables in, in the code based on some changes that you're doing. That's another space where we have a very, very good model that is being deployed as the default, again, in a, in a variety of IDEs. With the latest model. We're also able to do a more agentic kind of use cases, the kind of things you were referring to where maybe

• 41:54Speaker 00

  a developer or somebody who is not maybe even an expert can just ask the model to write code to build an app from scratch. What people call vibe coding, right? And with the latest release of the Mercury model, we're now actually able to support also those use cases. And because the models are so fast, the experience is actually a little bit different. Like you get answers quickly and then you, you know, if you want to still interact with the model, you want to provide suggestions. You want to ask the model to make changes, the workflow and the, it is much more interactive. And so we're seeing a lot of usage.

• 42:33Speaker 00

  I think based on the fact that people like to be able to interact more closely with the, with the LLM, provide suggestions. And this idea of the LLM will do everything in one, in one go. I don't even know if that's ever going to be the, the future. I think there is always going to be a need for guidance and interaction. And if you want to be our human in the loop, speed is the limiting factor. You want to maximize the number of interactions, the back and forth between the LLM and the developer. And that's where the future LLMs are shined compared to other rest models.

• 43:07Craig S. Smith?

  Yeah, that's fascinating. Where, where do you see this going? You're going to remain focused on code generation, is that right?

• 43:17Speaker 00

  Code is a big area of focus, but we have customers that are using the models for other applications that are latency sensitive. One big one is voice agents. So if you're building a voice agent, the typical pipeline involves ASR. Then there is an LLM that perhaps does sometimes does tool call it, maybe it will check the menu or it will check the calendar. If you're making reservations or, and then the LLM will suggest an answer that then will be passed through a text to speech system that then communicates back to the, to the user. And whether it is his customer support or is like educational agents. They are all kind of like built on, on a very similar kind of framework.

• 44:03Craig S. Smith?

  Yeah.

• 44:05Speaker 00

  Yeah, the LLM in the middle is a key bottleneck in terms of like the overall latency. And what we're seeing is that people are getting really, really good results by replacing the small auto-aggressive LLMs that they were using before with the diffusion language models.

• 44:22Craig S. Smith?

  Yeah, and going forward, is it a matter of, I mean, are you, are you, are you still doing research in this area? And is it a matter of scaling up the size of the models? Or you mentioned before, replacing the transformer element with, with some other architecture with Mamba, for example.

• 44:49Craig S. Smith?

  Is, is, are, do you see that there's some other direction that you may be going with?

• 44:59Speaker 00

  Yeah, for sure. I think we were just scratching the surface of what's possible with diffusion language models. That's why I'm so excited about these technologies that, you know, it's, everything is pretty new, pretty fresh. I think there is still a lot of low-hatting fruits.

• 45:15Speaker 00

  You know, a lot of design choices, I think, are suboptimal. We, we often just take whatever works for auto-aggressive models and we just use it in the context of diffusion language models whenever that's possible. But I think those choices are not necessarily the, the best ones, given that the architecture is different, given that the model is dramatically different. And so there is still a lot of opportunities for research, development, R and D on the models.

• 45:45Speaker 00

  That one of the things we're working on is a reasoning diffusion language model. That's a capability that auto-aggressive models have, and it's one of the ways they can drastically improve their quality, the ability to think before outputting an answer. And there are interesting ways of achieving those kind of capabilities in the context of diffusion language models. And it's one of the areas where we're investing heavily on, on R and D to come up with the first reasoning diffusion language.

• 46:15Craig S. Smith?

  Wow, that's fascinating. Yeah. And that would have a big impact on their use in agentic systems, is that right?

• 46:25Speaker 00

  Exactly, exactly. Because if you think about agents, yeah, there's often a lot of planning and reasoning involved. And those kind of capabilities become much stronger once you have a reasoning model under the hood.

• 46:39Craig S. Smith?

  And are you looking at doing multimodality? I mean, diffusion is already sort of the dominant player in image and video generation.

• 46:53Speaker 00

  Yeah, that's another reason we are so excited about diffusion language models is that, you know, we already know that diffusion are the best model for image generation, world models, video generation. If we want to be crack, figure out how to do text and code generation, this is kind of like opening the way for a single generative AI system that can handle different modalities and learn across the different modes. It's something that we're very excited about. And yeah, it's another thing that we are working on and hopefully we'll have some models and some announcements coming out.

• 47:28Craig S. Smith?

  Yeah, I mean, I'm going to be talking to Fei-Fei Li, your colleague or Erstwhile colleague, about World Labs, I think that's what she's calling it. Remarkable, this marble is a remarkable model. Is, can you talk at all about, you know, how diffusion L L Ms could fit into world models? Because world models are amazing, but what's really going to be amazing is when you have. A world model that can perceive the world and

• 48:14Craig S. Smith?

  understand the laws of physics and then combine it with language.

• 48:22Speaker 00

  Exactly, yeah. I think, I think there are very interesting opportunities once you combine different modalities, because as you said, yes, you could have a model that has been trained on on physics books and, and how works and, and simulation code and can write code to, you know, figure out what's going to happen and make predictions about the future. And I think that's just going to enable much stronger word modeling capabilities. Um, if under the hood, this, this model has been trained on not just the video data or just 3D data or, but really

• 49:02Speaker 00

  understands the world through all the available literature, all the available scientific knowledge that it should have access to.

• 49:13Craig S. Smith?

  Is, is there anything I haven't asked that, that we should talk about?

• 49:18Speaker 00

  I mean, I think, uh, one thing that is, I think, exciting and unique about inception is really that the models are available out there. Uh, it's not just the research prototype. It's a, it's a model that developers can use right now, businesses can, um, you know, build apps and, and they can replace uh, auto-aggressive LLMs with diffusion language models today. Everything is backhorse compatible, the API is the same. It's kind of like an OpenAI compatible endpoint. And so, uh, you know, this is happening and I think, I think your viewers might be interested

• 49:55Speaker 00

  in, in seeing that, you know, this is, and then be aware of this transformation and be, be prepared for the, you know, what might. Change, you know, the way they build things, they they develop apps and and so forth.

• 50:10Craig S. Smith?

  Yeah, and we started before I started recording, talking about hallucinations. How do you handle hallucinations? I mean, there's certainly been a lot of work done on the auto regressive side. Is it the same strategies for a diffusion L L M? And is is that a focus of, you know, refining these models or or?

• 50:42Speaker 00

  Yes, for sure. The the models are not perfect and they make mistakes. And yeah, we try to measure it using benchmarks as much as possible. And some of them, let's say, when you ask, you know, question related to question answer or general knowledge. We we we try to see, you know, where the gaps are. And then we we try to fix the issues by incorporating additional training data, additional

• 51:13Speaker 00

  preference data or other other ways of kind of like supervising the model to try to do the right thing. And I think it's a it's still an open problem, like even for auto regressive L L lands. Hallucinations do happen, mistakes do happen, people. And it's less and less as the capabilities improve. But I think, yeah, that that's a super high priority for us right now to keep improving the quality of the answers, the accuracy of the answers that we provide to our users, because that just makes the the whole experience better.

• 51:45Craig S. Smith?

  Yeah, so if someone wants to try Mercury, they go to remind us of the URL.

• 51:55Speaker 00

  Yeah, chat.inceptionlabs.ai. That's where you can find a chat interface to play with the model and platform.inceptionlabs.ai to get access to our API and you can sign up for a key. We give 10 million tokens for free to to our users to to get started. And then you start building some cool apps using really, really fast diffusion methods.