Speech Recognition Is Not a Solved Problem — Pavan Kumar Reddy

Pavan Kumar Reddy leads audio research at Mistral AI. He joins Tim Scarfe for a deep technical tour of Voxtral — and explains why the frontier of deployed voice is still a cascade of specialised models rather than one end-to-end system. IN PARTNERSHIP WITH MISTRAL AI: --- This episode was produced in partnership with Mistral AI. Mistral AI: https://mistral.ai/ --- The conversation opens on architecture. Voxtral Chat feeds a 3B Ministral text trunk with continuous embeddings from an audio encoder, passed to the decoder as direct token input rather than through cross-attention as in Whisper, so the model can answer questions about emotion, timing and who spoke when without an intermediate transcript to lose them. The real-time model becomes a dual-stream decoder that consumes audio and emits text at once, at a target delay down to 160ms, with slower streams in parallel for anything that can wait for more context. On generation, Pavan explains why Voxtral TTS predicts continuous latents rather than discrete codec tokens, traces the lineage from SoundStream through EnCodec to Mimi's split of semantic and acoustic codebooks, and places FSQ and flow matching in it. Tim presses on the priors underneath: why a mel spectrogram instead of raw waveform, what noise augmentation buys, and when acoustic overfitting becomes somebody's fine-tuning problem. Then the failure modes. Diarisation is emitted autoregressively inside the transcript rather than by a separate head, which makes streaming diarisation fragile — less context, late speaker changes, invented extra speakers. And because the architecture commits to what it has already predicted, one out-of-distribution mistake compounds into looping or skipped segments, which is what DPO corrects: the negative supervision pre-training and SFT cannot give. The last third is the argument Tim keeps returning to. Customers running voice agents over millions of sessions describe scaffolding, not a solved problem, with a sharp drop outside the top few languages. Cascades survive because each component stays separately adaptable, observable and constrainable. And voice alone is cognitive debt: absorbing information and deciding in one serial stream is harder than glancing at a menu. Voice becomes ubiquitous beside a screen, not instead of one. --- TIMESTAMPS: 00:00:00 Cold open 00:00:46 Why Mistral moved into audio 00:09:27 Inside Voxtral: trunk, encoder, dual streams 00:20:22 Speech that works in real time 00:30:52 How a voice becomes tokens 00:39:59 Flow matching, FSQ and the new codec 00:52:51 When speech models lose the speaker 01:03:23 Correcting hallucinations with preferences 01:12:12 Controlling synthetic speech 01:20:06 Why cascades still win 01:29:25 Speech in the wild 01:33:46 Audio models as interfaces 01:37:54 Why voice still needs a screen --- REFERENCES: paper: [00:01:42] Mistral 7B https://arxiv.org/abs/2310.06825 [00:09:38] Voxtral https://arxiv.org/abs/2507.13264 [00:14:41] Whisper: Robust Speech Recognition https://arxiv.org/abs/2212.04356 [00:19:11] Voxtral Realtime https://arxiv.org/abs/2602.11298 [00:21:52] Delayed Streams Modeling (Kyutai) https://arxiv.org/abs/2509.08753 [00:30:52] Voxtral TTS https://arxiv.org/abs/2603.25551 [00:32:38] SoundStream neural audio codec https://arxiv.org/abs/2107.03312 [00:34:59] Flow Matching for Generative Modeling https://arxiv.org/abs/2210.02747 [00:37:03] EnCodec: High Fidelity Neural Audio Compression https://arxiv.org/abs/2210.13438 [00:37:42] Moshi and the Mimi codec https://arxiv.org/abs/2410.00037 [00:39:05] Finite Scalar Quantization (FSQ) https://arxiv.org/abs/2309.15505 [01:03:33] Direct Preference Optimization (DPO) https://arxiv.org/abs/2305.18290 dataset: [00:46:14] Mozilla Common Voice https://commonvoice.mozilla.org/en/datasets organization: [00:50:47] Hugging Face https://huggingface.co/

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Speaker 00

  This episode is brought to you by Michelob Ultra. Good times mean competitions between you and your friends. Like seeing who can throw some pigskin the farthest or rush the field the fastest. Winner gets an ultra because it's not just a superior like beer. It's also giving fans superior access to their favorite sporting events. Superior is worth playing for. Visit michelobultra.com/superioraccess to enter. Enjoy responsibly. Copyright 2026 Anheuser Busch InBev Michelob Ultra Beer, St. Louis, Missouri.

• 0:30Speaker 01

  Are your ad campaigns lighting up the dashboard? But not the pipeline. That's bull spend and marketers are calling it out in dashboard confessions.

• 0:39Speaker 02

  My boss asked for results, so I opened my dashboard for the only positive sounding metric I had. Impressions.

• 0:47Speaker 01

  Cut the bull spend. See revenue, not just reach. LinkedIn delivers the highest return on ad spend of major ad networks. Advertise on LinkedIn. Spend $250 on your first campaign and get a $250 credit. Go to LinkedIn.com/campaign. Terms and conditions apply.

• 1:00Speaker 03

  I think a lot of people don't know how audio generation works. Voice is one of the primary ways humans communicate. Definitely came much before text was a thing. The world looked very different, I think, with neural codex and autoregressive generation, which was really cool that the field progressed so far. And when we made it work, it was pretty cool. It's an autoregressive architecture. So what you predicted already is in a sense committed. So it becomes context and you can't edit it out. It's like a voice, an end to end speech model, but primarily to serve as a interface to other systems.

• 1:45Speaker 03

  Yeah, I'm a research scientist at Mistral. I work on audio research at the moment. I lead the audio research team here at Mistral. I've been broadly focused on multimodal models at Mistral since I joined like two years ago.

• 2:00Tim Scarfe?

  This episode was produced in partnership with Mistral AI. So Mistral is it's a full stack AI company and you're in the thick of it. You're doing the actual research. What does that actually mean deep down?

• 2:14Speaker 03

  Yeah, definitely. The full stack description of Mistral applies to the whole company as in it's serving needs at different layers of the AI stack, I suppose. I think the company is most well known for the model layer. Since the original Mistral 7B, we've been releasing open models, both text models and over time multimodal models and reasoning models. That's one layer that we continue to focus on and one of the most important layer and the place I work on primarily. But on top of it, we have offerings at the application layer.

• 2:58Speaker 03

  We have Mistral Vibe, which is our coding offering, a chat app, and also a command line based interface. So there is a on top of it, we also have like Vibe Work, which is similar agent tech working experience. So that's at the product layer and we also have AI Studio, which is which has our API offerings and workflows where you can build agents, AI agents easily. And we also have Forge, which is a fine tuning platform

• 3:40Speaker 03

  and also other kinds of model adaptations which where you can take any of our open models, some of our non-open models as well, and just adapt them to your specific needs. Again, applies to both text models and multimodal models. And over the last year or so, we also have Mistral Compute, which is the AI cloud offering, which is bare bones compute. On top of which, many of the other work happens, including the research work. So it's a full stack AI company in that we do have some presence and offering

• 4:21Speaker 03

  across various layers of the AI stack because most of the enterprises which tries to incorporate AI usually require integration at several points in their stack. And we try to meet them where they are and give them the efficiency gains or productivity boost that they're looking for.

• 4:43Tim Scarfe?

  Very cool. And what's it like though? Just just being a research scientist at a frontier AI company. I mean, you must be doing, you know, you must be looking at lots of interesting research that that that the other folks are doing, going to conferences and so on. I mean, how is all of that work kind of being integrated into your workflow? What does your day in a life look like?

• 5:03Speaker 03

  Yeah, that's a great question. It's really fun to work at a frontier company, which is also a general frontier company as opposed to a task based company. Again, I think there are trade offs. It's not it's not a strict, one is better than the other. But for me personally speaking, it's always nice to have visibility into the whole work happening across the board. Even though I focus on audio, I closely model our work on text pre-training, post-training, reasoning, vision. And more recently, we have ventured into robotic foundation models as well.

• 5:43Speaker 03

  While I don't work across the board at any given point in time, it gives me good exposure to what's going on. And it also, I think, helps me incorporate the latest techniques because I think at this point, not just for audio, but across modalities and tasks, the approaches are getting converging and the paradigm of pre-training, post-training. There are subtle differences and specific nuances to specific niches. But at the same time, the broad techniques are super general. I mean, that's the success of the latest paradigm, I suppose.

• 6:25Speaker 03

  So an average day involves like doing a lot of my own research work, training models, running up latencies, investigating evals and stuff, but also following the work that's happening across the company.

• 6:44Tim Scarfe?

  Yeah, and I suppose it's cool as well because you folks get to publish a lot of your research. So you can you can kind of speak openly to the community. And a lot of your models are also open weights as well.

• 6:55Speaker 03

  Yeah, definitely. I think from the get go, we have the open weight ethos. Our very first set of model as Mistral 7B was the one which got us prominence. And we continue to open weight our most of our models, especially the largest models. And we the hope is basically we operate as one open ecosystem. We contribute to the ecosystem and take from the open ecosystem as well. And it's a the whole open frontier keeps moving. So it's also as a researcher, it's nice to be able to share our research work.

• 7:39Speaker 03

  It's like personally speaking, it's always great to share the work and also borrow developments outside of our company. And there's a lot of the ecosystem is quite big. So it's it would be a missed opportunity to not leverage the developments. And it's also always nice to contribute your own in your in your own small way.

• 8:03Tim Scarfe?

  Traditionally in machine learning, all of the different modalities were quite fractured into different expertises. So, you know, the the language people were one ecosystem and the audio people were another ecosystem. And you're working on so many modalities. And isn't it interesting that many of the same types of approaches work across those modalities?

• 8:24Speaker 03

  Yeah, it's actually, I mean, I've been working in machine learning and AI for, I think, at this point. Close to ten years, I suppose, and it's actually fascinating. It converged to a point where the same firstly, it's fascinating that architectures are so all of them, most of them are transformer based, not there's some variance across modalities as modality specific variance and now there are hybrid models with non attention based layers going on, but broadly speaking, the architectures have converged.

• 9:01Speaker 03

  And even in audio, which is a modality I started working on since two years ago, I suppose, at this point, the techniques are converging more and more to a unified approach. I think it just comes, the forcing function at least comes from the place that the way you use them is you want to use them as a single entity, like in many of these cases. At the end of the day, from a user standpoint, from an interface standpoint, it's a it's a model that you would like to give input through text, communicate through vision inputs

• 9:41Speaker 03

  and talk to it and hope it writes back or speaks back and in some cases generate images. So I guess it's a natural forcing function for people to explore how to make it all work in a single model because it's quite elegant way to do it. It's just, I was at least surprised when it started, not just as a nice to have, but it also turned out to be the way that that seems to be working really, really well.

• 10:12Tim Scarfe?

  Well, we should start talking about some of the audio work that you folks have been doing. Can you tell me about that?

• 10:19Speaker 03

  Yeah, I have to share our work. So we started working on audio last year and the very first model we released was Voxtral Chat. It's a audio input text to text L L M model. So you give it audio input and a textual instruction or it doesn't need a textual instruction because your question can be in the audio and then the model produces a text response. So the idea behind that is to have a general interface for audio understanding. So the model can do transcription, speaker segmentation or summarization or any questions

• 11:03Speaker 03

  like you can have a big audio document essentially, which is like an earnings call or a meeting recording and then ask it questions like what you would against a text document. So you can ask it about what was the topic of the conversation, who spoke when, did this thing happen? It's specifically useful when analyzing, for instance, podcasts, like if you want to, at what time point did the speakers talk about this topic, things like that. So that was the first model that we worked on. And then since then, our focus

• 11:42Speaker 03

  was to build foundational building blocks for audio agents to that. And we released a transcription model, a real time variant of that, and then a T T S model earlier this year. And we are continuing to work on improving those models. And the hope eventually is to build a end to end, speech to speech model in this space.

• 12:11Tim Scarfe?

  Yeah, really cool. And that first model that that you gave an example of, I guess you would call that an audio understanding model. And people might not be familiar with that because you know that there are Omni models that have audio as a capability and there are transcription models. And because this model has all of those capabilities built into it, you gave the example, you can actually say, when did Tim ask the question about the audio model? And it actually knows when certain events are and what time those events are. So that that's actually a new type of A I that I don't think many people have seen before.

• 12:42Speaker 03

  Yeah, definitely. People might already do this in a cascaded way. You basically take an audio document, get a transcription of it and feed it to an L L M chat bot. And you can ask it all kinds of questions that you just mentioned. This model basically does the same thing, but in an audio native way. So you don't need the intermediate transcription. So there are few advantages to handling it natively. I think the general principle of M L is the more you do end to end, the more you are removing the error propagation through a cascade like approach. And in this case, you can ask it about things like emotion,

• 13:23Speaker 03

  which is not necessarily captured in a transcription. Also, if you want to query about some information, which is not just the transcript, you need to have that information be captured in the intermediate format in the transcription case, like for instance, for the timestamp question you were asking, when was this topic, when did Tim mention this in the podcast? You were you are basically assuming the transcript has timestamps at the granularity you care about. So you need to capture very rich information in the intermediate representation. In this case, it's a transcript, I suppose, and it's always better when the quality matches

• 14:05Speaker 03

  up to the expectations to have it be done end to end, purely because you don't need to capture things ahead of time. It's the power of attention, I guess. You just focus on the things you care about by directly attending to the relevant aspects of the audio input in this case.

• 14:27Tim Scarfe?

  Very cool. Now, as I understand it, you folks use one of your text models as a trunk. And you know, like the promise of deep learning was always this beautiful idea that it's like Lego, right? We can compose all these different models together. So you can have a frozen trunk over here and then and then you can build, you know, an audio head on the top. But I suppose the question is, do you lose anything by doing that? You know, like what are the trade offs by building deep learning models in this compositional way?

• 14:54Speaker 03

  Yeah, definitely. It is compositional from an architectural composition standpoint. So just to take a step back and describe the architecture of this model that you alluded to. So we have the trunk, which is a three B text model that we train, that we call ministral series of models. And audio input is provided to the model through a audio encoder. But unlike, say, Whisper or models like that, where the audio input goes through an encoder, which is then fed into the decoder through a cross extension, here, audio encoder

• 15:36Speaker 03

  produces tokens, in this case, embedded continuous representations through embeddings. And then that fed into the main decoder model, just as a direct token input, similar to how you would feed text input. In the text case, it's a rather simple encoding scheme. You you send it through a tokenizer and you get token IDs and then you just have a embedding table. In this case, the encoder is a little bit more sophisticated. At least in the Voxtral Chat, the encoder is very close to Whisper encoder, although for the later models, we optimized it and adapted it. We just try to reduce the number

• 16:21Speaker 03

  of layers to the minimum number required for getting the performance. But in the first model, it was very close to the Whisper encoder. And the encoder is itself a transformer model. So it starts with extracting the Mel spectrogram and having a couple of convolution layers at the very beginning. And later, we have a transformer stack and a down sampling layer. So you basically get an audio token for every eighty milliseconds. So we are getting twelve point five tokens per second. With this, it's a fixed frame rate approach.

• 17:01Speaker 03

  And the the compositional aspect of it is you could train this encoder independently, which is very common in vision. For instance, vision encoders are trained with a large captioning dataset and then added into a decoder transformer like this and trained with the transformer pre-training task with vision text aligned documents. But in the audio case, rather at least in this model, we don't quite train this encoder in a separate phase with a different, different loss or different approach. It's a fairly simple

• 17:43Speaker 03

  recipe. And we always try to have simple recipes because it's, I think, stands the test of time and also easier to scale. There are a lot of benefits of keeping the recipe as simple as it can possibly be. So in this case, we just pre-train the model end to end with the encoder and the decoder on actually two basic tasks that we describe in the paper, a continuation task where we have interleaved audio text sequence. So there is an, imagine a podcast like this where it's, let's say it's segmented based on speaker. So if my

• 18:25Speaker 03

  speech is represented in audio, your speech would be represented in text and it's an interleaved training on the textual blocks. And also, we train it on the transcription like pattern for basic alignment between audio and text modalities. So since it's trained end to end, it's almost like a singular unit, although it's a compositional from how the audio is fed into the model. So I personally feel the line is pretty blurry. That's not, it's not that the audio encoder is solely responsible for audio understanding. It's clearly the joint optimization, which leads to the audio understanding being quite spread out in the model, although audio encoder is the first component which at least

• 19:08Speaker 03

  Uh. Red uses the at least presence the information in a way that the decoder can ingest it similar to text.

• 19:15Tim Scarfe?

  And and how in the architecture do you combine the two modalities? I I I think you said you summed them up. And what what would the alternatives be?

• 19:23Speaker 03

  Yeah, that's a good question. So in the Vauxhall chat, it's an interleaved audio text model. So at a given position, you either have audio or you have text. Uh. More specifically, if you have a audio document, you basically will have audio occupying um. The prefix and then comes the text part, which is first a text question and then the model generates a text answer. Uh. I think the summing part that you're mentioning is in the real time model. In this case, uh, that's a slightly different architecture where we uh, the encoder part is the same. But when the information is fed into the decoder, the model is processing audio and generating text at the same time.

• 20:08Speaker 03

  That's the reason why it's a dual stream model. Uh. And the specific task here is the model is expected to produce a transcript with an expected delay. Um. And the model is consuming the tokens it generated itself. That's the autoregressive component. And the audio stream is a read-only stream, since it's not generating audio. In this case, it's just ingesting the audio. And the way uh, both the text token and the audio token is presented to the model is by simple summation. You can imagine more complex fusion, but again, in the spirit of keeping things as simple as they can uh, get.

• 20:52Speaker 03

  We uh, went with summation uh, and it works just fine compared to other more fancy fusion techniques.

• 21:00Tim Scarfe?

  Yeah, so I suppose one of the themes that we're going to be discussing today is the trade-off between having like a big Omni model and building specialized models and using cascades. And this is a great example because when Whisper came out, it was an amazing model and there was loads of innovation in the open source community. And some people even hacked it to be a streaming model, right? So they were they were doing this kind of sliding window type thing. Because, you know, the obvious question is, well, why do we need to have like a a a streaming model architecture and a batch model architecture? I mean, maybe, could you just explain the difference between them and what the trade-offs are, you know, for for both of them?

• 21:37Speaker 03

  Definitely, and I'll prefer this by saying audio is interesting to me uh, also because the space is still evolving. Um. There is no uh, architecture which is like the end form, I suppose. Um. In in text too, that is true to certain extent, but in audio it is, I would say it's even more so. So whenever I contrast anything uh, with something else, uh, it's with the caveat that uh, it would probably look a little different next year or two years from now. But at least as it stands today, the the motivation behind streaming is to make it natively streaming.

• 22:18Speaker 03

  So if the model can natively, so the the property we want is for the model to continuously be listening. Uh, not listening chunks and we wanted to see how far we can push. Uh. There's also this approach, this approach is also inspired from the delayed stream modeling that Q-Tai published. And the key difference is uh, the encoder is trained from scratch in an end-to-end fashion in this case. And the idea here is to have uh, the audio fed to the model at the same frame rate that the encoder emits the tokens. In this case, uh, the encoder emits a token every eighty milliseconds. So the model is natively fed the audio uh, every eighty milliseconds essentially.

• 23:04Speaker 03

  And uh, one interesting aspect, I mentioned the model is expected to produce the transcript it's listening. But there is a subtle trade-off here. The model can produce the transcript after listening to the all of the audio, which would be the case with Vauxhall Chat by its design and Whisper originally as well. It takes all of the audio or thirty seconds of it and then produces the text. Uh. This model can do that too. It just would be, so the tokens it's producing when it's listening is like a a a filler token, like a streaming listen kind of token. And when it's ready to produce a transcript, it produces the text tokens corresponding to the transcript that it has just heard. And one extreme I just mentioned is listening all of the audio and producing.

• 23:49Speaker 03

  The other extreme is producing the text tokens it heard almost immediately. If it hears uh, New York and then it produces a text token corresponding to that word right after it heard New York. But you can, you can see how the this extreme case is actually um, ambiguous. Because in many cases, a full word could involve prefixes which are themselves standalone words. So the more you wait, the more uh, uh, the less uh, ambiguity there would be. And the higher, even theoretically, like this is assuming uh, oracle information, a model which does perfectly well. Even in that case, uh, there is ambiguity and hence producing errors.

• 24:33Speaker 03

  And the more you wait, uh, the less ambiguity and the uh, lower the chances of errors. And in this case, the expected delay that the model is supposed to have is actually a parameter to the model itself. We uh, condition the model with the target delay at hand. And the target delay is a target number of frames. It has to wait before it produces the word it just heard. Uh. So the model gives you a nice quality uh, latency trade-off that you can uh, control based on the application that you're building or whether you're feeding it to a downstream L L M or whether it's a subtitle use case where you're showing it uh, at the bottom of the video in a conversation like this. So that's flexible.

• 25:17Tim Scarfe?

  Yeah, it's so interesting because there there must be levels of latency where you get different types of degeneracy. So I'm I'm, you know, I'm imagining if you make it respond too quickly, um, at some point it'll just fall off a cliff. But there's also this um, more sophisticated form of of um, you know, text production where, as you say, it takes the context into account. To give you an example, when I do some A S R transcription and I'll get Claude to do a transcription refinement. And it can take the entire document into consideration and maybe even other stuff as well. And it will know, oh, it's mistranscribed Andre Andre Capathie's name and it's actually talking about this paper and there's a technical figure, you know, there's a technical term in in that paper. And and it's absolutely wonderful. And this is the kind of thing that practitioners can do, right?

• 26:01Tim Scarfe?

  So you could put a custom vocabulary in there or you could have varying levels of transcription refinement. So you could, you could have a fast path, maybe that refines within a five-second window and maybe like a a twenty-second path. And you you you see what I mean? You can just kind of layer on with engineering various forms of refinement to make it work better.

• 26:18Speaker 03

  Yeah, definitely. I think at the end of the day, uh, it's a system that you're building. Uh. Voice agency is the specific system uh, that is of most interest to us. But having said that, uh, the models are built to be applied in all kinds of contexts, including the completely offline case where you only care about uh, uh, performance at the end of the the quality. You don't care about latency at all. And uh, there are cases where you don't even care about the cost. You're happy to call many models, exactly like the case that you mentioned, where you're uh, post uh, like post-production task where you're taking a podcast recording like this or a earnings call

• 26:59Speaker 03

  and trying to do analysis over it, where this is not the biggest cost consideration. And you care about being as right as you possibly can. In that case, you can do all kinds of uh, refinement, like you mentioned, where you can bring a reasoning L L M in into the mix. And actually, the Vauxhall chat, uh, it was last year, but if we were to do again this year, that would be a reasoning model. So it would listen to the audio natively and then would not produce the transcript or analysis that you asked immediately. It can in its own uh, reasoning chain, uh, transcribe the necessary parts, uh, do a web search,

• 27:41Speaker 03

  uh, fetch appropriate context it needs to uh, disambiguate certain things that it is a little, finds a little inconsistent based on what was said. And then retranscribe in its own thinking chain. So this can, this can be done in a single model pass, I suppose. It's not even a complex system, but you can also augment it with other, like you can give tools to this model that can provide you context. You can condition it with the custom vocabulary like you mentioned. But custom vocabulary is applicable where the vocabulary is like finite and fixed. But in some medical domains, it could be uh, not infinite, but quite large.

• 28:18Speaker 03

  And in which case it can be a tool which the model can call and um, fetch appropriate context by its own uh, volition and uh, produce a higher quality transcript. That's definitely, that's definitely true.

• 28:33Tim Scarfe?

  Yeah, it's it's just so interesting how we have these cascades now. So I'm I'm a top one percent user of Whisper Flow. And I use it, I I talk to my agents more than I talk to real human beings apparently. And and it's so interesting. So Whisper Flow will do some refinement of what I say and then it would, and then it would go into my coding agent. And it just has this um, uh, almost clairvoyant ability to understand what I mean. So even if something was mistranscribed, it it will still understand what I mean. And you see what I'm saying, that we we will have voice agents and they won't, you know, yeah, they'll have some intelligence and they'll have some reasoning, maybe even adaptive reasoning. But then it'll probably go to another agent and the other agent will have more context. And it's it's almost as if coherence will just manifest in in the cascade.

• 29:15Speaker 03

  Yeah, uh, definitely. I think uh, Because now we have cascade cascaded systems are source of errors usually like in the case of ASR, LLM, TTS. But in the cascades that are emerging now with the use case that you are mentioning, for instance, the coding agents, but also the co-work agents where you are trying to leverage agentic systems to get not just coding, but other kinds of workflows done. And voice is a primary interface. Each of the components is also a smart system which tries to be robust against the mistakes made from the previous system.

• 29:57Speaker 03

  So it definitely like overall works quite well, but it's not a purely event when I suppose there are cases where if you if you are trying to like in the context of coding, if you're trying to be very precise about a a function name or a method like method or some other aspect of your code base that you are trying to fix. And it mistranscribes for whatever reasons because it doesn't have context. And if there are multiple entities which could match what you described, you can imagine how it can it can go wrong.

• 30:36Speaker 03

  So it's there's still value in trying to improve the understanding layer, whatever it is, transcription being the simplest case. But in an audio understanding model, it would be the holistic native audio input LLM or native audio input agent, I suppose. I think there is still a lot of value in trying to still bring down the the basic audio understanding quality. And that's what we're trying to do with both our transcription models by making them more contextual. So it can give all kinds of context that just as a transcription model. And you then use this transcription to feed it to whatever downstream system in whatever way you want.

• 31:16Speaker 03

  Or in a more end-to-end model, care about the specific cases where the failures are costly and try to address them.

• 31:27Tim Scarfe?

  Let me read this out. So it's a decoder-only transformer trunk predicting audio as autoregressive continuous latence reconstructed by a neural codec. So I think we should talk about the continuous latence and the neural codec because they're both very interesting. So first of all, why the continuous latence?

• 31:45Speaker 03

  Yeah, that's a good question. I think it came after several iterations. So we were trying to take a architecture for generation and that that was so the Woxer TTS model, which we released in March. The goal is dual-fold, I suppose, want to provide a offering from Mistral for voice generation, voice synthesis, for mostly voice agents, but also other use cases. And at the same time, for us to also do some research and come up with a architecture that we are happy with and continue to iterate on, both to improve the TTS model itself, but also to incorporate in a future speech-to-speech system.

• 32:30Speaker 03

  And the lay of the land is essentially neural codec based autoregressive generation seems to be a dominant paradigm when we started and it continues to be the case. But the key difference is a lot of models then were based on discrete token autoregressive decoding. And the attractiveness of discrete tokens is it fits well into the existing text token-based autoregressive frameworks. So if you so the neural codec itself would have a quantized bottleneck, which is producing discrete tokens. And examples of this is N-Code, X-Sound, Stream, Mimi.

• 33:15Speaker 03

  And it's not just discrete, it's not just one token. So similar to how I said, the audio encoder for the understanding model produced a embedding every eighty milliseconds. It's similar here, but instead of an embedding, it would be a set of tokens, usually corresponding to a residual vector quantizer. And you also have an audio decoder here or vocoder in some cases where which basically reconstructs based on these discrete tokens. And the model, this whole autoencoder is trained end-to-end to minimize the reconstruction losses and a bunch of other losses.

• 33:59Speaker 03

  And the discrete tokens are then used in an autoregressive decoder model, which is the, let's say, the TTS model, which takes text conditioning and produces the discrete tokens one by one. So once the to your question, why continuous? So at each time step, these discrete tokens, what it's not just a single discrete token like in text. It's a set of discrete tokens, usually an ordered list, I suppose, in the residual vector quantizer case. And the way many models approached is they have a architectural affordance just for handling predicting this multi-token set. There are many ways to do it.

• 34:39Speaker 03

  There is a delay pattern way to do it, but I think one of the more popular ways was to have a depth transformer. So it's also autoregressive in codes where at so there's autoregression in time and there is autoregression in codes. So we are trying to see if we can reduce this complexity around this autoregression in codes because the number of steps you need to do in the autoregression codes is the number of codebooks that you have. And in in our case, it was thirty-plus codebooks. So it's at every time step, you're doing a thirty-step autoregression again, although it's a smaller model.

• 35:18Speaker 03

  And given this consideration, we wanted to see explore approaches which are more controllable, which provide you a more delicate trade-off between the number of steps and the quality. And so we started exploring diffusion-based approaches. And the model that we ended up with was the flow matching head. So instead of a residual vector quantizer, which is a discrete set of tokens, we have a continuous latent embedding in this case. And the model takes the as this flow head takes as input the final hidden state from the decoder transformer

• 36:01Speaker 03

  and does the flow matching inference, I suppose, where it's it's doing a velocity prediction. It's very similar to denoising task, but slightly different in that it's predicting a velocity on a velocity field. And the inference is to do a integration over this velocity field, which comes down to just a simple summing of velocity with the time step over a fixed number of steps. And this provides a neat trade-off. You don't the number of steps is controllable and you can also there are also techniques to reduce the number of steps with a few-step techniques. So that's one of the reasons we picked it.

• 36:45Speaker 03

  But the continuous embedding is also attractive in that the discrete codes in some sense also have this connotation of communication where you're trying to reduce the bits of information that you're sending over a channel. But in in our case, that is not a consider we are not using this neural codec for compression or communication over a band limited channel or something like that. It's purely for generation purpose. So that bottleneck doesn't make sense in our context. And that's one of the reasons we also thought this would be the approach

• 37:27Speaker 03

  which allows us to expand the bottleneck and explore a more larger design space.

• 37:35Tim Scarfe?

  Yeah, it's so interesting. I remember the the N-Code paper, I think it was Meta. Yeah, I think was it about three or four years ago that that came out? And it was fascinating at the time. But can you just just explain a little bit more detail about what that paper introduced and what the lineage was?

• 37:50Speaker 03

  It's mainly residual vector quantizer based approach. So you have so you have a it's it's an autoencoder architecture where you take the waveform on the input side and generate the waveform on the decoder side. And you have a vector quantizer bottleneck, specifically this residual vector quantizer bottleneck. And then came Mimi, which is an enhancement on top of it. And one of the one of the aspects of this family of models is this semantic codebook and acoustic codebooks. And the key difference between these is the semantic codebook gets a distillation supervision.

• 38:36Speaker 03

  The motivation behind this is to keep the this codebook closer to the text space. And the order in which you predict the codes in the decoder TTS model or generation model is you generate the semantic codebook first. And the idea is that it's close to the text space, so it's easier to predict. And then condition on this predicted semantic codebook, you predict the acoustic codebooks, which are assumed to carry more of the acoustic information.

• 39:05Speaker 03

  Which is not, so you can have multiple ways to pronounce the same word. And there is this variance and this acoustic codebooks are assumed to carry more of this information. And first getting the word broadly right and then predicting all these. It's not as interpretable as I'm trying to say here, but that is the motivation behind it. But in our model, the key difference is we still maintain the semantic codebook in the first release we did. As a separate vector quantized codebook. But the acoustic ones, we made it a FSQ base quantization. And it is a numerical quantization.

• 39:48Speaker 03

  So instead of a residual vector quantization, here we have levels. So in this case, we had twenty one levels. So it's a thirty six dimensional embedding with twenty one levels. And since these are discrete levels, you can treat them as a continuous vector without the quantization. Or with quantization, you can actually treat them as discrete codes. So that's one of the reasons we picked FSQ. We actually were exploring discrete code prediction based approaches to begin with. And then later relax them to be more continuous as we build confidence that the quality of generation was either similar or improving with this relaxation.

• 40:31Tim Scarfe?

  There's a bit of a theme here that we're going to talk about, which is that when you build machine learning models, if you want them to be more sample efficient and work at smaller scale, there's a whole bunch of tricks, like engineering tricks that you can put into them. And I think, I believe this is one of them. But another one, for example, is you could in principle in an ASR model, just learn from the waveform data directly. Or what you could do is something akin to a short time Fourier transform with the, so the Mel spectrogram, I believe, is it's like a logarithmically, it's like a curve which is calibrated to human hearing with some discretization. And that's often used in machine learning as perceptual audio features for human dialogue.

• 41:14Tim Scarfe?

  And that is a form of kind of dimensionality reduction, I suppose. So you folks have used this. So I suppose it's an interesting trade off. Like presumably, if you had a huge amount of compute and a huge model and lots of data, you could just use the waveform data, but you decided not to. Can you explain the rationale?

• 41:34Speaker 03

  Yeah, actually, I think you covered most of the rationale pretty well. I think the viewpoint here being the model, especially when you're operating in small models, the encoder offers us to be a little bit more sample efficient, I suppose. And on the input side, there is no need to model the aspects of the waveform, which are not necessary for human perceptual understanding. And the Mel spectrogram based featureization is one such featureization, which is, which is still pretty broad, to be honest, because it is not a,

• 42:18Speaker 03

  I think in old school machine learning, you have very handcrafted features, which are known to help with certain aspect of the task at hand. In this case, this is still Mel spectrogram is quite general. It doesn't, it doesn't just capture the speech. It can capture a lot of different auditory features, I suppose, and background sounds. It is empirically shown to capture rich information. But it's still a lever on getting a little bit more sample efficiency. And you're totally right.

• 42:58Speaker 03

  I think as the models get larger and as you train these over large and large data sets, some of these become unnecessary and you can feed the waveform directly into the model. And even without an encoder, which is becoming more common more recently. And the belief also is that you're doing two, accomplishing two things by removing these things. One, you're letting the general purpose model find, I guess, a better point without your inductive bias, which is generally shown to work as you go to more end to end approaches.

• 43:42Speaker 03

  And secondly, the, the idea here is that

• 43:49Speaker 03

  as you, if you can feed in a more general purpose way, the, the scaling becomes more predictable as you go to larger and larger models, because you have one less component to worry about.

• 44:02Tim Scarfe?

  And let's talk a little bit about noise. So for example, you know, ASR models, they, they have varying levels of performance based on how much background noise there is. And I suppose there's a whole bunch of things you could do. You could do noise augmentation, you could do some kind of data preprocessing or voice isolation before the data goes in. I don't think you've, you've done that. It, it just works reasonably well out of the box. But, but can you kind of tell me about how you can make these models robust to noise?

• 44:32Speaker 03

  Yeah, definitely. I think noise adding, a lot of approaches involves augmenting, which is true with vision models as well. Augmentation is a way to expand the amount of data to cases and making the model in some way invariant to transforms of the data, which shouldn't matter for the task. In the object recognition case, for instance, if you have a flipped image, ideally it shouldn't matter and scaling of the image shouldn't matter. In the audio case, noise in some way is a similar transform in the sense that if you have speech that is being spoken,

• 45:18Speaker 03

  up to a point, noise shouldn't change what was spoken from a perceptual standpoint and hence the prediction task should stay the same. And we want to also make the model robust to various acoustic scenarios, which we might not have coverage for in the data. And this is a more automatic way to get these acoustic conditions simulated in the limited data we have and yet again work with the data limitations. But for the, at least for the initial versions of the model, we didn't have to do a lot of noise augmentation precisely

• 46:03Speaker 03

  because one of the things we tried to cover in our data sets is to cover various acoustic conditions. But having said that, we do do some noise augmentation, especially to cover the tail cases. And it does add like robustness to these models. And it's quite important, like I think when people report ASR performance and stuff, it's usually presented in a, on data sets, which are like more academic data sets. And it's not like they're very clean system. There are evals like, evals like Mozilla Common Voice and things like that,

• 46:43Speaker 03

  where the acoustic conditions are a bit more noisy, a bit more uncontrolled, because it's people recording on their laptop in all kinds of background settings. But still, when you deploy it in the wild, the diversity of acoustic conditions is like lot more. And usually models which are specifically tuned for a specific set of acoustic conditions sometimes struggle to generalize to these other scenarios. And you get models which shot drop off in performance. And that's, that's what we're trying to avoid.

• 47:16Tim Scarfe?

  Yeah, because I suppose even the data set selection and augmentation, that is an inductive bias. At, at the end of the day. And it's one of those things that, as you say, if you have sufficient diversity, then it will just learn to generalize. So, you know, at the small scale, it might overfit to certain acoustic environments, which means on this particular microphone, it works very well. On this particular microphone, it doesn't work very well. And I just wonder, how much work have you, have you folks done around that? Is this the kind of thing where you identify failure modes and you lean into them? So you say, okay, we need, we need to do more, you know, adaptation and augmentation here. Or have you just reached a sufficient level of data and diversity where it just kind of washes away?

• 47:55Speaker 03

  Yeah, that's a great question. I think the model is generally performant in wide variety of acoustic condition. And like you said, when we identify areas where it doesn't work well from, as a function of feedback from customers who deployed it in the wild.

• 48:13Speaker 03

  Uh. We try to uh gather data which is covering that distribution and try to improve the model. But yet again, I think this is one area where the model adaptation is quite uh important. And that's uh it also alludes to the popularity of fine tuning whisper in the community. Because usually when you deploy this kind of ASR model, uh by virtue of deploying, you have uh the data distribution, uh the precise data distribution that you care about. Uh it's it's the one that you where you are using this model. Uh so you're very well equipped to adapt the model to improve, especially even if

• 48:53Speaker 03

  you have a small reasonably small amount of compared to the large corpus which is usually used to train these models. Uh uh the small amount of uh exactly in distribution data and having some of it annotated to a high quality uh transcription, having an internal workflow where you can transcribe these uh audios. And that's one of the reasons why we have uh Forge, uh the model adaptation platform, where uh you can take any of our uh existing transcription models and uh with your own data, you can adapt them. And usually, one of the concerns people have have is uh uh some of this data might be

• 49:37Speaker 03

  very sensitive, especially if it's like meeting recordings in the context of a company or something like that. So you would like to keep it to yourself, I guess. And uh fine tune the model and have a uh variant uh which is very very much applicable to you and performs really well in your situations. You you don't care about how it does in other acoustic situation because you deployed in this very specific setting. You don't want, you don't need a general purpose model, I suppose, in that way. And uh if you're happy with the performance, you can deploy it in the context of your company or uh or wherever you deployed it.

• 50:12Tim Scarfe?

  Yeah, and can you tell me more about fine tuning? So i- is it one of those things that with great power comes responsibility? And you need to be extremely careful about, you know, still maintaining diversity in your fine tuning data? Or is this the kind of thing that you folks can do as a platform? So you can still inject more diverse data just to stop the model becoming degenerate? I mean, talk me through that.

• 50:31Speaker 03

  It's it's more so for the text models, but even for audio models, it's important to have uh diversity even when you're fine tuning, uh precisely like you mentioned, to avoid the catastrophic forgetting of the the general general purpose training the model has uh received in the earlier stage. Uh but at the same time, I think for a task like transcription, it's a little bit easier to fine tune. The recipe is fairly straightforward. Uh and people have been doing it for a while in the community. Uh especially, uh you see a lot of fine tune transcription models on hugging face, for instance.

• 51:13Speaker 03

  Uh because the transcription task, I think, is fairly well understood at this point. And the the the thing that you want to end up with a good fine tune artifact is having high quality data uh with precise acoustic conditions you care about, noise conditions and recordings and et cetera, along with like high quality transcriptions, which is very important because if you have mistakes there, uh it is especially since you're fine tuning, it's not part of a large uh large stage training. It's important to have high quality data and then starting from a model which is already fairly good at it.

• 51:54Speaker 03

  And that's one of the reasons we also pick text based trunks for our uh transcription models. And uh in in future, it would be a holistic audio capability as part of a large language model in general because uh transcription, while a lot of it is very localized, um it also benefits from just world knowledge. Uh I think for specific, it it's very obvious in the case of like getting entity names right. Um if the model is already aware of these entities, uh it and based on the context in which you are speaking, it can easily guess. Uh so starting from a good uh starting point and having a high quality data set are the most important in ingredients.

• 52:38Speaker 03

  But some of the uh general principles about uh having some diversity in the fine tuning stage to avoid forgetting the uh catastrophic forgetting is also applicable uh in our case. For text models, it's a lot more applicable. Uh you you do need to have a good mix, otherwise there is a get getting the details right is quite important in fine tuning. And that's where the platform part is also important because we would offer recipes which are well tested in various contexts and usually should work reasonably well out of the box.

• 53:15Tim Scarfe?

  Very cool. Now, let's talk about speaker diarization. Now, it's possible to do this in a specialized model, but you folks have built this into the model. So the the the model is actually outputting speaker change tokens as as part of their output. Um may maybe you should explain whether it's in your um streaming model and and your batch model. How do how does that work? And it it it must be quite a difficult problem. What what what kind of issues do you have with it?

• 53:40Speaker 03

  Yeah, that's a good point. Uh like you were alluding to, the way we do speaker diarization is uh as a autoregressive task. It's not a special head, it's not a separate stream. Uh in our offline or async uh model, the way it is usually done is you have the audio and then for a plain transcript, the model would just produce the text transcript. And in the context of timestamps, uh we just produce segmented uh transcript where you would have the start time of a segment and then the transcript and then the end time.

• 54:19Speaker 03

  And there's some special tokens to represent the uh time delimiters, I suppose, and the model would just predict the timestamp. For the speaker, it's basically one additional piece of information on the segmented timestamp. In in addition to the start time and the end time, the model also predicts the speaker ID. It's a fairly simple convention. It goes from speaker zero to speaker I, uh assuming there are I plus one speakers in the conversation. Since the model is trained, the the model is trained with the the same autoregressive loss, uh just with the data, uh with the speaker representation um uh incorporated in the case of the diarization training.

• 55:00Tim Scarfe?

  Yeah, very cool. And I I use a lot of ASR tech myself. So speaker diarization is a great example of something where prima facie, when you look at it, it looks, oh, great, this has done an amazing job. And then when you actually look at the speaker changes, you you see the failure modes. And um what I what I tend to see is is that sometimes it just takes a few seconds to get the speaker change. So it it'll get it, but it will it'll just mix it up by a few seconds. And and what I also see is that there are significantly more failure modes on the streaming models than the batch models. And what I tend to see is that they will create additional speakers. So the batch models will be quite good at sort of correctly identifying, let's say, four speakers, but the streaming models might kind of like pick up a fifth speaker and a sixth speaker when they were actually previous speakers.

• 55:43Speaker 03

  Yeah, the uh it's streaming models are fundamentally uh more challenging just because they're operating with a less operating with less context and they have to make do with whatever context they have. Like for in in our API and also in as a recipe for the streaming model that we have, the model provides flexible target latencies, like I was mentioning earlier uh in our conversation. So you can give it a target latency of, say, one sixty milliseconds, which is super aggressive. Uh and you can give it and you can have another stream open where you give it uh like a second or two second delay.

• 56:25Speaker 03

  And the idea behind this is for anything you would want to uh display to the user right away, like in the case of subtitle, you want to showcase the transcript with as less delay as possible, just to keep keep it relevant uh for what's going on. But at the same time, if this uh transcription is going into being logged somewhere for uh for uh uh future reference, in that case, you can use the slow stream to correct the mistakes on the fast stream. More context it has, the better it gets. And with this model, uh it's as simple as just opening a second stream because uh the delay is a parameter that is fed to a particular stream that is open.

• 57:07Speaker 03

  And to your point on diarization, yeah, it's actually quite challenging. I I I think it's far from solved, in my opinion, um especially multi-speaker, more than two speaker in the context of a meeting. For instance, uh even day to day meetings that happen in the company, where they're like four or five uh people speaking over each other, it's uh very hard to uh get that right. Uh the challenges are quite diverse because the audio model is like as a as a human, you also have a a visual input, you have a kind of uh, you know, the audio source, where it's coming from. Uh and you you you are present in one part of the room and you know where all the other

• 57:52Speaker 03

  people are located. So you have a lot of additional auxiliary auxiliary signals which you can leverage to uh disambiguate who is speaking when. And when you actually listen to these meeting recordings, uh I don't know if you ever tried to annotate, uh very painful exercise. And it is a, I actually like, you can emphasize with the models, they're they're, it's a really hard task to recognize who is who without like, you have no video, just just have to listen and and you're you're not listening to the people that you know, who you already know how they sound. These are people from a random meeting uh who you never met. And some of the people do sound somewhat similar and you don't know uh if they're the same person speaking.

• 58:37Speaker 03

  And then you have to uh take a step back and read the context and see, oh, okay, it doesn't make sense for the same person to be interrupting themselves, so it has to be a different person or something like that. So it's a very challenging task, especially since the way we cast the task is a audio only, single stream, collapsed uh way. So there there is definitely a ceiling that you can uh achieve with this constraint that you have with this. But having said that. Uh, the current performance is well below that ceiling, in my opinion, I think, especially in a more than two speaker, noisy environment and also regular speech, even in a two-person conversation and especially in more than two-person conversation,

• 59:18Speaker 03

  has a lot of overlap, um, which is, which is just a natural property of speech. It's not turn-taking, it's not I speak and then I abruptly stop or you only start after I completely finish my thought. That's not usually how, uh, regular dictation conversations go and that adds to the challenge because you'll have many, many segments of audio where more than one person is speaking. And not only do you need to recognize that more than one person is speaking, you need to recognize who all are overlapping and what each one of them is speaking independently.

• 59:56Tim Scarfe?

  Very cool. Yeah. So you know, the, the cocktail party problem has been a huge problem in speech tech for many, many years, uh, which is this famous setting where you're in a cocktail party and there's many, many different people talking. There's John talking behind me and we can focus our attention, right? We have two ears and, um, we, we, our, our ears are actually doing this kind of, um, time-based recognition. So to, you know, when, when the sound waves hit both of our ears, we can actually distinguish, our brain can distinguish, uh, different sounds. And, and as you said, you know, like the, these models, they, they don't have all of that additional awareness. So they, they, they can't do the kind of conversational understanding of the dynamics that, that we can do.

• 1:00:37Tim Scarfe?

  And maybe in the future that'll change, right? Because we, you know, we're, we're going to have models that have duplex and, and have much more information. But on that note though, I did notice, working with speech models about two years ago, some of the frontier speech models could do what I can only describe intuitively as, um, active speaker locking, which meant if there was cross-talk, it would lock on to what it thought was the active speaker and it would con- continue to transcribe that voice and ignore other voices. And maybe that was an emergent property of that particular, uh, frontier ASR model. But have, have you noticed stuff like that?

• 1:01:12Speaker 03

  Yeah, I, I think, uh, it also comes from the closed caption data, I suppose. Uh, it's usually transcribing the active speaker and the models which are, uh, trained from data. Uh, model, at, at the end of the day, I think a lot of properties of the model also can be attributed to the annotation quirks. And, uh, it, it, it usually explained with it. Not everything is explained right, but a lot of it is usually explained right. I think, uh, if you take a second look at the data, you'll see a lot of, uh, properties that the model is exhibiting can be attributed to the properties of the data set that you're trained it on. But having said that, uh, it is a challenging problem, but at the same time, the models don't need to be,

• 1:01:56Speaker 03

  the model, the whole point, I suppose, is that they are, uh, at least in principle, can be better than, uh, they don't need to adhere to the constraints of humans. Uh, they can be superhuman that way. Uh, and unlike, unlike us, uh, we can only focus at one speaker at a time. And we have this nice ability to decide who to focus on, which is pretty cool. But, uh, uh, models, at least as long as they have, uh, enough information, like in this case, if they have, uh, instead of a single channel or instead of a single mic, let's say, they can take input from several mics across the room and they have additional information where these mics are, uh, things like that.

• 1:02:40Speaker 03

  Uh, they can follow all the speakers, uh, all the time. Uh, it doesn't need to focus on a single speaker. In, in a more simple way, uh, in our, uh, model, we are trying to tackle this problem. When there is cross-talk or overlap speech, the expectation is the model transcribes both the speakers and identifies when the first speaker stopped and the second speaker started. And, uh, even when the start of the second speaker is before the first speaker stopped speaking. Uh, so this is trying to transcribe multiple speakers at the same time. But it's a limited setting in the sense that the input is still a single stream, uh, collapsed, uh, audio channel, I suppose.

• 1:03:22Speaker 03

  Uh, but it is, at least the goal is to make it, uh, superhuman. And, and it is actually does better than, uh, humans in, uh, quite a few settings, but there's still always room to, uh, go, go further.

• 1:03:34Tim Scarfe?

  Great stuff. Let's talk about hallucinations. So, uh, you, you, you remember back in the days of GPT-3 when it first came out and it would get stuck in these degenerate loops. But because it's, it's almost surprising, isn't it, that we can use an autoregressive model? And it's coherently not only transcribing things, but it's putting, you know, like, uh, time indexes in there and speaker change labels and, and so on. It's amazing. But there are, there is still occasionally hallucinations and you folks are using DPO to minimize those hallucinations. Can you explain that whole process?

• 1:04:10Speaker 03

  As you rightly pointed out, the hallucination problem, at least the degenerate generation problem can be somewhat attributed to the architecture itself, owing to the limitations of the existing autoregressive architecture. And once the model makes a few mistakes, uh, it tends to commit to those mistakes. And especially when those mistakes take the model, uh, out of its training distribution. Uh, that's usually the, uh, scenario where it goes into a degenerate mode of infinite, uh, generations

• 1:04:50Speaker 03

  or looping the same prediction or, uh, skipping a whole segment of transcription because it decided to skip one part of the transcription now. Uh, it continues to skip the whole part because it doesn't want, want to predict halfway. Um, again, owing to the, uh, out of distribution nature of, uh, its initial mistake, I suppose. And, uh, the way, uh, there are many ways to tackle this and DPO is a simple way, uh, which prov- which is, but at the same time, quite powerful to fix issues like this, which are, uh,

• 1:05:30Speaker 03

  they, they fall into a small set of categories and the recipe here is fairly simple. And you collect the generations from the model which have been degenerate and you generate a correct version of them and that would become a winner-loser pair. And the DPO, uh, uh, training process usually involves training on pairs of data like this directly, as opposed to having a explicit reward model. Uh, that is also a equally valid and effective way, uh, that also works. Uh, DPO is just a single-stage process and for, uh, fixes like this,

• 1:06:13Speaker 03

  uh, which, which you can consider alignment of sorts, I suppose. Uh, it, it, it is quite effective. So you would have a winner, which is the correct transcript and a loser, which is, uh, degenerate generation from the model. And usually you would like to, uh, limit to the distribution where, uh, the losers are actually a generation from the model. So it's more in policy of the precise model you're trying to fix. Like, you're not trying to generate, uh, infinite generation for scenarios the model never actually does, which is, uh, off-policy. And for all the category of errors, you generate pairs like this and it, it is quite, quite effective in, uh, fixing most of the problem.

• 1:06:53Speaker 03

  And same thing, uh, works with hallucinations or skip transcriptions or some family of transcription mistakes. It's not, it's a fine tool. It's not a way to, uh, support a new language or something like that. It's, uh, a nice post-training tool to, uh, fix, uh, small but important, uh, problems in the model.

• 1:07:18Tim Scarfe?

  And, and can you give me your intuition on, on what's actually going on there with DPO? Above and beyond, let's just say, using supervised fine-tuning. Because I mean, my intuition is, as you just described, you're helping the model develop a kind of conceptual space understanding. So this is good, this is bad, and, and it's kind of exploring around that space reflexively. And you're kind, you're kind of like helping it conceptualize in a way that perhaps wouldn't be possible in the base model or, or with SFT. I mean, is that a reasonable intuition?

• 1:07:50Speaker 03

  Yeah, that, that's a reasonable intuition and I, at least from my point of view, one of the primary tools that the DPO gives you is the way to give negative supervision. Uh, with both pre-training and SFT, the supervision is purely positive. You're just reinforcing what is the right thing to do. In pre-training, it's just general text corpus. And in SFT, the specific behaviors you want to get out of the model, uh, whether it's a chat model or a transcription model. But they don't offer a mechanism to say, this is wrong, like, uh, penalize this, like, uh, reduce the probability, uh, of this sequence.

• 1:08:28Speaker 03

  And DPO is not the only way, uh, even, uh, uh, many of the RL, RL techniques, RL adjacent techniques offer a similar mechanism. And DPO is one of the simpler, uh, variants, which provides you a mechanism where the winner gets, uh, reinforced and the, the loser or, uh, the one which is, uh, erroneous gets, uh, penalized, uh, by the objective function.

• 1:09:00Tim Scarfe?

  Yeah, that makes sense. So you're saying the most load-bearing thing, sorry to use a Claudism there, load-bearing, my god, I've been using Claude too much. But you're saying it's mostly pushing the energy landscape down on the negatives. So we have the base distribution, we push the energy landscape down. But is there another thing going on as well there? Is it pulling up the energy landscape? And is it actually learning and exploring more in the positives? Is that part of the story as well?

• 1:09:25Speaker 03

  Yeah, it's, so the loss, if you see the loss term of the DPO, it looks like a weighted positive loss on the winner. And then it's a negative loss on the loser. It's not, there is a weighted nature to it. And a nice formalism in the original DPO paper as well. And there is a lot more follow-up work trying to build intuition on what the algorithm actually accomplishes. But it is a margin, to me, it looks like a margin loss, which pushes the positives on the more positive side and the negatives

• 1:10:05Speaker 03

  to be predicted less by the model after the stage. Along with a component which tries to keep the model not too far away from the starting point. So you don't want to move too far from the initialization, because you want the DPO model to fix the subtle mistakes, while at the same time keeping most of the properties that you got from the SFT stage.

• 1:10:35Tim Scarfe?

  And how do you detect degeneracy? And presumably, you use that as a lens to synthesize or select these positives and negatives for DPO. So how does that process look like?

• 1:10:49Speaker 03

  It usually comes from many different places. The evals is one of the primary ways to detect this. You can define metrics for each of the degenerate subclasses, infinite generation being one, repetitive generation, or based on looking at the error patterns that you yourself notice or reported by the users of the model. You can, and there are some error patterns which are not captured by your evals, which is a good sign that you need to build a new eval to capture these cases, because it's likely to happen in future models as well. And once you capture the error patterns,

• 1:11:31Speaker 03

  then you would have a data generation phase, which would involve reproducing these kind of bad patterns and having, pairing up with a ground truth equivalent in those cases. And it totally depends on how you source this dataset. There are many, many, many ways to do it. You can have a ground truth, transcript, annotated dataset, and you can run your model. And you can take those subset of cases where it produced any of the, triggered any of the detector that you had for degeneracy and generate a paired sample out of that.

• 1:12:13Tim Scarfe?

  Very cool, very cool. Let's move back to TTS. So the technology is getting incredibly good. You folks have a TTS model. I mean, what fascinates me is like, where does the emotion come from? You know, like, where does, how does it even do that?

• 1:12:30Speaker 03

  Yeah, the model is, the way the model is trained is like the exact opposite of the ASR tasks. The input would be text and the output would be audio. In our case, the loss is a flow loss, which is an L2 loss for velocity estimation. And the model is trying to model a distribution of human speech from whatever dataset you have. And any reasonable dataset that you have where there is enough variability of human speech, which you should, we should see from any, any audio recording, I suppose.

• 1:13:14Speaker 03

  And the model makes associations, like any other deep learning model, between the textual semantic content. So the first level of mapping it needs to do is map the words to appropriate phonemes implicitly. Older TTS models explicitly did it, like phonemes were either generated or it's an intermediate stage of the model. But in the more recent versions of TTS architectures and the one that we actually released, it's completely end-to-end. And this kind of association happens somewhat latently in the model. And a first level of mapping is to just get the, what has to be spoken, the high-level thing.

• 1:13:58Speaker 03

  And then the next level would be a function of the semantic content of the text. You can have text based on the semantic content. You should be able to, it's not always true, because you can have neutral text, which depending on the emotion of the person can be spoken in different ways. But there's some text which is only spoken in certain way. Like, for instance, you can have sports commentary. So if you read a sports commentary, it would have lots of ups and downs, depending on what's going on the field. You'll have very excited intonation ups when there is a exciting event going on, like a goal being scored or a home run in the context of baseball. You can have all kinds of ups and downs in a game.

• 1:14:43Speaker 03

  And just based on the text, you should be able to tell reasonably unambiguously where the emphasis should be. And the models learn that. And for all the other places where there is multiple options, I think one of the aspects of the TTS learning is this one-to-many learning, which is somewhat, in one way, distinguishes itself from the ASR task, where it's like mostly a many-to-one mapping. In TTS, it reverses itself from one to many. It's a multimodal distribution, I suppose. And that's one of the reasons why techniques like flow

• 1:15:27Speaker 03

  or diffusion-based techniques work really well, because it models the multimodal distribution quite well.

• 1:15:34Tim Scarfe?

  Yeah, and is there a bit of a trade-off between kind of the naturalness and the variability? Because I mean, I'm sure many folks at home, you know, you go on 11 Labs and you generate a bunch of TTS utterances, and they vary quite a lot. And you know, humans are incredibly good at detecting change in tonality. And also, there's a lot of meaning in tonality. And what we don't want is to have TTS systems or voice systems that are disingenuous, right? So what we tend to do is we just do many, many generations, and we select the one that sounds most natural. But obviously, like, in real-time production systems, you've just got to take the one that's generated, because you can't go back and regenerate many, many times.

• 1:16:16Tim Scarfe?

  So is it, is there something that, you know, do you take a position on this? I mean, do we, do we want to have something that's quite stable? Or, you know, very human, but run the risk of it being disingenuous?

• 1:16:27Speaker 03

  Yeah, in the, in the case of, to answer the, start with the question around what is our stance on it? Our stance is making it customizable. So as a user, you get to choose which mode you want to operate the model in. And some of the customization, there are a few knobs to control the behavior of the TTS model. And the biggest one is the voice you pick from the system. And the model does support like voice customization. So you can give a recording of a person. The persona you want to give the voice is derived from the reference. So for short references where there is quite a bit of ambiguity,

• 1:17:12Speaker 03

  there is opportunity for more variability. But even there, you can control the behavior of the TTS system by having voice reference, which is quite neutral and quite safe and less variance. And on the other hand, if you're using it for a creative use case, which is not real-time, which is also a perfectly good use case for both our TTS model and many other TTS models, then you would actually want to pick a more variable, somewhat even wild variances in some cases, because you're looking for different candidates in your creative process to pick from. And that's actually a feature, not a bug.

• 1:17:55Speaker 03

  And so depending on the use case, you can pick. The property, as a research scientist and as a research team, when we are training the model, one of the goals that we want is to train a model which is general enough that it can represent both behaviors in the same model and control that behavior through the knobs, a voice being the primary one. But we're also planning to expose instruction-following-based control on how to generate these things. And And hopefully add generally add robustness to the model like there is an equivalence to

• 1:18:37Speaker 03

  infinite generation or hallucination in the TTS world is very similar to ASR. You can say the word that is not there or skip the word that is there. Or keep repeating the word and going to infinite loop. All similar problems exist, although the goal is to be much more robust to these problems. Because this model will be used in real time context and it's much more jarring to see these bad generations in an audio output as opposed to a text output because you have to listen to them. So we aim to make them much less prevalent in this. But based on your use case, if your tolerance is extremely low, because you're putting it

• 1:19:21Speaker 03

  in a real time context in a customer service agent or something which interfaces with consumers at scale where you have hundreds of thousands or millions of consumers, then picking a safe voice seems like the more conservative option to go. But that also depends, I suppose, because you can you can have a

• 1:19:46Speaker 03

  there are all kinds of audio agents now, not just for professional consumer service. There are personal use cases where people do, users might expect more variance from their audio agents, in which case it becomes a trade off based on your use case.

• 1:20:03Tim Scarfe?

  Yeah, and what kind of use cases are you expecting your customers to, you know, to derive from these models? I think if I understand correctly, you're not currently going for the long form generation. It's still quite short form.

• 1:20:15Speaker 03

  Yeah, it is a short form generation model. It can generate long form audio. It's just that long form audio has properties which are a little different from short form audio, like imagine generating a podcast or an audiobook. You like like it to stay consistent, but at the same time, you would want inflections long form consistency and variability in the long form. So in some cases, so that it's not monotonous. So the considerations are a little different and you can put together a system which can leverage the short form TTS to produce long form. Again, the TTS model is also contextual, similar to the ASR. And the hope is you give it more context, you can give it previous conversations, so

• 1:20:56Speaker 03

  it can continue the generation, so that can help you keep consistency across a long form generation. But the primary use case is short form generation because the deployment scenarios are voice agents and primarily enterprise focused voice agents, whether it be customer service or productivity assistance, which are customized for enterprise use case. Like I was mentioning, one of the things in Vibe Work and Vibe Code is ability for people to also make custom AI agents for their workflows.

• 1:21:37Speaker 03

  And the hope is you have a similar lever with audio agents as well, where you can customize an audio agent for all kinds of one off use cases in your enterprise context. You you can deploy them like for all kinds of use cases in an enterprise. So in these cases, there are all productivity assistance in some way, but they are aimed at a slightly different thing. So those are the main use cases that we are targeting with the system, which is productivity assistance and something which interacts with your external customers through customer

• 1:22:18Speaker 03

  support being the top one, but that itself is a big category with a lot of subcategories and there are adjacent ones as well, similar to that.

• 1:22:26Tim Scarfe?

  Very cool. Now, if I understand correctly, so you folks are about sovereign AI and allowing engineers to build systems from their component building blocks. So engineers would build cascades from your models, they would integrate it into their systems and they would wire them all together. That obviously that that gives incredible flexibility because it's running on premises and it's small models and they can fine tune it, they can control their data and all of that kind of stuff. But you know, is it also quite difficult as well? Do you provide recipes? I mean, how do you make it easy for folks to compose all these things together?

• 1:23:02Speaker 03

  Yeah, so one of the things we also have is a applied, science applied engineering team, which is also like a forward deployed engineering equivalent. And the main idea behind this is we do provide the building blocks for you to make workflows or proof of concept exploratory product, product slash applications in your enterprise context or any any other AI related tool in the context of your enterprise. And usually there are cases where you already have a preset opinion and experience on what

• 1:23:44Speaker 03

  you want to build and you're just looking for building blocks to just use in what you're building, in which case you it's kind of like a combination of self serve plus interaction with our company to get that going. Or in the in the other case, like model adaptation or customization or where it's a slightly open exploratory project. In that case, the usual workflow here would be to engage with our applied team and work

• 1:24:26Speaker 03

  together and our job would be to make sure your use case, the success of your use case and finding the best way to get it done with the tools that we already have.

• 1:24:40Tim Scarfe?

  Yeah, I mean, just to sketch out an example, if I if I wanted to make a customer support agent using this tooling, so I guess I would have some kind of database with intents in, I would be using a language model agent and then I would be using TTS to like, you know, to generate some kind of conversation flow. And then I would be using ASR to listen to the customer when they said something. But some of the complexities here, I guess, are when the TTS is talking, the ASR needs to be turned off because otherwise there would be a feedback loop. And you see what I'm saying? So so we would kind of construct this cascade, but there's a little bit of complexity there just just to stop the system from going crazy.

• 1:25:20Speaker 03

  Use cases precisely this kind of stuff. So you want to get started right away. You don't want to spend a lot of time building the scaffolding, which has already been somewhat solved. The cascaded system itself will have limitations owing to it being a cascade. But if you want a state of the art cascaded system today with the state of art components and the components we are building, hopefully are pretty state of art, especially for your use case. At the end of the day, you care about whether it's frontier for your use case, not frontier on some external benchmark.

• 1:26:00Speaker 03

  And you can adapt either of the components in the case, like we were talking about how to adapt a ASR model for your acoustic settings and the same with the TTS.

• 1:26:12Tim Scarfe?

  Yeah, because I think that this is the thing that I want to get across, that when you can do adaptation and set constraints, you can always strictly win. Because a lot of people at home, they might just be saying, oh, you know, why couldn't I just use an Omni model or GPT Live or something like that? And there's almost too much flexibility, right? So I'm I'm in large enterprise and I want to have an observability layer. I want to have hard constraints. I want to say when people are talking about this, it needs to be logged in the database or this needs to respect this security boundary, it needs to be running on my server because we're fine tuning it with my data. And and you see, we can we can do a lot of customization. And when we do a few iterations of that, we can actually make something which is strictly better for my particular use case.

• 1:26:56Speaker 03

  Exactly. I think the current voice stack, owing to it being a cascade, also gives you a lot of observability into the system, uninterpretability, because each of the components, the interface is natural language, which is quite, which makes it quite interpretable. And on top of it, these audio agents. The L L M component of it has a text counterpart in many of the cases. Like if you are deploying a customer service agent, you probably also have a text version of this customer service agent uh where people can chat with it over a text-based interface to get your problem solved.

• 1:27:38Speaker 03

  Like if you have a food ordering service and uh you probably already deployed a text-based interface for uh dealing with customer complaints on delivery problems or uh getting the wrong stuff. And you probably already have a safety system on top scaffold uh observability, traceability um and all other uh enterprise uh compliance layers added on this. And in in this case, for instance, you just want to add a audio interface to this system that you already have. And this allows for you to build it fairly easily because uh in the audio agent, your

• 1:28:23Speaker 03

  goal is to not uh a uh you could be, but in many cases, your goal is to just give a more pragmatic, practical interface uh to the system through audio. So you can make a phone call and interact with it. And the goal is to not have the most uh personal emotional conversation through this uh system, but rather just get the task task done. In which case uh it is a fairly powerful uh system uh to replace with a more uh end-to-end approach. Eventually, sure, I suppose, but today uh these of the level of flexibility and control you get out of these systems is uh uh quite nice.

• 1:29:07Tim Scarfe?

  And what does the frontier of voice tech look like today?

• 1:29:11Speaker 03

  Yeah, a lot of places, uh I think there's a lot of uh claims about even the basic A S R task being solved. But when we talk to customers, uh we spoke to a few customers just a couple of weeks back and the primary uh complaint is uh it's far from solved in the precise scenarios they're deploying it and it makes a ton of mistakes. Even for the most prominent customer service case, uh they feel it's not solved. They have to add so much scaffold just to deal with all the corner cases that you encounter when you deploy an audio agent over millions of sessions. So there are there are components around which is not necessarily model-related problems.

• 1:29:53Speaker 03

  Uh but there are more very much model-related problems as well, including things, basic things like the model not being good in languages. Uh it it has a sharp drop-off uh outside of the top languages. Again, owing to the data, there's a lot of English uh audio data in the wild. Um similar to it's a very similar story on the tech side as well. Uh but in audio case, it presents itself as the uh transcription quality going down significantly when you go off the uh top languages, uh when you go off the acoustic. Like, take an example where a audio system is deployed on a factory floor, which is

• 1:30:35Speaker 03

  fairly common uh use case, I suppose, be- because you want to provide a interface to control an automated system, not necessarily an A I agent, but I'm sure it would be A I agents uh uh going forward because they can help you uh solve hands-free uh where when you are already uh doing certain other job or uh managing heavy machinery or something like that. And in those cases, uh you can have uh all kinds of background sounds, noise, a lot of people speaking uh with each other, there's a crowd noise going on. Uh so in that setting, for instance, getting especially for an interface to a command

• 1:31:18Speaker 03

  system, you need to get it very much right. And uh in many, many cases, uh the current systems fall fall short. And that's one of the reasons why the customization is quite interesting. Um it's uh while we always aspire to incorporate more and more variability into our general purpose main model, uh there's always a data or distribution which is not captured uh uh by the uh which is probably so unique that it is only uh in your context that you see that kind of uh audio. And the model is reasonably good, but you can always make it better.

• 1:32:02Speaker 03

  Uh and since it's not unlike text models where I would believe it's also true with even any general purpose reasoning text agents that you can always make them better for use case, but the cost of doing so is so high. And the general purpose model keeps improving uh year over year where the the math changes and you'd rather use the general purpose model and hope for the best. Uh but in audio, that's you don't need to be constrained by that. Uh the models are fairly small, customizable, and you do have data to customize and the cost of experimentation is easy. You're not trying to uh uh fine-tune a one-T model here.

• 1:32:43Speaker 03

  So I think it's uh uh easy both logistically, where you don't need that much G P U compute to do it, and also easy from the data side. It's a fairly simple task. Uh it's it's hard to get the quality control right, to get the annotation uh quality high, but at the same time, the task is fairly easy to describe. Just write down what you hear uh for this uh specific piece of audio. So so yeah, I I think um in the wild uh performance is the main thing that is not not solid for a wide variety of settings and language coverage is uh one of the primary things I I I always hear when it comes to audio models.

• 1:33:26Tim Scarfe?

  Yeah, and I think that voice tech is going to become quite ubiquitous over the next five years. I think voice just seems to be almost a special case. I I don't know whether whether you would agree with that. It's something that people will increasingly kind of train and it will be embedded into their ways of work and and their life in a way that perhaps a large language model might not be.

• 1:33:46Speaker 03

  Yeah, it's a voice is uh one of the primary ways humans communicate. Uh it definitely came much before text was a thing. Uh and even today, there are a lot of places where voice is the main way people communicate. And even uh in text-heavy uh environments, I think voice is still uh maybe not the primary way people talk like in work context, but it's still like we have meetings all the time. Uh you you don't need to have meetings, you can just work over Slack. But it's a adds a level of uh depth which cannot be conveyed through just purely textual interfaces.

• 1:34:27Speaker 03

  So it's always been the case that voice would be a big uh interface uh uh for interacting with machines and may even other humans as well. Uh and it's always envisioned in like sci-fi and that's one of the goals. Uh I I worked on Google Assistant for a while and that was actually the goal there as well, for instance. But I think the previous generation of technology was uh not not quite there uh with a uh it even today's caskets are much more powerful than the previous generation of cascaded systems. And but I think uh today's technology is at a point where this very seamless uh omnipresent

• 1:35:12Speaker 03

  uh voice-based assistance uh is becoming more of a thing. And uh the cascaded approach uh itself can solve a lot of the problems, but I think that's where uh more end-to-end uh duplex approaches are also coming up because the technology is making it possible to build more robust duplex approaches. It's not sufficient to have like a prototype demo. Uh it needs to actually work in a uh real-time context and all kinds of weird uh corner cases that you would encounter on a day-to-day basis. And uh whether it be a real-time translator to make uh make it more accessible to talk to someone who doesn't speak your language or as you were alluding to as a language

• 1:35:57Speaker 03

  learning tool. Uh I mean, there is a primary thing where people try to learn language by reading it. It never happens that you have to talk to someone for regularly to learn a new language and with a uh voice agent uh which can actually uh capture the nuances of the target language that you're trying to learn and also capture the subtle mistakes you make in pronunciation or uh aspect of uh you speaking that language, you can improve much more significantly. So there's a lot of opportunity in language learning. And also just as an interface to uh A I agents uh for uh getting as a orchestrator to getting work done.

• 1:36:39Speaker 03

  Uh it probably the interface to this would look like how a human-human interface would look like in today's uh organizations where uh you're delegating work uh to other humans uh or collaborating with other humans to get your work done. And uh you speak to them uh many times through a meeting and I I I can imagine it might look very similar with the interfacing with the A I agents which are also getting your work done and you can describe what needs to be done uh in the in in an audio uh interface. And uh they can communicate back in audio. It's not the Exclusive way of communication, they can communicate through text or whatever other

• 1:37:23Speaker 03

  channels, but audio is definitely going to be one of the key pillars of interfaces, I suppose.

• 1:37:32Tim Scarfe?

  I don't know whether you would agree that in some ways, it's it's actually cognitive debt promoting. So I love listening to audio podcasts and it's it's a non-interactive stream of information and you know, audiobooks and it goes into my mind. And for closed-ended applications, it's wonderful. So like we have a, let's say, I build an application with a very clear interface and I'm talking to the voice agent. And if I already understand the interface and I understand the information, this is this is very efficient. But if I use a voice agent for doing coding or my email triage or researching a podcast, that's an absolute nightmare. And I'm trying to understand why it is. I think it's because when I have the text in front of me using an agent, I can very

• 1:38:14Tim Scarfe?

  rapidly kind of select interesting branches of inquiry and I can say, go there, go there. And it might just be because the voice agents aren't very intelligent and sometimes I don't trust what they're doing. So when I when I tell it to do something in Codex, I don't know whether it's using my existing Codex model. I don't know whether it's giving me an answer or whether it's routing the answer to the Codex model. I don't know whether it's making a new Codex session or using an existing one. And and I feel a little bit blind. So you see what I'm saying? It's kind of like magical, but there's also this weird new form of cognitive debt at the same time.

• 1:38:44Speaker 03

  Yeah, that's a great point. I think one of the things, there was one experiment that we did a while back, not here, but basically if you try to get people to order, make a food order, people try to make it through a pure audio interface and it's much harder to do it because when you have a menu in your hand, you see all that. So you are talking to the person who is taking the order, but you're doing two tasks at the same time, absorbing the information, making a decision, going back and forth and also communicating with the person. And if you do it, imagine not having the menu. So the person will read out, the person serving you would read out the full menu

• 1:39:29Speaker 03

  and now you have to like, like load it all in memory and you'll forget like, because new items are coming in under rapid fire succession. So it's just not the optimal way to do it. But it's at the same time, people still do, at least a subset of people still do enjoy interacting with the waiter and making the order while having the menu. I think the same philosophy applies to these voice interfaces too. It's still very nascent, so I'm sure it will evolve over time, but basically it would look like a mix where you have your favorite IDE or interface, basically a visual medium

• 1:40:14Speaker 03

  also, and voice is a augmentation or auxiliary thing on top. So you don't need to worry about, the visual medium provides you two things. It provides you a way to ingest information rapidly and go back and forth. And also it provides you confidence and proof of what it, what it says it's doing is actually what is being done. So you have like two ways to validate what is going on. So yeah, I think like there are places where pure audio stuff makes sense, like if you're wearing headphones or glasses or all these new new gadgets that people are exploring now, their audio only makes sense because there's no screen and you would probably

• 1:40:59Speaker 03

  not do certain things there, like probably wouldn't make, it's like the favorite example of making flight bookings, because it's a costly endeavor, right? If you get it wrong, you need to do a bunch of stuff to undo it. So that might not be the right task for that specific thing, but you can still do it in an audio interface when there is a screen at hand, for instance. So it would probably be an auxiliary communication medium on top of what you already have and which works best, people will experiment and figure it out. And to your other question on the audio models being less intelligent than the text counterparts

• 1:41:41Speaker 03

  today, I think that's also just as a artifact of the way they are developed today and there's no fundamental reason for them to be any less intelligent than text model, especially when it's a single model. And even even if that is the case with, I don't know how some of the other models are developed, I can only speak to the models that we are developing and it could be the case initially, but there is no fundamental reason for it to be any, any less intelligent or any different from a text-based model.

• 1:42:17Tim Scarfe?

  Yeah, I suppose so. I mean, it's all a trade-off at the end of the day because there's always a bigger model.

• 1:42:22Speaker 03

  Yeah.

• 1:42:22Tim Scarfe?

  So and it's not just the bigger model, this bigger model might have more context, it might have access to my memory system, it might be doing Google searches and whatnot. So that, you know, there exists some perfect cascade.

• 1:42:34Speaker 03

  Yeah.

• 1:42:34Tim Scarfe?

  And we're kind of trade, we're trading off intelligence and latency and a bunch of different things. But no, it's fascinating. I think we're going to have a lot of fun over the next five years, just kind of engineering all of these systems and figuring out how can we, it's quite interesting as well that, you know, maybe the reason our memory is so bad is because we've become so conditioned to having these visual user interfaces. And maybe when we use voice agents again, our memory will improve and it might be less less of an issue. Wonderful stuff. Pavan, it's been such an honor having you on the show. Thank you so much for joining us today.

• 1:43:05Speaker 03

  Thanks a lot for having me on the show. It was really fun chatting about audio and all other research efforts around Mistral. Yeah, hope to see you again. See you again.

• 1:43:22Speaker 05

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' health care to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House for Jody Pack.com. Not authorized by any candidate or candidate's committee.

• 1:43:52Speaker 06

  Your heart can tell you a lot about your health. Apple Watch Series 12 measures your heart rate every five seconds with the most accurate heart rate sensing and wearable. So your vitals app, now with heart rate variability, can tell you when something is off. And your readiness score can let you know when to rest and when to push. Hear the story in every heartbeat with Apple Watch Series 12. The features described are for wellness purposes only and not for medical use. iPhone 11 or later required. Based on Apple conducted study of heart rate accuracy, August 2026. Visit apple.com slash Apple Watch Series 12.