How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen

Tsung-Hsien (Shawn) Wen, CTO of PolyAI, tells Tim Scarfe why voice agents are harder than text agents. Voice adds time, and a good conversation depends on adapting to the person on the line, not just on reasoning to the best answer. Shawn describes an audio-native model (Dialog-RSN-1) that first predicts a turn-taking signal, then replies in text with citations, and writes the transcript last so enterprises can audit it. Along the way: training on real, noisy calls with synthetic noise added, and why over-cleaned audio made the new model worse. Latency, and what a voice agent should do while it thinks. Why a voice with a hint of regional accent beats a generic one. Why public benchmarks fall short for voice, why enterprises want to own their agent harness, and whether behaviour belongs in the harness or in the weights. The last stretch is about working with agents: cognitive debt, the shift from producing content to checking it, Wispr Flow, building tools that agents can use, and whether slop is in the eye of the reader. This episode was produced in partnership with PolyAI.https://poly.ai CHAPTERS0:00 Why voice agents are harder than text4:27 What enterprises want, and why PolyAI built its own model9:04 How an audio-native voice model works15:21 Training data, spectrograms and synthetic noise20:36 The cocktail party problem and the future of turn-taking25:21 Latency, adaptive reasoning and keeping callers' trust31:33 Voices, personality and the uncanny valley36:32 How do you benchmark a voice agent?42:00 Harness engineering and owning the intelligence45:09 Well-specified problems and auditable agents49:46 Weight adaptation and cognitive debt56:31 Agents at work: Wispr Flow, voice and tool building1:02:50 The next decade of voice, and what counts as slopREFERENCESThe Bitter Lesson: http://www.incompleteideas.net/IncIdeas/BitterLesson.html [9:05]Retrieval-augmented generation: https://arxiv.org/abs/2005.11401 [13:23]Mel scale: https://en.wikipedia.org/wiki/Mel_scale [17:16]Victor Zue: https://en.wikipedia.org/wiki/Victor_Zue [17:45]Cocktail party effect: https://en.wikipedia.org/wiki/Cocktail_party_effect [20:39]Speaker diarisation: https://en.wikipedia.org/wiki/Speaker_diarisation [21:16]

• 0:01Speaker 00

  This episode is brought to you by Michelob Ultra. Good times mean competitions between you and your friends. Like seeing who can throw some pigskin the farthest or rush the field the fastest. Winner gets an ultra because it's not just a superior like beer. It's also giving fans superior access to their favorite sporting events. Superior is worth playing for. Visit michelobultra.com/superioraccess to enter. Enjoy responsibly. Copyright 2026 Anheuser Busch InBev Michelob Ultra Beer, St. Louis, Missouri.

• 0:30Speaker 01

  Are your ad campaigns lighting up the dashboard? But not the pipeline. That's bull spend and marketers are calling it out in dashboard confessions.

• 0:39Speaker 02

  My boss asked for results, so I opened my dashboard for the only positive sounding metric I had. Impressions.

• 0:47Speaker 01

  Cut the bull spend. See revenue, not just reach. LinkedIn delivers the highest return on ad spend of major ad networks. Advertise on LinkedIn. Spend $250 on your first campaign and get a $250 credit. Go to LinkedIn.com/campaign. Terms and conditions apply.

• 1:00Speaker 03

  They want the agent to be super powerful like a human being, but they want the agent to be super controllable like a robot. Voice AI is very important because that's the native way that human talking to each other. And it's the most nature interfaces. Computers has not been at the same level as human in terms of voice conversations yet, but we do actually foresee that this should be the future in the next couple of years.

• 1:25Speaker 04

  Do you think that speech tech has become commoditized or is that just completely wrong?

• 1:30Speaker 03

  It becomes a lot more commoditized in a way, but also in a lot of different ways is still not. But modeling the real time conversation, it is still hard and it's not quite there yet. The bottleneck has already shifted from producing content to actually validating and auditing the content.

• 1:51Speaker 04

  This show is made in partnership with Poly AI. People think that, you know, voice assistants are a solved problem, but it's actually much harder than people think. Why is that?

• 2:02Speaker 03

  When ChatGPT came along, we thought that yeah, voice is probably going to be solved very soon. And I think a lot of people have the same idea. And then that's where you see a lot of rapper companies come along, just rub along ChatGPT, you know, piping those ASR and TTS together and then try to solve the problem. The reality is that it's actually harder than we think. And we kind of like, you know, settle on this idea that actually voice is, if you think about voice, the voice is actually one step to the real world. Where if you think about all the current innovation on the AI front is predominantly on the text and the digital world, where solving a PhD grade, you know, physics problem seems to be not a problem anymore because you have a really good documentations.

• 2:47Speaker 03

  You have a lot of like online papers you can research on. While in the in the voice scenario where you actually have, you have an additional like access, which is time, right? And then time is a very critical matter. I think we have not trained the model to be able to better understand that concept yet. And I think, and because of that one step into the physical world, we started to realize, okay, well, there's actually a lot more complexity when you add that, you know, time element into it. And also like with the humans is about the intelligence here is not really about reasoning. It's really about adaptation.

• 3:27Speaker 04

  When we spoke last time, you said that, you know, doing doing voice technology, it's a little bit like robotics, right? So it's incredibly easy for humans. You know, there's the famous Wozniak Coffee Cup Challenge, which is I want a robot to make me a cup of coffee.

• 3:40Speaker 03

  That's right, yeah.

• 3:41Speaker 04

  Just tell me a little bit more about that. I mean, why is it so difficult?

• 3:44Speaker 03

  Again, get back to the physical world situation, right? Because in the physical world for humans, we collecting data in the physical world very naturally. And we have a lot of data because we have visual data. We have like, you know, the body movements, your hands can reach out to things in the world. However, computers don't have that yet. Computer, all they have, you know, all the all the tools, it's all, you know, about how to use the computers, right? So I think now we are, we have trained the model to really good at understanding text and understand how to use tools in that virtual world. But in the physical world, the data is still very, very little. And that applies to robotics.

• 4:27Speaker 03

  That's why a lot of the robotic companies right now, as you say, is trying to, you know, grab the coffee from one place to the other. I remember one of my really dear friend and also a really good researcher from Cambridge. He was originally at Poly and he now left to start his own business. And he wanted to do a robotic company. I was like, what is the first application you are trying to do? It was like, I want the bot to be able to pour beer for me. That's the test number one. I was like, oh, wow. Okay. Well, because it's just really, really hard. And then you look at self-driving car, you know, Elon has been promising the world, you know, this is going to be, you know, fully commoditized.

• 5:10Speaker 03

  I think we started to see the side of it right now. I think voice is the very similar problem, just that we have so much, so many conversations between, you know, between each other, but machines just never be able to observe them.

• 5:27Speaker 04

  And what do enterprises actually want from a voice agent?

• 5:31Speaker 03

  Enterprises wants a branded voice. They wanted to be able to pick their identity. They wanted to, you know, the agent to exactly say their branded name exactly the way they wanted to. So that's number one. Number two, they want the agent to be super powerful like a human being, but they want the agent to be super controllable like a robot. So it's actually a very contradiction. And I think that's the difficulty of deploying AI agents into enterprises because the expectation of people towards AI is they just need to be as good as human. And actually as good as my best human in the team, right?

• 6:12Speaker 03

  And I think that's the, that's always the adoption curve is like, you know, harder to justify. Until, unless you can find a particular area, you're saying that, hey, here is a more about efficiency play. And then people started to adopting it and started to feel, okay, well, actually agent is actually better than I thought. That's what we are seeing from the enterprise adoption right now.

• 6:36Speaker 04

  And why did you guys decide to build your own custom model rather than using some of the existing ones out there?

• 6:41Speaker 03

  We have been doing research and building our own models for a very long time. We built our own speech recognition model previously. We built our own larger language models as well. Around one year ago, we decided that Cascaded is going to be outdated in the future. And the end to end model would be the future. And then we started, you know, to do a lot of experiment and try out. And we decided not to invest in our speech recognition models anymore because of that radical shift into the future. And when we started to look into the market, the reality is that all these latest speech to speech models, they are really good for the demos because I can show you really natural conversation.

• 7:23Speaker 03

  You'll be like, wow, that's, you know, singularity already. But once we deploy into the contact centers, it breaks in so many different places. The interruptions sometimes is a bit too aggressive. So if you are on the phone with an old lady and the old lady is the first time talking to the bot and she was just constantly confused and, you know, stop. And then the agent just keep, you know, speaking over her. That is like a level of frustration, you know? So we see a lot of these scenario, which is say that, hey, we probably have to really train our own model for these use cases because I think the frontier labs are not optimizing for what we care about.

• 8:07Speaker 03

  It's a real conversation turn taking for the business use cases. I think consumer use cases, I think you can build voice agents already. For enterprises, there's additional governance there. There's additional like, you know, enterprise requirements that we have to satisfy.

• 8:26Speaker 04

  And how adaptable could that be? You know, for example, if I'm talking to an old lady on the phone, I would adapt my speech. I would slow down a little bit. I would, I would give her more thinking time. And could the models be adaptive in that way?

• 8:37Speaker 03

  Yes. So that's why you have to collapse all these modules together into an end to end model or some sort of end to end fashion in a way. Because the traditional Cascaded system, you have a speech recognizer, you have a larger language model and you also, you have the turn taking on top. And that is actually making the end of the sentence detection. These three models don't really work with each other very well. And you have to tweak the parameters. In a traditional way, you can have another detector to say, oh, this is an old lady. So I have to tweak the parameter here and there, but it's all very crunky. And then reality is that it doesn't really work like that. And it doesn't really work very well as well.

• 9:21Speaker 03

  So therefore it's almost necessary to collapse all these pipelines together. So you have a large language model directly perceive the audio. And the audio can be, you know, slower pass, you know, the speech. It could be background noises. It could be cross talks. It could be a lot of these, you know, scenario you were run into when you actually have a real conversation with a human over the phone. Then you can collect those data so the model can adapt to that. So I think collapsing it is the first step. And then the initial model needs to be good enough so you can continue to iterate on that because you use it to collect more data as well.

• 10:04Speaker 04

  I suppose it all comes back to Rich Sutton's The Bitter Lesson, which is that, you know, in principle, maybe we could write the code to do this, but there are just, there's an exponential number of failure modes, like combinations of bad things that might happen at the same time. Like, you know, in this situation, we need to do this or here there's some cross talk or here there's some noise in the background. And just training a neural model end to end, you can, you can cover a lot of those failure modes. But I suppose though it still needs to be a cascade in some sense because it's not possible to embed frontier level intelligence in these voice models. But you could potentially draw a boundary around the voice intelligence model and then it could call out to an agent when it needs more, more intelligence. But I wanted to ask, so can you just sort of sketch out that the high level architecture for this voice model?

• 10:47Speaker 03

  Yeah, so the higher level architecture is that this is basically a audio native model. So the base model itself is a larger language model, so it actually output text. So we are not training models to do end to end speech yet. The reason is because enterprise is one level of control. So text is a lot easier to actually apply Gabriel on top, while speech is a lot, you know, fiddly to deal with. And so we train the model to output text. But then, at the same time, we allow the model to be able to collect A S R into his in input channel. So that means that the model itself natively understand audio.

• 11:30Speaker 03

  The audio you piped into the the model is that you are streaming audio chunks into the model. Then the way that we are structuring the model is that the model is doing multiple things at once. So first of all, the model would be based on the streaming in all the audio. It would predicting a first token. The first token is basically the turn taking signal. The turn taking is basically saying that, hey, is the speech, you know, just getting started? Or is it still, the user is still speaking? Or is already end of the speech? Right? So that initial token is very cheap to generate. It's just like the first token in the output. As audio comes in, we keep generating that token.

• 12:13Speaker 03

  Until the model generates, okay, this is end of the speech. Then we started to have the L M doing the product processing. But at that point, all the audio has been already pre-processed. Because as we encoding them and then predicting the first token, it's already in the embedding space. So you don't need to encode that sentence from scratch in that case. So then, the next, the next job of the model is to directly predict an output. So the output is a text, as we say. So it could be a response, or it could be a tool call, for example. And that is all streamed, like the typical HTTP API. So then we are streaming that response is back.

• 12:58Speaker 03

  And then, at the end of that response, we then continue to predict a few things later. A few things that, you know, we will also generate citations. So that, for example, if you have a knowledge base with a lot of different contents, the model will then generate a citation, say, hey, actually, for this response, I'm using this fact and that fact to actually make the prediction. So enterprise knows that, okay, it's actually has a reference there. Then finally, it also stream out the transcription. So we kind of reverting, reversing the order, because the model's job is not to predict the transcription, but we still wanted to predict the transcription for, you know,

• 13:43Speaker 03

  auditing purposes, debugging purposes and things like that. But it comes at the last, because you don't have to wait for the transcript to actually start responding back to the user.

• 13:52Speaker 04

  Even that, that's quite a nice, interesting trade off to talk about, because, you know, many folks would have used like the G P T live model. And that is a bidirectional waveform to waveform model. And, you know, it's really interesting. It just makes all of these, hmm, you know, like these weird pauses and sounds and, and it's, it sort of breaks the uncanny valley. But, but actually, that might be great for me at home, but in the enterprise, you need to have constraints, because you need to say, no, you're not allowed to talk about this, you're not allowed to talk about that. And conceivably, I suppose, you could run separate A S R on the output, and then you could do the retrieval, augmented generation and linking to the knowledge base and stuff like that. But there are just all of these trade offs that, you know, actually in enterprise, it's not necessarily what you want to do.

• 14:30Speaker 03

  Retrieval itself, I mean, you can also do retrieval with these native, audio native models as well, because the audio, but the retrieval mechanism is slightly different. You cannot use a traditional RAG anymore, because the traditional RAG is that you embed the user transcription, and then you do a vector search in your vector D B, right? But because now you don't have that transcription, what we need to do is that we will prompt the L M to generate a tool. The tool is basically a search query. So let's say, hey, now, I think that in order to answer this question, I need to generate a search tool, and that search tool needs to come with that query. It's pretty much like Cloco style search, which is like, you know, you, you know, people

• 15:15Speaker 03

  call it agentic search, which is like just another buzzword, but, you know, it's basically just like a tool call, really. The important thing is that our innovation is not in the model itself. We are innovating at the I O contract. So it means that, you know, how do we actually prepare the data? And how do we prepare the input and the mechanism of the, the, the input, the data feeding into the model? But also, how does the model output that data? And then we also, how do, how do we train the model based on these input and output pairs to actually behave exactly what we want? So, to be honest, to underline model, almost doesn't really matter all that much, because

• 15:57Speaker 03

  all our research team is doing is that, because we know, in this market, open source model come out every month, even every week as well. So we kind of wanted to maintain our data, our model, and then we have these training scripts that, you know, always getting ready, whenever there's new model come in, we will train on top of it, and then to see which one has the better performance.

• 16:21Speaker 04

  And voice data is much harder to collect than text data. How are you doing that?

• 16:26Speaker 03

  For us, we don't have to collect them, because we, our platform generates a lot of voice data. So because we have been deployed to enterprise contact centers, we have banking clients, utility clients, logistic clients, restaurants, hotels, you know, a lot of these like retail outbound sale use cases. So we have already a lot of the lost data. And lost data is exactly the kind of data we wanted to solve the problem for, because we serve, you know, predominantly enterprise at the moment. So therefore, that we kind of use leveraging a lot of this data. Obviously, getting agreements with enterprise clients to train on those models is non-trivial,

• 17:09Speaker 03

  because all these, you know, data privacy, GDPR, P I, the data things like that. But, you know, kudos to our legal team. We actually get quite a lot of our customers to agree to that, because, you know, they know that if they agree, you know, the model can continue to actually help to improve for them. And the way that we are training the model is that we will redec all the P I data, so that, you know, now, none of the customer data is actually export proposed. And we will produce a lot of the, you know, fake P I data as well, so that model can still learn how to identify names and addresses and things like that. But really, the data is actually just to let the model learn the mechanics of the

• 17:52Speaker 03

  conversation, rather than the actual user data.

• 17:56Speaker 04

  Very cool. And I wondered, does the model currently take in pure waveform data, or does it take in featureized data? I just remember I was doing audio stuff for my P H D, and back in those days, it would have been unimaginable to put in waveform data. And now, now it's simply a case of if the model's big enough and you train for long enough, you can.

• 18:12Speaker 03

  I think currently, the standard audio encoder is that you feed in a male spectrogram, right? So like, you know, it's still like this, you know, image, you can, you remember, like, you know, there's like different spectrums, and then you have different frequencies and things like that. It is still that. But then, to the model, you are feeding that, you know, segment by segment, right? And then to the model, it feels more like an image recognition task. I remember, because you're from the speech background, so I remember there was like a M I T P H professor, I think his name is Victor Su or something. He was, he was, you know, back at that time, like in nineteen nineties or something.

• 18:53Speaker 03

  He was like, you will be able to identify what the human speech is by just looking at the spectrogram. And he was like, this is, you know, exactly that particular, you know, syllable, whatever. And people were like, you know, what do you mean? There's not, nothing you can identify there. So, in fact, machine can learn those patterns really well from it.

• 19:14Speaker 04

  I love looking at spectrograms, because there are so many things, you can just immediately see speech, or if it's like a violin, for example, that you just see these horizontal lines, and you can see that there's a harmonic series. And if it's percussion, it looks like a sort of vertical bar in the spectrogram. But yeah, after a while, as a human, you know, that you've been looking at these for a while, you can just read it like a book. It's fascinating. And noise, of course, just looks like, you know, salt and pepper. But on the subject of noise, though, this is quite interesting. So how do you guys deal with that? Is this a kind of thing where you deliberately sample noisy data? Or do you kind of like augment your data and add noise to it?

• 19:54Speaker 03

  Yeah, so it's a very deliberate decision to add a lot of different noise into that training data set. I mean, to be honest, our training data set by itself contains a lot of different noise already, because it's a real conversation data. But we typically would apply another generated model on top of it. So say that, you know, now I wanted the model to pronounce my mandarin name, Zhong Xianwen. And then, by doing that, I also ask these T T S models to actually add these, you know, additional audio data we are seeing in the real production into it. So the generation of the outcome of that generated data set could be a robot pronouncing

• 20:37Speaker 03

  my name, while there's a background, you know. Baby crying in the background, for example. So these can be kind of like simulate in a way so that you can have a lot of different variety of that. Because, you know, thanks to the M- Merton generative A I, you know, uh, in models that you can do a lot of like very realistic data on that. So the answer to that is yes, we do a lot of this and it's a very deliberate a- approach. Because we want a model to be robust against all these different environments. But even in the training the data ourselves is already contain a lot of that. So the funny thing is that, you know, because we, in the previous technology, we kind of doing a lot of like, you know, denoising

• 21:18Speaker 03

  or, you know, doing a lot of these like pipeline sanitizations beforehand. All those additional components to this new dialogue reason model, it's actually making it worse. Just because, you know, like it's too sanitized the environment, it it it has not seen before.

• 21:36Speaker 04

  I remember for many, many years that there was this phenomenon called the cocktail party problem. Which is that, you you know, like when when we were at a cocktail party and there's many people talking at the same time. We can focus our attention on one person and our brain can just filter out the the other people. And A S R models could not do this. But but now increasingly they can do it because maybe internally what they're doing is is they're learning, well, this is speaker one and this is speaker two. And they kind of focus in on on speaker one. So so yeah, cross talk has always been a problem, but you know, what, how are you doing with with those failure modes?

• 22:05Speaker 03

  Cross talk in our model currently, it doesn't actually have a specialized mechanism for that. But the architecture is suited to actually tackle that problem. Just because, I mean, the speaker dialization uh technology was actually previously predominantly applied to speech recognition. And it's always assuming in a cascaded pipeline solution. So then you you you tagging this is speaker one, speaker two, speaker three. But it's ve- it's not a very natural way. Because as a human, you, if you are in a party, you're listening to so many people listening. You will not actively label, this is speaker one, this is speaker two, this is speaker three. You'll be like, oh, this is a- a- another person, that's another person I shouldn't pay attention to. Because there are so many of them, right?

• 22:48Speaker 03

  And then some of them would stop speaking and then you will lo- lo- lose them for a while. I think the traditional speaker like dialization is trying to solve a very different problem. But with a dialogue system, what you really need to do is that that the agent focus on the right kind of subject that it is talking to. So in that case, we just, be- because now you can combine the L M with the speech recognition together. So the L M can natively perceive uh there's a background like, you know, called cross talk somehow. I mean, to be honest, uh uh this L M is also promptable, right? So you can prompt it to say, hey, don't respond to uh the the background called cross talk apart from the major speaker you are speaking to.

• 23:32Speaker 03

  You can also say whenever there's a cross talk in the background, take a the a- active part of participation in that conversation as well. So technically you can do all this. The major difference is that, how do you train the model? So how do you collect the data? What kind of data you collect? And what is the instruction you instruct the model to do? So like, you know, for example, the, I think that you you you watch the G P T, you know, uh, live, uh, the demos where you have a lot of these, like, you know, old ladies coming along and then there's like, you know, the model will be able to identify new speaker. I'm pretty sure that in their, in their prompts, they are saying that, hey, now you are in this party, you know, party like conversations.

• 24:16Speaker 03

  And then you have to gradually identify speakers as they come along. So the model, once you fuse these model together, it has the ability to do that. In in the ma- then the major difference is that, how do you train it? What do you train it to to do?

• 24:31Speaker 04

  The turn taking thing in particular, we spoke about that earlier, it's really, really difficult. And I suppose we should imagine in in an ideal world in ten years' time, when this technology has reached the utmost maturity. Maybe we can have microphones in every single meeting room. And when there's anything private discussed, it it will know not to pay attention to that. And there might be multiple people involved in the conversation. And you know, you can address the agent and the agent can get involved and the agent knows who's talking. But um so so at the moment, we're we're kind of solving it on in the one on one case and we're we're solving this turn taking problem. But is is that a reasonable um estimation of of what the future might look like?

• 25:06Speaker 03

  I think the technology will be complete. I think, I think you were do for, you you know, prediction is right. I think the technology at that point, ten years from from now is a lot of time, you know. At the speed of all these technologies is improving uh right now, I think it's completely really, really reasonable for the future. I think, though, the only thing is that the voice modality, I mean, people say that it's the most natural mo- modality to interact with a computer. The reality is that we kind of have a several iteration of the of these kind of personal systems in the past, started with like, you know, Siri. And then you have Google Assistant, Alexa, you have the in in car voice assistant, whatever. Maybe the technology was not really mature yet.

• 25:49Speaker 03

  But then also there's just a little bit of this like voice is a lot more personal and private to you. And so then I think, I don't know whether the consumer behavior will be there already yet. Because that means that you have to surrender a a lot of your control to the agent. Which, you know, a lot of people would say that that's a great thing, you know, be- be- because, you know, the agent can do a lot for the for you. But then I I think there will be some consumer adoption curse curve that we still have have to climb. You know, even though the tech- technology is here.

• 26:21Speaker 04

  And can you tell me about how you guys did latency engineering? Now, you know, we're going to talk about this uncanny valley and all the things that we need to do to make it kind of, you know, for people to suspend their disbelief. Because they're talking to a voice agent and it's suddenly really good, but latency is probably one of the most important things. And when you go lower latency, there's always a bit of a trade off. You might be trading off some of the intelligence and you guys have best in class latency. So how have you done that?

• 26:44Speaker 03

  Latency, first of all, uh, the major thing is you need, to be honest, all these model processings, if you are using the right size of model. The model processing itself latency is not really a problem. Historically, the cascaded system struggling in waiting for to decide when is the end of the speech. Because there's like this very coarse, you know, pa- the parameter you can tune is that how long you wanted to wait until the end of the speech. And then if you are tuning to be too sure, it could be too aggressive and you are going to interrupt a lot of uh all the other p- people, for example. If you tr- tune it to be too long, you will be waiting for too long and a lot of people will be impatient.

• 27:28Speaker 03

  So everyone has a different threshold in terms of like e- e- expecting the conver- how fast the conversation should be. So I think that's the difficult one, because you are trying to set a programmatic metrics for every single u- u- user that is speaking to your assistant. Now, with this kind of model, you have a lot more flexibility on that. Because the model itself is now predicting based on the audio frame. And the audio frame contains the the how fast you speak as well. And it contains, it it it contains a semantic information. So if I'm pausing in the mid-sentence, the semantics ha- has not been completed yet.

• 28:09Speaker 03

  The agent knows, okay, I have to continue to wait until the the person finish the full sentence. So that is the major bit by collapsing the A S R and L M together. So then you have this more natural turn taking built in into the model. So the model is now making that pretty prediction by itself. So that's one way. A lot of the pre cache uh the the processing, as we talked about earlier, is another thing to actually, how to a- actually o- optimize it. The fact that, I mean, to be honest, the fact that we host our own models with our, the rest of our platform is another bit of a benefit. Because you don't have that, you know, network travel latency, usually uh impacting the most at the long tail use cases where, you know,

• 28:52Speaker 03

  the P nine ninety five is the ma- major pro- the problem for these net- network latencies. So a lot of this we optimizing in a way so that the agent can just, you know, respond very fluent back to you. However, now we are all, my team is adding more reasoning to the model as well. So, you know, the good thing about it is that once you save a lot of time, you can use that time to do more things. So now we are, the the, our our team is looking to adding auto reasoning to the model. So the model can decide, hey, based on this particular user query, if I stop my reasoning right now here, how likely I can give a good a- answer?

• 29:36Speaker 03

  And so we train our mo- models auto reasoning in in a way that we always challenge you to say, hey, like, you know, do you actually need to think that mu- much time? Can you, you know, keep it to like one second and then just re- respond back to the u- user? So the reasoning we are training for the model to do is a very latency budgeted reasoning. It's al- al- almost like, you know, you are, you are go to go go go go going to like a competition. Someone gi- give you a question and you started to kind of counting down.

• 30:08Speaker 04

  It's so interesting because sometimes when I interview people and I ask them a really difficult question, you, even though you're thinking, you still feel that you need to say something. So quite often, it's on a difficult question, sometimes the first sentence of their response is is just filler. Because they're obviously thinking and then and then they actually answer the question. And it's a similar thing here, right? That so so this auto adaptive reasoning is really interesting because like from a user experience point of view, how do we solve that problem? Do do we kind of make the model say, hmm, that's interesting and and kind of indicate that they're thinking? But, you know, because it might actually be a significant amount of thinking, maybe they need to delegate to an agent and they need to do some stuff and then they come back to you.

• 30:42Speaker 03

  There are a lot of tricks you can do in that scenarios, like, you know, in our real pro- the productions. You can add a lot of feeder sentences, obviously, but if you are repeating it too many times, it becomes a very repetitive. And then user will know that, you know, you are just, you know, bluffing them, right? We have another way is that you can put in the type typing sound the as well. So, you know, like there's like computer typing in a be- background, try to mimic that actual like scenarios. The re- the real thing that we are trying to solve in that scenario is that, you know, because you always have the back, the A P I latency you have to deal with anyway. And some of the enterprise backend is not the fat, the fattest, you know, backend by default.

• 31:23Speaker 03

  So you have to bake in a lot of lo- lo- those things to actually make sure user don't feel, you know, uh, because. The reality is that user has been talking to a lot of these automatic systems. And if the AI go dark on them over the phone, even three second or five second, you will panic. You'll be like, is the AI actually working, right? So in order to have that trust, you kind of have to provide this constant feedback. The good thing is that, and the contact center has a lot of way to deal with this. Agent will put you on a, on a mu- mu- music to wait while I'm thinking, you know. Maybe in a, in a, in a old world, human a- a- agent a- actually, you know, going to different places,

• 32:06Speaker 03

  talking to their colleagues, how to solve the problem. The AI agent will be put you, pu- put you on a whole mu- mu- music and is doing the reasoning. So, so I think there are a lot of tricks you can do there, but you know, articulately well is very important. Because the, I think the voice agent has the biggest e- e- issue in the contact center is that user don't have the trust in them, just because they got burned many times in the past.

• 32:33Speaker 04

  That's a really cool idea. So play elevator music while it's, while it's thinking. But I suppose that this gets on neatly to the next thing, which is, you know, what, what is the uncanny valley for a voice agent? And is it, because you know, we, we talk a lot about anthropomorphization and, and the risks of that. And is that something that we actually want in the enterprise? Do we want it to sound indistinguishable from a human?

• 32:54Speaker 03

  Different businesses would tell you different answers. But I think the majority of the businesses would probably hold the same position as Apple and as Google Assist, uh, Go- Google, for example. So if you think about Alexa is probably the person, the consumer personal assistant that goes the furthest to in terms of personalization, right? They give the Alexa identity. There's a very u- unique voice there. But that's as far as they want it to go, because if you, if this personal assistant is representing a particular company brand, and it's very hard to know how successful it would be.

• 33:37Speaker 03

  So a lot of times, you know, companies still try not to attach too much of a personality to it. And the goal is to just make the assistant solve the, solve the problem first, basically. So yeah, I think majority of the enterprise is still on more on the conservative side. We do actually have some customers, you know, as, for example, there's this cut, cut, there, there, this casino from the South America. They like the local, very southern American voices. You know, they like the agent to say howdy, well, to open the conversation, for example. To be honest, in real, in the end, they didn't launch that agent, because there's, they would be like,

• 34:21Speaker 03

  oh, that's, that feels a bit too much still. So there are like person, people would think about that, once they feel more comfortable about the technology. But I think we are still in the, in the phase where people are just building that trust, uh, uh, under the tech, tech, technology still.

• 34:38Speaker 04

  Yeah, I guess it's interesting how much we should or shouldn't chase the personality thing. Because the reason sometimes it sounds egregious is because that, that might be normal in Texas or in a particular location. But, but to me, it's not normal. But do you think in principle, there should be just a kind of generic voice? Or, or should, should the agent get to know you? And when it kind of knows how, how you talk and what you like, should it adapt to you?

• 35:02Speaker 03

  I can speak for what is happening right now, but I, I cannot really predict in the future what people would like. But currently, the majority of our customers don't like the generic voices. And also, the generic voices are in general perform a lot worse for the consumers as well. Just because the consumers would identify it's a robot. And once they identify it's a robot, immediately, they don't have a trust in it. Because we have these traditional IVR systems that has been deployed into, uh, uh, virtually in the America. You know, you know, say a few words, you know, like if you have a payment issue, say payment.

• 35:46Speaker 03

  So if you have, you, you, you wanted to, you know, transfer money, say transfer. And you say many times, you still didn't recognize you. And those voices are always very generic and also a little bit robotic. Just because the previous generational technology cannot really make such a nature sounding voice. So the generic voice right now, for consumers, they immediately connect it to that previous system they talked to. So then they immediately hand up, or they just, you know, try to bypass the agent as much as possible. We found the majority success is that you want, you don't want it to be genetic voices. You want it to be have a little bit of local tech, lo- local, you know, uh, regional take.

• 36:30Speaker 03

  So like, you know, for example, some like in, in UK, for example, a lot of the contact center or a lot of consumers like the Newcastle Castle voice for ob- obvious reasons, because a lot of agents are from there. So when they hear the Newcastle voice, they know it's a human agent. I can actually say my problem to them. They can solve it for the, for me. So I think that kind of voice choice is actually quite nuanced, but quite important. So it depends on which brand it represents and then what people e- expect a voice to be. But the best performing voices is usually the voice comes with a little bit of regional take. The funny thing is that in the early stage of us, you know, shipping those voice agents,

• 37:15Speaker 03

  we found that all the English voice were actually not natural enough. The most natural voice is a French voice, and then you force the French voice to speak in English. So it take a little bit of French accent is a imperfection of that voice, make it real.

• 37:32Speaker 04

  Oh, that's fascinating. We should talk about benchmarks. So you know, for years, there's, there's already been benchmark blindness. And many of the ASR companies, they, they kind of embellish their benchmarks and they all use different internal tests and different public tests and so on. But now in the regime of voice agents, it's, it's even more difficult to test this. So how, how are you guys doing that?

• 37:52Speaker 03

  Benchmark for the voice agent, I mean, benchmark for ASR, benchmark for LM, you, you, we have a lot of established standard. These are all like, you know, more of a particular module and you evaluate on that. The voice is really difficult to evaluate, because it's a voice, voice agent has the latency considerations. You have to reply fast enough. There's also quality of the response. There's also a reasoning task, how difficult, like, you know, this agent can handle the task. And how accurate you can i- i- identify personal names and, you know, in information like that. So it has too many, you know, uh, accesses.

• 38:34Speaker 03

  I think majority of the benchmarks that is public available out there doesn't really serve that purpose. And also, another problem is that a lot of these voice agent benchmark, you know, on the market are synthetic to a degree. Because voice is a lot more private, releasing those voice that, that are set for evaluation and then make it public. It's, it's non-trivial as well. So therefore, we have a lot of these, like, you know, it's, you know, these benchmarks on the market, but not really sufficiently represents what a good voice agent should look like. And I think the industry as, as a whole is still figuring out how to best evaluating it.

• 39:16Speaker 03

  And the reality is that voice agent moving towards so much into the real world application. Sometimes it's also hard to say there's a really good benchmark and solve all the problems. So we are try, we, our benchmark is built on the real data set we have in the conversations with our consumers. So we build it in a way, so it measure a lot of different important accesses. We wanted to make sure the agent response time is a factor into the benchmark. Because a benchmark, a voice agent benchmark without latency consideration is not a real one.

• 39:57Speaker 03

  Because user cannot wait on the phone for sixty seconds for, for, for you to come back with an answer. So that is a very important element. We evaluate on the, the understanding, like, would you be able to actually understand what the users say? And then transcribing it accurately. So the ASR transcription is still one of them. And then the tool and the traditional, the LM, you know, the evaluations, instruction for the following, the quality of the response, are you using citations? Are you using tool calls and things like that? So there's a lot of different evaluation we, we, we run on. So currently, our model is evaluate on that benchmark.

• 40:42Speaker 03

  It is the best models, of course, is internal benchmark right now. But our plan is to, in the next months or two, releasing that benchmark for the market and for the, the research community to be able to also evaluate on, on it as well. Because we do believe that having a more realistic benchmark to evaluate on would be a good thing for the overall industry to continue to move forward.

• 41:10Speaker 04

  One thing that I'm thinking of is that this might be an example of subjectivity. So, so there are some objective things, right? So, you know, just quality and trust and tonality and stuff like that. But, but for example, it, it, it might be cultural specific, task specific. So for example, you know, in this particular application, someone in Newcastle, they might like it. Someone in France might not like it. So, so what would you do there? Would, would you, would you sort of like, you know, statistically aggregate over lots of different situations? Or I mean, how, how would you approach that?

• 41:40Speaker 03

  Currently, the benchmark is actually very objectively look at the quality of the response and more at a factual level, right? The actual end user experience level, I think that's an even harder problem to measure. I think for measuring that, we can argue that we need a benchmark data set to evaluate that. But the reality is that if you're building a agent or assistant towards that level, it's almost at a production grade already. The production, the production agents, you have the end user to tell you already. So the way that we are also measuring this in the real production is, hey,

• 42:24Speaker 03

  does the user want it to engage with the agent? And does the user want it to be able to, uh? To be able to, you know, does a user actually finish the task in the end? So there's a lot of other the production related metric you can measure there. But that one is a lot harder to measure because it's not just about the model performance. At that level, you have to couple the model with the harnessing together. So it's that entire agent evaluation and that is even harder task.

• 43:00Speaker 04

  Can you talk me through the complexity of harness engineering? I mean, the reason this is interesting is so many enterprises, they're adopting AI. And the main form of adaptation is harness engineering. And it's an incredibly good thing and it's an incredibly bad thing because everyone is building their own harnesses and they're adapting their skill surfaces and prompts and essentially everyone is reinventing the wheel and doing different things. So, you know, there is an argument to have some kind of like platform and standardized ways of doing this. I mean, how do you deal with that problem?

• 43:31Speaker 03

  This is the market demands, right? Because I think enterprises, enterprises acknowledge that artificial intelligence is going to be a very important technology for them to own. Not just buy, they wanted to own them. They obviously wanted to own the entire brand, the entire like, you know, like ideally models as well. But they know they cannot own the models because they don't know how to tune the model. And it costs a lot and they have, they need to have a lot of GPUs, they don't have the budget for it. So they gave up on the models, they don't want to own the models anymore. There was like, hey, we can switch to different model APIs, that will be okay. But they wanted, they still wanted to own the intelligence themselves.

• 44:15Speaker 03

  The only way for them to own is to own the harnessing. So that's why I think a lot of enterprises up in to build their own harnessing. Because everyone is considered that this is such an important technology, they need to in-house. And the harnessing is the only thing they can do to in-house currently. So I think that's kind of like the market sentiment right now. I do actually think not harnessing, not every harnessing you should own yourself. Because if you are building, it depends on how critical the task is. And to be honest, I think voice agents and the voice facing the customer. I mean, we talk about like voice is actually a harder problem than just general reasoning.

• 44:58Speaker 03

  Because the adaptation itself is very important and it's very critical. I don't think enterprise value that part as much. They value the reasoning part a lot more. So we'd like to talk to our customers is that, hey, you don't actually need to own the voice agents or the voice agent harnessing itself. You can actually have this voice agent as the front door to your customers. And you can own your brand in the background. So then you have the voice agent that can respond very quickly. But then it can delegate a task to your mastermind behind the scene. So I think this is probably what we are going to see in the next couple of years is that

• 45:42Speaker 03

  enterprise, we realize that they don't have to own everything. But they have to figure out like, hey, they have to draw the boundary. This is how I feel comfortable giving away. Some people will probably just always stay at, oh, I really need to own the harnessing because I wanted to do a lot of customization myself. And then I'm comfortable to use the model. And that's okay as well, but I don't think there will be the majority.

• 46:10Speaker 04

  I love thinking of software engineering as a specification acquisition problem. Apparently there's a whole theory around this, which is software engineering as theory building. But many folks at home would have experienced this, that when you do vibe coding with agents, there's a, there's a kind of Rubicon moment where your software is sufficiently well specified that an autonomous agent, when it sees problems, can just fix things. So it's almost like if you can get it to the point where it's well enough specified, then it just kind of converges from there and that's the magical moment.

• 46:40Speaker 03

  Exactly. The beauty of, of working for these enterprise conversations or business conversations, there's kind of a good look like, right? It's actually relatively well specified. Because in enterprise conversation, the major thing is that does the customer's problem got solved in the end? And then does the customer feel good in the end? Is it a good conversation? And how fast you can solve the customer's problem? So the good thing is that, you know, despite that in our human to human nature conversation, there's no gold standard because a chit chat, if it's a good chit chat, it will actually change the relationship of the two.

• 47:23Speaker 03

  But in the business conversation, there's no good chit chat, you know? It's just like, did you solve my problem or not? So it's a much more well-defined problem. So that's why I think we see a lot of success in applying coding agents on top to be able to actually automatically optimize it.

• 47:42Speaker 04

  Very cool. And I guess one of the issues with agents, I mean, this has been a dream of mine for ages, you know? What if we could almost build an artificial organization with some agents and some humans? And then we have to solve the coordination problem between them. So we have some agents that are specialized in front office and some that specialized in back office and coding. And they're all just sending messages to each other, fixing things. That's very exciting, but it's quite problematic as well because there are all sorts of failure. How would you approach that?

• 48:08Speaker 03

  It is, and I think the enterprises don't feel comfortable about it yet. Despite that, we have a lot of vibe coders and you have a lot of AI native companies. It's like rushing to build all these video feature. And to be honest, a lot of these products are really, really cool. But I think enterprises are feeling very threatened by that. Yesterday, we actually have one of our largest utility customers talking to my team. My team is really freaking out because they say, oh, in the future, all the PRs got merged into the repository. We need to have a review and the review cycle is two weeks. And then we have to make sure that everything got merged into production is, you know, auditable and then reliable.

• 48:49Speaker 03

  And in the past, it wasn't really the case because we were just getting standing out the agent and making the agent really, really good for their use cases. But now they are applying that control on top. And I think they have been holding for a very long time. And now they feel it's a time to do it because the agent is really good right now. But the thing is that just because enterprises wanted to have their visibility, their guarantees, their, you know, that everything is auditable. So you kind of have to build around that as well. So a coding agent just directly spit out PRs and auto merge would not fly with them.

• 49:29Speaker 03

  That's why in our platform, we also build a lot of visualization components of the agent. It's a, hey, this is what the flow looked like. So if the agent enter here, this is what the processes look like. So they still have a general sense of, oh, the agent is actually executing these processes one by one. There's a bunch of tools and these tools are exactly what the agents can do. So then it, anything that is outside of that tool set, the agent doesn't have the ability to do that. There's a citation module of the model so that every response comes with a citation. So they can audit trail that. And I don't know how long we will have to have this because we kind of live in a world where

• 50:14Speaker 03

  you have a lot of vibe coding and coding agent can do things very fast. But we are still dragging this traditional SaaS enterprise, you know, like no code, no code flow with us. I think now the good thing is that these no code flows is no longer the primary interface for people to build. It's just for you to audit the actual processes of the agent. So I think we are kind of shifting towards that way. But that UI component and visualization is still very important for the enterprises.

• 50:46Speaker 04

  Adaptation is intelligence. So these models, they are being adaptive. But at the moment, we've been talking about kind of skill surface adaptation and code adaptation. The great thing about weight adaptation is that you can in principle share this model in the organization, you know? Because if you're adapting code and skill surface and you've got agents everywhere, it's a little bit brittle in the sense that, you know, I can't really take my code here or my skill here and share it with an agent somewhere else in the organization. So like, how do you think about this adaptation problem in general?

• 51:16Speaker 03

  There's always this debate is like, where do you want it to, you know, change the behavior and where does the customization should go to? I mean, to a certain degree, we agree with that principle because we do actually train our own models as well. Because we agree that there's just like inherent behavior. Like, of course, you can try to solve from the agent harnessing side. But even if you try to solve it there, you're probably going to put a lot of red tabs around it. It's probably not going to be good enough for your use cases still. And if you have the ability to alter the weight, then you have a lot more control over what you're building.

• 51:58Speaker 03

  So I think it's more of a philosophical questions, like, you know, to what degree you think you should have a control? Because there's definitely some danger in terms of going that deep. It's like, yeah, you can control the weight, but maybe you don't know how to do it the best way. You end up with a suboptimal solution. So for the expert like them, I'm pretty sure that, you know, that they know what they are doing. But I would say that for a lot of the enterprises, they probably, that's probably not their primary area of expertise. So I think that will require some help in terms of like, you know, even if they wanted to fine tune the model.

• 52:38Speaker 03

  Or I usually, what I would suggest them for a lot of our customers that start with the harnessing. And if you really cannot get around that, then you can think about, you know, using different models or tune your own models. But that should be a second result, you know, that should be like, you know, your number one priority immediately.

• 53:00Speaker 04

  Another thing I wanted to talk about is this notion of cognitive debt. So I've been using OpenAI, you know, they've, you can now link your phone to it. And I can control all of my coding agents. And it's amazing, I can, I can go down the shops, I can take a walk along the road. And in some ways, it's amazing, in other ways, it's not. And the reason it's not is when I go back to my computer, I realized that it's sometimes the agent was doing the thinking when I thought, you know, one of the, you know, the voice agent was doing the thinking, one of the underlying coding models should have been doing the thinking. Sometimes it wasn't using my established agents, it was just creating new ones and it was just, you know, it was a little bit messy. And also the cognitive debt is, it's really important sometimes just to see the visual information in front of you.

• 53:43Speaker 04

  So how do you address that problem?

• 53:46Speaker 03

  It's a huge problem. I mean, it's a huge problem, not just for the product development and our customers, is also a huge problem for us organizationally. Because like, you know, all our teams are, you know, adopting Clouco right now. And we have a lot of people, you know, integrating to all the Gmail, G Suite and integrating to Slack channels. People now started to, you know, sending Slack channel calls, sections with each other. If people are not careful, they're sending a war of texts to the other person. And the other person will be like, oh, like, I have to spend so much time reading it.

• 54:25Speaker 03

  And over, over time, we see more people just say, okay, now, you know, have Clouco read all the texts and then summarize it back to me. It's hard. It's the problem, you know, like you, we, humans don't have that much capacity. I mean, the bottleneck has already shifted from producing content to actually validating and auditing the content. And that has been the problem my engineer has been telling you since the beginning of the year. Like, we have so many, so many calls to review. And I don't trust agents to do the code reviews yet because, you know, I don't feel comfortable for them to just merge the code. So I think we are definitely entering that stage.

• 55:06Speaker 03

  And I think we, we'll probably need a lot of education for people is that, hey, especially in the coworking scenario, you have to educate people that, hey, you have to help people to summarize your point. And also, don't send people the AI slob that you have not read yet. You have to read it yourself and then really understand and agree with what you send, not just like, you know, throwing that out to people. So I think it's a real problem for the problem.

• 55:37Speaker 04

  I think encoding is particularly acute because you're just dealing with something of very high complexity, there's lots of background knowledge. Because it feels like in a customer service agent or something like that, both parties understand enough about what's being spoken about.

• 55:50Speaker 03

  Yeah.

• 55:51Speaker 04

  And is that part of it? Is it something like, if, if we both understand it, then it's fine, because the voice agent, it's a little bit low resolution, but it's pointing to things that we already know. But in the domain where we don't know something, then it, then there's that cognitive debt problem.

• 56:04Speaker 03

  Voice agent has less of that problem because by nature, those voice conversations are a lot shorter, right? Like, it doesn't actually generate a war of texts. Like, our average output token size in the production agent is probably about 50 tokens. It's not a lot, right? So consuming those is very, is relatively simple. It's the, it's the reasoning behind it. It's like, if you are probing a larger agent to do the reasoning task, where it generates a lot of reasoning tokens, that's actually more of a problem. Usually, voice agent by itself is not a problem. Voice agent has a different problem, is that it doesn't have the sufficient bandwidth

• 56:47Speaker 03

  in that communication channel, because the voice require very ping pongy, very short, that kind of conversation. So it has a slightly different problem. I think you, the, the task you are doing is actually very interesting, because I have been thinking about, you know, using voice agents to do vibe coding as well. That would be really cool. I think that would probably require a little bit of harnessing into it, where actually make sure that, hey, voice agent always service the necessary like high level bullet points, rather than just pass on a simple sentence per se. But I don't know whether it will be easy enough to really for you to understand the full picture still, because it's a lot.

• 57:31Speaker 04

  What I really love about voice agents, or just agents in general, is that I can do insane amounts of context switching. So I can be doing my email triage in one, I could be doing video editing in another, I could be doing coding in another. And it's, it's really made me change my workflow. So now I have a breadth first search workflow, where I'm just kind of like popping off one, popping off another one. And, and there's good and bad, because in a sense, I'm not going really, really deep on things, I'm just kind of doing many things at once. But the beauty of the agents are, as we were saying earlier, these, these domains have become well enough specified that the agent can have some autonomy and I just sort of like feed, feed information to every single one. So it's, it's kind of changing how I work. I mean, have you, have you noticed that?

• 58:13Speaker 03

  I build a lot of these agents as well. I mean, currently, my workflow every day is, there's only once, I mean, probably two software, three software I open, Claw, Slack and Chrome, you know. Chrome, sometimes because I have to open Chrome, so I can log into some software, so my agent can do a computer use on top of it. So it's not really I'm using myself, it's the agent, so the agent can access to it as well. So it's already changed my workflow completely.

• 58:46Speaker 04

  Yeah. And do you think this is a mindset thing? So obviously, like, you know, we are engineers. And what, what I did as quickly as possible was transform all the work I do into, you know, an agentic representation, basically. And, you know, because the million dollar question is, we want to get to a point where voice agents are used for basically all work that can be done. And I mean, how do you see that, that transition happening?

• 59:11Speaker 03

  Voice is a very good input modality. Whether the voice agent itself is the best working, you know, intermediate layer, I'm not actually sure, to be honest, because like, you know, I'm being, give you example, I've been using Whisper myself, I'm pretty sure you use it as well, it's a very convenient ASR tool.

• 59:30Speaker 04

  Wispr Flow?

• 59:31Speaker 03

  Yeah.

• 59:31Speaker 04

  Yeah, I love it.

• 59:32Speaker 03

  It's amazing.

• 59:32Speaker 04

  Yeah, I'm in the top 1%.

• 59:35Speaker 03

  Yeah, exactly. Because it's so convenient, you can actually just talk to the agent and the agent, I mean, despite that, you know, I'm not a native speaker, so then the transcription is not always accurate. But, you know, those agents understand those, you know, errors still. So then, you know, it's not like, you know, I send it a voice memo to someone and then they see that there's a lot of mistranscript or they get confused. Agent don't have that problem. Agent would do the reasoning and say, oh, this word should mean that, this word should mean that, okay, well, I'm going to do it that way. However, I am a user that heavily using it. I use it even at work, because I was like, at work, you know, you put your headphone

• 1:00:17Speaker 03

  on and people are sitting next to you, and there are people already taking calls on, you know, on their desk already. What's the problem of putting headphone on, talking to an agent? I got judged by so many people, that was, what are you doing? That's so weird. I was like, no, I'm just working. So I think the voice, as I say, voice that, voice modality has this like privacy layer, where some people just don't feel nature doing that. Even me talking in the office, just talking to my agent, people feel weird. That's the same argument as the meta glass, right? You can wear it and then you can actually talk to your agent, talk to agent, I got one as well.

• 1:01:02Speaker 03

  I started using it, people were like, what's going on? Because I was like, hey, meta, play the music, take a picture. And there was just like, people feel it weird. So I think there's this user adoption issue. So I would say, we are rare, because, you know, we are really agent built, we really like to actually work with that new modality. But I think a lot of people probably still not quite there yet.

• 1:01:29Speaker 04

  We are engineers and what I do is I build tools to help, to help tasks become agent enabled, whether, you know, so for example, video editing, you can't use an agent to edit something in DaVinci Resolve, or sometimes I make 3D animations in Blender, you can't really, you can do that with an agent, but it's always going to go wrong. So what I do is I build a tool, I build a mental model, so, so an abstract model of the domain, you know, and then I build an agentic CLI and I put a skill prompt in there and, and then my agent can do it. So I'm always in this modality of tool building to make an application amenable to agents. And that feels like the gap at the moment, because like, my mum wouldn't be able to do that. So we want to get to the point where they can almost demonstrate.

• 1:02:13Speaker 04

  So they say, this is something that I do, and they can put a demonstration in there and they can describe abstractly what their thought process is, and then they can build an agent interface, and then from there, the agent can do it and you can correct it when, when it makes mistakes.

• 1:02:25Speaker 03

  I have been building quite a lot of tools for my internal teams as well, because people don't know how to do them. I think one of the most useful tool I build for the team is building these like internal wiki stuff to, you know, track all the product features, functionalities, sales, you know, like, how do we pitch to customers, that kind of thing? And then continue to iterate on that. And also build another agent that goes online research to search, you know, suppliers, vendors, competitors, you know, market trends, and then just building that into the video wiki. So then, that is the context engineering, I'm trying to actually do that for them.

• 1:03:08Speaker 03

  There's also a lot of tools you can build around it as well, like, you know, access to your, you know, email, your Slack channels, and then, you know, I'm going deep, you know, because Claw has a lot of these default connectors, I almost throw out all those connectors, because I don't think they are, first of all, fast enough, second of all, is actually not powerful enough. I want to go straight to the, to, to, to the source. So now it's like really powerful. So, yeah, I think this just require a lot of that thing. And then I think people, a lot of people just wanted to use it, you know, they don't necessarily have the mindset of building something and use it yourself, they're kind of waiting for people to build for them.

• 1:03:51Speaker 04

  And what do you think the next decade of voice research is going to look like?

• 1:03:54Speaker 03

  The voice agent need to be increasingly more end to end. And in that end to end, how, I mean, number one, turn taking must be end to end and collapse into the entire pipeline because that is the single most annoying bit when it comes to engineering and the most difficult bit to actually get it right when it comes to engineering. So that is almost, I mean, thi- this is, i- it's already happening with a lot of these mo- models, right? And then we were starting to see in the voice modality, we'll probably have to start to branch out. There should be a consumer voice for consumer entertainment, day-to-day use cases, you know, eh eh Alexa, Google Assistant of the world.

• 1:04:37Speaker 03

  There's a enterprise voice use cases, started with the enterprise contact center, more of a customer su- su- support and professional use cases. It could be als- I mean, the, i- it could be also some like meeting note takers and then also like, you know, an agent help you to coordinate these meetings, could be in that scenario as well. And then it could be also becoming the internal productivity layer o- of that voice, you know, between, you will call the colleagues. I don't know what that would loo- look like yet, but I do actually think that the voice channel that require for consumer and enterprises are very different.

• 1:05:18Speaker 03

  One requires a lot more professional, a lot more like, you know, uh, the, uh, the regu- regulatory, the, the audible. The other one is like really about fun, because conversation is also supposed to have fun some- sometimes. So I think there should be these two different b- the bran- branches coming out. And then we will see there are more hardware devices to be able to carry these voice agents because the voice is a natural mo- modality for these I O T d- d- devices. We'll probably have, see a second wave of I O T, you know, coming along. How successful o- of that will be, I don't know, but I think I'm ri- really excited about that.

• 1:05:59Speaker 03

  I actually be- buy a open source ro- the robot recently, just like, you know, more like a voice speaker ro- ro- robot, but you know, it's not, you, you can program a lot of it, but you know, it's still like very crunk, cr- cranky, but I think that all these things would become a lot more pro- prominent in the next ten years.

• 1:06:16Speaker 04

  I can imagine a future where the technology becomes so invisible that we can train the system, where it's almost like teaching a child. And then I suppose a, a really important thing, certainly in terms of organizations, is how we share those learnings.

• 1:06:31Speaker 03

  Currently, it does actually feel like the technology is still very visible because there's still a lot of learning curve and then A I, I mean, our, in our organization, people have been talking about A I slop because it's very, really visible, more and more pe- people becoming like sending A I slop m- message to, to each other. So it is still very, very visible. I do actually agree that eventually it should just becoming, you know, invi- visible or in a way that it just blended in, into your organization. I think that will probably still require a bit more iteration in terms of sharing the context, sharing like who you are and what the, the organization is about.

• 1:07:11Speaker 03

  I think despite that, we have done a lot of internal knowledge, uh, you know, uh, uh, uh, detecting, there's still a lot of principles that we have not writing into those M D files yet. So I think there's still a lot to do and some, you know, we are probably running out of these low hand, handing fruit now because some of the, some of the things that they, they are actually so personal to ourselves or to the organization is more, you know, s- spiritual, you know? It's about more like, you know, this kind of constitutions or some principles that you probably don't print it out, you know, e- everywhere. Because the funny thing is that if you have the agent to ingest these cultural va- values

• 1:07:54Speaker 03

  of your company right now, sometimes the agent just come up with random ideas and then just overly fo- focus on one thing, but actually it's not about that. Despite we write a culture va- values down like this, there's a reason why these come about and those reason a lot of time is not injected into the con- context. So agent, over time, focus on the wrong things and just be, because in their ways somehow they think that out of these six, thi- thi- this one is the most important one, but that's actually not true for the organization. So we kind of have to share a lot more in order to get to that, you know, y- y- invisible, you know, state that you, you are talking about.

• 1:08:37Speaker 04

  The slop thing is very interesting and so my definition of slop is generation without competence. I'm using memory um, systems, you know, I have lots of um, kind of like adaptive specialization, skill adaption and so on. So it always has significantly more context, increasingly it knows what I, you know, what my preferences are. Um, I've grounded it with, with the correct technical referenceable materials and, and so on. So the, the, the fact of the matter is that there's, there's two modes of A I. There's A I that has sufficient context and, and it knows what to do and, and that A I does not produce slop.

• 1:09:11Speaker 03

  Thing slop, unfortunately, is probably a little bit more subjective in a way that I engineer and context engineer my agent so that the agent doesn't feel like slop to me, but it may feel slop to the other person who is actually reading my m- message because in his context, these things I care about probably doesn't really matter all that much to him. Or maybe he is not aware of this is important yet. So then when they read it, they thought it's a slop because it doesn't actually see the con- context behind the reasoning. So there, that is a bit subjective problem. So I think we have this internal debate or a- argument is that should I care about whether

• 1:09:53Speaker 03

  I generating slop and then just sharing with the colleagues or, well, the fundamental debate is that should a human being reading those things anyway? Or should a human e- being just ask agent to read it for them and then summarize in a way that, that is better suited for their, for the, for the focus, you know? So, so like we are, I al- al- I almost landed in a way that, oh, I should probably build a skill for everyone's agent to, based on their personal context, summarizing whatever message that, you know, people send to them so then they can extract the useful information for them.

• 1:10:33Speaker 03

  So in that case, you wanted to encourage the, the output side to be as rich as possible. So then you send a wall of text, that's okay, because the other side, there's a A agent summarized it for that particular individual. But I kind of feel like this is running in circles. It's like we human communication now, me, agent in between to help us to cherry pick the useful information for us.

• 1:11:01Speaker 04

  John, it's, it's been amazing having you on the show. Thank you so much for joining us.

• 1:11:05Speaker 03

  Thank you, yeah.

• 1:11:06Speaker 05

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid and risked three hundred fifty thousand Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by H M P, thehousemajoritypack.com, not authorized by any candidate or candidate's committee.

• 1:11:38Speaker 06

  Your heart can tell you a lot about your health. Apple Watch Series twelve measures your heart rate every five seconds with the most accurate heart rate sensing in wearable. So your vitals app, now with heart rate variability, can tell you when something is off. And your readiness score can let you know when to rest and when to push. Hear the story in every heartbeat with Apple Watch Series twelve. The features described are for wellness purposes only and not for medical use. iPhone eleven or later required. Based on Apple conducted study of heart rate accuracy, August twenty twenty six. Visit apple.com slash apple watch series twelve.