⏭️ Forward Deployed: Voice AI on what works in 2026

Watch the full show on YouTube: https://www.youtube.com/watch?v=MwNvowwcZOo

• 0:00Speaker 00

  All right, we're in the remote studio. This is a special one because we're launching a fourth podcast on data and space. People don't necessarily keep track, but we cover different things. Basil came across my radar because you're doing all these dinners and gatherings and panels and podcasts on for deployed engineering. You hosted the FDE track at AIE and it did super well. So welcome to the pod.

• 0:24Speaker 01

  Yeah, thanks.

• 0:24Speaker 00

  So you've been running your podcast for a while. What made you decide to focus on FDE? And what's your typical self-intro?

• 0:32Speaker 01

  Yeah, so I started as a product manager at Intuit. I was specifically working on Credit Karma for a couple of years. I worked at a small venture studio after that. And then I ended up starting a consulting business called Exaflop Labs, where we were working with retailers, insurance companies, that sort of thing, just building agents essentially. This was like two years ago. So it just felt like a natural extension of a lot of the product work that I was doing earlier. It was like, hey, I'm working with customers. They have a specific type of project that they want built for some use case. And I'm going to understand what they want. I'm going to make trade offs and I'm going to help them build that. So yeah, I just felt like a natural extension of what I was doing. And so back in January, we started working with a couple of private equity firms. And a lot of people were just like, hey, like I'm following on what's going on on Twitter. What's going on on LinkedIn?

• 1:17Speaker 01

  I don't know what's marketing BS and what's not. So I was like, why not just bring on people who are working on the cool stuff at cool companies? We will talk about what they're working on, like we'll do a deep dive and we'll record it. We'll post it and hopefully people can take learnings away that they can apply to their businesses. And that's why I ended up starting doing a lot of the fireside panels and podcasts that I've been doing since then.

• 1:39Speaker 00

  Can you give like rattle off just your greatest hits, like what you've covered?

• 1:43Speaker 01

  Yeah, so our very first episode was on the future of agentic engineering. So we brought on companies like Factory, Cognition, Composio, Sound Grab. We've done panels on voice agents, which is actually the one that we're going to be showing here. We've done some on computer use agents. We've done agents in the enterprise. So we've done a ton of stuff just to get nitty gritty into the details of a lot of this stuff.

• 2:04Speaker 00

  Yeah, and I guess, you know, you picked the first feature, you picked the voice agents one. What are we about to listen to and what stood out in particular?

• 2:14Speaker 01

  Yeah, so we brought on some people from Decagon, Vapi, Retail Daily and a company called Smallest AI. Basically, we just talked about like, what is this state of the art in building voice agents? Because that's a very hot use case. I think Sierra even talked about this. This is like one of the most competitive markets in AI right now is like building voice agents. And so I thought it would be useful to talk about how they're actually built. And some of the things that, you know, engineers who are building in this space still have to contend with. So for example, we talked about how like the state of the art right now in building voice agents is a cascaded pipeline. So we talked about how it's like a three step process. You go like speech to text, LLM, then text to speech. And we talked about like, why is that versus why don't we just have a voice to voice model? Like, why don't we use that? Like the reality is those just are not super reliable right now.

• 2:58Speaker 01

  We also talked about like the trade offs that you have to make when you're building voice agents. So you have to trade off like the intelligence of responses that you're getting versus the latency. So you can get very intelligent responses, but you're also going to trade off. It's going to be a slower response. And so for different use cases, that might be good or bad. We also talked about like reliability. So there are some LLM companies that may not be like super reliable in terms of like their infrastructure. And so like, what do you do when Opus goes down? Like you need to have a waterfall of models to basically pick it up so that your voice agents don't just stop working. We also talked about like turn taking and how like it seems like a trivial problem where, hey, like you're talking to a voice agent. And let's say I pause. So how does the voice agent know that, hey, like I should, I should interject and I should actually like respond to you?

• 3:42Speaker 01

  Versus, oh, the person is just taking time to think. So turn taking is actually like not a trivial problem to solve. And so we talked a little bit about that. And then also we even talked about like how I think Pipe Cat built a voice agent for the World's Fair, right? So we even talked about that a little bit. And how like, I think, I mean, you can talk about this a little bit, but I think like you guys built a voice agent to like, if anyone calls the voice agent, it'll answer questions about the conference. And then you got a ton of like real world information and turn that into a benchmark.

• 4:08Speaker 00

  I mean, to be clear, I have no idea, actually. I never looked at analytics, so I have no idea how many people actually called. Probably, you know, people just kicking the tires. You know, like it's not, it's not that serious. Like we, it's a conference, people know what it is. But you know, I think that's a good deployment of use case. Daily is a sponsor, so why not? I, you know, I would say that vast majority of voice stuff is support. And like, yeah, oh my God, there's, you know, so many call centers out there, hundreds of billions of dollars spent on this stuff. And they're all, you know, bad. So hopefully we can sort of raise the state of the art, which, you know, I think that's why Decagon and all these things, people exist. Like to me, you know, it's not as interesting if a bunch of people who are obviously selling you voice pipelines, telling you that voice pipelines are the state of the art, like, yeah, duh.

• 4:53Speaker 00

  But, you know, having, having the people actually focus on customer support and all those things. Also basically conclude like, yeah, like the models are not there yet. They may never be. And actually, this is just the way that you got to do it. And the people that really feel it are not the researchers, because researchers just always want bigger models to solve everything. And the product engineers will try to do it. But the FDEs, the people actually dealing with customers will be like, dude, they made this like horrible mistake. Because this can never happen again. How can you guarantee me that? Well.

• 5:23Speaker 01

  Yeah, yeah, yeah. So like, actually, we even talked about that a little bit. So like, we talked about like the inbound versus outbound use cases. Because we even did that once, we like ran an outbound use case. And like what you find is whenever somebody like picks up the phone and you do an outbound use case, a lot of the times they just hang up as soon as they realize it's a bot. So we even talked about that, like, how do you, how do you solve for that?

• 5:41Speaker 00

  All right, that's a good teaser. You know, I admire your work. I'm excited to feature you on the InSpace. We'll be doing more in the future together, but this is just an intro to FDE for at least, for at least the InSpace audience.

• 5:54Speaker 01

  I think this might be a good place to give everyone a one on one lesson on what does a simple voice agent architecture look like? So maybe that's a good question for Bruin, because you guys run Pipe Cat, so.

• 6:07Speaker 02

  Yeah, so there are many ways to build one nowadays, because the complexity is the nature of things. But the simplest one, which is called a cascade model, is typically voice input. It comes through a transport. It could be WebRTC, phone call or web sockets. It goes into the, into a speech to text. The transcription happens. There can be some additional models right there for background noise removal, voice isolation. So if there are a few people in the foreground, it will isolate to one person, remove the background noise. There's a turn detection model. Unlike a text where you know when you're done, you press enter and the message goes.

• 6:50Speaker 02

  And the LLM starts to like execute with the voice. There is no, it's not walkie talkie or push to talk. So there is no additional information apart from like, if you're looking at the face, you could actually figure out that the person stopped speaking. But typically you use something like a voice activity detection and a smart turn model to like figure out that the turn is complete. If I pause mid, mid-sentence, that's different from like if I finish speaking and pause. You may use a speech to speech model right there. Then when you have the text, you, from the transcription, then you send it to an LLM. The LLM may respond with an inference.

• 7:35Speaker 02

  It may actually tell you, it may infer to tool calls, it may do a bunch of things there. You get that. When you have the full output, you text to, like you do text to speech, DDS. And it's usually faster than real time. So then you have to like figure out how to like stream it back to the end user. It will go over the same path, transport that, that you spoke over and then it'll play it back on your speakers. But things are getting more complex as people, you know, this was like 2024, 25, people wanted to do this. If you're doing an outbound use case, like you do an outbound call, debt collection is quite commonly a use case for this. You get a call, they tell you you have like an unpaid bill.

• 8:20Speaker 02

  They ask you if you'll pay it or, you know, give you a certain set of options. For that, the bot has really like, if you go off the rails, like the human goes and starts saying random stuff, it can just drop the call. It's not obligated to stay on the call in any way. So guardrails are easy. It knows what the inputs it can accept. It knows what inputs it can, it doesn't need to respond to. It flips when you have an inbound use case, because the bar has no context, like has some context of why it exists, but doesn't know why you, the person who's calling it is. If you're a mom and pop shop, like a flower shop or, or like a barista or something like that. Okay, there's only like finite things that you can do there. But if you are more complex, let's say you're Amazon, right?

• 9:05Speaker 02

  You're like a billion products on your platform.

• 9:09Speaker 02

  And you have different policies for different things. Is it a, you know, is it a physical item? Is it a cheap physical item? Is it an expensive item? Is it an electronic item, like all of that? So it needs to figure. And then when you said something like, you know, you can't put all of that inside a single prompt. Because it will, like LLMs are, you know, goofy in that sense that always remember the first four percent and the last four percent and everything in between. They kind of forget. So if you have a return policy right in the middle of that context, it's very likely it'll hallucinate. So you have to do more smart things and that's where you have like more models come in.

• 9:54Speaker 02

  You can have a compaction model, like we're all using coding agents now, right? So you know, like, if you're using like a one million context model, like at 25%, you should become nervous at that 25% mark. It's nowhere near 70% mark, but at 25% mark, you're like, okay, should I start compacting and like saving my work? So that this thing does not go off the rails, right? The same thing, I think voice has been like one generation of ahead of coding agents. Like we've all in the last two years solved things that felt like so alien, but very tangible for us. Which now when we see coding agents, you're like, man, compaction, we, we were doing compaction from day one.

• 10:36Speaker 02

  Like we knew like after five turns or 10 turns and your context was only 250 tokens or 250,000 tokens or 50,000 tokens. You know, models were really small like before. And at 10,000 tokens, they would like go off the rails. So, so a lot of our work is, I think, about like making sure that the bot does not hallucinate. And trying to like, keep track of the conversation as the, the turns progress. And the turns are fairly short, because the, either way, the bot is more likely to speak more than the human did. So you have to like kind of keep track of like, what did the human say? Where are we in the conversation?

• 11:17Speaker 01

  Yeah, so I guess this is a question for one of you three. So do you guys use this like cascading model for your guys' voice, voice agents?

• 11:28Speaker 03

  Yes, that is one of the major offerings that we have cascading in speech to speech.

• 11:33Speaker 01

  Yeah, I was gonna ask, like, that sounds too complicated. Why not just go voice to voice? Like, why not? Like, why do you have to have this, like, crazy three-step process?

• 11:42Speaker 04

  I feel like the easiest way to answer this is you can call into a, you know, really nice voice-to-voice demo and you're like, wow, it's like listening to me laugh and it's like responding to my tone and it's so snappy, it's so fast. But then, you know, I tell it that it asked me what day I want to schedule my appointment for and I say, you know, next week and it says, great, is that June 10th? And I'm like, no, it's, the year's 2030. And it's like, you're right, it is 2030. So let's schedule this for June 10th, 2030. And I'm like, great, sounds good. And so from that perspective, I am actually very curious, especially on the, on the smallest side, how that's evolved over time because we are very much keeping our eye on how these speech-to-speech models are performing. But overall, I

• 12:25Speaker 04

  think our current stance is the cascading model just allows you to enforce so many more rigid guardrails and just tight control over the ability to say, hey, this input is going to go through the same supervisor models to detect for, you know, prompt injection or social engineering, then it's going to go down the conveyor belt into intense selection, then we're going to optimize context by checking conditions ahead of time to say, you know, in this complex process that we're going through with this user, maybe this prompt is applicable if they're this type of customer, but this prompt is applicable if they're that type of customer. So let's, you know, figure all this stuff out ahead of time,

• 13:07Speaker 04

  compact and compile a good system prompt for our message generation model and get a response back. And then we can take that response and we can check a whole bunch of other things, right? We can check, is this grounded in truth, right? Is it 2030 or is it 2020, whatever year it is, 2026? And then it goes back out over the line, right? And so the obvious constraint to that is how do you make that performant, right? How do you parallelize as many of those steps in the conveyor belt as possible? I think the last six or so months has been a really amazing feat from our engineering team at least to find and shave off like 10 milliseconds at a time across every single part of this pipeline to make it feel snappy, even though there's a lot of things going on behind the

• 13:49Speaker 04

  scenes that, you know, you don't necessarily have to do with a speech-to-speech model. So that's at least my take. But yeah, I am curious for, for the rest of the group's thoughts on this.

• 14:03Speaker 05

  Yeah. No, I think you brought a question around like cascaded versus speech-to-speech. Like, let's talk about like, why did people start thinking about speech-to-speech? And so the initial idea was simple that, hey, if you convert speech to text, it's gonna lose the emotional information, right? So if you say, hey, you might be saying it in a sad way or an excited way, the bot is gonna answer in the same manner, right? So that was the obvious reason people started. I think at some point of time, at least at smallest, like, that has evolved to speech-to-speech is a more natural way in which the human brain operates. So for example, when you do the cascaded thing, you do speech to text,

• 14:48Speaker 05

  then you send the prompt to an LLM and then it responds, right? So we call that asynchronous architecture, like it's happening one after the other. But our brain is thinking while listening. So as I'm speaking to you, you're already forming your thoughts and if I'm talking for too long, you'll interrupt me, right? Or you might be taking notes in the backend, right? So you might be essentially doing tool calls while I'm speaking to you. And so the whole idea is that if you ever want to pass the Turing test of how the human brain operates, you need something that is working asynchronously and can not just understand emotions, but also operate, like, take in speech natively and give out speech natively

• 15:30Speaker 05

  asynchronously. And so that's why we've been sort of building Hydra. Now Hydra is our speech-to-speech model. Now, in terms of accuracy and interpretability and all those things, I think whenever there is like a new architecture that comes out, it's often sort of good in one parameter and then regressed a little bit in other parameters, right? Like, so for example, the speech-to-text accuracies for just a speech-to-text model might be way better than the encoder of a speech-to-speech model that you have. And so I think the challenge is that while you made progress on the, you know, making it

• 16:12Speaker 05

  more natural and operate more human-like, et cetera, et cetera, how could you keep the accuracy bar the same? And so I think a lot of that comes down to interpretability of such models, like, because you don't want it to be a black box. So how can you understand where it is lacking? And so that's a lot of research that we do and how to make speech-to-speech models more interpretable. The other constraint is, I think, initially the speech-to-speech models, like I think Sesame, et cetera, that came in, that they were just speech-to-speech, like, they literally took in speech and gave out speech. Ours is like multimodal. So it takes in speech and text, gives out speech and text both. So it can do tool call and take in text parallelly. So if you want to put guardrails,

• 16:56Speaker 05

  if you want to do all those things, that constraint does not go away. What we are also seeing is there is still, at least in enterprises, a lot more cascaded deployments compared to speech-to-speech model. But speech-to-speech, I think, will be like an eventual future. That's my opinion.

• 17:17Speaker 02

  So my take is that when you have like competing things, you end up in a hybrid. And I think the answer for the midterm will be some form of hybrid. Because the speech-to-speech are improving, there are parts of the conversation loop where you would say like, oh, my use case of my workflow is complex. I'm going to use multiple LLMs anyway. So for the active loop, like I'm talking to as a human, you called me, I'm answering your questions. But whenever I have to do some kind of lookup or something, I delegate to another LLM, which will then do the cascade stuff. So the voice in the first part of the loop keeps running. And then you have like interesting kind of semaphores or like you can think

• 18:02Speaker 02

  of those threads, you want to interrupt the, the speech-to-speech model because you realize that this is a complex question and you want to, before the speech-to-speech response, you say, actually, you cannot answer this question, right? And delegate to the cascade and so on and so forth. So typically, I kind of say, there are many use cases where, you know, if you wake me up in the middle of the night and ask me a bunch of questions, there are definitely some class of questions that I can answer without thinking. And we all do our jobs in a certain way where we can be in autopilot like 50% of our time, right? So as these use cases become emergent and, you know, you're fully deployed with

• 18:43Speaker 02

  a customer doing high volume, you could essentially train specialized models that fully understand like that 50% use case very well. And you could, you could always ask the model like, hey, what is my return policy? And it'll say like, in the simplest case, this is my return policy and it applies to like 70%. And I know which SKUs or products it belongs to or it applies to. And if you ask me outside of that, then I have to like do all the crazy work. And if I don't, then I answer it, right? So.

• 19:17Speaker 01

  So on these cascaded pipelines, what, what models are you guys using? Let's say of the frontier models of the, you know, GPTs, of the, of the, what, OPIS and Sonnet, are you guys using the latest ones? Or are you using, you know, like GPT-4? Because it's like the right balance between speed and intelligence?

• 19:40Speaker 02

  If you turn off thinking, then you can actually use any of the like, if you want a fast model which responds to like the prompt and does not need to think, because if you want it to think, you actually start to think about paralyzing the work because you want the, the first model to be super fast. I still like my Gemini 2.5 really well. Like, it's so fast. It's like, the 3.5 which they launched is like, okay, it's slower, but the 2.5 is so good. Even the Haiku is really, really good. They're still slower than what you would expect, but yeah.

• 20:15Speaker 01

  How do you parallelize that pipeline? Because it feels like, oh, well, I do have to first know what text the person said. And then the LLM needs the text to do anything. And then you can only generate speech once the text has been generated that you want it to speak.

• 20:31Speaker 06

  You just spend the, you know, you can send the speech. Like in the speech to speech model case, you would send the speech directly to one place, fork it into another place, run the STT there. And if you want multiple models, the text output from the first forks into multiple LLMs. You can think of it as like a waterfall, like, you know, it comes down.

• 20:54Speaker 01

  Because you don't know what the output exactly will be.

• 20:57Speaker 06

  Yeah, you better put a gate at the bottom, which would be in XR or whatever fancy thing you want to do, which is like the, if all of them say we are right, then you put another LLM downstream, which one is better. Like, you can go really crazy. If you love computer architecture, 20 years, we had nothing to do. Now you can do all this crazy stuff from 20 years ago.

• 21:21Speaker 01

  Yeah. So how do like accents and different languages all fit into this? Maybe Sudharsan, because you guys are building speech to speech.

• 21:31Speaker 05

  Yeah, so at least for speech to speech right now, we are just fully focused on making it more intelligent in English and, you know, getting it really good in English. Like, we don't want to do any other, we don't want to introduce any other variables because the technology in itself is, I would say, quite frontier. Asynchronous is not yet like mainstream, et cetera, right? But in terms of like cascaded, yeah, we've seen like a lot of demand in. So we have a lot of presence in India, so we see like a lot of demand from India. We see a lot of demand from Latin America, et cetera. I think US is more like mostly English and then there's some Spanish and then there's a lot of accents to it.

• 22:12Speaker 05

  And I think, yeah, US, at least, we have not had any troubles in terms of technology. I think it's, yeah, I mean, noise cancellation is probably the last mile of problem that is pending in terms of handling those things. But yeah, that has nothing to do with accents. I'm curious what you guys have seen.

• 22:34Speaker 06

  Maybe I can add some examples about that. Because on multilingual, I like to definitely leverage Cascade Model because there are multiple levers you can pull. To talk about an example, it's a customer we were deploying for Japan. So you could typically go, you know, to your 4.1 and OpenAI or your Deepgram for transcriber, and that typically worked well. Now, the challenge became on the voice piece. So it turns out, what we thought it was a state of the art for something like English, Spanish, Portuguese, those type of, that are typical in the US, didn't work at all.

• 23:16Speaker 06

  So one of the benefits about this is that you can swap certain pieces that doesn't quite work for you. At least on Vapi, how we solve it is that we let customers bring their custom text to speech server. So they actually have like, there's some startup, there's a lab in there, and there's typically, you will find that in every market. Arabic is also really hard to get it right in pronunciation of brands, addresses. And that really, we don't, cannot build expertise and optimize for every single use case. So that's one of the reasons why I like that as well.

• 23:54Speaker 01

  Cool. Also, actually, question on the like Sesame model. Yeah, like they came out with this like really insane like demo, and then I've never heard of them since. Like what, do you guys know like what happened and what's going on?

• 24:07Speaker 05

  Yeah, I think, so they're building hardware is what I understand. Like, they are putting those, the voice into some sort of a hardware. I'm not sure if it's glasses or what are they working on exactly? But I know a few people who got hired there and they're all sort of focused on the hardware and voice sort of backgrounds.

• 24:27Speaker 01

  Yeah, the answer I got was that the CEO had already made a ton of money and he just really wants to play around, basically. So I guess he can do whatever he wants. Cool. So is it standard practice for, in this like cascaded pipeline, for you to just give, just have one giant like system prompt that you're giving the LLM on, these are all the rules for how you should be responding? Or is it more of a, like a workflow approach? How do you guys think about that? What's the standard best practice right now?

• 25:06Speaker 03

  So I think it highly depends upon the use case and also the size of your prompt, right? So I think I have a counter approach to Verun in terms of that. Like, usually in inbound, I find you know, you're usually having dedicated lines and you know specifically where the workflows are going to go, right? And so in these cases, if you know a little bit more about and you can predict where the conversation is going to go, then you can have a more of a node or graph builder kind of approach. But if you don't know what's going to happen, typically, I think in outbound, yes, there is a dedicated like message, but what the caller says back, you never know. Because they could just be frustrated, they could be angry, or they could be, I don't know, like annoyed.

• 25:51Speaker 03

  And in that case, when they're erratic, when a person's erratic, you never know. So having it in a way that is like all in one, one prompt allows you to have a more like a central brain in terms of, I can pick up components from my prompt that may not have been in a dedicated flow, but actually can be utilized to enhance the response. And then on top of that, you have to think of your knowledge base and knowledge, the things at your, at your expense, right? Like, in an inbound flow, maybe you know exactly when you need to pull a specific part of your knowledge. So you don't need to extract every single piece of context, right? Whether it's your websites and documents.

• 26:34Speaker 03

  But sometimes in outbound, you should utilize all that information to get a better response and you will rely more on cosine similarity and stuff to make sure that, hopefully, your retrieval is good enough to respond to them.

• 26:51Speaker 04

  I think from my perspective, the limiting factor has been and will continue to be, although it's getting a lot better over time, just to, you know, some of the earlier points made, the ability for these frontier LLMs to be able to take a massive wall of text and not just read the first three lines and the last three lines, but actually be able to properly reason and like localize themselves into, you know, we are halfway through this procedure and these things have already happened and these things haven't happened yet and these things are true and these things are false about the situation. And so therefore, like, this is the one thing that I should be really zooming in on right now, right? So from our perspective, I think we kind of take an approach that depends on the customer.

• 27:35Speaker 04

  But we do, within our execution engine, have the ability to make some of these rules or prompts or guidelines more situational so that we can, you know, check if these are relevant right now before we even include them or don't include them in the system prompt. And so that was our solution that was especially relevant, you know, like seven months ago when the LLMs really were just like consistently skipping step 4A1. But we have found over time now that sometimes the reliability of deciding when these things are relevant or not is actually worse than just giving all of it to the model, because the models have improved a lot since that time. And so now we're going back, I think, swinging a little bit back towards, just give the

• 28:18Speaker 04

  model everything and more or less it will figure it out. But at the end of the day, you know, with non-deterministic systems, it really just ends up being customer by customer. We have to really understand their use cases and at the end of the day, just build it and then test it a whole bunch of times and then use LLM as a judge to tell us, you know, did it work, was it reliable? Or do we need to add a little bit more preprocessing and context optimization ahead of time in order to make it reliable? So the, the real answer is most likely always just test and find out and then test again and then find out again and, you know, it goes on forever.

• 28:54Speaker 01

  I'm assuming that, yeah, maybe the models are able to handle those edge cases now, like just in a giant prompt, but that probably also increases latency. So first, could you talk about like, what is a good latency look like for a voice agent? And is that even true that, yeah, the models are getting better, but they also require more time? And so for the end user, that's actually a bad thing?

• 29:20Speaker 04

  Yeah, that is a really good question. I mean, I think the definition of what like benchmarking latency stats look like that are good have changed a lot and expectations are, you know, getting better and better and better from our customers. Overall, I think it is also important to mention that as much as fillers and contextual fillers is something that we don't like to hear, just as a noun, it is normal for, in normal conversation, for humans to say a few words while they're thinking. And so we've actually found that when we have zero contextual fillers, sometimes it actually feels more rigid than when we don't. And so we tend to find that having the, the right amount of fillers that kick in when,

• 30:03Speaker 04

  you know, we are experiencing some sort of latency because of some very large prompt or some very complex thing that the model is thinking through, most commonly, you know, running a tool call where our customer's API becomes the bottleneck and we're waiting, you know, five seconds for something to come back. Being able to say, hey, just give me a sec, looking at that, okay, cool, here's, here's your answer, right? There's actually nothing wrong with that when we've, when we've tested with our customers. Obviously, the, the limiting factor in that case then becomes how good are your fillers? And how natural do they sound? And so that's been a place where we've invested a lot of time as well to say, how do we create an execution engine where we really don't need fillers, but in those moments where we do, they're really good? So that's at least my take.

• 30:44Speaker 04

  But again, I know folks have probably a lot of opinions here about latency and how to find that trade off. Because in my opinion, I honestly think that that's the hardest problem to solve in voice deployments is just that trade off that's always going to be there of like performance versus stability and reliability in, in general, so.

• 31:05Speaker 06

  Yeah, I agree with all of that. Some of the ways that we think about it in VAPI is offloading some of that instructions. So break it down. Sometimes you don't really need that, you know, 10, 15 step workflow in the in the prompt. One example I can talk about is this collections use case. We do allow users to capture, businesses to capture credit card information, but not every call is about that. Some people already have a payment method maybe added or that. So one way of doing that offloading into a specialized agents. So we can start thinking about multi agents and there are many architecture and patterns in there.

• 31:48Speaker 06

  Is to constrain the instructions that you give it to to that particular moment of the conversation. And once that achieves its goal, it can go back to that more free form approach with has loaded, like your longer context. Something we're also been experimenting lately and it's more about the Garel space is to your point where you can have, think about it as a waterfall, you can actually stream to multiple places. So what if you have streaming down to a small language model, which can do inference in a very little time. So think about classifiers to maybe collecting intent.

• 32:32Speaker 06

  While the pipeline is still maybe adding a filler word as well to keep a consistent experience. But kicking off a background process that is still intelligent, still thinking. So those are still things that we'd like to invest a little bit more research on. So yeah, constantly investing in that.

• 32:56Speaker 03

  I think one thing that we haven't also factored in in terms of the latency trade off is cost. Right? Now that you're putting such a big, large prompt into your LLM, you have to think about those calls where many people just hang up and predominantly calls just hang up 10 seconds into the call. Right? Now you're having to pay for all those tokens to just get inputted in. So that like Stephen mentioned, splitting out your prompt, you are also playing to that strength of lower latency and lower cost.

• 33:31Speaker 02

  Let's speak just to the benchmarks. So we publish turn based STT benchmarks, LLM benchmarks and TDS benchmarks. The whole point of that is that as I think today, Nematron 3.5 launched and there's already an ASR benchmark out, did very well, the Nvidia one. So the idea there is that all our tooling is open source. So with the STT benchmarks, the TDS benchmarks and the LLM benchmarks, all the turns are actually open. So you can run it against any new model that comes out. You can run it locally, you can look at the outputs from from our graphs and compare that to what you're getting on your infrastructure, if you're self hosting or elsewhere.

• 34:18Speaker 02

  So that's one important aspect to like, consider. If you know what your turns will look like, you can just update or, you know, just download our benchmarks, update the turns and see how that like goes for your own flows.

• 34:34Speaker 02

  Usually for the LLM stuff, it's really useful because you can throw the STT and the TDS and just run the LLM inference loops to like check against a customer. So yeah, I think evals are really important. And if you have like, while talking to customers, if you can evaluate like what type of conversations or like what type of workflow they're going to have, like those are real direct inputs into like what you can directly start testing, even though you're not live with them or even they've not shown any intent. So I think that's where like the SEs and the forward deployed engineers can actually come to for like you. We all have cloud running in the background, right? So you can just say, here's what we had a discussion like, can you build an eval?

• 35:19Speaker 02

  Like based on our eval suit, like, can you like just run a smoke test with the conversation information that we already have? And that helps a lot, I think.

• 35:30Speaker 05

  Yeah, just just adding to that, like we are seeing, so we, while we focus on models, we also have all the models sort of orchestrated on our platform. So you could build a voice agent on smallest. We are seeing a lot of customers who just pair our models with our own like small language model. It's called Electron. We fine tune one of those to make it work for real time voice use cases and see a lot of folks using that with better knowledge basis memory, etc. Over GPT four or 4.1, you know, the popular real time models. Just A, from a cost perspective, it's lower. B, from latency perspective, it's like way lower. And then just more reliable, I think, I'm not sure if everyone faces this, but like

• 36:15Speaker 05

  OpenAI APIs spike and then you have no control on those latencies. With, if you sort of have a self hosted model, you could scale up, scale down based on your requirement. So that's one thing we are seeing a lot with our customers.