[AIEWF Preview] Gemini in 2025 and Realtime Voice AI

As part of our AI Engineer World’s Fair preview, we’re releasing a special cross podcast recorded with Sam Charrington of TWiML AI at last week’s Google I/O! TUESDAY: Shrestha and Kwindla’s workshop: https://www.ai.engineer/schedule#milliseconds-to-magic-real-time-workflows-using-the-gemini-live-api-and-pipecat TUESDAY: Kwindla’s workshop: https://www.ai.engineer/schedule#building-voice-agents-with-gemini-and-pipecat WEDNESDAY: Shrestha and Kwindla’s talk: https://www.ai.engineer/schedule#milliseconds-to-magic-real-time-workflows-using-the-gemini-live-api-and-pipecat WEDNESDAY: Kwindla’s keynote: https://www.ai.engineer/schedule#-voice-keynote-your-realtime-ai-is-ngmi THURSDAY: Logan’s keynote: https://www.ai.engineer/schedule#a-year-of-gemini-progress-what-comes-next Catch all the speakers at AIE (both workshops and talks): Logan Kilpatrick: https://www.latent.space/p/chatgpt-gpt4-hype-and-building-llm Shrestha Basu Mallick: https://www.linkedin.com/in/shresthabm/ Kwindla Hultman Kramer: https://www.linkedin.com/in/kwkramer Full Video Episode This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:04Speaker 00

  Hey, I'm Sarah Chant, and welcome to another episode of the Twemo AI Podcast.

• 0:08Speaker 01

  And I'm Srix. This is a special episode of the Late in Space Pod with Twemo at Google I/O. Welcome.

• 0:15Speaker 02

  Thanks for, thanks for being here. Thanks for hanging out with us. I'm excited.

• 0:19Speaker 01

  Logan, you are, you were our first guest. You came back remotely a few months ago, and now you're back. You're sort of the, a lot of the face of the AI studio, basically, like that, that a lot of people are using. I'm using it. And, you know, I think it's a really welcome change for people, like, being more accessible with, like, the rest of the Google suite. And Stressa, you've been, I actually don't, super know you all, like, I just generally have you pegged as, like, PM of the API team with a particular focus on live.

• 0:46Speaker 02

  Stressa runs the show behind the scenes. Behind this is the less public face running the show behind the scenes. Model launches, the live API, generally all the stuff that's happening in the API is, uh, Stressa's hard work, so.

• 0:59Speaker 03

  Thank you for that, Logan, but I think everyone knows who really runs the show. There's public evidence there. But yeah, I, I work with Logan and a few other excellent PMs, but I lead the API side of the house.

• 1:14Speaker 01

  There's a lot of announcements. I think a lot of people have done their recaps. What are you guys' personal highlights over I/O?

• 1:20Speaker 02

  I'll break the rule and I'll give two that are not the sort of big, big flashy ones. I think the two that I think developers are going to be super excited about, one, Thinking Budgets coming to 2.5 Pro. So, and you can also, you'll be able to disable Thinking as well. So if you just want 2.5 Pro as like a raw, non-reasoning model, we'll have that, uh, hopefully in early June. And then Thought Sumories. So I'm, we, we've had this debate internally about, like, do we need to show full thoughts? Do developers want full thoughts? I think developers say they want full thoughts. We have Thought Sumories right now as a sort of step in that direction. It'll be really interesting to find out and get the feedback around, like, what are things that work, that work with Thought Sumories? What are the things that don't work with Thought Sumories?

• 2:00Speaker 02

  I was reading some threads last night about, like, Thought Sumories are now live in Cursor as well, and people were sort of reacting to, you know, having summaries versus not full thoughts. So it'll be interesting to see, but I'm, I'm excited for both of those things. Thought Sumories are live now. Thinking Budget for 2.5 Pro will land with the GA model in a couple of weeks.

• 2:18Speaker 03

  Yeah, and I should say we already do have thinking budgets in 2.5 Flash. I do think, you know, with all of the other features that we are releasing on top of our thinking models, summaries, budgets, I think this is our way of, you know, you have the models, but then we want to give developers as much control as they can on top of models. But coming back to your question about my favorite feature, it's really hard to pick because like all of these features we've been trying to push out for weeks. But I think native audio output is a personal.

• 2:51Speaker 00

  I was just saying that with Quinn, yeah.

• 2:52Speaker 03

  Yeah, yeah, is a personal highlight. I actually, Quinn and I have been playing with it together for a bit as well. I think especially with all the, obviously the voices sound great. The fact that it can switch in and out of languages. So Matt, Bill, also our boss, actually has a demo on Twitter where it actually speaks Klingon, even though that's not an officially supported language. But, you know, I speak Bengali, just being able to, for it to switch into and out of Bengali and English, that's been special. And then if I get to pick another one, it's, I'd say we released a new tool called URL Context. And the idea is that you can use it by yourself or pair it with search

• 3:35Speaker 03

  to retrieve more in-depth information from webpages in a way that's respectful of our publisher ecosystem, of course. And I think that this will unlock new use cases, like if people want to build their own version of a research agent, which is something developers ask us for a lot.

• 3:51Speaker 00

  Yeah. Yeah. Worth mentioning that just prior to I/O, there was a ton of new, interesting new capability, including the update to Gemini 2.5 Pro, as well as the implicit context caching, which I know a lot of folks are waiting for.

• 4:03Speaker 02

  We made implicit caching happen. I think there was lots of feedback that people are like, explicit caching is nice. Like, there's definitely use cases where it makes sense, but people want implicit caching. So I'm happy, passing the cost-saving on to developers. You don't have to do anything, it just works right now and you're saving money. It's a great outcome.

• 4:21Speaker 01

  I don't want to manage that myself.

• 4:22Speaker 02

  Yeah. There are, there are people, like, I think if you, there's so many use cases where, like, you're just doing chat on the same stuff over and over again. And for those use cases, you know, you want to be able to explicitly cache the thing and make sure, guarantee your cache it so that you save money. So I'm happy we have that.

• 4:38Speaker 01

  Is there any behind the scenes of like, what makes caching hard? Or anything that people don't appreciate about caching as a general concept? Because I think this is a very important pricing paradigm that people need to really get behind.

• 4:50Speaker 02

  Yeah, that's a good question. I think there's a trade off between like all of the dimensions of caching, which is around like the sort of latency, because in some cases you're getting latency gains. In other cases, it's like, how much, you know, what's the cost for Google? How much stuff do you want to cash altogether? So we could have an entire episode and get a bunch of the caching people. It's like a good example of like an infrastructure problem to be solved. And a bunch of the folks who we work with love working on this problem. So we should do a deep dive episode.

• 5:17Speaker 01

  Yeah, I want to shout out that you've been doing more video stuff. You have your own podcasts as part of your Gemini work. You've also been doing a video with people on the team.

• 5:27Speaker 02

  Yeah, it's been fun. We had, so we should do a caching episode.

• 5:30Speaker 01

  Exactly. You did the long context one. People loved it.

• 5:32Speaker 02

  Your reception was very, was very positive about the long context one. So thank you. That was the first time that we did like a more deep technical discussion with folks on the team. And Nicolay is awesome. And we actually just did one with, we did one with Shrestha about the live API, which I'm excited about. We did one with folks on the team about the multimodal capabilities in Gemini. We're going to do a pre-training one, hopefully, which will be really cool. We've got a bunch of people who are excited to talk about that. So there's a bunch of them in the works and it's, it's fun to make them happen and have those conversations.

• 6:00Speaker 01

  Yeah. And my underrated pick is Gemini diffusion.

• 6:03Speaker 03

  Yes.

• 6:04Speaker 02

  Yeah, yeah, yeah.

• 6:05Speaker 01

  It's not underrated.

• 6:06Speaker 03

  I'd be underrated for all the love it's getting.

• 6:08Speaker 01

  Yeah. So like, apart from speed, I wonder like what the potential results of a diffusion language model could be.

• 6:16Speaker 02

  Generative UI, generative UI, this is the way the generative UIs happen is through, through this experience. The UI bit, just like being able to like, say, I want, you know, build the UI on the fly using code based on what a user does. So like you have no pre-compiled notion of what your website is. And as a user goes through, as they click buttons, thousand tokens generate. And it just like makes that UI for you.

• 6:39Speaker 01

  Interesting.

• 6:40Speaker 02

  I think that's going to be possible. I mean, I think there's a lot of work to productionize, make Gemini diffusion like actually a high quality model that meets the bar for us to bring to the world more generally. But I do think that's going to be the, the killer use case will be like this generative UI experience that doesn't exist today because the models just take too long to generate tokens.

• 6:59Speaker 01

  Yeah.

• 7:00Speaker 00

  For me, it was really the role that audio and video are taking throughout a bunch of independent product releases, from the generative models to the live API to the on-the-fly transcription and translation. It's, I think, kind of foreshadowing the role that that's going to play in a lot of developer applications.

• 7:24Speaker 03

  Yeah, transcription, actually, even before we released Native Audio, now, of course, you get text and audio interleaved in the output. But transcription used to be one of the biggest use cases we had on the live API.

• 7:37Speaker 00

  What are you seeing as the challenges for folks getting started with live?

• 7:40Speaker 03

  Yeah, that's a great question. I think firstly, awareness, right? Like, people knowing that we have a live API. That's why we're doing this, talking to you folks. I think some of the areas where... So, we were actually the first to market with also video input. But one of the areas where we've been getting a lot of feedback is in session length. Anybody who's been trying to put this in production, like, when we started, you could do like 15 to 20 minutes of audio, I'm sorry, and about 5 minutes of video. And so, we've been putting in a lot of knobs for developers, and we can talk about that more if you guys want, for people to have a sliding window or decide what resolution they want to send video in,

• 8:25Speaker 03

  but, you know, to basically increase the session length. And then tool calls, that was another area where we used to get a lot of feedback. Again, we were very proud because we introduced tool chaining first. So, you could chain search and code execution to all kinds of analysis. But then, we've had to do a lot of work in improving function calling, improving the performance of search, and we continue to push on that.

• 8:48Speaker 02

  I've got a quick one on this too, which is, I think the level of commitment you need to make to the model provider in the world of the live API, like, I do think for developers, it's a higher bar. If you look at, like, what does chat completions or, like, what does, for us, generate content provide from just like a text modality perspective? It's like, it's a pretty lightweight thing. There's a lot of model providers that have that option. Like I could switch to a different provider if I end up not liking some model provider, which I think is good for the ecosystem. I think if you look at a lot of the live API infrastructure right now, like you really do need to commit that you're like gonna, you know, there's it's not easily interoperable between different model providers. Like everyone's infrastructure is all bespoke and different. So like it is a, it's a different level of commitment that you need to have to like really bet your company

• 9:33Speaker 02

  or your business or your product on the live API, which I do think is a challenge for developers to sort of make that level of commitment in this like fast moving AI world. But I think hopefully there'll be like some level of like similarity and you'll get some model agnostic infrastructure to help make that, you know, make developers feel a little bit, a little bit easier about being able to move between models potentially.

• 9:55Speaker 03

  I could go on and on, but if you have say more complex workflows, then one of the things is being able to change the system instructions at every step of your workflow. And so, yeah, so onboarding some of the more complex use cases with the live API has been a work in progress as we've released like new features.

• 10:13Speaker 01

  So what kind of complex workflows are we talking about?

• 10:15Speaker 03

  You know, like we have people who are building, say, gaming agents, but like which have multi-states, for example, in them. We have a lot. I mean, this was a famous demo at Next, but we have folks who want to, you know, customer support agents, of course, you know, they can, the sessions can last for hours, right? Then there's a lot of use cases around people showing a certain screen.

• 10:40Speaker 02

  This is the coolest use case, honestly.

• 10:42Speaker 03

  Yeah. And I was referring to like the famous demo at Next where Shopify showed how to set up a DNS using Cloud Play, right? So in certain cases, especially the longer your workflow runs, like you might have to go from one state to another state and might want to change their SI. Or if you handle it, hand it from one agent to another agent, you might have to change the system instruction.

• 11:04Speaker 00

  When you're thinking about building voice-based applications is speech to text and then processing with a standard LLM, would you say that's like a precursor to the Live era or are these two distinct paths that are still viable and that you still see being viable going forward?

• 11:23Speaker 03

  That's a tough question and I'm still like, no, we have both out. I do think perhaps eventually for most use cases, as these audio-to-audio architecture models get better, a lot of use cases will probably transition to that. But you know, when we talk to our developers, they still very much like those componentized components. So that's why we also put out two new text-to-speech models at I/O. Not available through the Live API yet, but really high-performing, controllable, promptable text-to-speech models.

• 12:04Speaker 02

  I have an angle of an answer to this question, which is I talked to Cori this morning, who's our boss's boss, the CTO at DeepMind. And Cori had a really interesting take, which is just around like, what makes, one of the main things that makes what we're doing at Google with Gemini different than what a lot of the other labs are doing is like, we're here to make one model and like, that model is Gemini. And like, I think, I think you do need to, to trust this point, like, to make the capabilities work in some cases, like, you do need to have these forks that like, go off and make that capability and harden it and then find a way to bring it back into the mainline model. But like, we want to make one model and it's the Gemini model and like, not have the sort of splintering of all these different capabilities.

• 12:46Speaker 02

  And we've done a good job of, I think, thinking the reasoning stuff was like, the best example of this. We had those, they were separate from the mainline Gemini models so that those teams, the research teams could go and hill climb and make progress and not need to be constrained about like, how do we do this without having there be collateral damage on other capabilities like multimodal or something like that. But the teams went and did that and then they find a way to sort of bring the capabilities together. And oftentimes what you see is there's tension in bringing them together, but it's the root.

• 13:14Speaker 01

  Really exciting thing is what happens when you bring the capabilities together and like 2.5 Pro with reasoning is a great example of this where like multimodal with video understanding ended up like having this huge like it's having this beautiful moment. The model is like soda out of the box. Because of all the reasoning capabilities that were baked in. It wasn't because they like did a bunch of stuff to make video understanding really good. It was just like an artifact of bringing and merging those capabilities together. So I think that is like a North Star for Gemini models makes makes a ton of sense.

• 13:44Speaker 03

  I agree with you and that's what I said, right? Like I think eventually a lot of use cases will end up on Gemini will end up on natural voice. But I think in order to foster development, like we have these offshoots from right to right. We have our Imagine models for image generation, even though now another I O well slightly pre I O announcement, you can do interleave text and image within Gemini also, right? And it unlocks.

• 14:09Speaker 01

  Yeah, but those are different models, right?

• 14:11Speaker 03

  Those are different models.

• 14:12Speaker 01

  One is autoregressive, the other is diffusion.

• 14:14Speaker 03

  The other is, that's what I'm saying, right? But for a lot of image generation, image editing, high quality photorealistic use cases, developers are still using Imagine, but then, you know, slowly but surely we're bringing those capabilities into Gemini as well.

• 14:29Speaker 01

  Whoever's watching this, we had a mid I O switch because obviously, you know, there's a lot going on here. It's not AI shape shifting. I know, I know. But we also have Quinn, actually, who made this podcast happen. But you're a founder CEO of Daily, welcome. I'm a big fan of all things voice and audio. So it's fun to be here with you and with Trista. Quinn actually runs the Voice AI Meetup in San Francisco. Like you are basically consistently the leading sort of community builder. And you're very generous of your time and knowledge. I really appreciate that. And obviously also recently you started Pipecat, which is this open source framework for voice orchestration. Which has really great support for all the Gemini models. You wanted to say something about the relationship with Gemini and Daily?

• 15:13Speaker 03

  I just wanted to say that it's been a very, very fruitful partnership with Daily. They've been our partner since the launch of the Live API. And you know, a lot of their feedback that they continuously get has been, you know, instrumental to the success of the Live API. So both Daily and Live Fit are, were partnered with them.

• 15:34Speaker 01

  Quinn, I think, I think, you know, we had, we had a little bit of a prep for this. You also wanted to dive into a little bit on like the cascade of models in Gemini Live.

• 15:43Speaker 04

  I mean, I think Shrestha has taken a really interesting approach designing these APIs. So you talked about components a little bit. You talked about how you want to be able to do things both in the Live API and in the more sort of traditional chat API. And you've got, originally you designed the Live API to have audio in, but then it's a separate text model, the Notebook LM models, audio out. What was the sort of driver for that originally?

• 16:05Speaker 03

  I mean, that at the time was, we wanted to hit a certain quality bar, a certain latency bar. And you know, Notebook LM was already out and the TTS models that were powering Notebook LM were very, very good. But we wanted an aspect of native. So it was native audio in, but TTS out. And we still have that architecture available through the Live API. But then now we just released audio to audio architecture.

• 16:30Speaker 04

  I mean, the infrastructure for this stuff is so interesting because you're always balancing latency, cost, output quality. There's no free lunch.

• 16:38Speaker 03

  Yeah. And other things like multilinguality. Coming back to your question earlier, Sam, a lot, we had a lot of users asking us for, say, better German language support or something, which hopefully now we've delivered on with these models.

• 16:54Speaker 04

  Yeah.

• 16:55Speaker 01

  But now you have audio to audio in the, in the Live API as well.

• 16:58Speaker 03

  In the Live API only is where we have the native audio output models.

• 17:03Speaker 00

  Continuing to pull on the component versus single model thread a little bit. When I think about voice, I think about it as being an area where to deliver solutions, you need to surround that strong model with a lot of voice-specific infrastructure that is, you know, I'm imagining challenging the scale. So, Shrestha, can you talk a little bit about that? And maybe we can have Quinn talk about that from his perspective.

• 17:30Speaker 03

  So, the first thing that comes to mind is, of course, the voice activity detection models that we have. And we've done a lot of work like finessing that model server side. But we've also learned that we need to provide some knobs to developers. So now developers can actually tune the sensitivity on our voice activity detection model as well as, you know, how much of the prefix, like how much of a time duration at the beginning, at the start or stop of saying things. And we also have a mode where now you can, where you can disable our voice activity detection and bring your own. But I think the larger point that you're touching on, Sam, that I do want to mention is it is really, really hard to bring all these

• 18:14Speaker 03

  components together and still get latency down to where it needs to be, you know, in the 500 to 700 millisecond range. Like, it's one of the hardest things we've had to do with the live API.

• 18:25Speaker 04

  What we see is that the shape of building these real-time voice agents is a different set of developer problems in the shape of, you know, non-real-time or text mode things. One of the fun things about partnering with Shrestha and DeepMind is we work on this open-source framework that people use to build these kind of production voice systems. And so, we try to solve problems at the framework level like turn detection, like context management. As the models get better, as the use cases get more clear, some of those features migrate from the framework into the APIs, which makes life easier for developers. The use cases, at the same time, continue to broaden out. And so, there's more things for the framework to do. So, we're sort of filling the top of the use cases, building blocks, developer experience funnel, and pushing down as we all get better

• 19:10Speaker 04

  and we all figure out what this new world looks like.

• 19:12Speaker 03

  And maybe this is also a good segue into web sockets versus web RTC, Quinn.

• 19:17Speaker 04

  Yeah, you know, there's so much infrastructure, like one I for my whole career, I've been building, you know, large scale, low latency network stuff. What we saw from my perspective, when we started to see the possibilities of voice AI was, you need this packet routing like down underneath the inference layer.

• 19:32Speaker 01

  Oh, my God.

• 19:33Speaker 04

  There's like the AI inference stuff, but then there's the just how do you move the audio? And increasingly, video around the internet. And so there's a whole new generation of developers who are interested in these networking protocols because voice AI and now real time video are so interesting, which is super fun for me because like I've always thought moving packets around is one of the most fun things you can do on the internet.

• 19:53Speaker 00

  Yeah, seven layers of the OSI stack.

• 19:54Speaker 04

  Exactly, exactly. Yeah, at at pretty, pretty demanding real time latency, this rest is saying like, human beings expect you to respond in a conversation in 500 milliseconds or so. And if we're talking to an AI, we don't relax that assumption. We bring our assumptions about human conversation into that experience of interacting with an AI.

• 20:15Speaker 03

  Yeah, or not respond. So.

• 20:18Speaker 04

  It's a great point.

• 20:18Speaker 03

  Yeah, so like one of the features that we've pushed out, a little more experimental, but we love for people to test it is what we're calling proactive audio. And it's available only in the native audio, in the audio to audio architecture right now. And what this feature does is it's trained not to respond to irrelevant audio.

• 20:40Speaker 01

  Okay, so it's like a refusal kind of.

• 20:42Speaker 03

  Yeah, or you could call it directionally like semantic voice activity detection, right? So basically, yeah, like let's say I'm talking to the AI and then Quinn comes and asks me a question and I respond to Quinn, it'll know when not to respond. So.

• 20:58Speaker 00

  I saw that in one of the demos, the AI seemed to ignore a background question from someone else in the video.

• 21:04Speaker 04

  I think there's two threads to pull on there. One is that's another great example of things that we had to work really hard at the framework level to implement. It's much, much better if it actually migrates down into the model or the API. The other is part of the magic there is this semi separate feature, but I think they're multiplicative of now your models can actually recognize two different people just based on their voices. You and I replied that.

• 21:29Speaker 01

  They have to, they, yeah.

• 21:31Speaker 03

  This is not officially supported yet.

• 21:33Speaker 01

  Okay.

• 21:34Speaker 03

  The model just does it.

• 21:34Speaker 01

  But just try it, right?

• 21:35Speaker 03

  Just, I mean, you try it and give us feedback.

• 21:38Speaker 04

  Is it okay to talk about it? Because it's, it might be my single favorite thing you can do with these models that you previously have not been able to do.

• 21:45Speaker 03

  I mean, you can talk about what you've observed. I'm just saying it's not a thing.

• 21:49Speaker 01

  Not officially.

• 21:49Speaker 00

  And what specific models are we talking about? Because speaker identification and diarization has always been really hard for these models.

• 21:56Speaker 03

  All the, it's called, gosh, like model naming now has become, so it's called the native audio dialogue. You'll see it in the live API, but that's the model.

• 22:06Speaker 01

  Excellent.

• 22:07Speaker 03

  And then, you know, and to your point again, Sam, about architectures. One thing that we launched on the cascaded architecture that we hope to eventually bring to the native audio as well is asynchronous function calling. So earlier, the way it used to work is if you wanted the model to do a function call, you'd have to wait for the response. And now you can set a non-blocking parameter and the model can go off and execute the function in the background.

• 22:33Speaker 01

  Love you. I love you so much.

• 22:34Speaker 03

  Yeah.

• 22:35Speaker 01

  Yeah, that's great. We do have to wrap up. So I think one fun thing that we can do to wrap up would be a wish list for like next year's I O. What would be one thing that you would wish? It doesn't have to come true, but you know, wish happens with Gemini.

• 22:48Speaker 00

  Well, I was hoping for Gemini 3.0 at this I O. So maybe Gemini 5.0 at the next I O.

• 22:54Speaker 03

  Tell us what you mean by Gemini 5. What do you want in Gemini 5.0 then? I'll let Quinn go.

• 23:01Speaker 04

  I'll just put on my hat as representative of a big community, people building this stuff, more and more languages, because AI is global and there are so many communities all over the world that are starting to do this stuff.

• 23:12Speaker 01

  Can we do language, Laura's? You know, it's, it's hard to stuff everything in one language and it's in one model. Yeah. Okay. But yeah, and then.

• 23:19Speaker 04

  But they're, they're building one model, as they said. They're building the one universal model.

• 23:22Speaker 03

  That would be a boring answer, but I think really more and more. No, languages, I mean, wasn't I telling you earlier? Like, we officially support 24 languages, but you can try talking to the model and cling on and it'll respond to you. So I think we'll get there way before next I O. But I just think more and more capabilities into the main model is what I would say. I'll have to think about this.

• 23:46Speaker 01

  Yeah, yeah, it's a fun parlor game. But it also helps people align as to what is possible and what's coming up. Thanks for your time, everyone. This is very hastily organized, but I'm glad we can make this happen. And it's nice to actually see Sam in person. Same.

• 24:00Speaker 03

  Yeah. swyx, you think I am totally the PM for the live API? But we didn't get to talk about some of all of the other releases as well.

• 24:09Speaker 01

  We'll save that for your talk at World's Fair. And you guys are all speaking and we'll be podcasting as well.

• 24:15Speaker 03

  Sounds good, yeah.

• 24:16Speaker 01

  All right, that's it. Thank you so much.