Ep 48: Co-Founder/CEO of LiveKit Russ d'Sa on the Best ChatGPT Voice Use-Cases, New UX Paradigms in AI, and When Voice Makes Sense

For this episode of Unsupervised Learning we spoke with Russ d'Sa, co-founder of LiveKit, a company at the forefront of voice AI technology. Russ thinks of LiveKit as a “nervous system,” powering the sensory interfaces humans use to interact with AI – including the Advanced Voice feature in ChatGPT as well as applications like Character.ai, Spotify and many more. Russ talked about when voice makes sense as an interface, the exciting new UX paradigms on the horizon, the intersection of voice and robotics and Anthropic's Computer Use API.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Welcome to Unsupervised Learning. Today, we had Russ DeSau, the founder and CEO of LiveKit on the show. We talk about the future of voice AI, how LiveKit is powering applications like ChatGPT Voice. We talk about how the keyboard and mouse will be replaced by the mic and camera, and much, much more. Super fun episode. Thank you so much for joining us.

• 0:21Speaker 00

  Well, Russ, thanks for being on Unsupervised Learning.

• 0:23Speaker 01

  Thanks for having me, man.

• 0:24Speaker 00

  Maybe to kick it off, LiveKit powers some incredible applications, OpenAI, ChatGPT Voice, and many more. As someone that's been deep in this world for a while, how do you personally use ChatGPT Voice?

• 0:37Speaker 01

  A few different ways, actually. I'd say my favorite way is sometimes I'm driving up and down like 101, 880, those terrible highways in the Bay Area during rush hour. But I'll just pop an AirPod in, one AirPod specifically. Okay?

• 0:55Speaker 00

  Yeah, gotta stay safe.

• 0:56Speaker 01

  Yeah, gotta stay safe.

• 0:57Speaker 00

  Street legal.

• 0:58Speaker 01

  Right, right, right. So I'll pop an AirPod in and I'll just ask Advanced Voice. I'll say, hey, I want to learn about some frontier tech. Teach me something about a topic that I don't know, kind of in technology. So it'll be like, oh, you want to learn about quantum theory, quantum entanglement, quantum computers, CRISPR, even some basic things, too. Like, the other day, I was talking to Advanced Voice about just how does lightning work? And try to understand what's happening during a storm. And so the amazing part about this use case is you have this tutor that you can just ask any question to, right? Like, I know y'all have probably spent time in a college auditorium

• 1:43Speaker 01

  where the professor says, okay, any questions? And you just sit there and you're like, I got some, but I think everyone else already understands this, so I'm not going to embarrass myself. It's amazing how freeing it is to just ask whatever, quote, dumb question you have. And this model has the entire world's knowledge, but it just doesn't judge you.

• 2:04Speaker 00

  Jacob and I certainly have a lot of dumb questions.

• 2:06Jacob Effron?

  Yeah, I feel like we started this podcast as our own. We didn't need the the advance in these models.

• 2:10Speaker 00

  We also now know where Russ gets all of his um his fun facts from, so.

• 2:14Jacob Effron?

  Yeah, we were saying before the episode that you have just a wealth of fun facts. And so I guess that makes sense now, given how you spend this commute.

• 2:19Speaker 01

  Yeah, I gotta I gotta keep building up that fodder, right, for for these kinds of conversations. I'm sure we'll get into some of those.

• 2:25Speaker 00

  Um, maybe for our listeners, can you explain, you know, how specifically LiveKit works with ChatGPT Voice and kind of how that um that whole fit workflow kind of flows through?

• 2:35Speaker 01

  So that first piece, there's a LiveKit SDK that sits on the device.

• 2:41Speaker 00

  Mm hmm.

• 2:41Speaker 01

  Um, and what it's responsible for is being able to access the camera and the microphone of that device. When the user says something, it captures that's that speech and it sends it over um LiveKit's network. What we really have is like this edge network that uh it's servers all around the world and they all kind of communicate with one another and form like this like mesh network fabric on the backend, where a user says something on their phone, wherever they're located in the world, that LiveKit SDK on their device hits a network, transmits their audio through the network to what we call an agent on the backend.

• 3:20Speaker 01

  Our agent's framework as um kind of like an application server in the in the web application kind of um. As a web, yeah, the metaphor to a web application, the agent's framework is kind of like an application server.

• 3:34Speaker 00

  Mm hmm.

• 3:34Speaker 01

  So, you know, in this case, OpenAI builds uh an agent using our framework on the backend. That's sitting there as a receiver that is getting that audio data coming from the user's device over our network onto the server in that, you know, that agent receives that audio.

• 3:52Speaker 00

  Mm hmm.

• 3:52Speaker 01

  That audio then, depending on whether so with. The traditional voice mode before advanced voice, it was going through a step there of speech to text. So it's getting converted, that audio from the user, what they said is getting converted into text. And then that text is getting sent into the LLM. And as the tokens stream out of the LLM, they're getting converted back into speech. And then they're sent back over LiveKit's network to the client device where they're received by that SDK, LiveKit SDK on the client device and played out through the through the phone or the desktop computer or website. In the case of advanced voice, that speech goes from the client over our network to the agent.

• 4:37Speaker 01

  The agent then sends it directly into, through like the real time API, this web socket connection to the actual GPU machine. It gets sent from there directly as audio into GPT-4O. GPT-4O is trained, kind of has a joint embedding between like text and, you know, speech tokens, where the inference is done directly on the audio embeddings. And then speech is then generated by GPT-4O, comes back out through our network to the device and played out.

• 5:14Speaker 00

  Very cool. Yeah, you've previously described LiveKit as the nervous system. Maybe kind of in this context, can you say a little bit more about why LiveKit is the nervous system?

• 5:25Speaker 01

  We originally started the open source project around really connecting humans to other humans right around the world. And so the obvious use cases for that are video conferencing and live streaming types of applications. But then when we started working on voice mode, it was very different. It was this, this was connecting a human being with, with a machine, right? And the realization was that if a company like OpenAI or Anthropic, Gemini, et cetera, there's a variety of people working on foundational models. If they are building AGI, right? What is AGI?

• 6:06Speaker 01

  I was just listening, actually, to the Strategory podcast this morning, and they had a segment with Steve Jobs. They just had like a little speech thing that he said back in the 80s around how they were looking at the speed of all of these animals, and humans came way far down the list. But then the person who came up with this list also, and was tracking all these speeds of these animals, I can think the condor is number one, they also said like, okay, well, but if you take a human being and you put them on a bicycle, then they're the fastest. Yeah. And what it's illustrative of is that humans are tool builders, right?

• 6:48Speaker 01

  We build these tools that amplify our abilities, right? And so if you think about what AGI is, AGI is, it's tool builders building tool builders. Like, that's ultimately what we're doing, is we're building the tool builder, a synthetic human being. And so if you have a computer that is effectively like a human brain, it responds to you and acts like a human and all of that, the way you interface with that human-like computer is probably not with a keyboard and a mouse. I mean, humans, I O, the way that we're talking to each other right now, we're using our

• 7:31Speaker 01

  eyes and our ears and our mouths to communicate. If you have a computer that's just like a human, you're probably going to communicate with it the same way that you communicate with other human beings. And so that's, the eyes, the equivalent there is a camera, the ears are microphones, the mouth is a speaker. You know, those are all of our senses. And so if OpenAI and Anthropic and the foundational model companies, if what they're building is really the brain, what LiveKit is building is a nervous system. You have to be able to take information from those senses. Transport them to that brain and then take the result of that brain and transport it back out. And so that's kind of what we're, we see ourselves as building.

• 8:10Jacob Effron?

  What does the world look like when we, you know, I mean, maybe walk me through like a, an office looks like when we're, we're no longer keyboard and mousing, but just have these, you know, interacting with everything around us in the, in the same way we do humans.

• 8:22Speaker 01

  I'm not even really sure what like the office of the future is going to look like, whether that's just like a desk in my, in my living room or that I'm on the go. I'm like always in the office and I'm like interacting with these agents, um, you know, just via my phone or, or et cetera. I mean, I think the nature of work is going to change drastically.

• 8:39Jacob Effron?

  Maybe your desk will be a Jarvis Iron Man style.

• 8:43Speaker 01

  Yeah. I mean, that's an interface.

• 8:45Jacob Effron?

  Always Iron Man. I always bring back Iron Man and Taylor Swift to come up on the podcast.

• 8:48Speaker 01

  Okay. All right. Well, we, checkbox for, for Tony Stark.

• 8:52Jacob Effron?

  Stay tuned for Taylor Swift.

• 8:53Speaker 01

  Yeah. Yeah. Okay. I'm, I'm ready. I think that, uh, that's definitely one interface that I think, uh, is going to happen. Um, I think like creative tools are going to become more kind of voice, uh, based, more interactive, more multimodal in general. I actually send around this video to people of like there's someone on YouTube did like this, uh, compilation of all the different ways that, uh, that Tony Stark interacts with Jarvis across the movies. Um, and, uh, and I think that that's definitely going to happen for creative tools, right? Um, where they'll do a lot of like these transforms or operations, um, on the, the actual assets, uh, but that you'll be kind

• 9:35Speaker 01

  of the, uh, the orchestrator of it, right? Or the, um, the maestro that's kind of like, uh, you know, ultimately kind of shaping it, but not really doing all the mechanical work, uh, of the shaping itself. I think that's one way that it's going to change, uh, how we do work and what the office of the future will look like. But I think that, uh, one debate that's come up recently is kind of like co-pilots versus agents and.

• 10:03Speaker 01

  I honestly think that it's going to be a hybrid of the two. How do humans work with other humans? Like, we have these analogies today. And even in the workplace today, there are some coworkers that go and just completely own some part of your business and drive it forward. And like, there are tasks that they go and they do, and they're autonomous. And then in other contexts, sometimes I'm pairing with someone. Sometimes I'm building something with someone. Sometimes I have meetings to go through a design of something, right? And oftentimes, there are multiple stakeholders in that meeting. And so I think that it's going to shake out that way, too. I think that there, I think the main difference, you know, ultimately will be that

• 10:44Speaker 01

  you'll be doing a lot of this with AI and maybe not as much with humans.

• 10:50Jacob Effron?

  I mean, obviously, it seems like you're, you know, you're behind this kind of voice taking over most things. You know, I imagine there's some circumstances, whether it's privacy or, you know, people, sometimes people consume information or retain information better, reading it like, I guess, in this future, you articulate of kind of us all interacting via voice. Where are the areas where voice doesn't permeate, I guess, as you think about things going forward?

• 11:12Speaker 01

  Again, kind of drawing a, drawing a line towards kind of how do humans interact with each other? We still text each other. We still read text on the internet. We still type into boxes in places. I think that there's going to be places that we continue to use text, right? It's like text, voice, computer vision. Video generation is a bit farther down the road, and we'll see how that pans out, I'm not sure yet. But, but computer vision, for sure. Like, we use our eyes, we're visual creatures. That's going to be an important part of the, the future, for sure. But I think that text has its place. It will always be used in some fashion. To put, use a very concrete example.

• 11:57Speaker 01

  Imagine that I do have a voice interface to a restaurant, right? And I wanna order food, or I want my agent to or assistant to order food for me. There's one version of this where it understands me perfectly. It knows my preferences and I'm like, okay, yeah, like get me the regular thing that I order from mixed or whatever, the salad that I always order, right? And it'll go and it'll do that and all of that. It'll be perfect there. But imagine it's like a new restaurant, right? That I haven't tried before. Or a new type of cuisine that I haven't had. And the AI doesn't know exactly what I want. But it's still is helpful and it's part of that that flow. You don't really want it to like

• 12:42Speaker 01

  read the entire menu to you. That's just not a good UX, right? I think like the future is definitely it. And it's a kind of like a hybrid, right? Where voice makes sense in certain contexts. Like the obvious one is hands-free interfaces, right? Like where you're driving or you're cooking or et cetera. Those are the obvious ones where voice is just a natural modality.

• 13:05Speaker 00

  Are we gonna need that? Are we gonna be driving and cooking? Like, you know, I wonder.

• 13:08Speaker 01

  I was gonna say, to be honest, I'm like, okay, well, maybe we'll have a Tesla bot or a figure robot. It's like cooking for us, so maybe and maybe we won't be driving.

• 13:16Speaker 00

  Even in Waymo, it's fun to set the the music by by voice. Then you don't need to touch anything or I don't know.

• 13:22Speaker 01

  I mean, I think like hands-free, things that are like far away, like those are pretty obvious, right? Any of the use cases that you might have for like your Siri or Alexa type of device, those are other ones too. And so I think there are scenarios where you don't want voice or where it might be a hybrid, where it's voice plus like some text or plus like some generated on the fly UI that is delivered to you and you can just like tap a few buttons. I think what is unique, though, about how the interfaces will evolve is

• 13:57Speaker 01

  One thing, you know, as an engineer, like my whole career, there's been this like oscillation around the idea of a thin client, where you don't have to have this super powerful device, thousand dollar computer in your pocket, and it can like render all kinds of crazy UI. And you have thousands of apps with all this different UI. And then like my dad or my mom like, you know, might get confused by like this app versus this app. And it's you're kind of at the mercy of like the developer's design, you know, chops around the UX. And my previous company, we were doing like pixel streaming of mobile apps. And we were trying to like build this thin client dream a little bit.

• 14:42Speaker 01

  And what I kind of realized with ChatGPT is like, well, what is the kind of native human interface to each other when we're not next to each other? Like, we're talking, right? But when we're not talking, like, what is the interface that we use to communicate with each other? It's chat. Like, it's texting, it's Telegram, it's WhatsApp, it's Twitter, it's Slack. Those are all like chat interfaces. You have a text box and you have a stream of messages, right? And so you'll see people tweet sometimes like, oh, I'm so sick of like these chat interfaces with AI. I want different interfaces. And it's like, I understand that, but also like, what do you use many, many, many times a day with other human beings? You use a chat interface, right?

• 15:26Speaker 01

  Like, and so what if in some ways, like, what if the chat interface is the thin client dream? Where it's just one UI to every single type of application. And you have voice in there and you have on the fly generated UI. And you have text as well. And so like all of these modalities, but it's kind of in this very familiar interface. Where you interact with any of those applications.

• 15:52Jacob Effron?

  You mentioned these kind of, you know, potentially novel interfaces that are hybrid, like voice and text. Have you seen anything cool like that yet that you think does a good job of this?

• 16:01Speaker 01

  You know what? I haven't. I was, I was talking, I was talking to a friend about this, actually, where we were talking about where the interfaces are going to go. And I think what you see today is you kind of have like you have ChatGPT and you have something like Cursor. And you have advanced voice, right? In ChatGPT as well. And The way it's it's working today is really this, I I call it like you're interacting with applications based on a mode. So, okay, I'm in my car, or I'm sitting on the couch and like my device is far away from me, I'm gonna use voice, right? I'm in like a voice context,

• 16:44Speaker 01

  right? Where voice makes sense. I'm in a crowded space, so voice doesn't make sense, I'm gonna use text. Or, I'm sitting at my computer and I'm in VS Code and I'm gonna like use, you know, uh, I'm gonna use text, I'm typing already. I think there's a lot of like really interesting kind of UI explorations that are gonna happen over the next few years, that treat the modality more fluidly. Like when I think about when I pair program, when I was at Twitter, I pair programmed with someone, they had a very strong pairing culture. And so I paired with someone for like almost a year and a half. Like every day, we would write code together. And when

• 17:29Speaker 01

  you look at that kind of experience, you know, swap out the person I'm pairing with for an AI, when you look at that kind of experience, it's mixing modalities like on the fly, like we're both looking at the screen, so there's computer vision there, right? Where I'm typing and maybe I'm driving and then I'm also asking a question to the person who's, you know, my co-pilot, who's sitting there in the sidecar and they're like, oh, I think you made a mistake there. I'm like, oh, where? And then they'll take control of the keyboard and then they'll start to like correct the code. It's this very like mixed modality kind of interaction. And we just haven't seen that in applications yet. It's very much been these kind of two bifurcated experiences. And I think over time, we're gonna see a blending of those things, especially with

• 18:13Speaker 01

  computer vision too, as it comes into the mix. Oftentimes, people use their eyes and, you know, speech and hearing in concert. And so, yeah, it's really exciting.

• 18:25Jacob Effron?

  Obviously, we have these like much, you know, OpenAI just released this kind of speech to speech model. I thought there's a lot happening in the audio world. What are you seeing out there in terms of like some of the use cases for, you know, some of these speech models?

• 18:36Speaker 01

  Yeah, I think that what OpenAI is doing, and others as well, right? There's Gemini Live, there's a few, Character AI, Perplexity.

• 18:47Speaker 01

  They're building these voice interfaces to their systems, right? So you can interact with them using your voice. You can look up information, ask questions, use it as a tutor or a therapist in some cases. It's also useful for that. So these are kind of what I would bucket as emergent use cases of voice. There aren't a ton of companies right now that have just massive scale, especially on the consumer side around AI. And so that's why I kind of call them emergent. They're pushing the envelope. They're doing new stuff. I'm sure you all saw like Anthropic's computer use API that came out. Super cool stuff. And so these companies are like really pushing the envelope of like where this is all going to go.

• 19:30Speaker 01

  From an interaction perspective, from a capabilities perspective. What's also happening though, is that there are these low hanging fruit kind of places where there's existing dollars. They are at massive scale already for voice. And it's more of like a margin optimization game than a kind of like a user value, user experience kind of exploration. And that's in really like the telephony space. So or telecom, like any, any kind of use case where you have to like call a, call a number and you talk to either a human on the other end

• 20:15Speaker 01

  or an IVR phone tree type of system.

• 20:20Speaker 01

  AI is, is very rapidly going into that space. There's companies like Sierra and Parloa, et cetera, that are going into that, that space. And really disrupting it by integrating AI into that space and, you know, reducing costs over time.

• 20:38Speaker 01

  And I mean, the telephone is a voice native system that's been around for, I don't even know, 50 plus years. And, and so, you know, there's billions of calls that are happening around the world every single month. And so that's an area where I think there's like a immediate, high penetration like use case for, for voice and AI-based voice.

• 21:04Jacob Effron?

  Yeah.

• 21:05Speaker 01

  Is, yeah, any kind of, any kind of experience where you're picking up a phone and talking to someone.

• 21:10Jacob Effron?

  Is, is like latency a blocker there? Like, what are you kind of seeing around that? Obviously, I know there's different amounts of, of, you know, inference these models need to do and maybe more complex tasks that require reasoning. But I think, you know, it, we've seen kind of the chatbot customer support stuff explode. And I'm not, maybe we're on the precipice of it, but I'm not sure we've quite seen the voice stuff really take off yet.

• 21:29Speaker 01

  Yeah, it, I think there's a few reasons why we haven't seen it completely take off yet. So the chatbot use case has been around for a while, right?

• 21:39Jacob Effron?

  Yeah.

• 21:40Speaker 01

  It's definitely been around for a while and it's not as latency sensitive as the, the support, customer support to use like just one kind of massive industry that is telephone dominated today. In the customer support space, yeah, we, I mean, there's higher penetration for these less latency sensitive workflows. I don't think, though, that it's latency that is really the reason. I think it, it really comes down to there is already a system that works at scale where it's a sufficiently large risk to go in there and like swap out the engine and expect that

• 22:22Speaker 01

  you're going to get the same NPS or customer satisfaction out of this new system. The other, the other part about it specifically for customer support that is important is the AI models. They're not perfect. They don't get it right all the time yet. They still hallucinate. There are still kind of these evaluations that have to be done and bulletproofing. And so, you know, because that's the case, because you, they're not perfect, you often have to have human in the loop. So it's not like you can just completely shut off your, your human contact center. You have to have humans still that are on standby or ready to jump in

• 23:06Speaker 01

  and correct the agent or take over for something it can't do. And so I, you know, the latency, when we first started to explore building, you know, an AI that you could talk to, this is like at the start of 2023, when ChatGPT, the website had first just come out, no API or anything. And we were starting to explore that and build a demo there.

• 23:27Speaker 01

  That latency back then, someone asked me this recently, I think it was in conversational terms, something like four seconds, right? It was like crazy high. And now you have with GPT-4O and the real-time API, the latency is down to, I think, like when you use LiveKit plus that, it's like 320 milliseconds on average, right? So it's like a tenfold decrease. And human level speed in kind of conversational turnovers, that's around 300 milliseconds. So we're getting to this point now where it's fast enough, where any faster is not even super beneficial. In some cases, I built this, this voice assistant using Cerebras. And Cerebras's inference is just so crazy fast.

• 24:12Speaker 01

  I think it's like 100 milliseconds or something they can do, or a little over that they can do inference. And, and the model was like responding too quickly.

• 24:21Speaker 00

  Yeah.

• 24:21Speaker 01

  It's like people would say something and the model would just start to speak already. And it's like, instead of like you being able to interrupt the model, the model's like interrupting you. And people were like, hey, can you make, they would like wrote into us. And they said, hey, can you make it like respond slower? So I think latency wise, we're already kind of at the threshold where it's, it's good enough to replace these use cases or to be, you know, to be viable for this, this use case. But it's more about systems integration that I think is, is a bigger challenge.

• 24:56Speaker 00

  Yeah. And a lot of those integrations could. Be much slower on the backend, right? Like if you're a customer support system that needs to go access someone's information to figure out how to resolve a ticket and that could take much longer.

• 25:08Speaker 01

  Totally, yeah. That's a great point. Like something that you also have to do that I'm not sure how many people think about with these support-based use cases that happen over the phone is you have to go and like update systems like on the backend, right? You gotta like move the record through Salesforce or you know, mark something here. And a lot of these systems too are kind of bespoke, right? They build their own custom software or they use an obscure type of, you know, vendor for whatever they use to track tickets and stuff like that. And so who's going to be going and like updating those systems? I think the ideal is that the AI model can also do that, which you know, brings the

• 25:49Speaker 01

  computer use API that Anthropic put out into context a little bit, but we're still, we're not there yet, but we'll get there.

• 25:58Speaker 00

  Yeah. What'd you think of the computer use stuff? I think you guys have done some multimodal type demos.

• 26:03Speaker 01

  Yeah.

• 26:04Speaker 00

  And yeah, I was just curious to get your thoughts on that.

• 26:05Speaker 01

  Yeah, we're really excited about it. Maybe a couple of months ago, we were at an AGI house hackathon in Hillsborough and I think it was Andrew Ng was hosting like this AI agent, open source AI agent hackathon. And we put out this, this new kind of beta API that we haven't really even written about at all, which is a service where you can, we run like effectively virtualized or cloud or call it headless Chrome instances, right? So you run browser instances in the cloud and then the agent can actually hook into,

• 26:46Speaker 01

  like you build an agent with LiveKit's framework, it can hook into one of these browser instances and it can start to do work with it. It has a playwright interface, so it can like commandeer that browser and like, you know, load webpages, click on buttons, fill out forms, all of that stuff. And there are other folks that kind of build this stuff too, this specific piece of infrastructure. What I think is also unique about the version that we have is When the agent is stuck, it can actually stream the browser as video to a client application, to a human user. And this is kind of lineage to our, my previous company that was kind of doing pixel streaming and interactive like kind of applications, mobile apps by streaming them to your, to your phone and then streaming the inputs back up.

• 27:31Speaker 01

  You can stream this browser, the agent can stream it to you and say, hey, I'm stuck. I have a password field you gotta fill out or which of these items do you want? I'm not sure which one you want. And you can actually go and like click into that browser and you're like clicking on video pixels, but we're replaying all of those events back up in the cloud. And it's like you have an interactive browser instance, a shared session that's streamed to you, but that the agent is operating on and you can go and unblock it and like, you know, nudge it in a certain direction. And then it can go back off and continue doing its work. And so when I saw the computer use API, I was like, oh man, this is, this is kind of exactly in line with how we've been thinking. The way that I describe what we're doing with that kind of infrastructure

• 28:17Speaker 01

  is that previously LiveKit had given AI the ability to see, to hear and to speak.

• 28:27Speaker 01

  What we're doing with this feature that I just described is we're giving now AI the last sense that I think it makes sense for us to give is the ability to touch. Right? Because like, what do we do when we use our mouse and keyboard? That's a bit more abstracted away, but what do you literally do when you like take out your phone? You're touching applications. If you look at like the actual code, it's like touch events, right? Like you are touching the application and manipulating it. And now what we're doing with this tech is we're giving AI the ability to also touch, right? To manipulate these applications. It means something very different in the context of humanoid robots. That's like touching physical things, but these are more around kind of the capabilities that you're giving AI

• 29:11Speaker 01

  within like a consumer device is the ability to touch applications like you can. And so, yeah, we're not going to work on smell and taste, probably.

• 29:20Jacob Effron?

  I don't know what that would look like.

• 29:22Speaker 01

  I don't know what it would look like either, but, uh, but yeah, I'm excited about it.

• 29:26Jacob Effron?

  What do you like pay attention to in the voice world? I mean, I feel like, um, you know, we hear obviously there's like 11 labs and like some of these like text to speech models. There's, you know, folks like Cartesian doing like kind of entirely new architectures. Like what's on, what's on your radar these days?

• 29:40Speaker 01

  I think the, the area of focus for me, um, probably most recently has, it's largely been in telephony. Mainly because there's just an insane amount of workload, um, in that space. There are so many phone calls that are happening, um, that are kind of starting to get automated away. Just to use an example, like insurance eligibility lookups. This is something that like, I'm not even sure how many people know this happens, but you know, where your hospital is like calling your health insurance company to like make sure that, you know, you have coverage for a particular procedure that you need done.

• 30:21Speaker 01

  There are like just millions of these that happen and, uh, and folks are starting to like build AI-based, uh, you know, systems that go and like, not just do like the kind of the, uh, human calls to, but where the AI calls out to a human, right? And reaches out to a human, it's the other direction. And, uh, so these are, I think there's so much, uh, happening there that I'm start, I'm, I'm paying a lot of attention to that, kind of in the, in the near term. In the longer term, I think the thing that happened that I was really excited about, um, and I was just waiting for it to happen was kind of, uh, the, the OpenAI kind of, uh, GPT-4O kind of fully multimodal model.

• 31:04Speaker 01

  Where you have a model that is trained on speech, trained on text, and can take in any combination of those modalities and then output any combination of those modalities. Another interesting thing that I've been following specifically around like Cartesian, um, and, you know, these, uh, kind of different architectures from the, from the transformer specifically. I am interested in kind of how does voice?

• 31:34Speaker 01

  How does kind of the voice workflow play out as you get models that are capable of running on device? I don't exactly know how it's all going to shake out, but it's something that I've been kind of following a bit more casually, but very interested in.

• 31:51Speaker 00

  Just to that point, like, how do you think about what will run on device? Cartesian, obviously, they have some great models that run on device versus, you know, kind of bigger ones that may be more intelligent or whatever in the cloud. And like, how do you think about what may happen where and realizing that it's kind of early days right now?

• 32:09Speaker 01

  Yeah, it's early days. So I think that there's, I think it's really interesting how the split is gonna, is gonna play out. One area where I think the split is gonna be super important is in humanoid robotics. So, or in robotics in general, but, you know, everyone's talking about humanoids. So we'll just talk about.

• 32:30Speaker 00

  Thanks, Elon.

• 32:31Speaker 01

  Yeah, yeah, exactly. Appreciate it. You know, with humanoid robots, you, I mean, if you look at like the figure robot, for example, they mentioned how, I believe they do some like planning and reasoning, they do that kind of in the cloud. I think maybe they also use OpenAI, I'm not 100% sure, for, for that piece. But then they're actually on-device models that do a bunch of like the kind of reflex action and an application of that planning or reasoning, maybe like even just the, you know, the kinematics and the movement of the, of the robot. And so that's like one clear example of like, I, it does not really make sense to me to

• 33:14Speaker 01

  put like the movement or the application of, once a plan is developed, put that like in the cloud, or even put it at the edge, the edge here meaning like where the internet backbone is, not. Edge device.

• 33:30Speaker 00

  Not on device.

• 33:31Speaker 01

  Yeah. Um, I think you want that stuff running on device, right? I mean, if your robot is walking and then there's like a car coming and it's like, you go, wait, I gotta go to the cloud. I gotta step back onto the sidewalk, you know, it's, uh, that's just not gonna fly. Um, and so I think ultimately the way it shakes out is there will be some that's done on the device, there will be some that's done in the cloud. One thing I like to, I like to kind of draw an analogy to, again, kind of with humans, is there is no human analogy for a model that runs, like if you think of me as a device and you as a device and Jacob as a device, like, none of us have all of the world's information in our heads. And when we need to go look up information, we do inference in the cloud. Like, we go and we take out our phone and we like look up some stuff.

• 34:17Speaker 01

  Or when we need like our router rebooted, or if, you know, or if we need our router fixed, right? Um, and if rebooting it doesn't fix it, you gotta go call like, uh, you know, an AI contact center and it's, and talk it through what your problem is and that's cloud inference, right? Like, your local model is not gonna understand like the latest Linksys spec and how to, how to fix issues there. So I think that there's going to always be some inference that is happening in the cloud. And there's also gonna be some that's happening on the device as well. And that's just kind of the same analogy that happens in, with humans. The thought for me, and I don't know how it's gonna shake out, is do you want that cloud model to respond immediately?

• 35:01Speaker 01

  Or do you want your local model to say, you know what, I don't know yet? Let me go to the, let me go and ask my friend over here. Or do you want that friend to like always be listening and then respond? I mean, let's just start from like the ideal scenario if there were no resource constraints.

• 35:23Speaker 01

  What you'd probably do is you'd probably like parallel path both. You'd have a local model, you know, doing inference. If inference was free, let's say, and then you would, which is the ideal scenario. And then you'd have also like the cloud model doing inference. And the fastest responder effectively with a real answer would be one of the two. And so even in that case, and again, I don't know how it's going to shake out ultimately around resource constraints, but in that world, you're kind of doing both.

• 35:50Speaker 00

  Yeah.

• 35:50Speaker 01

  Right? And, and, and so then, and then there's all kinds of other stuff too. Like if your model makes a mistake, um, do you, you need to like have a record of like what it said, what was said to it, what it saw, if in a computer vision scenario, um, for legal reasons, for generating, you know, updated training data, for like data labeling, you know, for, uh, you know, erroneous examples, et cetera. So you're going to be sending that data to the cloud in general is how I think it's going to shake out. There are going to be some privacy, you know, sensitive use cases where I think people

• 36:33Speaker 01

  will want something that's happening purely locally on the device. And it's, yeah, it's locked to them and, and we'll see how that shakes out. But even like the new Apple intelligence stuff also sends to the cloud. It's just a very, very secure, super secure cloud.

• 36:49Speaker 00

  We usually end with a quick fire round. Uh, and so we will ask you a series of questions, kind of classic fire round. Um, and maybe to kick it off, what's overhyped and underhyped in AI today?

• 37:04Speaker 01

  Overhyped, transformers. Underhyped, or call it under researched, um, is spiking neural networks. I've been like really interested in that lately. It's this kind of a analog, uh, neural network design that is, yeah, apparently it's like more, it's modeled more closely after the human brain and how neurons actually interact with one another. There are some tricks to it. Uh, you like can't exactly back prop through it as far as I understand. But there are some techniques for how to train them, but it's harder to train them than transformers.

• 37:50Speaker 01

  But I'm I'm super excited about them in the context of that they're kind of perfect for for audio and video signals or data and processing it. But remains to be seen whether they can be trained properly, but I'd call that underhyped.

• 38:08Jacob Effron?

  What's one thing you've changed your mind on in AI in the last year?

• 38:11Speaker 01

  What have you changed your mind on?

• 38:13Jacob Effron?

  When we started, I think we were we thought there'd be applications that had like genuine moats and we were just like, oh, it's like, yeah, this is six months after, you know, GPT three or six months after GPT four. And so of course, we haven't seen any moats yet, but like in a year or a year and a half, like there's going to be some real ones. And I think increasingly, it's like the underlying models change so much that I think really the only thing you can build is just like be super embedded with your customers, like understand what they need and build super fast. Like it just it changes way too fast to have anything more than that. And so I think we're we were so focused on like unique data assets and like, you know, like ways to and I still think some of that stuff is real, but I think I've pivoted a long a long way away from that to just like really fast teams, really deeply embedded

• 38:58Jacob Effron?

  in their customers, which maybe sounds trite and obvious at the time, but I feel like was when we started, well, a year ago, we were very much not thinking that way.

• 39:05Speaker 00

  Got it. Well, I think the moats of the application layer will be very similar to traditional SaaS, right? Like there'll be something that's hard to build and hard to get into the hands of a bunch of users.

• 39:17Jacob Effron?

  Totally. But it's like when you talk about a consumer company or something or an infrastructure company, it's you have this like great sounding moat that's like sounds so smart on a piece of paper. And when you talk about a SaaS company, what their moat is like it never, you know, it's like a million little things they do super well.

• 39:28Speaker 01

  Totally. Yeah. I asked you that question to try to like give me ideas as well or like jog my thoughts around like and I and it ended up doing that. So thank you. I would say that the thing I changed my mind on in the past year on AI is the speed of penetration of AI. So I think that a year ago, Even just as an example, like I was thinking that in a year I would be shocked if like people were not using ChatGPT. There's a lot of people that aren't using ChatGPT. I mean, I know it has a lot of users, right? Two hundred million weekly actives, or that's the reported number, two hundred plus. But I'll meet people on a regular basis that have heard of ChatGPT,

• 40:12Speaker 01

  right? Because it's in the news everywhere, but they don't use it. And they don't really use any form of AI, or at least kind of like the contemporary forms of AI in their, in their work. So it's a, that's a thing that I think is, was surprising for me is that I thought more people, it would have penetrated more rapidly and deeper in, in the last year.

• 40:37Speaker 00

  You have high expectations. I think it's the fastest growing consumer product of all time, but not.

• 40:42Speaker 01

  Totally, yeah, totally.

• 40:44Speaker 00

  Which AI startup are you most excited about right now besides LiveKit?

• 40:48Speaker 01

  It's Tesla. The reason why is I just like, I borrowed my co-founder's Tesla. So we went to, we went to the XAI Grok Hackathon, and I built this voice interface that could control your Tesla. So it could like open its doors and honk the horn and all of this stuff. And most people didn't even know that Tesla had an API, an official API that you can interact with the car. So we did this thing and then I took his Tesla and I like had it drive itself at 10:00 PM at night on like a Saturday on the highway from San Francisco all the way

• 41:29Speaker 01

  down to Menlo Park. And I gotta say, I mean, like it was magical the first time I used ChatGPT. But I don't think that like Tesla self-driving gets enough like just holy smokes. Like this thing, it drove me all the way to my co-founder's house.

• 41:52Speaker 01

  It's like on small streets, like navigating around construction and going on the inside lane and like driving super close to the the wall on the 101. It's it was wild and I was just, this is a marvel of technology. And and I think you take that like with the car and then Tesla bot. I mean, it's just, it's it's like sci fi dreams right there. So I'm I'm just, yeah, I'd say I'm most excited about them.

• 42:17Jacob Effron?

  I feel like self driving cars like the most visceral like experience with tech you could have. It's the first time anyone does it.

• 42:23Speaker 01

  Oh, yeah, that was my first time. It was amazing.

• 42:25Jacob Effron?

  Um, I love when people are from in from out of town from San Francisco, like showing them that like putting them away more. So now Waymo has expanded other places, but it's really, uh, it's really powerful. Good party trick. Totally.

• 42:35Speaker 01

  Totally.

• 42:37Jacob Effron?

  Obviously, you're you're, you know, in the picks and shovels game right now. But if we were to, you know, to say, hey, tomorrow, you got to go start an AI application, what would you go build?

• 42:49Speaker 01

  This question round. Um, I'm gonna, I'm gonna say, I think there's a bunch of different things that I might go try and build. But an interesting one is I would go build a video game. I'd go build a video game that like has a really, I don't know, novel way of like interacting with NPCs. You know, there are these games like Elden Ring and stuff like that that have like these expansive worlds and all these different plotlines. And and I think it's video games of the future are going to be just amazing in the way that you'll have these open worlds, but then they'll also be filled with like

• 43:33Speaker 01

  these dynamic, you know, characters that are lifelike and the stories can change. It's like kind of like choose your own adventure, but you know, with an infinite set of possibilities or permutations. I think that would be, that's gonna be incredible. And I'd love to work on something like that.

• 43:51Jacob Effron?

  And you can talk to the characters.

• 43:52Speaker 01

  Of course. Yeah. I mean, that's almost by default. I think that didn't talk about that much. But I think like voice AI applied to video games is gonna be, it's this thing that maybe that's under hyped, actually. Okay, I'll change my answer for under hyped. I think video games are just gonna be incredible in the future, where you are interacting, using your natural human inputs with with these characters and.

• 44:17Speaker 01

  In different types of worlds. Yeah.

• 44:19Speaker 00

  Maybe to give you the last word, where can folks go to learn more about LiveKit and the exciting work that you're doing? What would you kind of point them towards?

• 44:28Speaker 01

  Probably like the number one place is to our GitHub. Most, the vast majority of everything that we do is open source. And so we have a variety of different repos and tools and projects there. GitHub.com/livekit. But of course, our website as well, livekit.io and on X. X.com. I always want to call it Twitter because I worked there. I did not have Elon buying the company I worked for and then changing it to X on my bingo card. But yeah, X.com/livekit.

• 45:03Speaker 00

  Awesome. Well, thank you so much for joining us. This is super fun.

• 45:06Speaker 01

  Thanks for having me on, guys.