Pod Insights Summation with Auren Hoffman LiveKit CEO Russ d'Sa - voice AI and the future of human-machine interaction About this episode Russ d'Sa is the founder and CEO of LiveKit, the infrastructure company powering real-time voice for OpenAI's ChatGPT, Character.ai, and numerous AI applications. In this episode of World of DaaS, Russ and Auren discuss: The evolution of voice AITurn detection and conversational dynamicsReal-time infrastructure for AICopilots vs autopilots for the future Looking for more tech, data and venture capital intel? Head to worldofdaas.com for our podcast, newsletter and events, and follow us on X @...
Transcript 0:02 Speaker 00 Hello, fellow data nerds. Welcome to World of DAS. I'm your host, Orin Hoffman, CEO of Incubate and GP of Flex Capital. Discover more episodes, get weekly data as a service news, original content, articles on data and more at worldofdaas.com. That's world of das, D-A-A-S, dot com. Hello, fellow data nerds. My guest today is Russ Sasa. Russ is the CEO of LiveKit, an open source platform powering real-time audio for applications like OpenAI's ChatGPT Voice Mode and Character AI. He was previously an engineer at Twitter and 23andMe. Russ, welcome to World of DAS. Thanks so much, Orin. It's great to be here. I've got my character hat on here. Oh, nice.
0:45 Speaker 00 It's murdered out in black, a little hard to see, but it is there. Yeah, it looks good, actually. I think I want one of those. I'm jealous. Privilege of working with them, yeah. Get that swag. Why is voice in for us so hard? And there's so many companies working on it or in and around it, but why is it just so hard? At a high level, it's hard because it's just a departure from the way that the typical web application works. It's just very different. So when we started to work with OpenAI, it was partially a surprise. I built a demo at the start of 2023 that you could use WebRTC, LiveKit's infrastructure that we built. It's built on top of that protocol. I built a demo
1:27 Speaker 00 where instead of texting with ChatGPT, you could talk to ChatGPT. And it was kind of strung together, and it was a little glue and tape, but it worked pretty well, and the latency was pretty low for the standards of that time, TTS and STT. They were still pretty high latency. What's STT mean? Speech to text is STT, and text to speech is TTS. And things have changed quite a bit since the start of 2023, but the vast majority of voice interfaces since then that have been built where you can interact with an AI model, the original Voice Mode that OpenAI released, they They use what we call a cascade or a component based model.
2:09 Speaker 00 What that means is that the user says something, the audio streams to the backend where it is converted to text. Is it converting while a person is talking? So if someone talks for 30 seconds, is the first five seconds converting while they're going? Or how does that work? That's correct. Yeah. Even like the first 200 milliseconds, 300 milliseconds is being converted on the fly. Oh, wow. And it's using some of the same dynamics where it can predict words to have a sense of what word it might be based on that, based on what you said in the past. So these are neural nets. They're just smaller neural nets than an LLM is. And they're kind of very tuned for the specific task.
2:49 Speaker 00 So for speech to text, it's tuned for converting audio into some kind of text based transcription that accurately captures what the person is saying. And so it's doing that on the fly. And that's why sometimes if you talk to older systems like Google, I remember Google now has existed for a while on Android phones. And when you would talk to it in a voice mode, you would see the translation or the transcription of what you're saying changing in real time. And then eventually it kind of gets finalized where the model has a confidence level based on all of the things that you have said in the last second or two, the most likely sentence that you uttered.
3:32 Speaker 00 But at every stop, 300 milliseconds in, 800 milliseconds in, it's kind of changing sometimes or updating words within that sentence because it's not quite sure with a certain level of accuracy as to exactly what you said. And it's looking at the surrounding words all the time. In any case, so that audio is streaming in, it's getting converted into text. That text is flowing into an LLM. The LLM is spitting out tokens in a response. And as those tokens are streaming out, it's going to another neural network that is converting those tokens from text back into speech and then streaming that speech out to the user. And so this stringing together of the user speaking into a
4:14 Speaker 00 Then into an LLM, then into a text to speech, and then back out to the user's device where it's played out. That's called the cascaded or component model. I think at a high level, the reason why it's challenging even just to do this and perform well for one user is there's a lot of variables at play. One is where is a user located? Where are your models located in the world? Do you have to cross a network or a long path over a network to get the user's audio to the agent on the backend that is going to process that audio? And then you have to go through one neural network to convert it into text and then another neural network to process that text via the LLM and then another neural network to convert it to speech. And you got to do all of that with as low latency as possible.
4:59 Speaker 00 And usually you're using different services. So you might be using cloud service X for the speech to text part and then another LLM. And then you have many concurrent conversations you're having and it's all real time too. So it's not like a web application where you make a request, you wait for a while and then you get a response back. You have to have a constant connection that is going, constantly be streaming the audio, figuring out when the user is done speaking. There's a lot of conversational dynamics at play here that make it quite a departure from the traditional web application paradigm. I found that when I'm talking and then I a little bit ramble or um, um, um, it messes it up somehow. And it's like, oh, sorry, I didn't understand.
5:42 Speaker 00 I have to like rerepeat everything, which is unlike a real conversation. If I don't really know what I'm talking about in a real conversation with the human, they'll like wait for me to finish. They'll be polite or they might jump in and help me a little bit. Oh, I'm going to guide me or if I'm searching for a word, they might like jump in with something. But the LLM doesn't really know what to do. If I haven't really thought out what I'm going to say, it might mess up. Is that a Common thing that I get afflicted with. It's very common, so you're not alone in that. I think a lot of people experience that. What I think is one of the hardest problems in voice AI, and it's not the only thing holding it back, but I think it's one of the key things that are holding back massive or mainstream adoption
6:26 Speaker 00 of voice-based agents for interacting with services and applications like we will in the future. And that is this concept known as turn detection. What you described there, that falls under the umbrella of turn detection. Turn detection is knowing when a person is done speaking or done uttering their thoughts, but also knowing when is an appropriate time for you to interrupt, if at all. It might be that the right approach for that particular conversation you're having with someone and the thought that they're sharing is to just wait until they're finished speaking. Or like I'm doing right now, ask a clarifying question or something. Yes, and you just interrupt there with an or and it felt like the appropriate time to interrupt.
7:08 Speaker 00 And I saw you interrupt and I kind of held back and sat back and waited for you to say what you were saying. I could see how this is so hard. Oh, my gosh. I'm now appreciating how hard this is. It's extremely hard to do and humans do it automatically. I think in the future, the foundational models will as well. Any problem really out there where humans do it well, but it's hard for you to describe exactly how you're doing it is a great candidate for machine learning and neural networks. But it requires a change in the architecture of how LLMs are set up today. So to give you an example, there's a couple of companies out there. So QTi is a small company in France, really strong research team there,
7:50 Speaker 00 who built this model called Moshi. It's a small seven billion parameter model. But one unique part architecturally of this model is that they have this dual channel setup where audio is streaming in. I'm not an expert at this either, so I'm not going to explain the full thing. But the model basically has architecturally a wiring or a connection, a connective tissue or a joint attention mechanism between the input channel and the output channel.
8:18 Speaker 00 Fact is, audio is streaming in, the model can also optionally stream audio out. And so it can decide based on that input that is streaming in, whether it should interrupt or whether it should wait, kind of in a similar way that a human does. They're kind of monitoring the inputs constantly while also thinking about what they might say at the same time. And then choosing whether dynamically, whether to insert themselves into that portion of the conversation or to wait for another moment. If you want to see a similar approach in practice, you can try the full duplex mode in Meta AI's new app. And that is this kind of dual channel thing as well.
9:00 Speaker 00 And when you go and you try it, you'll see that the actual AI or agent in Meta's app can make these dynamic decisions about whether to interrupt or not. And the latency is super low because it's not even converting to text as an intermediary for that particular application. It's processing the audio directly. That's the way it's going to go in the future. And it's going to be kind of widespread done this way is my guess. But we haven't gotten to the point yet where we have enough audio-based conversational training data to build a large model that can do this. Why not? I mean, you have movies, you have podcasts, you have all these different things that are out there.
9:44 Speaker 00 I'll be talking like an Aaron Sorkin movie or something. I think to do it well, you have to have a wide diversity of human to human interactions, both in group settings and one-on-one types of interactions. Because just to use an example here of what you mentioned, if you take like a podcast, even that we're doing right now, there is a different kind of social contract. Yeah, it's like an interview, which is weird. This is not a normal interaction. If we were just going to dinner, I wouldn't just be like interviewing you. We would be having more of a conversation. Yeah, and if you were to look at the actual turn-taking pattern in a conversation like we're having right now, it's different than two friends sitting down for coffee and just on a Friday,
10:28 Speaker 00 talking about what they're going to do over the weekend. There's just a different back and forth and a different cadence to how they're talking. And you really need like a diverse representation in your training data to be able to do this. At a human level precision. I use these voice agents quite a bit. And what I find is that it's pretty good at one on one, which is way better than, let's say, if I was calling for customer service. I don't want to like use the phone tree. If someone just answers immediately, hi, I'm Mary, the AI. Can I help you? That's great. And I've had a few scenarios where that's just been an incredible service. Where it's a little weird is when there's more than one human on the call. Then it really doesn't know what's going on.
11:14 Speaker 00 And sometimes even like my son and I will be like talking to, let's say, chat GPT. He's interested in football. The three of us are having a conversation, two humans, one AI. And it doesn't know when to jump in. And it doesn't even know its place. Should it be in listen mode? Is it him and I having a conversation every once in a while? Can you like correct us with a new stat or throw something in or? It's super choppy. And so that adds another element of complexity to this problem, which is speaker diarization. Speaker what? Diarization. And that is the problem or the challenge, maybe the objective, of figuring out who is actually talking in a group setting. Is it your son? Is it you? Is it your friend who's also there? They may have just walked in the room and left.
12:00 Speaker 00 Yeah. And then also across an entire conversational context, what's been said so far in the last 10 minutes? Who has said what? And labeling the correct speakers and carrying through context. Your son might have said something and the AI might want to respond to that particular point that was said earlier, not something that you just said. We haven't yet seen this now because of course, one on one is a much easier problem to solve. But there is going to be this future where you have AIs that are attending Zoom meetings for business calls or things like that. Or another clear example of this group dynamic that gets even more complicated is when you want to put AI NPCs in video games. And in video games, you might have multiple human players and one AI. You might have multiple NPCs and one human.
12:44 Speaker 00 You might have multiple NPCs and multiple humans. How do you work out all of those conversational quirks and dynamics such that it? Feels natural to interact with these characters in the video game in the same way that you might interact with a human. It is a really hard set of problems and we're kind of just starting to get the turn detection piece correct for one-on-one conversations, but then yeah, we have to solve speaker diarization and then we have to do turn detection in group settings. And so it is quite challenging. There are some companies out there that have the data sets to potentially solve this more easily than others. Zoom and Discord have just a ton of
13:26 Speaker 00 multi-user conversational data between many humans at once that could be used for training these models, but of course there's security and privacy considerations there at play. But yeah, it's a hard problem to solve in terms of like the overall conversational dynamics that humans have. Where are we going in the short term? Like a year from now, what should we expect? I would say that a year from now, you can expect that turn detection for one-on-one is solved. I think we're definitely going to get to that point within the next 12 to 18 months. I lean more towards 12 than 18. We're starting to see a lot of progress already. Like if you look at Meta's new model,
14:06 Speaker 00 it's really impressive in how conversational it is, how it can understand when to interrupt. I think another piece that comes maybe right after doing turn detection for one-on-one conversations without any visual input is adding visual input to the mix. So you'll notice that for turn detection, when you're talking on the phone with someone, humans still kind of mess up. Sometimes they talk over each other and Yeah, you go, you go, you go. Oh, yeah. Exactly. So latency is an aspect of that and that's going to be embedded just in having conversations over computer networks versus having conversations in real life where the latency is almost instantaneous. So that is always going to be a somewhat of a persistent issue.
14:49 Speaker 00 Life, it can help kind of tune that down on the network propagation side or transmission side, but when you add A video input. Or you can see like right now, when people aren't watching the video, if they are, they can see me like leaning in a bit. So you can see I'm probably gonna talk. Exactly. There's like nonverbal cues that humans automatically pick up on that help inform whether it's time for you to talk, or whether it's an appropriate time for you to interject, et cetera. And so absent that visual or nonverbal cues. And also, like, you might have a sense that the other person's a little bored of what you're saying, and so you might just stop talking. A hundred percent. And then I think another element here that I didn't talk too much about, but I think will kind of get encoded into these models
15:33 Speaker 00 with enough training data is that human conversations are self-healing. So if you start to talk over me, but then you see I continue to like speak, and I almost like reject your interjection, you'll kind of back down. Especially in a group setting. I want to jump in with something humorous, but then you can't always find the right time to jump in when you're talking with four or five people or something. Exactly. And then I might be saying something and I might realize, oh, oops, I wasn't supposed to say something here. And then you'll kind of taper off and then like wait and then finally share your point or continue. And these changes, by the way, are happening. We didn't talk really about latency, but these changes are happening within 200 milliseconds.
16:14 Speaker 00 And so that's really the threshold that, yes, some turn-taking can be longer, but it can get as low as 200 milliseconds. Like, a human can react to these things almost immediately. And so we got to get the overall end-to-end down to that level to make this feel truly at the fidelity of a human-to-human conversation. Interesting. So we're gonna have the turn-taking is gonna be solved essentially in the next 12 months? I think it will be solved to the degree that I don't think that the latency is gonna get down to 200 milliseconds in the. Next 12 months, the full end to end. What is it now when I call like a chat GPT?
16:54 Speaker 00 So for GPT four O, it can get as low as 300 to 320 milliseconds for the end to end turn latency. However, I would say it probably settles more around 600 or 700. And that's why it feels a little bit unnatural. That's why it feels unnatural. I think there's another component to it that is kind of hidden underneath, which is how long does the inference take? There's a transmission or propagation latency, which, you know, is going over our network in the chat GPT case. And that's pretty consistent. The part where you see the variability comes from the speed of inference. And so that can vary widely. If you're using a llama hosted on Cerebrus, for example,
17:39 Speaker 00 you can get that latency really low. I mean, you can get it down to like 150 milliseconds. The trade off there is how big is the model? How smart is that model? And what's the quality of the response? One of the problems with chat GPT is it just is often overwhelmed. So even if you pay for like the top, like I do, often you have to reload it or it just doesn't work or something like that. Like it's super buggy on that side of things. The inference still needs a lot of work. The variance between like the P 50 and the P 95 is just so wide. That's what contributes to that average moving up for its response speed.
18:19 Speaker 00 That's one part. So I think turn detection in the sense that today, how it's often annoying where you talk to chat GPT. If you just wait even a moment, let's say you use chat GPT to practice being an interviewer with you for a podcast and you have the questions ahead of time. You're going through and trying to practice how you're going to respond to things. And then you pause for a moment to like, collect your thoughts or to formulate how you're going to word something. And then all of a sudden chat GPT starts to talk to you again. I think that those low hanging fruit of turn detection is going to be solved. I think it will start to feel quite realistic for those one on one cases where you're talking to a customer support agent or something like that, doing interview practice. I think it will get solved for those. I don't think we're going to have
19:03 Speaker 00 all of the speaker diarization and conversational dynamics between humans and 200 millisecond back and forth or repartee between two humans. I don't think that's going to be there in 12 months, but I think. Within eighteen to twenty four, it definitely could be. Now, beyond voice, I assume you're using a lot of AI tools for a lot of, whether it's personal things or for business. What are some of the things that you're using it for that maybe the average person isn't? Or maybe the average listener of this, that might be technical, but something you're doing for fun or for personal use or for business use. My voice example here is really around education. So the way that I learn about new topics now is using ChatGPT advanced voice mode.
19:45 Speaker 00 And I set it up while I'm on a commute or on a drive somewhere, I pop in an AirPod. And I just have a conversation. I can ask this AI model literally anything about a topic that I want to learn about. And you think about ahead of time, like, okay, I'm going for a drive. I kind of think about this thing ahead of time, what I want to ask, and I have a sense of the types of questions I want to ask. Sometimes. Cause I find when I do it, that's how it works. When I find a little bit meandering, it's harder. So if you're meandering, it's definitely harder because of the turn detection. I think if the turn detection gets better at understanding how to respond to you. Like we trained a turn detection model of our own internally, and it works quite a bit better than the existing.
20:27 Speaker 00 But it's not a final solution because it doesn't take into account how you're saying things, just the content of what you're saying. In addition to the audio signal, and it's not perfect, but it's definitely a step improvement over what is existing in the ChatGPT app. Once it gets better, I think that you'll be able to have more of a natural meandering conversation where the topic you discuss with ChatGPT can be a bit more spontaneous. But I think for me, it's a mix where sometimes I know what I want to talk about going in. How does lightning work? What I'll do is I'll actually ask it, hey, what kind of frontier topic in science or technology can we talk about? Give me some suggestions. And then it'll be like, oh, well, are you up to speed on the latest in quantum computing?
21:11 Speaker 00 And then I'll be like, actually, I'm not. I've never understood qubits and how they work. And we'll go deep on qubits. It's really about like in any good interview, coming up with the right questions. So hard. We've been doing a few meetups in SF recently, and it's the first time where I've been the person on the other side. Who's like really doing the interviewing. I have so much respect for you and other folks that are kind of in the interviewer seat because I realize asking the right questions, being the DJ in a sense of this session is a lot harder than just. Answering questions, it's crazy. I haven't had a good, really long conversation with chat GPT or something like that, but it is super helpful when I've got a specific thing I don't understand.
21:53 Speaker 00 I don't understand this thing about options, trading or something, and it could start explaining it to me. And then I can also level set it pretty quickly. I understand that piece of it, but not this piece, or you're getting a little bit too high level. Can you get a little bit down? Get it more as a 12th grade level for me or something. And then it can level set pretty quickly and get me to the answer. But where I need to get much better in is getting it to help me ask better questions somehow. Like, hey, what questions would you ask? And maybe it could come up with some really good ones or something. I see, yeah, that could be definitely used for research and things like that. And I'll sometimes use it to, in that way, not through voice, I'll use it through text for that. If I'm going to add some questions.
22:35 Speaker 00 This last meetup that we just did, we had Justin Uberti, who's the head of real time at OpenAI, and also an advisor to LiveKit, but also the creator of the WebRTC protocol, which LiveKit is built on top of. And I wanted to understand, how can I like go deep on some of these questions based on his career? And what he's done and places he's worked and companies he's started? And tying it into what he's working on now at OpenAI? And I enabled deep research and I said, hey, put together like a talk track for me. It's going to be about a half an hour talk. What kinds of questions should I dive into or should I consider? And it was pretty good about that, but I do agree with you that one thing that is tricky is having a few back and forth and refinement process on a conversation like that.
23:20 Speaker 00 It's a little bit more one shot today. Especially over voice. Especially over voice. It needs to be a little bit more collaborative. And you know what's kind of getting in that direction is these coding agents. Also, a common AI use case for me is I use Cursor to build stuff and sometimes Vibe Code, sometimes Copilot on code. But I use it and the Cursor agent experience is a little bit better at having this multi-step, going deeper and deeper on a code base and starting at a high level and then drilling into certain parts. It's a bit better at that, but it also still struggles in that you have to start a completely new chat at a certain point because it rabbit holes on one thing. And it's hard to like move it out of some kind of local maxima or minima.
24:05 Speaker 00 I don't know, depending on which direction you want to think about it. From, I think it is still a weakness. When I watch people who are really good at using a cursor or wind surf in like an ID standpoint, or maybe using CSS better with like a bold dot new or lovable or something. When I watch the people really good at that, they are iterating with hundreds, sometimes thousands of prompts where they're asking it to change. Hey, can you please move this? Or can I, can you move this? Or can you change this to a darker blue? Or this didn't work out, or now I don't understand. It's almost this very weird conversation over text that they're having with this thing.
24:43 Speaker 00 And it's very strange, very small number of people are going to be good at this prompt back and forth. I think that that's a bug, not a feature. If you remember the early Dolly beta versions, if you were really good at prompt engineering, understanding, oh, I gotta put like Nikon something in here and some random photo, Ansel Adams, this, these random words, and then all of a sudden it spits out something beautiful. That's maybe kind of how I conceptualize or relate the current coding agent incantations for prompts that you have to come up with. The whole text based thing, I should be able to like circle your face and it's, hey, can I make this nose a little bit differently?
25:28 Speaker 00 Or change this thing. And it shouldn't like redo the whole thing. And ideally, like if I want just the eye color to pop, it should just make the eye color pop. A hundred percent. And that's where we're going to go. I mean, it is going to converge on Jarvis from Iron Man, but we're not there yet. This is the early innings for that stuff, too. Where else do you think these assistants are going to evolve over the next few years? I think that there's going to be two paths that you're going to see emerge, already are emerging to a degree. But again, it's still early days for either one. I think on one side, you're going to have the copilot and the copilot is going to be there with you to help with creative work.
26:09 Speaker 00 So I'm envisioning something, I'm working on an image or a painting or something like that, or a digital painting, I guess. Maybe we'll have humanoids that also can do physical painting with brushes, but you're working on some piece of art or it might be a song. It might be a movie, et cetera, and you are really the director of this. And the co-pilot is helping you with some of the mundane tasks or the mechanical tasks that are kind of embedded or part of that creative work that you're doing. And so it's your partner in crime through that kind of experience or creative process. And we're going to see one whole path of applications that are infused with AI that is your co-pilot,
26:51 Speaker 00 not so dissimilar from like the early versions you see now with a lovable or a cursor or a windsurf. And then on the other side, and this one's a bit earlier than these co-pilot examples, is the autopilot examples. And I think the autopilot version of this is where it's less oriented around creative work and more oriented around mundane or rote or automatable tasks that you really don't even want to do anyway. I think human beings love to be creative. I think that that's embedded within us as a species. I think there's other parts of work and labor that we'd all rather not do, to be honest. Even like copying, go to this thing, copy the numbers from here, put it in a spreadsheet,
27:35 Speaker 00 send this report to this person or something like that. There's not always an API for all those things. Does anybody actually want to do those kinds of things? And so I think in the autopilot use case, there's going to be a bunch of applications or agents, I think is like the new word that we're using to replace the word application. These agents are going to go and you can effectively tell them to do stuff for you. You can tell them what task you want them to go and tackle, and they'll go off and they'll do it. And they might ping you back and ask you for clarifications, or they might end up being blocked on something and they need your advice or guidance on how to proceed. Or they might need you to type in a password or unlock something for them or give them privileges over a resource that they don't have access to. And that's going to be the other.
28:20 Speaker 00 Way that you interact with this technology. It's really hard to figure that out, though. A very small example, my doctor, I got my blood test. Someone in his office takes each thing from the blood test and puts it in some sort of Google Sheet to track it over time and track these metrics over time. Someone has to go in and literally go do it and have to know. And sometimes the blood test changes what it is. You have to know what's what and what is. And then it's got to be HIPAA compliant, so you can't send it to OpenAI. It would have to be a local model that would live on your local things and stuff. It sounds like it should be easy, but he's tried eight different services. None of them work yet and he's worried about getting it wrong. He doesn't want to like scare the client if it's not right.
29:05 Speaker 00 There's a lot of challenges and on the security compliance side and red tape that have to be adjusted for or dealt with in some way. I don't want to say circumvented because that's not really a great word or doesn't connote the right kind of approach to these kinds of things. But they have to be taken into account. Some of the constraints within certain problem domains, like finance is one example. Healthcare is another great example. But I think once that is navigated over time, once we kind of figure out how to integrate AI into these workflows, to use the example with your doctor taking these blood tests and looking at them over time. What it unlocks, I think, is going to be very profound in terms of value.
29:47 Speaker 00 So very directly tied to the blood test example you used. I had a blood test or a series of blood tests for my kidneys because I was working out a lot, eating way too much protein. That's what the internet told me to do. And I had my blood tested and they said, oh yeah, you have like too much protein in your blood and your kidneys aren't filtering out. And we think your kidneys are like a 70 year old's kidneys. And a bunch of tests proved that I was fine and there was nothing wrong. Thankfully, thankfully, knock on wood. But what's interesting is that if you looked back at my blood test from when I was 25, 15 years or so years ago, I was already showing a naturally higher sign of protein in the blood. Before I was doing any of this crazy diet or working out.
30:31 Speaker 00 So if somehow they just realized that they could have circumvented all that. Exactly, or they would have been able to contextualize what I was showing at 25 with what I'm showing now at 40. But what they weren't doing, what the doctor wasn't doing was because the doctor's a human and they just can't scale to the number of patients they have. The only time it got on their radar that this number was creeping up or that it was already high or too high. Was when it went from being like green into the yellow or whatever color they associate zone. And they're like, oh, wait, this is too high. But it turns out that that entire range is supposed to span ages zero through 70 or 80. And they're not contextualizing it to someone who's 30 or someone who's 41 or whatever.
31:13 Speaker 00 When you can add AI or computers to this mix and they have the context and they're compliant and all of that. All of a sudden you can unlock more value for patients because you can scale software better than you can scale human beings. It's not like to say a doctor isn't valuable. It's just to say that they have limited time, whereas a computer doesn't eat, computer doesn't sleep, runs 24 hours and computes way faster than a human brain does. Now, on the founder front, one of the things you've been pretty vocal about is having founders not raise until they have product market fit. Unpack that a little bit for us. There are founders that have raised when they don't have product market fit and they've succeeded. So I don't think that this is like a hard and fast rule that you always must follow.
31:58 Speaker 00 I think from my own experience, I tend to think of raising money as a responsibility. There's pension funds that invest in VC funds and my partner is a special needs preschool teacher and she has her money in pension. So I think of raising money as a serious responsibility. I think of it as a job to bring a return on capital to the folks that are taking a bet on me. And so I think what's tricky for me anyway, and the way I think about raising capital. And the responsibility associated with it is that if you don't have product market fit. Then you don't really have an indication or a confidence that you can deliver a return. And so when. You raise money before you have that confidence.
32:41 Speaker 00 It puts a lot of stress on the founder, or at least it puts a lot of stress on me in my experience, where I suddenly feel like I am under the gun. And if I had the capability or capacity to be able to not raise, to experiment a bit, to find something that has at least some indication of product market fit, maybe not full-blown product market fit, but some indication or signal. Then I have the confidence level where I can say, okay, I'm going to start to dump gasoline on this fire or on this small spark. And I think from a stress perspective, you're just in a much better mental state as a founder. And so in the past, I've done companies where I've raised money,
33:26 Speaker 00 sometimes quite a bit of money, before I had really any idea what the company was going to do. I just wanted to start the company to be a founder, to say I was a founder of something. We've been there. LiveKit was the first time I didn't do that. And it's worked out a lot better for me, both in terms of the company's success or progress and in terms of my own mental state of running the company. What other advice would you give to founders? The main piece of advice that I would give to founders is keep in mind that we don't have a lot of data points. A lot of the advice that you get as a founder is coming from a good place. It's well-intentioned, but it's two things. One, it's folks that are trying to pattern match to what we've seen before.
34:08 Speaker 00 And we just don't have that many data points about what it takes to make a successful company. And then there's also a lot of other variables at play, like the market at large and the space that you're playing in and competitive landscape and your own team and how fast or quickly you can iterate and react to what's happening in the macro that's out of your control. There's a lot of variables and we try to come up with like some kind of mathematical equation that maps X to Y, X being the inputs or many X's, X1, X2, X3, et cetera, being the inputs. And then Y being the output as to what you should do next. And I just don't think that we have that much data to say with any precision about what you should exactly do. So I would take the advice that you get that is sort of... Sort of pattern matchy with a grain of salt.
34:51 Speaker 00 The other thing that I think is really important for a founder to bear in mind is that nobody understands your product, your space, what you're doing better than you do. When you go into a VC pitch and you feel super nervous because these folks are investing large amounts of money and they seem like they know a lot, they actually don't know as much as you do about what you're working on. They're there to learn and get up to speed. In fact, if they do, there's a problem. Exactly. If they do, there's a problem. I've also had founders come to me and say, oh, well, I'm thinking about bringing on this person as an advisor to help guide the product or serve as a product advisor, head of product. I'm like, if you need to do that, I think you're in trouble. Yeah, you have to be the PM. You have to be the PM.
35:34 Speaker 00 You are the number one visionary PM for what this product is going to be and where it's going. And you cannot farm that out or outsource it. And so think about advice in that way. Don't put too much stock in it. The corollary to this is think first principles. I know it gets said a lot, but nobody knows what the right choice is better than you will. Incorporate learnings, incorporate that information, but make a first principles decision. Reason about it, not based on what other people do. Reason about it based on the information you have and make the best decision, given what's put in front of you. And then kind of react to what you learn as a result. It's good to build the muscle of being comfortable with making mistakes, acknowledging that you
36:17 Speaker 00 did the best you could with the information you had, and then doing something differently now that you have more information. So I think that that's another piece of advice I'd give. On your X bio, you describe yourself as a late bloomer. What does that mean and why is that? There's an interesting juxtaposition in my career. I've been pretty early to stuff, early to some ideas. I was, you know, in the fifth batch of YC. I was the second front end. Seventy fifth employee at Twitter. I had joined twenty three and me. I interviewed there and got a job offer, but turned them down and do Y C when there were five people. And then I went back after my Y C company failed. And I went and I joined them as like the thirtieth or thirty fifth employee.
36:58 Speaker 00 I think I've been early to things that have been done by other people. So I think I've been good at looking and seeing what is out there and selecting into things that I think are going to become big movements or big parts of the future. But then when it comes to like, starting my own company, I think I've been pretty immature. It took me five tries to really learn what responsibilities and thinking thought process is required to be a good founder. To kind of learn some of the stuff that I just told you, like around first principles, thinking and not putting too much stock in what other people tell you or the direction other people tell you, you should go with your product.
37:38 Speaker 00 Thinking strategically about go to market. How do you prioritize different product initiatives properly? And which ones are going to be the things that in the near term, you have to weigh in on versus waiting for other parts that might pan out in the long term and pushing those off despite them being these shiny objects, knowing when to grind versus when to go for the green field opportunities. It took me many tries to learn that. And I don't think all of it was just from like doing startups. I think some of it was due to just getting older and learning more things adjacent and going through life a bit more. But that's why in my X bio, I talk about early, early, early, but then as a founder, I think a late bloomer for sure.
38:20 Speaker 00 You mentioned twenty three and me. Why wasn't that company more successful? It's so many good things going for it. It had the mind share and the market share. It was early. So many people I know were customers and maybe tens of millions of customers and everything. It has all these ways of learning from one another. It had a founder that has billions of dollars, personal net worth, all these things you think would have made it successful. Why was it not successful? There's a lot of different things that can contribute to a startup not working out. There's like a thousand different ways you can get killed as a founder or your company can get killed.
38:58 Speaker 00 But I think fundamentally, there is one thing that in my opinion, held 23andMe back from being successful. This is something that I even said to them, including Anne, she's wonderful, by the way. But back in the day, even after I'd left, and that was fundamentally, science moves slower than technology. 23andMe was really early in the game, doing this personalized genomics and personalized medicine as like a goal. And I think it's a wonderful goal. I think it's going to be a thing that happens and within the next 10 years, we're going to have a lot of the things that I think Anne and the team there envisioned in the future, even back when I was working there. And I think timing is really important.
39:42 Speaker 00 And what's tricky is that 23andMe came in, the first player, a lot of people you know have all done 23andMe and they've captured the market share. And we were Oprah's favorite product back in 2008. We were Times invention of the year in 2008 or 2009. And I remember just being inundated with these purchases of the spit kits and we were having to go and work overtime packing boxes. It was great energy and it was a great moment, but I think that when it comes down to it, you get your DNA sequenced and you take a look at what we can say about your genetics at that time. And it just wasn't that high impact, except for a few individuals. So like BRCA1 with breast cancer, APOE with Alzheimer's, LURK2 with Parkinson's.
40:26 Speaker 00 For a very small set of those markers, genetic markers, we can say something very impactful to your life. But for the vast majority of people, we can't really say anything. We don't really know how to combat those kind of things. And some of the mitigations are Brian Johnson-esque. It's like, you've got to do this thing for like 40 years and then maybe you'll ward off the Alzheimer's onset and maybe you won't. And it's one of those things that's really hard to commit to something like that. Without having like a degree of precision or confidence that it's really going to work out if you do make that investment. And so for 23andMe, the big challenge was we have
41:06 Speaker 00 the technology, we can sequence the DNA, we can do it affordably, but what we can say about what your DNA says and the kinds of like actionable advice we can give you is still very limited. We don't know enough. Maybe we need AI to like go and analyze a full genome. If we start marrying that with outcomes and marrying that with, okay, people who did these things got this or people who got this drug differently and stuff like that. The databases never got merged. You'd have to do it anonymous way where you protect privacy. It just never happened. I feel like we could get there. We have the data now. Like now we just merge it in. I would opt into that in a second. I think everybody who did that would opt in.
41:52 Speaker 00 There are some complicated dynamics at play. They had the FDA thing that came down on them. They were the first company really that had to deal with how do we work with the FDA and bring genomic information to light? How do we avoid being a diagnostic tool or interpreted by the FDA as a diagnostic tool? And there was a lot of red tape that 23andMe kind of trailblazed. And so that's one thing that slows things down, this integration of systems. There's some fear there, too. If you're a bit cavalier about it or a bit more aggressive, are they going to interpret it in such a way that they shut everybody down? And then it's like, uh, this is why we can't have nice things scenario and then no genomic companies can do anything. And so I think that that's one problem. The second thing is the fundamental part that held back 23andMe. To use an example of what I think
42:37 Speaker 00 is a good situation or a good setup is Elon Musk is kind of like a master of this particular phasing or staging of his companies. So with Tesla, the mission is to electrify the world and reduce or eliminate our reliance on fossil fuels. The way he's going to get there is not by doing that directly. It's first, he's going to make really, really nice cars that people want to buy. And the money you make from that is going to fuel, almost literally, pun unintended, your foray into robotics and into. Power walls and into solar panels. And then those things are going to cost money and you're going to make more money from that and get to the next stage. And that's what he's done with SpaceX too. We're going to colonize Mars and become a multiplanetary species.
43:22 Speaker 00 While we're not going to do that straight away, we're going to like take other people's payloads into space and sell satellite internet. And maybe do civilian space travel and maybe replace airlines for traveling around the earth. And there are all these kinds of things that help build the company up over time, even from a revenue perspective, such that you reduce your reliance on having to raise external capital and become more self-sufficient. And you can see OpenAI is moving this direction too, and towards their path to building AGI. And this is something that 23andMe really just didn't have fundamentally is, and this is my take from the outside. There really wasn't this organic strategic path of how to layer in value over time as the science progresses.
44:05 Speaker 00 One thing that I remember talking to them about was like, hey, you guys should work on like your own fitness band or your own health-based product that you can constantly monitor and layer it into your genetic information. And the direction Apple Watch is starting to take, though I would never want to compete with Apple in retrospect. But there just wasn't like a staged approach to building this business over time on your way to this ultimate mission of personalized medicine for all. That to me is a fundamental failure. Interesting. Before we started talking, you had mentioned you had like some interesting coincidence or something like that. I'd love to hear it. One of the first companies, if not the first company that I interviewed at when I was doing my BS in computer science, at Davis, was this company called Rapleaf. One of my first interviews, if not my first one, right out of college.
44:51 Speaker 00 I don't think that we met. I think that I interviewed with someone there, but rightfully so. I was being a jack of all trades and a master of none, part product, part engineer, not good at any one of those two things. And so I I don't blame Rapley for not giving me a job. I had a call with you, at least I have a call here. I don't know if it was with you or not. February 13th, 2007, which is the super early days of it. That was right before I did Y C. Yeah. Okay, yeah, it was a four or eight phone number. I don't want to say the phone number because it might still be your phone number. I think my mom has that phone number now. We swapped numbers. She had an unlimited data plan.
45:30 Speaker 00 I'm just looking at it here. And then later on, I see we had a overview. So that's crazy. Oh, my God. So you're in my old school emails going back. I am. I thought it was so wild that I was coming on the show and I'm like, oh, my gosh. Oh, yeah. So it was a recruiter forward us to you or something. It was either recruiter or Craigslist ad that I had applied to. Looks like a recruiter forwarded to us and then somehow, I think you and I did talk. I don't have the notes. I usually have my notes. But it looks like we did talk back in the day, 2007. That's awesome. This is a great coincidence. Crazy, huh? All right, last question we ask all of our guests. What conventional wisdom or advice do you think is generally bad advice?
46:14 Speaker 00 And you're not allowed to say follow your passion because that's 90% of what people say when I ask this question. Maybe I sort of answered it, though, in a different way earlier. But the conventional advice that I think is bad advice is don't do what people tell you to just because they tell you to. First principles thinking, every single person that you talk to and ask for help, they're well intentioned and they want to help you. But everybody's story is different, including yours, and advice is not going to directly apply. It's like if you go into an exam in school, like for computer science or for math, and there's going to be homework problems
46:56 Speaker 00 that are similar to the questions that are on the test, but the numbers are changed or the situation is tweaked. You have to incorporate your learning. From those problems, but also understand how to adapt your answer or the way you go and solve that problem to this unique situation that's put in front of you, these tweaks that have happened to the problem. And that's an analogy for how you take other people's advice. Take it for what it's worth. It's learnings from a different scenario, maybe an adjacent one, or maybe a similar one, but incorporate that into your own thinking. Don't take anybody's advice wholesale. Derive your own answers. That makes a lot of sense. It's interesting because most people are really good people and
47:37 Speaker 00 they're trying to help you. And so you meet these people, they're evangelical about a very specific thing, meditation, or protein, or some sort of diet, or workout routine, or something. It really is because it worked for them, just like religion. It worked for me, so therefore I'm recommending it to you. Or this type of interview, way of interviewing people, or this type of way of running a business, whatever it is, I'm evangelical about it because it worked for me. But just because it works for me, doesn't mean it's going to work for you or your company, because we're all in different situations. As you mentioned, we have different DaaS, our company has different DaaS. That meditation might not work for you, just because it worked for me.
48:18 Speaker 00 Life is way too high dimensional to be able to really say that somebody's exact pattern is going to apply to you, and you're going to get the same results in the end. I don't know how much time you spent on Reddit with people that do exercise, but one thing you'll notice in a lot of threads where someone that knows what they're doing when it comes to exercise, or weightlifting, or working out, is they'll say, why MMV? Or this is what worked for me. It doesn't mean it'll work for you. And then they'll list out what their routine is. And I think that that kind of applies to really all advice. Take it for what it is. It's an anecdote, and figure out how elements of that anecdote apply to you, and which ones don't apply, and then make the best decision for yourself and your unique
49:04 Speaker 00 That's what I think. I love this. All right. Thank you, Russ Assaf, for joining us, World of DAS. I follow you at Dassaf, D S A, which is an amazing X handle, which you probably only can get if you're early there. That's an incredible three-letter X handle. I definitely encourage our listeners to engage with you there. This has been a ton of fun and really interesting. It's been wonderful to have the conversation, and I really appreciate you giving me the opportunity to come on here and ask in the questions. And really nice that you looked up that old email and were able to connect almost 20 years later in this capacity. Yeah. Crazy. Yeah.
49:42 Speaker 00 Thanks for listening. If you haven't already done so, please subscribe to World of DAS wherever you consume your podcasts, YouTube, Apple, Spotify, and more. And please help us get discovered by leaving a review and check out worldofdaas.com. That's worldofdas, D A A S, dot com. And of course, connect with me on Twitter, at Auren, that's A U R E N. Would absolutely love to hear from you.
Previous US Secretary of the Army Dan Driscoll - modernizing military tech & fixing procurement Next Crunchbase CEO Jager McConnell - predicting acquisitions, value of data, AI Page updated November 3, 2026 · transcript and summary are machine-generated