The world of voice AI, with Mati Staniszewski of ElevenLabs

Mati Staniszewski is the co-founder of ElevenLabs, the research company making audio accessible across languages and voices. He sits down with John to discuss the "voice Turing Test" and why AI has conquered text but still struggles with conversational speech. They discuss the future of human-computer interaction, including why we still can't get our phones to read a PDF properly and the massive potential for voice agents in everything from farming to healthcare. Mati also opens up about ElevenLabs’ rapid ascent to an $11 billion valuation and gives a behind-the-scenes look at how Ukraine is using their tech for digital government services. Timestamps (00:00:27) How audio models work (00:08:52) ElevenLabs business model (00:17:50) The conversational Turing Test (00:21:01) Link by Stripe (00:26:02) Cascaded vs speech-to-speech (00:31:53) Universal translation (00:51:41) Designing an AI-native org

• 0:01Speaker 00

  Matty Stanishevsky cofounded Eleven Labs in 2022, and has since scaled it to the $11 billion leader in AI audio. He's credited with capturing the humanness of speech through realistic emotional inflection, and they're now expanding into everything from agentic workflows to music.

• 0:17Speaker 01

  Thanks for having me.

• 0:21Speaker 00

  The only good place to start is, describe to me how, like, I know how an LLM works at a high level. Describe to me how an audio model works. Like, if we were Carpathia-style, looking to, you know, build a toy one from scratch, how does it work?

• 0:36Speaker 01

  In early days, you tried to replicate it exactly how you replicate it with the human body. So you would try to completely try to reproduce a machine, analog machine, that would create a vocal tract effectively. Then that progressed into trying to create effectively like a digital signals for speech. Bell Labs was one of the first to try to like, create a structured set of signals that will represent the speech. And that is the first precursor to what we would do today. Then you would try to stitch in phonemes, effectively different sounds of how we speak humans, and then try to concatenate them together. It's another important part in that equation where you would, based on the most probabilistic approach of the next word,

• 1:17Speaker 01

  you would effectively try to bring the phonemes from your library of phonemes and bring them together. And then down to the modern history, where now we effectively do similar like neural nets in other domains. So you predict the next sound, based on, of course, the context of the previous sounds. If it's a streaming speech, if it's, let's say, a context of audio, you will use combination of predicting of the phonemes, but you also use the contextual text element of that work. And here, credit to my co-founder, Piotr, who effectively came with that new idea of how you can now create voice models, which are both reliable, high-quality, quick, where you would bring a lot of the ideas from transformer models, from diffusion models, into the speech space.

• 2:01Speaker 01

  So that prediction of the next token in the phoneme space wasn't something that was possible. You might be, you spoke briefly about this of like how you kind of operate. On the text, on the waveform space, there's also Mel spectrogram space. So like usually you do text, Mel spectrogram, waveform.

• 2:15Speaker 00

  So what's a spectrogram space?

• 2:17Speaker 01

  It's like a visual representation of how the speech sounds across pitch, across energy. And then you transform that into waveform.

• 2:24Speaker 00

  Got it.

• 2:24Speaker 01

  So like when WaveNet came along and TACOTron models, they would effectively use text to Mel spectrogram. So that visual representation. And then how you decode and encode that into the waveform to to bring it across. And Pyotch figured out how to like abstract some of those steps and decode and encode them a lot, a lot better. So that predicting of the next phoneme was one of the big piece. And second big piece was how do you bring that context into the equation? So what I mean by context is if a voice actor was reading a textual copy, you would know that, okay, this is a dialect sequence, I need to produce a dialect. If it's a happy sentence, I might need to pronounce as a happy sentence. But kind of what happens before and after

• 3:03Speaker 00

  Yes.

• 3:04Speaker 01

  comes into the equation and you need to bring that across. And then there's a last big piece. So voice model has the sound of how you intonate the given fragment. But the second big part is the voice itself of the characteristics of accents, of style, of prosody across that voice. So when you actually try to vocalize something, when you create that voice model, you turn text into audio, you need the text, you also need the voice reference of how you want it to be spoken. So here was kind of the second big innovation. So apart from context is how you decode and encode those features. So when Bell Labs came with their initial representation of speech, the big piece there was you would have effectively hardcoded parameters for that speech. With 11 Labs models.

• 3:49Speaker 00

  Heartcode parameters for enthusiastic speaker, British accents.

• 3:53Speaker 01

  Exactly, exactly, that kind of stuff of like the set of pitch elements that you can select, set of energy spectrograms you can select from. And in our approach, effectively, you would Give the the the model open-ended ability to select what those parameters should be. So it's not going to be British, Polish, Spanish, English speaker. But the model will deduce them themselves. The same for for other set of parameters that are not hard-coded. Whether it's the enthusiasm, whether it's the sadness, etc.

• 4:21Speaker 00

  You're saying kind of Britishness is an emergent property

• 4:24Speaker 01

  Exactly.

• 4:24Speaker 00

  in your voice models.

• 4:26Speaker 01

  Exactly. But yeah, and those kind of those two big parts. So encoding and decoding of how you create the voice.

• 4:32Speaker 00

  Yep.

• 4:32Speaker 01

  Super hard problem before and figured out too. How you then construct it in a sentence, how you get the context across, you can predict the next phonemes.

• 4:39Speaker 00

  Yes.

• 4:40Speaker 01

  So how you bring them them together in a reliable and stable way while doing it quick. And these were kind of the two first big innovations in the voice models that continue to today.

• 4:48Speaker 00

  But OK, so if LLMs reason about text and, you know, word subparts, tokens, as the way they think about the world, what is the equivalent of a token in a voice model? You mentioned phonemes a bunch, like what is that representation?

• 5:04Speaker 01

  So we do, we we store, we store the the voice embedding effectively for the speaker.

• 5:09Speaker 00

  Uh huh.

• 5:09Speaker 01

  So you need that reference when you produce and create the speech.

• 5:13Speaker 00

  Yes.

• 5:13Speaker 01

  Of course, in the input to the voice model, you still get the text and you bring the the speaker and and and coding. And then when you produce speech, you do operate on the waveform or on the or effectively on the phoneme level of of that of that speech. And then when we kind of go the opposite, so of course.

• 5:31Speaker 00

  What is a phoneme, fill in my understanding?

• 5:33Speaker 01

  It's like a syllable deconstructed even to smaller elements.

• 5:37Speaker 00

  OK.

• 5:37Speaker 01

  And these are effectively like the human sounds you can produce.

• 5:40Speaker 00

  Got it.

• 5:41Speaker 01

  So these would be like the the most close to that representation. But of course, in our models now, it's going to be a combination of not only operating on phoneme level, you also operate on the text level, you operate kind of in both in sync. Because when you are predicting the context, you need to understand

• 5:57Speaker 00

  Yes.

• 5:57Speaker 01

  how that sentence will get constructed. And especially if it's more of a streaming, real-time use case and like a voice agent setting, you need both parts to to work across. So it's similar to how you operate on the token level on the on the text side. We operate on the token level on the audio side.

• 6:12Speaker 00

  It feels like a big part of the magic of Eleven was your voices were much more human sounding. How did you accomplish that?

• 6:21Speaker 01

  So I'll kind of give you a a quick, quick synopsis of what, how we think about the models on the text to speech side today. In any model, you need, you need architecture, you need compute, you need data. So architecture innovations were one thing. The data part was the second big thing. With audio, you will have a lot of audio data available, but frequently you will not have it annotated in the right way. You won't have which speaker is speaking when. Some of the what is annotated, but the how isn't. So like as we are speaking now, what's the emotions that we use? What are the actions that we use? So we would invest a lot internally on effectively creating our own data labelers, our own team to be able to create those data sets that will be better.

• 7:05Speaker 01

  And that was a combination of, of course, like semi-automatic techniques and then manual techniques. And actually, a lot of the models that we did afterwards actually span out from a lot of that research too. So Speech2Text model initially was a model we did for ourselves because the models on the market just weren't good to annotate that data. And then another brilliant researcher on our team was kind of be able to construct it so we could span it out as a model that we brought to the customers.

• 7:30Speaker 00

  So you've just been doing useful stuff in voice and that has emerged a whole bunch of products that you might not have expected because you find you're building useful stuff.

• 7:39Speaker 01

  Exactly, exactly. And that's kind of combination of data of being able to do it, automatically create a team that's coached on voice, on how to describe it.

• 7:47Speaker 00

  Yes.

• 7:47Speaker 01

  Because most of the labelers out there just aren't as well versed on understanding the audio and voice. Helped us a lot to bring that back. And then, of course, deploying those models in production, seeing how customers interact with them, having them annotate all of the data helped us refine those models over time. A very interesting thing on the side. So we spoke about kind of the speech representation. The first guy who created the speech representation is a guy called Kempeland, Von Kempeland. So he created those analog machine that would represent effectively a human vocal tract and try to produce that sound. He had spent decades on that and that kind of started producing vowels. But that's the same person that created a chess machine, the first viral, let's say, chess machine

• 8:31Speaker 01

  that would kind of simulate playing chess.

• 8:35Speaker 00

  This is a mechanical Turk?

• 8:36Speaker 01

  It was called Turk.

• 8:37Speaker 00

  Yeah, yeah.

• 8:37Speaker 01

  But exactly, but the kind of crazy thing behind it was operated by a human.

• 8:42Speaker 00

  Yeah, yeah.

• 8:42Speaker 01

  And it was all a fluke.

• 8:44Speaker 00

  Yes.

• 8:44Speaker 01

  And that's where the mechanical Turk from, which actually we use in our kind of data labeling production to make that work there.

• 8:52Speaker 00

  Yeah, yeah. And sorry, we kind of jumped right in, but if you describe the Eleven Business today, people think of you as the speech company. How should they actually think of your business to the extent you can describe the big areas? Text to speech, speech to text, voice agents, just like break down the business for us.

• 9:08Speaker 01

  Cool. So in like the nutshell, describe Eleven Labs as a research and product deployment company. We build foundational audio and voice models and then build a platform for businesses to transform how they communicate with their customers, with their employees. And that will apply through AI agents from customer support, sales, hiring, training, all the way through to marketing and storytelling for our creative tools. And in that set, we've created all types of foundational audio models, so text to speech models for producing speech, speech to text models that work over 100 languages and happily beat others on benchmarks, all the way through to conversational models

• 9:53Speaker 01

  of how you loop them together, to music, to other domains of audio. And then, of course, beyond the models, when you actually bring them to production, that's where the second level of the platform comes in, where that meets the businesses on the specific use case. So on the agent specific example, it would be how you now connect those models to the knowledge base, to telephony, to the integrations that you need to perform the actions, how you evaluate and monitor the agent that behaves in the right way, how you build the right safeguards. On the creative side, on the marketing side, it's how do you create a good ad so you can create a good video voiceover for one of the campaigns, how you create an article that's narrated with a specific voice that represents the brand in a good way.

• 10:38Speaker 01

  So that's where we combine the models and understanding of the customers we work with into one, call this a platform.

• 10:44Speaker 00

  Every platform company has this question about how far they go into applications. So how do you think about where you go horizontal and power the whole ecosystem versus where you develop applications? Because you can imagine there being a whole ecosystem of... Closed captioning tools that screw up that again are built on the 11 Labs tech. Like it's not necessarily a space that you would have to go after yourself.

• 11:06Speaker 01

  I think the big difference between in your kind of question today, we we we we see ourselves as a platform where if you're building a horizontal use case in your business, a great place to come. If you have a lot of domain specificity, that's where I see a lot of kind of the application companies forming over time. Where they will, where where that's the specifically not the spaces we will go into.

• 11:28Speaker 00

  And I think it also is interesting when the tech is moving as quickly as it is here. It's one thing, you know, with SaaS where you get these like vertical specific providers. But I would imagine one of the biggest risks for you guys in being intermediated is if there is, you know, like in this example, a closed captioning service that is on a two versions, old version of 11 Labs and hasn't upgraded. That's a problem because you want people to be using the latest and greatest model that you've developed. And you'll be kind of deploying new capabilities every week. And I presume that's part of your thinking is that just when it's moving that quickly, you need to go direct in a lot of cases.

• 12:04Speaker 01

  That's right. In the in the closed captioning, like, you know, here already now we know that our services is going to be able to tackle like 99.9% of the cases that customers have. And then there's like added benefit of we work with healthcare customers where we will create custom models for those customers. Where it will get that transcription perfectly.

• 12:24Speaker 00

  The context is the tricky thing in closed captions where like we talk a lot about a lot of technical stuff on this.

• 12:31Speaker 01

  Yeah, for sure. And that's where you need like effectively like a dictionary of ways that you detect before beforehand, which which as we work with the businesses, we know we need to, we need to embed in that creation process.

• 12:42Speaker 00

  We're talking about kind of products here. And one thing I noticed is that LLMs are amazing. And, you know, you have the usage stats of ChatGPT and Gemini and all the popular LLMs where they're working and people use them a ton. It feels like there's a big product overhang when it comes to voice, where the leading edge voice models are incredibly capable. And yet I was driving home the other day and I needed to read a PDF. When I was driving, and so I said, okay, I'll just have my phone read the PDF to me. And you can kind of try and hack it with like iOS screen reader, but it doesn't really work with the scrolling. And then in theory, you can upload a Gemini, but you're trying to get it to not summarize it. And it actually just hung when I tried to press the like, read this to me button.

• 13:25Speaker 00

  And so there was no way I could get my phone to read me something, which seemed like a fairly basic, you know, feature. And, you know, all cars advertise voice control. And yes, it sucks in separately, you know, if you want to, you know, input something to the navigation. Just no car has a good version of that. Yeah, maybe Tesla does. And, and, and, and so why does it seem like with L L Ms and Claude code and everything? We are using all the capabilities of the intelligence, whereas with voice, we're like living 10 years ago somehow.

• 13:57Speaker 01

  Well, I'm thinking what I agree with the premise that we are 10 years behind.

• 14:01Speaker 00

  In the lived experience of people day to day, like they're using series transcription, which has gotten better.

• 14:05Speaker 01

  Yeah.

• 14:05Speaker 00

  And it's still way behind the leading edge.

• 14:08Speaker 01

  Yeah, like there is definitely a piece of like, like we, I think the technology in many of those cases is ready. There is a deployment gap to what you are saying.

• 14:18Speaker 00

  Yeah.

• 14:18Speaker 01

  Like automotive or some of the big companies are not adopting that quickly enough for bringing that into the production. But like plenty of different problems that you need to fix along the way. I mean, the, the, the quality of voice models for them to actually sound good. Like this, this is only like last three, three years thing.

• 14:34Speaker 00

  Yeah, there's three years.

• 14:35Speaker 01

  It's, it's a three years thing.

• 14:36Speaker 00

  Cars have over the air software updates now.

• 14:38Speaker 01

  So, so that's three years for the first voice model that can narrate.

• 14:42Speaker 00

  Yes.

• 14:42Speaker 01

  Text async. Two years ago, you can start seeing the real time version of that and not really like it's, I think the real break was like a year ago.

• 14:49Speaker 00

  Okay.

• 14:49Speaker 01

  Where you could start seeing that in production. And then I think over 2025, the big piece that hasn't been possible is how you connect now the real time voice interaction with something which I think you're referring to it like it has context of what you want to do. What is the material that you want to read? How does it connect to set of your preferences from the past?

• 15:08Speaker 00

  Yes.

• 15:08Speaker 01

  And gets that across. I think that's like only recently became possible and where we've seen like kind of the big adoption across the enterprises leading on the technical side.

• 15:17Speaker 00

  Yes.

• 15:17Speaker 01

  I think this year it should be in the automotive side too, or the, some of the applications.

• 15:22Speaker 00

  Okay. So you think we'll start seeing kind of great voice models in cars this year?

• 15:27Speaker 01

  This year for the on cloud use cases, like on, on, on car, in car. So like without connectivity, not yet. Twenty ten, there's deployment, of course, gap of like how you bring that into the gaps. But I think like the next two years, three years.

• 15:41Speaker 00

  How about the PDF reading use case? Like you should.

• 15:43Speaker 01

  That should work, I think, yeah.

• 15:45Speaker 00

  Well, but how should I have done it?

• 15:47Speaker 01

  We've, we've, so back in the day, we, I'll preempt this with a story to queue Eleven Reader. But we, we, we had this problem. We have so many audiobook authors come into Eleven App. So twenty twenty three, we released first software. We had a lot of creators and then a lot of audiobook authors or book authors that tried to, couldn't afford professional narration and wanted to create an audiobook. However, none of the companies accepted AI audiobooks.

• 16:12Speaker 00

  And you can't sell an AI audiobook on Audible or something?

• 16:14Speaker 01

  Exactly. So Audible would like block AI content. So we had no choice. Like, we need to create an avenue for them to.

• 16:20Speaker 00

  Because there was no distribution for AI audiobooks.

• 16:24Speaker 01

  Exactly. So we created Eleven Reader. And that kind of came with functionality where you can upload your PDF, you can upload your text and have it read out loud with a number of incredible voices. So whether it's Sir Michael Caine, all the way through to a state and working together with Sir Richard Feyman, where you can.

• 16:41Speaker 00

  And so you're working with the Sir Michael Caines of the world?

• 16:43Speaker 01

  Exactly. And then you can actually read it out loud. And that kind of works extremely well. So that works. Now, how can you do it?

• 16:50Speaker 00

  I do want everything read to me by Michael Caine.

• 16:52Speaker 01

  It's a, it's a great voice.

• 16:53Speaker 00

  Yeah. Shouldn't you guys have a consumer app where I can just do the common voice things? Like, I want to be able to have an Eleven app on my phone. And then if I upload a PDF to it, it can do the common things that I would like, such as have it read it to me.

• 17:06Speaker 01

  Yeah, that's, that's exactly your own reader. So that works.

• 17:08Speaker 00

  Okay. The phone makers allow third-party keyboards. Do you think they, do they allow third-party transcription engines? Will they, do you think?

• 17:17Speaker 01

  The phone makers, you said, right?

• 17:18Speaker 00

  Like Apple and Google?

• 17:20Speaker 01

  Yeah, they.

• 17:21Speaker 00

  So the OS maker?

• 17:22Speaker 01

  Yeah, not, not, not all of them. Android, with Android, you can work through it. It's like, you know, variations of like Nothing, Tech and others.

• 17:29Speaker 00

  But again, I feel like if you had a popular Eleven app that allowed for transcription, people would use it a bunch. And maybe eventually Apple would say, oh, we should allow third-party transcription engines if that's what people want.

• 17:39Speaker 01

  I mean, there's, it seems like they might be going in that direction, right? And I recently, they announced they will open up the Allemeco system. Hopefully, they will do the same with Voice Assistant, which is kind of similar.

• 17:48Speaker 00

  But again, I think rational to do when it's moving so quickly.

• 17:50Speaker 01

  Yeah.

• 17:51Speaker 00

  The voice assistant.

• 17:54Speaker 00

  Paradigm is one of the oldest paradigm, you know, UI paradigms in computing. Like the open the pod bay doors, Hal, from nineteen sixty nine.

• 18:04Speaker 01

  Yeah.

• 18:04Speaker 00

  I will claim it's not working yet. So Siri doesn't have the intelligence. And then on Gemini and ChatGPT and those apps, I mean, I want to use the voice mode, but I don't know about you, it just doesn't work. And so like, sometimes I'll be using my phone and I'll use the iOS keyboard transcription to type in the field and then like say a bunch of stuff and then send it off. But this suggests to me that consumers really want voice mode that works. And yet it's just not working yet for the major LLM apps or for anyone. Why isn't it working yet?

• 18:41Speaker 01

  It is pretty hard to do because you want, you want two things. You want, you want to be able to say things that you want, but you want sometimes for it to execute it, sometimes to wait for you to like finish and add something in a sentence. Sometimes you want it to be interactive, so it asks you questions back to clarify and get some of the additional detail. And all of that is actually pretty hard. Like that's where kind of the magical, like ideal version of a voice agent for us comes through, where you need the speech to text element, you need the transcription side. You need, you need then the kind of the turn taking mechanism. So like, when do you finish sentence? When, when is it likely based on silence, based on ways, likely on the context? And then sometimes you want it to speak back and clarify, or at least give you the text back to clarify and then maybe execute set of instructions. So that problem is still very hard.

• 19:26Speaker 01

  So I agree with the claim that like this orchestration side has not like passed a true conversational agent Turing test where that behaves as you would expect from another person, where you can say.

• 19:37Speaker 00

  Yeah, that's the simple way of saying what I'm saying is that we have passed the Turing test with text LLMs a long time ago, or actually nowhere near that on voice LLMs. So it's kind of interesting how that's a final frontier.

• 19:47Speaker 01

  Yeah, I feel like it's it's it's going to work in like specific domains like in customer support call.

• 19:53Speaker 00

  Yes.

• 19:54Speaker 01

  Passes the voice to us, works well. And like, let's take another like spectrum of that. An interactive gaming experience, like a truly interactive as you would have with another human in that game. It's like, you know, so hard and and and further out there, we haven't passed it yet there.

• 20:09Speaker 00

  Yes, yes.

• 20:11Speaker 01

  Yeah, but I think that's a combination of like, you know, like even like a simpler version of of within that like. Sometimes you might give a response immediately back. Sometimes you need a tool call to get additional information from the database. How you orchestrate that. So like, that's probably the most common thing we see as we work with some of the companies out there is. You want those systems to orchestrate extremely well, where if it's conversational use case, pretty simple, you can root the agent to speak with. But if you need to authenticate, if you need to pull additional information from the database, what do you do? How do you handle that graciously?

• 20:41Speaker 00

  That's great.

• 20:42Speaker 01

  And yeah, and and to that extent, I would agree. That's just getting getting getting there. And we'll hopefully see that we our goal is to like pass the voice tuning test in all those cases. Or the tuning test for all conversational agents outside of voice two. And I hope we will be there in the in the next year or so.

• 21:02Speaker 00

  For subscription businesses, a lot of revenue is lost in that last few seconds before the checkout. Someone has to get up or find their wallet, or they mistype their card number, or they hit an error. They just give up and you lose the sale. For a company like Eleven Labs, adding hundreds of thousands of subscribers, even a tiny bit of friction like that, it would really add up. But that's why Eleven Labs uses Link from Stripe. Customers save their details once, and then they can check out in seconds across more than a million businesses with saved credentials. So if you want a faster checkout for your customers, you should turn on Link from Stripe.

• 21:37Speaker 00

  Are you guys working on personalized?

• 21:41Speaker 00

  Voice transcription, where it feels like part of the way we're making it hard for ourselves is when I speak to Siri, I have a bit of an accent and so it sometimes has a hard time understanding me, but my accent doesn't change. And so it could just get good at listening to John. But my understanding is it's not, it's just like running the global voice recognition model. And I'm guessing it's the same for 11 Labs where you're running the global voice recognition model. But again, you have an accent. And so if someone's understanding, like if you walked up to someone in a coffee shop and said two words, they might have a hard time understanding it because they like, they're not putting it through their Matty Polish accent filter. And so where's this going with like actually interpreting the person that you know to exist on the other side?

• 22:24Speaker 01

  Yeah, I have a, I have a very tricky one to detect. So my voice is frequently used in the test.

• 22:29Speaker 00

  Ah, you're part of the test suite.

• 22:30Speaker 01

  Yeah. For text to speech, for speech to text, for like everything.

• 22:33Speaker 00

  Yeah, yeah, yeah.

• 22:33Speaker 01

  I mean, like, yeah, it's pretty tricky.

• 22:34Speaker 00

  But again, trying to parse your voice in a global model is just making life hard. It's like have a Matty specific model.

• 22:40Speaker 01

  Yeah, so, so on the, on the speech to text by transcription, exactly. Like the big part now that we are bringing in is you have two parts. One, effectively like a person or a voice specific detection, which is true for the accent side, but it's also true for a crowded room. So that's, that's where we have, we have incredible research team that's like able to continually do both the accuracy high, but also, also, also like add things like speaker detection, of course, noise reduction. But then the second part is also keyword detection. So there are specific words that you would want to say in those settings that you want to effectively monitor for. So we spoke about, you know, like, let's say I'm going to the coffee shop and order things.

• 23:23Speaker 01

  The set of actions I would, like the coffee shop would expect me to do.

• 23:26Speaker 00

  This information theory, it's like there's a, you know, they can just listen out for the coffee words.

• 23:30Speaker 01

  Exactly, exactly. And then try to like match it to the closest proximity.

• 23:33Speaker 00

  Yeah.

• 23:34Speaker 01

  So like both things will help in, in the, in the setup where you have my voice. Perfect.

• 23:39Speaker 00

  Yeah.

• 23:39Speaker 01

  You can decode it, encode it on that. If you don't have my voice, or even if you want to like double amplify it, we, we already support like effectively a keyword detection. Which is useful for like real time setting and async setting. So back to like cheeky pine transcription. You could effectively pre pre generate that from the previous podcast and like look for a set of words that you would use.

• 24:01Speaker 00

  Yes.

• 24:01Speaker 01

  Traditionally in that.

• 24:02Speaker 00

  And so how hard, okay, so you do the keyword detection already, but how hard are the, I want to get superhuman transcription performance by feeding it an hour of Matty audio before it listens to Matty. And then it should be able to do a much better job transcribing. Is that just a really hard research problem?

• 24:18Speaker 01

  No, solvable. We think we can, we can, we can roll it out in one of the next versions, which is like hopefully in the next, in the next months.

• 24:26Speaker 00

  So you think this year you're doing.

• 24:28Speaker 01

  Yeah, for sure.

• 24:29Speaker 00

  Person specific transcription.

• 24:30Speaker 01

  Person specific transcription, like we can already direct speakers extremely well. So like if we are speaking and of course dissimulate who is speaking when.

• 24:38Speaker 00

  Yes.

• 24:39Speaker 01

  Which is like in transcription side, a part of accuracy, dirertization is one of the harder problems. And we do that extremely well. And now it's going to be like effectively what you're saying, like fine tuning based on the speaker that I want to listen to.

• 24:51Speaker 00

  Yes.

• 24:52Speaker 01

  Which we know will be important. I mean, like in healthcare setup, such an important part. You're in an operating room, you're a doctor, you want to say a command. Then you want to really be able to listen to that one person specific piece.

• 25:03Speaker 00

  Yes.

• 25:03Speaker 01

  You have a hardware device at home, let's say it's a pilot that helps you control the TV. Here too, you will want that to listen to you versus, let's say the family roaming around. Or maybe you want it to everyone. So like you could decide that, but in many cases you want to be able to also specify that.

• 25:21Speaker 00

  Okay, that's really exciting.

• 25:22Speaker 01

  It's great because there's like still so many unsolved research problems that we can, yeah.

• 25:26Speaker 00

  Like there's just breakthrough after breakthrough coming in the, in the domain of voice models. How about on the flip side, when it comes to speech generation, you know, the zoom touch up my appearance feature?

• 25:37Speaker 01

  Yeah.

• 25:37Speaker 00

  I've always thought about that in the context of voice where. Should you offer a de-accenting filter for voices? Or like even there's one podcast that I like to listen to. But the voice is a little mumbly and I always thought they should put it through a demumbling filter. Just like make the

• 25:54Speaker 01

  Or like slow it down.

• 25:55Speaker 00

  Yeah, make the enunciation a little better. But all these things, again, like photoshopping an image. There's no reason that the, like, have you thought about voice to voice, basically? Rather than voice to text or text to voice?

• 26:07Speaker 01

  Yeah, so there are kind of two big parts. One, on the speech generation side, similar, so many innovations still there. There's like a wider piece and that's like the, we released the V3 model that I kind of were solving that for the first time. It's like, can you control speech? So you can have the text to speech, you generate something that sounds emotionally great. Previously, until, until end of last year, effectively you would rely on model to decide what's the best performance. You could regenerate it, but then ultimately model decides the best performance. So that's where the controllability came in, where we can finally give it cues of say it in a slower way or change how you deliver the dramatic pause or kind of any cues that you give. And to be able to do that, you need architectural changes and the data that we kind of created over time,

• 26:52Speaker 01

  where you annotated what was said and how it was said, so you can actually train the model to do that. So today, finally, you can have both speech generation or entire voice agent experience with what we call expressive mode, where the agent knows the emotions on the other side. So if the person is stressed, it can react and be reassuring. And that's generating a limb response on the reassuring side and response in that set of emotions too. And that breakthrough was super hard to do. And that, of course, stretches to a lot of what you said. It could be some version of speech enhancement in their real time or in a post setup to change how that's delivered. And that's relatively recent innovation and it's like, we know it can still be so much better.

• 27:35Speaker 01

  Like the edge cases of how you want to describe it is pretty large. So that's one. And then the second part of the question, which is a huge question, the speech to speech models. So as you said, our approach, as you think about voice agent, conversational agents, is effectively a cascaded approach. We use transcription and speech to text, LLM, text to speech, and orchestrates all of that together. And then you have a speech to speech, which kind of goes directly from speech and there's a speech response on the other side.

• 28:02Speaker 00

  And when we say speech to speech, is that the idea that it doesn't go through text, doesn't include in the intermediate?

• 28:08Speaker 01

  Exactly.

• 28:09Speaker 00

  Oh, interesting, okay. For performance reasons, for accuracy reasons?

• 28:13Speaker 01

  You'd usually do it for latency.

• 28:14Speaker 00

  Okay. It is faster to run a model that does not have to transcribe and then generate?

• 28:19Speaker 01

  Exactly. It's quicker, but on the flip side, you lose reliability.

• 28:23Speaker 00

  Yes.

• 28:24Speaker 01

  You lose all visibility into the parts of the pipeline. And emotionality, we think you can deliver on both sides extremely well, and maybe you can make it more controllable, too. So today, we are optimizing heavily on a cascaded approach.

• 28:35Speaker 00

  I'm sorry, a cascaded approach is?

• 28:37Speaker 01

  Is the speech to text, going through the text layer.

• 28:39Speaker 00

  Okay.

• 28:40Speaker 01

  Going through the text layer. And as we work with a lot of the businesses and enterprises, they will need that visibility into what happens, they will want to execute certain tasks on top of that, they want a good visibility into each of the steps and great accuracy of all the models. But beyond that, they can abstract away what's the LLM layer, what's the intelligence layer, the integrations are easier in that system. So that's where we are betting a lot of the research work of how you can make that great, and we think we can make that great. And speech to speech, as you think about maybe more of like a companion version of the applications, that's where that will flourish, because maybe the hallucinations aren't as important, but the latency is a little bit more, and maybe hallucinations are even a feature. And maybe in the future, future, just to finish that part, you will have like some

• 29:24Speaker 01

  version of combination of the models, that for like low complexity, easy models, you will have speech to speech, and for like higher complexity, you will have the cascaded.

• 29:32Speaker 00

  Okay, so I was gonna ask about this, like, you know the way there is research on how the invention of writing changed humans' brains and just like changed the neural pathways in ways beyond kind of the actual written language? Do you observe that speech to speech models think differently than cascaded models?

• 29:58Speaker 00

  Like it sounds like they're dumber.

• 30:00Speaker 01

  They are definitely dumber. You need smaller model. You cannot.

• 30:03Speaker 00

  But that's interesting, right? That like forcing models to reason about text. I mean, I know they just have much more in there as well, but they're smarter.

• 30:10Speaker 01

  Yeah, but it's like, you know, like if you are going speech to speech, usually you will use smaller models. So it's still quick.

• 30:16Speaker 00

  Yeah, I see. So it's also just a model size thing. Yeah. Okay. But are there interesting differences beyond like correlates like size?

• 30:24Speaker 01

  What I can say is like slightly different to your question. The people interacting through voice and the performance we see for like how they interact with with with the business changes just by nature of interacting with voice. A good example, you can contact 11 Labs and register for for your interest. You go through the form and at the end of that, we supplemented that that instead of going through the form process, you can speak with our agent and leave more details. And what happened are two things. One, people were actually much more keen to leave the forms through speaking with the agent. So we would go through the form a lot, a lot easier. But second, they will be a lot more open ended in terms of what the use case are.

• 31:03Speaker 01

  So they would start giving us information about the wider set of use cases, the complexity of the use case. So like the writing out was tedious and tricky.

• 31:12Speaker 00

  This is like an open ended adventure.

• 31:13Speaker 01

  Open ended. You could ask follow up questions. You can clarify, but people were just more at ease and could like trust the system. While doing that, that it's, it's, it's working and that kind of helped us a lot. And then free, which maybe is like more of a technological barrier. It also works across all languages. So now we have leads from like all parts of the world coming in and leaving their details. So we did that use case. And now we have few different companies building their SDR versions of that too, to help who will help them capture the leads. Coming, coming in from, from banks all the way to actually one of the automotive companies that like leaves that where people are just more keen to speak through voice.

• 31:49Speaker 00

  So yeah, I want to ask about this kind of the second order effect. You have, you know, you talked in the past about how growing up in Poland. I guess the dubbing of TV shows, they were cheap and so they would only have one voice actor for a TV show. So no matter all the parts, male and female, they're like, I love you. I love you too. You know, there's like one voice. Doing all them. And now, you know, thanks to better voice models, you'll be able to just have like really good voices AI-generated for all the dubbing. Because again, it's not like it's taking jobs from great dubbing that was happening previously. It was like awful dubbing, you know, happening in Poland previously. So that's like one example of the second order effects. What are the other second order effects you're seeing of ubiquitous good text to speech, speech to text?

• 32:34Speaker 00

  It seems like across a broad array of languages. Because whatever about in English, just this didn't exist in Polish or Irish or, you know, pick your language.

• 32:44Speaker 01

  One, the breaking down the language barrier, we, you know, the kind of the inspiration came from, from the movie side, but it also applies in any, in any communication setup. Like, could in the future, could I travel to another country and speak, speak Polish or speak English? And that, that language isn't being understood in the local native language. Like from Hitchhiker's Guide to Galaxy, this version of the bubble first. Exactly, that you can like actually understand the world. And voice, of course, will be an interaction layer. But similarly, all of us will have our own kind of extension and voice agents that can help on our behalf. And there is like very clear and, and, and great examples of that of people that lost their voice and can get it for the first time, for the first time back. We see that everywhere, whether that's people that lost it due to ALS or throat cancer that can get it back.

• 33:28Speaker 01

  Just recently, there was an example of a patient that had Neuralink. We worked with them to bring the voice that that person could speak with their own voice back to the, back with the, with the family around. We worked with, with, with, with the lady that lost her voice before, before she got married. And, and then finally, technology became possible. We, we were able to recreate that voice. And for the first time, she could replicate the marriage ceremony and speak the vows together, which was like such a heartfelt moment, probably like the most important from all the work that, that we do.

• 34:03Speaker 00

  When you guys, you guys talk about voice agents. Is a voice agent just the idea that you have?

• 34:12Speaker 00

  Some long running or persistent agent that is going out and interacting with the world through voice and so customer service would be one example of it. You know, in the other direction, your claw going and making you a restaurant reservation and actually calling up the restaurant, is that is that kind of how I should think about voice agents?

• 34:31Speaker 01

  That's right. It's exactly whether it's like the reactive side of being able to interact with the customer or the proactive to call it back. We recently had a very interesting one, topical because it was a Guinness related one where there was a there was a developer developing a Gindex effectively.

• 34:47Speaker 00

  Oh, I saw that. They were calling all the pubs in Ireland to check on the price of a pint.

• 34:50Speaker 01

  Yeah, you could like ask that and or report information.

• 34:53Speaker 00

  That was built, the Gindex is built with 11 Labs technology.

• 34:55Speaker 01

  It was built with 11 Labs too. So like people could actually could do both sides, could proactively reach out, reactively reach out, all was captured for voice and yeah, and then kind of 3000, 3000 different entities could report their prices and and get that across.

• 35:12Speaker 00

  Have you, by the way, hooked up your open claw to 11 Labs? Is the open claw 11 Labs combo something that a lot of people at 11 are doing?

• 35:21Speaker 01

  So, yeah, as you know, the open claw will like kind of look for the most popular tools frequently where it tries to hook up. So 11 Labs is one of the recommended ones.

• 35:28Speaker 00

  It's actually the top option for voice. Can you tell me a bit about the business of voice models where I think people have an intuition around big L L M's where there are these very expensive training runs and yes, they kind of appreciate it quickly, but there's so much usage that all of the models trained to date have, you know, paid off their training runs and then some. And then there's this kind of ever larger CapEx going into, I mean, a lot of it is inference these days, but also training. And so people have some intuitions from the L L M world. I'm curious just how I should think about voice for one, how expensive is training the voice models? Is the expense in the researchers?

• 36:10Speaker 00

  Is the expense in the the training runs? And I mean, the economics is presumably kind of simple where it's just per per usage. But yeah, just talk us through the business.

• 36:21Speaker 01

  Yeah, definitely cheaper than the L M and image video models, particularly smaller models.

• 36:28Speaker 00

  Okay, so the models are smaller.

• 36:29Speaker 01

  Smaller.

• 36:29Speaker 00

  What's a parameter count for a leading edge voice model?

• 36:33Speaker 01

  Few billion to low tens of billion parameter models.

• 36:37Speaker 00

  And for context, I think the, I mean, kind of like, you know, CPUs moved away eventually from gigahertz as like the metric as they moved to more cores. I think we've mostly moved away from just raw parameter count, but I think the leading edge LMS are in the hundreds of billions of parameters.

• 36:52Speaker 01

  I think the leading ones, yes, but of course, you know, you have the variations that you will use at lower scale. So CapEx is still pretty high. We've, of course, raised recently a half a billion, at 11 billion valuation.

• 37:06Speaker 00

  Makes sense.

• 37:06Speaker 01

  Makes sense to continue, to continue being able to build the best models in the, in the world. Researchers, you know, of course, you want to, you want the best people in the world. I think we have, we have those people working in, in audio, in, in microfounder who, who is, who is leading that work. So that's, that's, that's definitely a big piece of like, not financially, but even like how you keep the ambitious deployment. So you kind of continue building leading models helps you attract more talent and, and building that. And then on the how we serve it, so of course inference is correlated with how the models are used. And, and for us, like, we've seen incredible, incredible growth across the work. Mostly this is charged per, if it's input text or text to speech, it's usually per,

• 37:50Speaker 01

  per, per text token. If it's voice agent or, or transcription, then it's per minute. And we see that kind of being the, the bigger part, but usually like broadly it's per token basis. And of course, as we work with businesses, it's like an annual agreement, the bigger the spend, the bigger the comments, the bigger the discount to go get across. The way we usually do is like when we have a new model, we try to give it at cost to a lot of the customers so they can experience the best. It's still usually like not as reliable.

• 38:19Speaker 00

  The newest thing is often the most expensive, whereas you make the newest thing the most economically attractive one.

• 38:25Speaker 01

  We try to make it attractive, so the customers are like, you know, like it's more expensive for us than any previous generation. We don't like, the quality is higher, so we try to keep the prices still competitive.

• 38:36Speaker 00

  I see you subsidize it, but it's inherently more expensive for bigger models.

• 38:39Speaker 01

  Exactly, exactly. And over time, we might do some tricks to optimize it. But like, we want the customers to like experience, because of research, the big thing that we've realized, we've seen is the reliability of the model in the early days might not be there. And then two, you, people don't even know what's possible with that model. So you kind of want the widest set of distribution, so people can show the world what's possible. So you can have it, of course, as a distribution mechanism, learn yourself what to improve, what to change, and then, and then get it out there.

• 39:10Speaker 00

  Are the voice models just getting bigger and bigger? Like, will we have voice models in the hundreds of billions of parameters? Or have we found, like, it seems like for certain types of model architectures, there's like an upper limit on like the natural size. Have we found that upper limit for voice models?

• 39:26Speaker 01

  It's, it's, it feels like for specific use cases, like say audiobook narration, you probably found that size, you probably don't need to stretch it too much bigger to, to make the quality as much higher, but for certain use cases, that will probably grow. The, the thing that's, that's, you know, like I hesitated on the question is, in a cascaded approach, you probably will not see like dramatic size changes. You inherently want the models to be quick and reliable, you want to orchestrate them in a smart way. In a fused approach, probably that will get into like tens, hundreds billion parameter models, because you kind of combine, of course, the L M side and the voice side, so that will get bigger. But on the just voice, I think it will keep, keep being small.

• 40:07Speaker 00

  Okay, but there are certain domains where, yeah, we'll see bigger models. That's so interesting.

• 40:11Speaker 01

  Yeah, yeah, yeah.

• 40:12Speaker 00

  It, it is amazing how, it does seem fun from a research point of view, how there are still these various unsolved aspects and how you guys are just making technical breakthroughs and then releasing them down the product pipeline. That's like a really fun stage of a company's lifecycle.

• 40:27Speaker 01

  For sure. It's like fun because we, it feels like we can do innovations on both sides. There's like so much on research side, so much on product side. And then like the, the kind of, you know, ultimately the, the, the biggest part is how we deploy it to, to the customers where like S and B will have very different dynamic than, than the enterprise. It's not vendor SaaS relationship where you just give the product out there for the biggest companies out there. But you, you are more of a partner in their AI transformation part. So you want the resources to work alongside them to, to work on the frequently very new use cases that were impossible. To help, help create and bring those voice agents to production. So that's like a big, big, big shift.

• 41:08Speaker 01

  But the biggest focus is, is how we, how we bring the conversational agents out there to, to the businesses around the world.

• 41:13Speaker 00

  So when you say bring conversational agents is the biggest priority. Is this for customer service type use cases? Like what are the most popular use cases for conversational agents?

• 41:25Speaker 01

  Yeah, like we want to be a partner for like full interactions between business, businesses and their customers or their audience. I'm saying the, the audience because that will apply in support. Support is the easiest one because that's where it's most ready. But like, and that's maybe the big difference to how we see ourselves to some of the other companies in the space is, this can also apply to sales. You can, you can have the proactive side of reaching back. You can have AI SDR versions of that. And then you can have all the way to the marketing use cases where we are your partner for, for working on, on, on, even outside of like the conversational agent space of how you create a great marketing campaign.

• 42:01Speaker 00

  Sir, how does break down between, you know, we had Des Trainer from Intercom on here and they have Finn, their agent. And it's a thing on the website that you can go talk to. And he described a very similar phenomenon that you described, which is you start maybe thinking, oh, this will help me answer customer support queries, but it becomes like a generic UI for the, you know, for, for the website where it's a box you can type in to go do things and understand things. And so why wouldn't you read the docs and design your integration, you know, whatever? And so will I have like one for text and then one for voice? Will you guys do text too? Will just, how does that?

• 42:43Speaker 00

  Cause it seems like this is also succeeding at the text level with Finn and Sierra and all these things.

• 42:48Speaker 01

  The places where we know we will, we will be able to provide the biggest value is like where ultimately today you will have more. Either a big portion or most of the interactions coming through voice. So if that kind of intersection is there, that's where we can provide higher value. And of course, if you need a text chatbot there, that's like, if you fix the voice agent, you'll have fixed text piece inherently as well. But the place where we do optimize today is going to be like, how do you select the right voice for the right customer interaction? How you pull that in the, in the pretty complex case of what you mentioned earlier, of like how you orchestrate that to pause or look for something deeper into the docs? How it can be extension of entirety of the business?

• 43:31Speaker 01

  So not only in support, but across entire of a user journey. But the bottom line is like, we want to be able to provide you across entirety of the interactions. Voice is usually a big part of those interactions. And yes, we need to solve the integrations, we need to solve the knowledge, we need to solve text as part of that. But like, we wouldn't, for example, go into what I think will happen in a lot of those cases, like very deeply into reasoning version of those use cases, where you maybe need to like the multi-touch.

• 44:00Speaker 00

  Yeah, yeah, yeah. And a lot of complex actions.

• 44:02Speaker 01

  A lot of like financial analysis of like, that would be not something we optimize for.

• 44:08Speaker 00

  Can we talk about your revenue ramp? You're just one of the fastest growing startups period of the past few years. What's your most recently announced revenue figure?

• 44:16Speaker 01

  Most recently announced was end of 2025.

• 44:18Speaker 00

  Whatever number you want to give us.

• 44:20Speaker 01

  So most recently announced was 350 at the end of 2025. But the best proof of the technology working. So recently we were announcing our work with Deutsche Telekom and T-Mobile with Revolut, with Klarna, with Meta, with IBM, a wide set of use cases. And this quarter was kind of one of the best for enterprise growth, where we had the first quarter hit 100 million in an additional ARR growth, which is crazy.

• 44:45Speaker 00

  In net new ARR?

• 44:46Speaker 01

  In net new ARR.

• 44:47Speaker 00

  Okay, so if you're thinking this quarter was 100 million net new ARR and 350 million at the end of the year. I'm no mathematician, but it's up in the 450 million range. So, and that's versus this time last year, that's a several-fold increase. Just was? What's working? Like, from the outside, I would assume that there is really strong cohort growth within accounts. And then you seem to have self-serve and enterprise businesses that both contribute a lot. I don't know how big self-serve is, but as a user, I like to be able to fiddle with 11 labs and not have to, you know, go talk to sales. But maybe you can just talk about what worked to reach 450 million plus of ARR so quickly?

• 45:28Speaker 01

  Yeah, so exactly, so we are over 50% is now sales led on enterprise.

• 45:32Speaker 00

  Yeah.

• 45:33Speaker 01

  And, you know, like, I think largely that the technology that like powers a lot of their agenting interactions just became reliable at the same time as high quality over last year, year and a half. So like, that's, that's, that's, you know, frequently, you know, you know this extremely well, you will, you will start the account and then, and then, of course, it continues expanding and we see, there is definitely land and expand motion across 11 labs. We, we bring.

• 45:59Speaker 00

  And what does that expand look like? Is it like new departments? Is it just the usage starts taking off when a customer expands?

• 46:05Speaker 01

  Both, but usually the first, the first part too, it's like, we try to make it very easy for our customers. Maybe that kind of against ourselves where we give the technology a pretty attractive economics because we so much belief in the technology providing value. So you can actually try it and test it and then within that.

• 46:24Speaker 00

  And you think you'll make it up in usage, basically?

• 46:26Speaker 01

  Exactly, that usage, the kind of comment continues increasing because you know it's providing value. And, and then it's so much easier to make that a choice.

• 46:34Speaker 00

  Yes.

• 46:34Speaker 01

  And then, of course, cross-department pollination is there too. And it's like, you know, our work with Deutsche Telekom started with marketing side. So we did Magenta work and POKTAS generation, then it kind of expanded to customer support. And then it expanded to us working on the, in, in agent across the entirety of the network. So people can call in and have the agent. So like, you could see those step changes, step changes across. But we are now 470 people as a company. So we keep, keep on growing, but some of the things that stay consistent is small teams. So we have less than 10 people teams for each of the product or research initiatives.

• 47:13Speaker 01

  Or even as you think about sharding some of our go-to-market strategy, those will be smaller teams, understanding the industry in depth, understanding the market in depth and going independently and going quickly. So that, that definitely contributed largely to that. Two, especially on the biggest enterprises, what we found works is, and it's like we have the full spectrum, self-serve, PLG motion, that helps drive distribution, drive

• 47:38Speaker 01

  kind of awareness of Eleven Labs. And on the completely other spectrum, we have the high-tight for deployed engineering working side by side with the customers to customize the entirety of their work together.

• 47:50Speaker 00

  Why did you guys do self-serve? Because I presume you have a lot of competitors where they have tech and it's behind a contact sales forum and you have to go talk to an SDR and then talk to an AE, blah, blah, blah, blah. And you guys just offer the tech available. And I'm a huge believer in this. I mean, a huge part of Stripe's growth has been driven by the fact that we just made Stripe available to anyone and built a lot of product around that adoption pattern. But so many companies seem to skip it. So I'm curious how you guys came to it.

• 48:18Speaker 01

  So many reasons, so many reasons. I think, you know, the quick ones that come to come to mind is feedback loop. You just, you can have immediate understanding of how good your technology is. Two, which is an extension of that. We stand behind our tech. We believe it's the best in the world for models, for voice agents, for deployment. So we want people to experience that. And I think you do that the same in Stripe, where the best version of the technology is available to everyone, which is so attractive to like actually try it out. We like always try to make everything we build for the highest end use cases, bring it back to the ecosystem free. Frequently, the newest of the use cases, you know, for enterprise, you will need reliability, you need compliance, you need

• 48:59Speaker 01

  the scale, which we deliver. So frequently, as you develop new technology, it might not not be ready for a lot of those parameters, but it's only ready for developers and SMBs. And we love what they are doing because they are showing us the future and effectively helping us find a trajectory of where Eleven Labs should go.

• 49:17Speaker 00

  I'm totally convinced on that. I'm just always amazed that more companies don't pursue it, or it feels like they're really shooting themselves in the foot by not. Like, did you guys self-serve on Stripe? Or did you?

• 49:27Speaker 01

  We self-serve on Stripe.

• 49:28Speaker 00

  Yeah, for example, you know, Eleven is a huge company and yet you started on Stripe on a self-serve.

• 49:33Speaker 01

  You're kind of like initially, and it's like, you know, we were two of us at the beginning. You try to see what's working in the industry, but you try to think from first principles. So you want to try it out. You want to understand how it works. So the more friction elements before you're trying it out, the less you trust whether it's available, whether there'll be additional payment that's hidden behind some of those steps. So you don't want to go through them. So it's so much.

• 49:55Speaker 00

  Speaking of Stripe, do you have any Stripe feedback for us? Anything you want us to fix?

• 49:59Speaker 01

  My most common feedback until recently is like, why don't you pay as you go usage-based billing type version? But one of our finance lead, Maciek, I know I was speaking with your team, and that was the day before. He's great. He was like thinking about it for a long time. He's great.

• 50:17Speaker 00

  Was it the thing who said like, you guys should buy Metronome?

• 50:19Speaker 01

  You should buy Metronome. And then the next day, Metronome acquisition was announced. So now you have it. So that is my most common feedback. And we'll be launching. That's a good announcement for this podcast. We'll be launching usage-based billing to everyone.

• 50:33Speaker 00

  But sorry, I'm shocked you. Oh, as in previously?

• 50:36Speaker 01

  Pay as you go, pay as you go.

• 50:37Speaker 00

  Okay. Previously, you had it on an enterprise basis, but everything on the self-serve basis was like plans?

• 50:42Speaker 01

  So we had the subscriptions, yeah. Subscription plans, you can go over them.

• 50:45Speaker 00

  Yeah.

• 50:46Speaker 01

  But now we are launching a full pay-as-you-go experience. So you can just try out Voice Engine, which is effectively this orchestration loop all the way through to any of the models directly.

• 50:56Speaker 00

  Going back to self-serve, I think a new thing in AI is that all self-serve products should have pay-as-you-go as an option. Maybe you want to have like a subscription with someone limited tiers, but I don't know if you had the experience of like you're using Claude and like you're typing away your queries and eventually you hit some rate limit. And it's like, sorry, you've hit your usage limit. And you want to be able to do the thing that you can do in Claude code, which is just pay per API. It's like, fine, I'll pay for it. And it's kind of very funny as a consumer to not have the option to pay more, to use the product more. And so, yeah, I think every AI product will need, you know, they probably want to have some all you can need, most of what you can use, subscription with limits, and then the ability to pay for overages. So it sounds like that's what you're?

• 51:37Speaker 01

  Yeah, exactly. That's what we, what we're doing.

• 51:41Speaker 00

  The other thing I want to ask you about is, I feel like all CEOs of larger companies today are trying to figure out. How do all these AI advancements change the nature of the organization? And how do you redesign your organization a bit around all this new intelligence? And so that could be about what the scaling factor is of like the number of people you need to do the work. But it also should be like, do you need more senior people? Because they're better able to direct the AIs. And the AIs are maybe kind of do the work of what previously would have been junior people. Do you need more junior people because they're going to be more AI native in how they work? Do you want smaller teams? Do you want bigger teams?

• 52:21Speaker 00

  How do you actually go do the process engineering of, you know, your finance teams should be using Claude extensively. But like finance teams did not historically, you know, have a lot of home-built software. And so there's all these questions that are floating around. And you have very rapidly built a much more AI-native company. And so I'm curious what lessons we should all be learning from Eleven Labs as a large business recently built. And so without the baggage of decades of, you know, how we've always done this.

• 52:52Speaker 01

  Yeah, we started with 2022, which is a year when the two topics of the day were crypto and metaverse. So just before, and then of course AI flow started. Exactly, exactly. But we could like had the privilege of like kind of scaling through the world when it was all happening for us, Woodworks. And we like really believe in that being the big part of the future. The first is small teams, like keeping the teams small and super flat. So like, can you have, both me and my co-founder will have over 15 direct reports each that we'll work with. And most of those people will have that same scale of direct reports.

• 53:28Speaker 00

  Okay, so your span of control is way larger than a traditional company. Normally it would be aims, you have double that. And obviously that's an exponential.

• 53:34Speaker 01

  Exactly. And of course, you know, there are some teams which in the short term might not do that. But ultimately, that's where we think it's going to be headed. It's like roughly 10-team size within each of those of those work items.

• 53:45Speaker 00

  And startups, no offense, but like startups often have pretty wacko management ideas. Like there was a funny tweet, Laura Grant, the confidence of a, you know, early-stage startup founder blogging about their management theories. But like, you think this is not a startup effect? This is an AI effect where basically.

• 54:00Speaker 01

  No, it's definitely a little bit of startup effect too.

• 54:04Speaker 01

  I think it's like it's hindsight, hindsight benefit.

• 54:07Speaker 00

  I'm cancelling our Stripe changes, yeah.

• 54:10Speaker 01

  No, no, it's like I need to pre-end it. I'm not kind of, you know, it's the hindsight of this may be working. We'll see in next five, ten, five to ten years.

• 54:19Speaker 00

  Much flatter org.

• 54:20Speaker 01

  Much flatter org. So it works for us, might not work for all the companies. And there are some parts where like go to market, we still are trying to figure out what's the best way. But small teams, flatter org, and I think there are two paradigms, but like generally people being more technical. Or if not technical, even in non-technical teams, having a technical resource. So, you know, we will have a person in ops or in talent that will, we have effectively a tech lead for that team.

• 54:45Speaker 00

  Yes.

• 54:45Speaker 01

  That helps them automate a lot of that, that, that work and helps up-level the rest of the team too.

• 54:50Speaker 00

  Yes.

• 54:50Speaker 01

  So there are kind of two parts that are helping.

• 54:52Speaker 00

  Okay, so talking through this in talent or something like that, is it that you are building your own software? Or other companies might have bought software like a Workday or a Greenhouse or something. Is it that they are using the existing software you have better? Is the process that would be spreadsheets in a traditional company are built with software? How do you kind of use the software in these sorts of organizations?

• 55:16Speaker 01

  Yeah, like there, sometimes, but we still use a lot of like the, the traditional vendors. Like one pattern is of course eliminating everything, like making the, the data explorable for you to be able to interact with it. Of like who's in the pipeline, what worked, who does the best references, like all of that, all of that works, so you can double down on that. But two, it's frequently things that you manually do, that a lot of the, like there is a gap between where, where, where the agents are today versus what you could do if you have the technical skill set. And a good example is like, how do you scrape all the right profiles to be able to reach out to the right candidates? So you like analyze whether it's, you know. How much I should want to say, but but I try to detect specific things that we know worked.

• 56:01Speaker 01

  So you bring that across to the to the to the to the people. On go to market side, like there's just so many things you can do with with with additional amplifiers. You know, it goes from understanding what case studies are relevant and creating a good pre read for you before you go to the meeting. Through creating the A I S D R experience that we spoke about to creating an entire deck experience. So you have like a pre populated deck with the right numbers that is customized to that customer, which you want still the person to go through and and and develop. But ultimately is is in there. So there's plenty of those additional things that you know will amplify the work of the people around. Potentially replace some of those those easier tasks that are done.

• 56:46Speaker 01

  And then there's like, you know, we we wanted for people to explore the culture at eleven left. So created a voice agent that people can speak with and see what's the culture, but also get prepped for the interviews. I think across many of those teams, like additional additional benefit of of what they can do. Interesting piece. So of course, in Ukraine, we're done going work. They need to rethink a lot of how their development, their systems, their support works for the citizens across the country. And people are in the in the war zone. They don't have the same access to the information. They can't rely on the same phone lines. They cannot rely on the same physical services around the country. So they've developed effectively a central.

• 57:22Speaker 00

  Your employees in the Ukraine.

• 57:24Speaker 01

  We we we had a few, but they reached out because they were developing their central map called Dia. They developed it over the years, but now with war, they were double downing of how this can doubling down on how this can be a way of supporting the citizens. And of course, there's a easy part of like how you create a first agenda government where you have a help with the benefits and what's happening on the front line or education. So that's delivered to everyone or healthcare so you can book your your check up or appointment. So like how you create all of that. And of course, we we we traveled to Kiev. We worked with them on on bringing that and making that available for voice so everybody can access it. But the thing we've learned while being there was that model of what we speak about where you. Have technical resources in each of the, each of the teams.

• 58:08Speaker 01

  They actually have the same in every of the ministries. So every ministry had technical resources working on creating that agentic version of their work. And then it was like a central digital transformation team that would like assemble this all together to deliver that for the central citizens support, which I thought was brilliant.

• 58:25Speaker 00

  That's very tech forward by Ukraine.

• 58:27Speaker 01

  So tech forward, like the most advanced set of work we've seen. So we got a little bit validated, like, okay, maybe technical resources in each of the teams is a good idea. And that works heavily for us. And, and you know, you, you mentioned some of the other parts, like do you hire the senior or younger? The like main thing we try to filter for. Of course, the culture piece is so important. You can scale people. But it's scaling culture is much harder. So, so, so like you want to optimize for that being right. And in our case, it's first principles, taking ownership, striving for excellence, but staying humble. And the main thing that's kind of in that ownership part that I think works well for the AI world is agency. Like if you are, if you have that agency to explore, regardless of where you are in the experience cycle, it's, it's going to be a tremendous amplifier to your work.

• 59:13Speaker 00

  My biggest takeaway from all this has been that around agency where I feel like high agency people are the winners of the advances in AI and within organizations, low agency people will lose out.

• 59:28Speaker 01

  Yeah, completely agree. I, I probably the most proud thing that Pieter and I are is as we scaled eleven labs, the people that are at eleven labs has been like just the culture and seeing the expansion of the culture where culture builds the company now rather than any single person or any single product builds the company. That was probably the biggest validation and, and happiness. And there is a kind of the other angle of that where I think people are like striving to be incredible in their craft, in their work. But at the same time, have fun in a lot of their work and that kind of combination of, of, of, of agency and, and just enjoying what you do is, is probably the best, the best thing we've been able to do today at eleven labs.

• 1:00:09Speaker 00

  Well, it sounds like a really fun stage, like we were saying, interesting research breakthroughs, really fast growing business. So I'm sure you're enjoying it. Andy, thank you.

• 1:00:16Speaker 01

  John, thank you so much.