AssemblyAI CEO Dylan Fox on 120 Million Voice Conversations a Week

Dylan Fox is the founder and CEO of AssemblyAI, the voice AI infrastructure platform behind AI notetakers, medical scribes, drive through ordering, contact centers and humanoid robots. AssemblyAI went through Y Combinator's first AI batch in 2017 and now serves over 1 million developers with roughly 100 million API calls a day. We get into the volume numbers, why weekly conversations are up over 800% in three years, the three macro trends pulling voice into everything, why coding agents turned small businesses into API customers, how data mixture drives 75% of model quality, the humanoid robot speaker problem, on device models, and the disclosure question the entire voice agent industry is avoiding. "It's very easy to optimize for a public open source benchmark. It's very hard to optimize across all these real world applications." We cover › 120 million weekly conversations and 4x YouTube's daily voice volume › Coding agents expanding the TAM by 100x › Context aware voice models and the McDonald's ordering problem › Why humanoid robots can't disambiguate speakers › Whether voice agents should disclose they are AI Dylan Fox: https://x.com/YouveGotFox Molly O’Shea: https://x.com/MollySOShea Sourcery: ⁠https://x.com/sourceryy 𝐄𝐏𝐈𝐒𝐎𝐃𝐄 𝐋𝐈𝐍𝐊 YouTube: https://youtu.be/xAF7HkX2DXI 𝐒𝐏𝐎𝐍𝐒𝐎𝐑𝐒 • Brex https://brex.com/sourcery—The modern finance platform, combining the world’s smartest corporate card with integrated expense management, banking, bill pay, & travel. https://brex.com/sourcery • MongoDB https://mongodb.com/ai–Millions of developers and more than 65,200+ customers across industries, including ~75% of the Fortune 100, rely on MongoDB for their most important applications. With integrated capabilities for operational data, search, real-time analytics, & AI-powered data retrieval, MongoDB helps organizations everywhere move faster, innovate more efficiently, & simplify complex architectures. https://mongodb.com/ai • AssemblyAI http://assemblyai.com/sourcery–Millions of developers use AssemblyAI to power their voice ai applications and features. One API gives you access to best-in-class speech-to-text, voice agent, and speech understanding models for both pre-recorded and real-time audio. Granola, ClickUp & HeyGen are scaling with AssemblyAI - get $50 of free credits today at http://AssemblyAI.com/sourcery http://assemblyai.com/sourcery 𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒 (00:00) Dylan Fox, Founder & CEO at Assembly AI (01:06) AssemblyAI's voice traffic just beat YouTube by 4x (03:33) $100K in GPU credits that started it all (06:39) Why Voice AI is inflecting right now (10:11) AssemblyAI's infrastructure-first strategy (12:41) Handling 120 million weekly calls (14:49) How AI agents are rewriting Internal Ops at AssemblyAI (17:05) The Sovereign AI problem nobody's solved for voice (18:26) Is the Keyboard and Mouse finally dying? (22:33) The problem Humanoid Robotics hasn't solved yet (25:14) The real problem behind translating languages (29:27) Can AI actually talk to animals? (31:24) The real reason Open-Source benchmarks lie to you (34:20) Building websites in chat rooms as a kid (36:48) On-Device AI models are about to change everything (43:03) The people who shaped Dylan's career

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The amount of weekly conversations that Assembly handles through our APIs every week is up over 800% over the last three years. On a peak week, there'll be something like over 120 million voice conversations going through our platform. Over 2 million hours of voice was a little over 4X the amount of daily volume that's going to YouTube. So they're getting more accurate, they're getting faster, more controllable, more capabilities. The newest voice models that we just launched, you can give them context about the environment they're operating in. If you drive through a McDonald's and you order with your voice, that voice AI system has no clue that it's taking a McDonald's order. But with our models, you can give them context on, hey, you're taking a McDonald's order. You want to focus on the person that's ordering, ignore the kids that are screaming in the background.

• 0:41Speaker 00

  And so we're seeing this huge inflection now, like you have enterprises, small businesses that can build with API infrastructure now. Our TAM has just increased by 100X because we're not just selling to engineering teams within product companies. It's now like anyone.

• 1:06Molly O'Shea?

  Dylan Fox, welcome to Sorcery.

• 1:07Speaker 00

  Yeah, thanks for having me here.

• 1:09Molly O'Shea?

  So we're gonna have a very fun conversation on voice. We were just gabbing off camera about how much I love voice and how I've used it to start this podcast and do all these interviews. Specifically, like use the transcripts that I capture from recordings into different kinds of formats. I started it for investing and so I would take that, I would repackage it, I would make like open source expert calls and open source memos and all this kind of stuff just from capturing that data. Now, you have built the infrastructure for how that even works, so I want to get into that. I want to get into everything from the business model, how you've built out the products. You've been doing this for a very long time and there's a lot of new entrants to the category.

• 1:51Molly O'Shea?

  So it'd be great to learn what is BS and what's not.

• 1:55Speaker 00

  Yeah, yeah.

• 1:56Molly O'Shea?

  As well as like areas that you're most excited about. But before we start, I guess it'd be great to kind of capture and illustrate just how big Assembly AI has gotten, how much data you train on.

• 2:08Speaker 00

  Yeah, so the, the demand for voice applications and the applications that companies are building on our infrastructure is, over the last two years in particular, really started to explode. One stat I was just looking at before I came over here was like, the amount of weekly conversations that Assembly handles through our APIs every week is up over 800% over the last three years.

• 2:32Molly O'Shea?

  Oh my god.

• 2:32Speaker 00

  And so now, you know, on a given week, on a peak week, there'll be something like over 120 million conversations, voice conversations going through our platform, over 2 million hours of voice, which as of December of this, of this past year was 4X or a little over 4X the amount of daily volume that's going to YouTube. Which when I found that was like, wait, I need to double check this because, you know, that can't be right. But yeah, the, the volume is huge. There's, you know, almost 100 million API calls a day coming against our API. About a million developers, a little over a million developers on the platform now. And, and, you know, 40, 40% of those developers signed up to the API last year.

• 3:15Speaker 00

  And so while we started this a long time ago, it's really over the last like two, three years where voice and it's accelerating, inflecting voice is becoming just a core part of software and increasingly hardware too. And so we're seeing that through our platform because we're the infrastructure really under, like, under all of it.

• 3:34Molly O'Shea?

  So if you look back over time, you guys went through YC.

• 3:38Speaker 00

  Yep.

• 3:38Molly O'Shea?

  Did you think that this would be the reason why voice would take off and people talking to their phones and talking to their computers and that kind of thing? Like, what did you think?

• 3:48Speaker 00

  Yeah, I mean, I did, which is why, you know, I, I've spent so much time on this because we were actually the very first AI batch in YC when we went through YC. And so it was Daniel Gross, who, if you know of Daniel now at Meta, he started the AI batch at YC back when we went through in 2017. And it was me and like five other companies and we got, you know, $100,000 in GPU credits. That was like our perk, which, which now seems like cute, right? It's like, wow, you know, that's nothing. But back then it was like, wow, you know, $100,000 of like Nvidia K8, K80 usage, like this is amazing. But the, you know, the, the AI ecosystem back then was just like in its infancy.

• 4:30Speaker 00

  Like I was going to the very first like TensorFlow meetups, just to put it in perspective, like it was, no one was really using AI in production yet. But for me personally, I had got an Amazon Echo and was like really into voice interfaces and talking to this hardware that worked well. Because my experience with voice prior had been, everything was terrible. And so then I went looking to find like APIs that I could just in my, you know, spare time back then like build with and I couldn't find anything and all the technology kind of sucked at the time. But I felt like, okay, over the next 10 years, this is going to get so much better. And when it does, if it's easily accessible to developers like a Twilio or a Stripe,

• 5:13Speaker 00

  you're going to have so many creative people build really cool things. And so now, you know, that's what gets us excited when we see, you know, our customers like Granola, right? Granola building these like amazing products that are helping companies and professionals like run their businesses. You know, we have customers like Tolans that are building, we were talking about it before, like AI companions that hundreds of thousands of people are talking to for like hours every day in the healthcare space. Aethelis, Camer, it's a big health tech company, building AI scribes for doctors. And it's really cool to see what people are building. So I think that, to go back to your question, like, I always viewed this journey as similar to self-driving cars where, you know, it wasn't a question of like, oh, is the product market fit going to come?

• 5:57Speaker 00

  It was like, no, the technology just sucks and as it gets better and better, you're going to continuously cross these thresholds where you open up more, more tranches of the market. And that's still happening and there's new macro trends that are happening that are accelerating, accelerating that further beyond just the like core technology getting better. I can talk about those, but that was very much like the, like thesis behind the company, right? It's like for me, it was like, this technology is going to get better. It's going to enable like so many new applications and voice to be something you can really build with and like a core, a core like modality across, across, like in our lives. And it just felt like something really cool to work on.

• 6:39Molly O'Shea?

  So what are those core macro trends?

• 6:41Speaker 00

  So there's really three things that we see that are causing, you know, everything, all our metrics like inflect. The first, which is what we control, is voice models are getting just much better. So they're getting more accurate, they're getting faster, more controllable, more capabilities. You know, the newest voice models that we just launched, for example, you can give them context about the environment they're operating in. Giving example of that is like, you know, today, if you drive through a McDonald's and you order with your voice and there's a voice AI system there, that voice AI system has no clue that it's taking a McDonald's order, right? It's like, it could be, it has no context, it's like fresh.

• 7:22Speaker 00

  But with our models, you can give them context on, hey, you're taking a McDonald's order. You want to focus on the person that's ordering, ignore the kids that are screaming in the background. And those types of capabilities make them much more accurate. And then they're also faster and they're lower cost, so they can be more widely deployed. So you have these voice models that are just getting better across these key dimensions. And that's like what we control. But the other two macro trends, the, the one is, the, the number two in this list of three is like, you have this ecosystem of AI infrastructure getting built out. So you have these reasoning models, you have vector databases, you have other AI models across other modalities.

• 8:03Speaker 00

  And so now when you're, you know, a company wanting to build something like drive-through voice ordering or AI note-taking or healthcare, you don't just have the voice models, you have like the rest of the infrastructure you need to go build the thing you're trying to build. And that has, you know, been accelerating over the last couple of years. And then the third, which is new, is coding agents. So we're now seeing companies sign up, spending a lot of money on the API. They're like, not even two years ago, like 18 months ago, like I would never think of them as our target customers because we're API platform. So we, you would think like, oh, you're selling to engineering teams, product teams.

• 8:47Speaker 00

  But now everyone's an engineer because they have coding agents, right? And so like, we saw this like small business, like lawn care chain sign up. And, you know, we're like, what do they do with the API? And a lot of these small businesses are, are automating parts of their back office using lovable or Claude Code or Cursor or whatever, Replit. And, you know, those coding agents are building on our infrastructure. And so we're seeing this huge inflection now of like, you have enterprises, you have small businesses that can build with API infrastructure now. So I, I think about this as like, our TAM has just increased by 100X because we're not just selling to engineering teams within product companies.

• 9:31Speaker 00

  It's now like any, anyone. And we're seeing some global enterprises now come in and build their own software for a bunch of different use cases and they're doing that on top of our infrastructure. And when you, when I go talk to them, like, why are they doing this? It's like, well, they have. You know, two people working on this with a bunch of coding agents. And so those three things, it's the voice models are getting better, like the broader ecosystem of AI technology. And then the third is anyone can build this stuff now with, because of coding agents are really causing voice to be way more widely deployed and across just like so many different use cases.

• 10:11Molly O'Shea?

  So for people that are new to this space, it would be helpful to explain your differentiation between the likes of like 11 Labs.

• 10:21Speaker 00

  Yeah.

• 10:21Molly O'Shea?

  Obviously, Sierra is in a different bucket, too. But like, how do you fit in within that world?

• 10:26Speaker 00

  Yeah, so we are 100% focused on voice AI infrastructure. So we don't do anything at the application layer. Where we focus, you know, is the models. So we create models that are amazing, voice models are amazing for all those use cases I spoke about. So healthcare, we have medical focused models, drive through voice ordering, contact center and note taking. We then build out the inference around those models. So to handle 4X the amount of YouTube volume on a, on a day, 120 million conversations a week and growing over 100% year over year. There's a lot of infrastructure we have to build out to make sure everything scales, is available, is

• 11:10Speaker 00

  like super fast, is low cost for customers. So we build out a ton of infrastructure on inference around our models. And then we also do a lot around the orchestration layer. So if you want to build a voice agent, if you want to understand speakers, if you want to translate data, we have a ton of, a ton of software at the orchestration layer that, that companies can leverage. And then, you know, above that, it's we have this amazing developer experience, agentic coding experience. So agents can easily build with our stuff. And when we say infrastructure, I think a lot of times people think like, oh, you're just creating the model weights, right? Like you're just creating like models. That's a part of it. But those other parts are equally important to us.

• 11:54Speaker 00

  The inference, the orchestration, the developer and agent experience. And so we're 100% focused on that. And so, and across like all the different use cases that we focus on and making sure everything's like tailored for those. And so the reason, you know, most AI note takers are using assembly as the voice infrastructure is because our models are the most scalable, right? If you have, you know, tons of people, millions of users that you're doing live note taking for, we have the most scalable models in the market that, you know, are super low cost, high accuracy, very fast, can burst to huge amounts of traffic. You don't have to like stand up servers as a customer. So all of that is like the area that we're focused on and we're enabling companies to

• 12:39Speaker 00

  build on top of that.

• 12:42Molly O'Shea?

  How did you get it to become so efficient?

• 12:47Speaker 00

  You know, years of engineering work, people like Ben at our company, we have an amazing team of engineers, of researchers that just operate so closely to customers that they really understand like how this stuff is being deployed. And that's the biggest difference between us and like a lab at a bigger company. So, you know, we have a lot of, a lot of researchers, research engineers that will come to assembly from a bigger company. And they're so far removed from the customer. So they just create their model, they benchmark it, and then it's like, okay, I'm done. But you have to know, all right, who are the customers? How are they using this? What do they care about? What errors are, what errors like break their application and what errors like don't matter? And then how do you optimize both your models and your infrastructure for that?

• 13:31Speaker 00

  So we, we've just spent years and years building out the infrastructure to make this stuff work. So infrastructure that's like cross region and cross cloud and stuff that just will really scale. But it's a, it's probably like half our work is spent just making our infrastructure more and more scalable.

• 13:51Molly O'Shea?

  This episode is brought to you by Brex, my favorite. You become what you spend on. And I refuse to spend my time on work that shouldn't exist. Expense reports, receipt chasing and manual closes. The companies building what's next from Vercel, OpenAI, Anthropic, Granola and Deepgram all made the same call. They all run on Brex. Brex is the intelligent finance platform that combines cards, expenses and banking into a single stack with agentic finance built in. AI agents that handle expenses automatically, enforce policy before spend happens and close your books in minutes. That's why Sorcery runs on Brex, so I can spend time on building and not

• 14:34Molly O'Shea?

  busy work. It's time to get Brex AF. Learn more at brex.com/sorcery. That's B R E X dot com slash S O U R C E R Y. Bye. I feel like between like, well, the SaaS apocalypse is one thing, but even leading up to this point with AI helping companies restructure, whether it's on the kind of internal side with developers or on the outward facing side in marketing, that kind of thing, I'd be really interested to hear how you've re-architected the company internally to do that.

• 15:14Speaker 00

  Yeah. It's a great question. I mean, we're, we're AI native company. Like everything we do is, is AI native. And so what I mean by that is like, I have, even me personally, like I have my own agent that like I built, right? I call it like my Dylan Claw on my computer. And it has my meeting notes and it can make slides for me and it can do, do everything. And so I just on boarded someone new to the company. And I was going through my slide deck and it's like, oh, I was talking through a part of the deck that was like a bit clunky. And then because my agent has access to the slides and can make new slides and was, you know, has the notes from that meeting powered by assembly. I was like, hey, go create a new version of the deck that is a bit easier to walk

• 15:59Speaker 00

  through and like, look at the, the transcript from that most recent onboarding to make some changes. And I want to like merge these two decks together. And then like 20 minutes later, it's done. It's perfect. And I'll have to edit it. We have a, a agent that has access to all the company information. So our metrics and the whole company uses it. It can submit PRs and GitHub across our products. And so I think like, you know, we've always stayed very small, you know, we're only, I think we'll be 80 people at the company pretty soon. So we're, we have huge scale for the size of company that we're at. And we try to be intentional about that. But, you know, I think, I think for us, like, we've just really leaned in to try to, you know, use AI like

• 16:42Speaker 00

  across everything. Like, you know, the most recent example is we had Claude rebuild our whole website off of Webflow and just deploy it on Vercel. And now anyone, you know, people in Slack can just make changes to the website. So you get off a customer call or like, oh, this part is, needs updating. And it's like, boom, you can just change it. So it's things like that that have, I think, helped us to like leverage AI across the company.

• 17:05Molly O'Shea?

  Sovereign AI has been a huge topic and you mentioned some of your customers that are in categories that are a bit more critical and sensitive for data. But we are seeing this across the board because you can get copied by the labs.

• 17:18Speaker 00

  I mean, yeah, so you can deploy our platform, you know, on premise. You can run it all self hosted and that's totally private. And we do see an uptick in that for sure. I think that, you know, it's this mix, like in, voice is so early where, you know, we, we, the models and the capabilities are getting better so quickly that it's still hard to like, to branch out. You know what I mean by that? Like, like we've seen some customers, maybe six months ago or a year ago, they'll take an open source model and fine tune it or something. And then, you know, they're now like, okay, this is like shit. This is like really behind and it's like nightmare to maintain and I don't want to work on this. And so they'll, you know, they'll, they'll come to

• 18:02Speaker 00

  assembly. I think voice is much earlier where we haven't seen as much of that as you're, you're probably talking about in like, yeah, if you're a bank and everything's going into Claude or something, it's like, you know, wait, all my proprietary kind of knowledge is going out. But we are working on ways for companies to create like their own custom models, deploy them on our inference or run them, you know, on their own cloud. But it's, it's earlier there for voice, for sure.

• 18:26Molly O'Shea?

  I was thinking about it because when we were at the Raze Summit, this was a hot take that Max Cook, who's a sector head at Code2 said, he thinks the keyboard and mouse is over. We've been through so many different form factors of computers and the human like machine interface that's like, no, we gotta close the books on the computer keyboard and mouse.

• 18:46Speaker 00

  It's funny when you go into offices, everyone has like the gooseneck microphones and they're like whispering over. Um, I

• 18:54Speaker 00

  I think you're gonna have, so here's like what I genuinely see like, voice is a, I would think of the word like modality or interface, but I don't even think there's capture, like voice is a, is a reliable form of, let's just call it like data capture now. Like for the first time probably ever in the past year, it's like a reliable form of data capture. And so what does that mean? That means like you can actually talk to your computer now. You can actually have it listen in, you have your computer listening on your meetings and take notes for you. Doctors can use it during your patient encounter to update the charts. We have a startup. It's called Zero AI. I don't know if you've heard of them. They're building on the, on the platform.

• 19:34Speaker 00

  They're creating an AI for field service tech. So if you're a plumber, HVAC technician, they have an app you can run and it will listen in on your service visit and then give you feedback after for like sales coaching. And it's helping those, those field tech salespeople be so much more successful because like they're actually getting feedback and they never were before. So they're earning like 20% more take home pay or something. You know, we're seeing it with like toys and with hardware where you can like talk to toys now, which is cool for kids. You know, me as a parent, seeing that stuff is really cool. So voice is this, is this reliable form of data capture. And I think when people talk about voice AI, a lot of times they're talking about like, like AI voices, right?

• 20:19Speaker 00

  And that's a part of it. That's a big part of it. Like you have voices that are sound great and they're amazing. They're so much better. But the other part is like, can the computer, the machine understand you? And that voice capture part is that reliable? That's solid? Does that work at scale? But that, that is the case now. And so I actually think you're gonna see, you know, like touchscreens didn't replace keyboards. It's like you have both. You have the, you know, on a coffee machine, like there's the rotary dial, like, and there's the button, the touchscreen menu. And I think voice is gonna be another dimension. That's probably the way, the way I think about it. Voice is gonna be a new dimension. And, but it's gonna be an additional dimension. And so for me, like I like to type, but I also like to talk to my computer. I like to do both things.

• 21:04Speaker 00

  And I think for, you know, where I get excited is like consumer electronics that you're gonna be able to talk to. So like your TV, like it's still crazy that like most TVs, you can't just be like, hey, put on, like open up Apple TV and put on, you know, this, this episode of whatever.

• 21:21Molly O'Shea?

  They are always listening. I don't know if you saw the succession.

• 21:23Speaker 00

  No, no, I didn't see that. No.

• 21:25Molly O'Shea?

  Always listening.

• 21:26Speaker 00

  Yeah, but like, it's crazy that doesn't work yet. And so I don't think that means that computers are just gonna be like a screen and you just talk to it. Cause I actually think we'd get tired of talking, but, um, but I think it's gonna be a dimension that's added to everything. And so that over the next couple of years will happen. And I, I think that's really cool. Cause I think that will allow for us to be freed of like our addiction to screens, right? Cause your input to most computers now is like the, this like touchscreen. It's like smaller and smaller touchscreen that like just captures you. But when, but with voice, that enables more passive hardware and more passive computing experiences where

• 22:11Speaker 00

  it's like you can talk to things and you can be walking around and you can be free from being like a prisoner to the, to the device. So that, that's what I think is really cool. And that, that's for sure coming. We have some like big consumer electronics companies, like actively engaged with us, like implementing our technology right now for things like that, which is like really cool.

• 22:33Molly O'Shea?

  What about humanoid robotics?

• 22:35Speaker 00

  Those two, you know, I think that, you know, every time I think about like the Neo or the figure, I think about like my, I have a four-year-old and a three-year-old and I think about like one of those in the house with them. Because we have a Matic robot and they like love to, to, to mess with the Matic robot. They broke one. It was like, cause they, they will always, they'll always go run over to like turn it on. Cause they've realized like you can press the button and turn it on. And then they like sit on it and they mess with it. They try to like ride it around. And yeah, I don't know what the weight is of a Neo or something, but, uh, but yeah, I think, I think you'll have humanoid robots. Obviously the way you're gonna wanna interact with the humanoid robot is by talking to it. But this is where what we do is so important is because, I'll tell you one of the main problems the humanoid robots face today.

• 23:21Speaker 00

  Cause a lot of them are using our APIs.

• 23:25Speaker 00

  If you have three people standing next to the robot, it doesn't know who, who to listen to. And it has a hard time disambiguating who's saying what. And so it all just gets kind of jumbled and merged. This is a big problem.

• 23:35Molly O'Shea?

  Is it using listening data, not video data?

• 23:38Speaker 00

  Yeah, exactly. And even if it's video, like if you're not facing it, maybe you can't see, right? So like, it does have the cameras, but you know, you, you really need to be able to, like for you, like if I, if I, for me, like if I close my eyes, I can still tell, you know, who's saying what, right? But for voice agents, even if they're over the phone, this is a big problem with them today. You know, I, I last week or last month, like called a voice agent. And I called a restaurant. It was a voice agent that answered. It was like AI that I'm talking to. And I know it's an AI, even though it tries to trick me, cause like I build this stuff. But my son's in the back and he's talking and it's like tripping up the voice agent. Cause it doesn't know that that's a background speaker. It just hears speech and it's like, I need to respond to this. And those problems are still not solved yet.

• 24:22Speaker 00

  These voice models, and that's going back to like the sovereign AI stuff. I think right now we're still in the phase of like, I need this stuff to work. And I just need to like, cause there's, there's such an opportunity if you can be the first to get it to work. And we're still in that like get this stuff to work phase. Probably that will be the next, you know, 18 months. And, uh, and yeah, I think that like, yeah, across humanoid robots, consumer electronics, like you're gonna see voice as this dimension that you just like expect. And what, the example I think about is, you know, when you see pictures of like kids, like trying to like swipe on TVs or something? Cause they just expect everything is a touchscreen. I think it's gonna be similar where, you know, in like

• 25:05Speaker 00

  five years, kids are just gonna like talk at things, expecting them to be able to, to understand them. And that's like the, the like shift with like computing in general that we're gonna see.

• 25:14Molly O'Shea?

  What has it been like to expand the models into translating different languages?

• 25:20Speaker 00

  That's a hard, it's a hard problem and it's less about,

• 25:25Speaker 00

  it's less about like the science of it. And it's actually more about like the culture of it, right? So like, our head of research is from, is Japanese, from Japan. And there's like a lot of, you know, there's, there's a lot of like opinions about like how, you know, like how you like, think about it as like policy alignment, policy alignment for how you want it to show up for a native speaker. My wife is Danish and so I always have her test our Danish models. And I'm like, well, is it good? You know, like try it. And she's like, yeah, but like this, you wouldn't really do. And it's like these details, uh, but they're important details. And so I think it's less about the, the science and it's more about can you get those details right?

• 26:05Speaker 00

  And more and more you wanna have a single model and that can do all these different languages, right? And so our latest model, for example, it's a single model. It can do 20 different languages. You don't have to tell it. It just will figure it out. You can, it can switch across those languages in real time.

• 26:24Speaker 00

  But if, you know, if you, if you fix this one thing, then you might like break this other thing. And so it's that like equilibrium that you're trying to make in these models and, um, and you know, that you have to have this like kind of deep understanding of each language. And it's why actually, if you look at like local voice AI vendors in, you know, certain countries, like they actually are typically the best. Because it's, you know, they speak the language. They understand it and they can more quickly identify like which data is good and stuff to train on. That, so that, that's a big part of it is like that, like policy alignment and like the language expertise per language that you really need.

• 27:01Molly O'Shea?

  Would you acquire those companies? Like how would you get that?

• 27:05Speaker 00

  Yeah. I mean, a big, a big part is like, you know, we, um, we try to build a team of those experts. We have a lot of customers that, you know, help out and service those experts and give us that feedback. We partner really closely with customers. A lot of them like are not working in sensitive applications. So they opt in to, to letting us use their data to improve models and their feedback, which is great. So it's a mix of all those things.

• 27:32Speaker 00

  But that's, that's really where, you know, there's still a lot of opportunity to differentiate at these models. And I think for voice, a lot of people will think like, oh, voice, like what people have been telling me forever is like, oh, voice is, you know, isn't that solved? Like, isn't that just like commoditized technology? But like, it's definitely not because there's, there's so many gaps across languages, across different, different verticals and applications and domains that there's big rooms for improvement and still.

• 28:02Molly O'Shea?

  In AI, you need to know MongoDB. The database platform developers love and built for the agents you're running. MongoDB stores, searches and reasons over your data in real time. With vector search and embeddings from Voyage AI, all in the same system. No separate pipelines, no stitching together 10 different tools. It's why 75% of the Fortune 100 and leading AI native startups run on MongoDB. Build and scale from your first user to billions of vectors. Go to MongoDB.com slash AI to learn more. That's MongoDB.com slash AI to learn more. Bye.

• 28:43Molly O'Shea?

  Assembly AI is a voice AI infrastructure layer millions of developers build on. They build the industry's best speech to text, voice agent and speech understanding models that serve as critical infrastructure for companies like Granola, Haagen, Ashby and ClickUp. Their speech to text models lead the industry in accuracy and quality. And their speech understanding models help you go beyond transcription by uncovering insights, identifying speakers and highlighting key information from voice data. You can get started today at AssemblyAI.com slash sorcery and get $50 of free credits to start building voice AI products. That's AssemblyAI.com slash S O U R C E R Y. I have a hard question.

• 29:28Speaker 00

  Yeah.

• 29:29Molly O'Shea?

  How far away? I'm gonna try to deliver this without smiling.

• 29:37Molly O'Shea?

  How far away are we from translating dogs and cats?

• 29:41Speaker 00

  Dogs and cats, yeah. I don't think voice will be the interface. I think it's gonna be, I think it's gonna be more like, uh, like MRI scans or something like some, some, sub vocal. That, that's the thing I really think about here. Will we talk about computers? I think probably the ultimate interface is, is, um, it, it's just reading your mind, right?

• 30:00Molly O'Shea?

  That's true.

• 30:00Speaker 00

  So you just plug in some USB stick and like put it on your hand and then it just reads your mind. And then it's just like, it's unlimited bandwidth to the machine. So that, that's probably the, the solution for the dogs and cats.

• 30:11Molly O'Shea?

  I don't know. We both said, we both watched Project Hail Mary and he did translate an alien.

• 30:19Speaker 00

  Yep.

• 30:20Molly O'Shea?

  So.

• 30:21Speaker 00

  I know. Yeah, I was telling you this before, but you know, as I was watching that part, I was like, this is way too easy for him to train this model. Basically like you would need like a lot more data, uh, but yeah.

• 30:34Molly O'Shea?

  So where, where did he go wrong with that?

• 30:36Speaker 00

  With, uh.

• 30:37Molly O'Shea?

  With Rocky?

• 30:37Speaker 00

  With, with Rocky? Well, he solved it. I mean, he, yeah, they were able to talk. Uh, but yeah, he did. They, they did turn Rocky into more of like, as we were talking about, like a dog. That I think, uh, like a very intelligent spacefaring entity.

• 30:51Molly O'Shea?

  Yeah, that was a weird realization because I was like thinking about it. I was like, they both left their planets. I mean, Rocky's supposed to be like 100 or 200 years old.

• 31:00Speaker 00

  Yeah.

• 31:00Molly O'Shea?

  Right? And like, he's an adult. And he has a lot of responsibilities. But like, I mean, both of the characters were quite playful and I think that's why the movie was so great.

• 31:12Speaker 00

  Yeah.

• 31:12Molly O'Shea?

  But it was definitely super weird.

• 31:14Speaker 00

  Yeah, yeah, yeah.

• 31:15Molly O'Shea?

  Why was Rocky the dog?

• 31:16Speaker 00

  Yeah, they made him like, like, uh, like Grace was like the, the like parrot. But it's probably good for, for viewers.

• 31:25Molly O'Shea?

  Now I'm like totally off tangent.

• 31:28Molly O'Shea?

  Okay, so I guess like to get deeper into how you train based off of the hundreds of millions of hours.

• 31:36Speaker 00

  Yeah.

• 31:37Molly O'Shea?

  Of data that you have, how does that really work?

• 31:40Speaker 00

  Yeah, um, my experience has been, um, it's really probably like 75% of it is like the data that you're training on. Like I would say for any AI model, it's like, there's always like, you know, like these, these like step functions and like algorithms and architectures and stuff, but the data is just so important. And so a huge, we spent a huge amount of time on data, trying out different data mixtures, training different models. And, um, you know, we release model updates like every couple of weeks. And that's the big benefit that our customers have when they're building on our platform is like, it's literally like constantly getting better. It's not like every six months or every year there's an update. It's like there are constant improvements going out.

• 32:25Speaker 00

  We have a ton of different model versions. We have a ton of different APIs, which we just launched like a different API today for different use cases for dictation and like push to talk type use cases. So we're constantly innovating on the model training part. So much of it's about the data. Is the quality good? Is it aligned with like what users want? And, you know, I'll just give you an example of that, like for, for the speech to text task, like some applications don't want to pick up the background speakers. Some do. So like, we have some customers who are, you know, looking at police body cam footage for like legal discovery. They're running that through our platform and they want to know like everything that was said by every single speaker, like leave nothing out, right?

• 33:10Speaker 00

  But if you're McDonald's taking a voice order, you don't care about anything other than like the person in the front seat placing that order. If the kid in the back screen screens like an extra large Frosty, it's like, you know, you, you want to be smart enough to ignore that. And so there's all these different expectations of these models. And that's where I think the alignment of data to the use cases is really the process knowledge that, that is required to build like really good technology. And it's why like, if you go online, you look at some of these open source benchmarks, it's like, oh wow, all these models are performing similarly. But it's very easy to optimize for like a public open source benchmark. It's very hard to optimize across all these real world applications. And so we spend a ton of time on evals.

• 33:52Speaker 00

  We have like a whole team internally that works on evals, constantly evaluating our models across like a million different metrics, million different data sets, across all these languages. And it's so much of it's about the, the data.

• 34:04Molly O'Shea?

  How is the team constructed? How many employees do you have?

• 34:07Speaker 00

  So we're, yeah, we're almost 80 people at the company. Uh, it's pretty much like almost entirely like product research and engineering and forward deployed engineers that are just like working really closely with customers.

• 34:20Molly O'Shea?

  We didn't really go into your background too much.

• 34:22Speaker 00

  Sure. Yeah, yeah, yeah. I mean, it's interesting. I sometimes I'm like, wow, I've been doing this a long time. I like, really where it goes back to is I taught myself how to program in college. And actually like way before that, as a kid, my brother would build computers like in the, in our basement. And I would be on like, like IRC, if you know of IRC, like.

• 34:43Molly O'Shea?

  Totally, yeah.

• 34:44Speaker 00

  Yeah. No, these like, I would be playing these like, uh, video games and in these like chat rooms with like random people and like building, you know, websites for these like teams we'd make. And so I was like, always on computers and then kind of got into programming and taught myself how to code in college. And I loved the idea of being able to build something and these like, these like hard, like hard problems in computers. So that got me into like natural language processing and machine learning. And that's where I really got interested in this like natural language, understanding voice area. And it just turned out to be this really cool emerging space where, you know, we get to work with all these like cool innovative companies and build new technology.

• 35:27Speaker 00

  And so for me, that's the really cool thing, like building this. I think every company has a DNA and like a reason why they started, uh, or why it started. And for, for me, it was like, I love developer tools, like I'm developer by background. Like that's kind of always what I was passionate about. And giving like other people tools to build on and seeing like what they can do with it is so fun and cool. And every time we see, you know, some startup builds something on the API and then they like raise the series A and stuff, it's like really cool to see. At some point I want to try to, uh, figure out like what's the cumulative total funds raised of like all the startups on Assembly?

• 36:11Speaker 00

  Cause just to show like how much demand is going for, to, towards all this stuff. But yeah, my, my background's really about building things.

• 36:17Molly O'Shea?

  So that's why your infrastructure layer versus application.

• 36:20Speaker 00

  Yeah, exactly. I think, you know, companies have their focuses and their, and their DNA. And for us, it's like infrastructure and we're really good at scaling stuff. We're really good at creating technology and that's our, that's our DNA. And, and that's ultimately like what our customers are, are buying from us. It's like, hey, we're, you know, we're, we're, uh, this amazing infrastructure platform. We're like the AWS for voice capabilities. That's the, the like product vision that we have.

• 36:48Molly O'Shea?

  I know some people can't think this far in advance because AI has made things, time windows very short.

• 36:54Speaker 00

  Yeah.

• 36:54Molly O'Shea?

  But what are you most excited for in the next 12 months?

• 36:57Speaker 00

  Yeah, so, um, 12 months is, it's actually, I feel like it's easier to predict like 24, 36 months and it is 12. I think over the next year, we'll see a lot of, a lot more consumer applications, hardware and software, where voice is a core dimension. So toys, games, um, consumer electronics. And I think that's going to be really cool because I think that's gonna, you know, like, uh, one, like free us from our phones, two, create this more futuristic environment where technology like blends in more. You know, because right now a lot of technology, a lot of technology like,

• 37:42Speaker 00

  it's kind of like when you see a fridge that has the custom panels, you know, in a kitchen, it like blends in and you don't notice the fridge. It's like, where's the fridge versus like the fridge that's just like stainless steel and this like object there. And I think that, yeah, voice is a way to create like technology like that, that just kind of exists. Cause you don't need to be able to access it. You can like talk to it and it can talk back to you. So, um, uh, I think consumer is going to be an area where there's a lot of innovation. We're, for example, working on on-device models. So models that can run like on a phone or on a, you know, really low powered piece of hardware. You know, like a remote control for a TV or something.

• 38:26Speaker 00

  And that's going to expand a lot of, a lot of these applications. So that, that's part, and then.

• 38:32Speaker 00

  I would say there's something that like we all in the voice industry have to reckon with, which like, which is this, um, you know, there's this interesting difference where every text-based agent that you talk to, customer support, in an app, everywhere, at least my experience has been. I have not encountered a single one that's like trying to convince me that it's a human. It's always like, hey, I'm, you know, the AI SDR, I'm, you know, the AI support, you know, agent. I can help you with this. Voice is different where like today when you're building a voice agent that you're talking to over the phone or something, for the most part, you're trying to trick the,

• 39:17Speaker 00

  the goal is like to try to trick the human into believing that you're, that it's also a human. Right? You know, there's, there's exceptions to that, like Tolans that we talk about where it's very clearly a, you know, there, there, it's like a AI avatar, like you, it's like a character that you're talking to. Um, and I think that as an industry, that's something that kind of figure out. I think right now you're trying to trick the user to believe you're a human because, you know, that's the way to mimic intelligence. Everyone's used to these like really shitty voice experiences that just like don't work. So they bail right away. Right? And we see this. If our customers, if they're building a voice agent and you disclose upfront that you're an AI,

• 40:01Speaker 00

  people just hang up versus if you don't, people continue. Um, and I do think that, I do think that will change and I think we're going to want that to change because I don't know about you, but like I want to know, is this a human or a agent? I got this really weird call.

• 40:17Molly O'Shea?

  Would you act differently?

• 40:19Speaker 00

  Yeah, I would. I had this really, I had this really weird call and this is now like starting to trip me out, but we recently like had to move some stuff. So we're at a moving company and I got a call and it was, it was like, I was like, after I was like, that was a weird call. I was like, is that, was that, I think that was an AI, but it was like, it was like with an accent, the AI had an accent. It was like really trying to trick me that it was a human and it was just, the result was like, this is weird, uh, that I felt. And so I think that with voice agents in particular, um, there are something we still have to figure out about

• 41:01Speaker 00

  the UX for, of AI that you're talking to, to balance the, hey, this is intelligent, but I'm not trying to pretend to be a human and trick you experience. Cause if you ever talk to a voice agent and then like two minutes in, you realize you're talking to AI, it's just like a weird experience. And I think that's something that, that we still have to figure out.

• 41:25Molly O'Shea?

  I feel like at some point we should just like assume everything is AI though.

• 41:28Speaker 00

  Maybe. Imagine you call some like medical helpline and then you're like asking for help. Hey, when our kids were born, I used to call the like nurse line for the hospital to ask questions. And like, imagine now I'm doing that and then like two minutes in, it's like, you know, I realize I'm talking to AI and be like, wait, what is going on? Versus like, no, I want to call like and talk to like a human nurse, you know, I want to know, you know, I want the options.

• 41:55Speaker 00

  But I don't think we figured out how to balance that today. And I think today, when the mark, when the market and the industry talks about a good voice experience, they think like, oh yeah, it should like match like a human. And I think you can have good pacing and a good conversation. It can be natural, like with a robot. That should be, there's examples in, in, in science fiction where that's, that, that's been achieved. And I think we could, we could figure out a way to do the same.

• 42:23Molly O'Shea?

  So I know we covered a lot of topics. Is there anything that we didn't cover that you want to touch upon?

• 42:27Speaker 00

  I feel like we covered a lot of it, you know? Um, yeah. If you, uh, yeah, the, the new models that we're building that have the ability to take in context are like really cool. They're the first type of models that can, voice models that can do that. But no, we, it was, it was great to, to cover the full range.

• 42:46Molly O'Shea?

  It seems kind of obvious though. Like if you have models at McDonald's, wouldn't, they don't know they're at McDonald's? Like that's crazy.

• 42:52Speaker 00

  Yeah. Yeah. The voice models haven't been able to do that yet. And, um, and you know, we, we shipped like really like the first one that can a couple of weeks ago.

• 43:02Molly O'Shea?

  That's super cool. Yeah. Okay. So as we close out, there's one question I have to ask. This is a Brex question because they're all about performance, spending smarter, moving faster. I like to think that for personal performance, it's kind of who you surround yourself with. Some people say it's like the five closest people, like who's either mentored you, who's a close friend, like who's been inspiring? Who are those people for you?

• 43:25Speaker 00

  Yeah. I really put it in like three buckets. Um, uh, like family, friends, and then it's going to sound so corny, but like really our investors and, you know, on family, like my wife's an entrepreneur too. She's founder and that's been like a blessing. Cause you know, that, that, yeah, always able to like get advice from her and talk to her. Friends, you know, I've been able to, to meet, um, over the last couple of years, like other founders that are in the same stage and phase. And, um, that, that's been amazing. And I think having like finding peers

• 44:05Speaker 00

  that you can just like be super open with and transparent with is super helpful. But then in terms of investors, like we have an amazing group of investors that has really been along for the ride, you know? And I think about like Keith Block and Smith Point that invested in our company at our, our series C. Um, they're, you know, operators from Salesforce. They've started their own VC fund and like, they're just, you know, so I think like Steve from Excel, Steve Lauflein, Rebecca from Insight, like they're just always, um, uh, like pushing the company and pushing me and, in, in great ways. And they're, they're amazing people. So that I'm not even trying to like be cheesy when I say, oh my.

• 44:49Molly O'Shea?

  No, I kind of feel like that's a hot take. I feel like a lot of founders are like, you know, I don't need the investors. Like I'm good.

• 44:55Speaker 00

  Yeah, they're, they're great. And I mean, I think we've been really, really lucky to have a great group of, of people around the company. So yeah, I think friends, family, um, you know, investors have been, have been, uh, have been super helpful for me along the, along the journey.

• 45:11Molly O'Shea?

  Amazing. It's a good place to end it. Well, Dylan, thank you so much. I appreciate you sharing everything on Assembly and the early days to even the craziness that's happening now. And I hope pretty soon you can start translating dogs.

• 45:22Speaker 00

  Yeah, we're working on it. April 1st, come, come check it out.

• 45:26Molly O'Shea?

  People, yeah.

• 45:27Speaker 00

  Yeah, yeah.

• 45:28Molly O'Shea?

  Thank you.

• 45:28Speaker 00

  Yeah, cool. Thanks for having me on.

• 45:30Molly O'Shea?

  Huge thank you to the entire Raze team for an incredible event. And thank you to Brex, MongoDB and Assembly AI for making this trip and series possible. If you enjoyed this conversation, you're going to love the rest of the Raze series with Tony Kim from BlackRock, Scott Wu from Cognition, Andrew Feldman from Cerebras, Rodrigo Yang from Samenova, Michael Hurlston from Momentum, CJ Desai from MongoDB and many, many more like our hot takes that we did at a secret location that you can find on X, YouTube and Instagram. Subscribe to Sorcery on YouTube for more conversations with the people shaping AI and join the free newsletter. You can also do paid at sorcery.vc for weekly insights on AI, robotics, enterprise software,

• 46:16Molly O'Shea?

  consumer semiconductors. Did I say AI? AI again. And everything that's coming next, like funding announcements and all big things in tech. Thank you. Bye.