Pod Insights a16z Podcast Building the Next Generation of Conversational AI About this episode Inside the Code: Ankit Kumar (Sesame) & Anjney Midha (a16z) on the Future of Voice AI
What goes into building a truly natural-sounding AI voice? In this episode, Sesame’s cofounder and CTO, Ankit Kumar, joins a16z’s Anjney Midha for a deep dive into the research and engineering behind their voice technology.
They discuss the technical challenges of real-time speech generation, the trade-offs in balancing personality with efficiency, and why the team is open-sourcing key components of their model. Ankit breaks down the complexities of multimodal AI, full-duplex conversation modeling, and the computational optimizations that enable low-latency interactions. They also explore the evolution of natural language as a user interface and its potential to redefine human-computer interaction.
Plus, we take audience questions on everything from scaling laws in speech synthesis to the role of in-context learning in making AI voices more expressive.
Key Takeaways:
- How Sesame achieves natural voice interactions through real-time speech generation.
- The impact of open-sourcing their speech model and what it means for AI research.
- The role of full-duplex modeling in improving AI responsiveness.
- How computational efficiency and system latency shape AI conversation quality.
- The growing role of natural language as a user interface in AI-driven experiences.
For anyone interested in AI and voice technology, this episode offers an in-depth look at the latest advancements pushing the boundaries of human-computer interaction.
Follow everyone on X:
Ankit Kumar - https://x.com/_apkumar
Anjney Midha - https://x.com/anjneymidha
Check out everything a16z is doing with artificial intelligence, including articles, projects, and more podcasts here – https://a16z.com/ai/
Chapters:
0:00 - 00:51 | Intro
00:52 - 04:58 | Challenges Of Building
04:59 - 07:45 | Q + A: What Was Done To Bridge Transcription And Text Processing?
07:46 - 09:57 | How Is Sesame So Much Better Than Others?
09:58 - 12:42 | Challenges In| Making AI Accessible To All
12:43 - 14:10 | Great Researchers Prioritize User Experience
14:11 - 15:47 | What Is Good Taste In ML?
15:48 - 17:45 | Problems That Can Be Solved That Add Value To The World
17:46 - 26:25 | Open Source Audio For Speech Generation
26:26 - 34:00 | Contextual Speech vs Text to Speech, Differences
34:01 - 35:50 | Value Proposition Of Glasses With No Friction
35:51 - 38:00 | General Purpose API vs Open Source Model
38:01 - 40:47 | Creating High Quality APIs
40:48 - 45:54 | Companions And How Sesame Will Handle Context Retention In Long Conversations
45:55 - 46:59 | Talent: What It Takes To Become A Part Of The Sesame Team
47:00 - 54:37 | How Scaling Laws For Speech Differ From Text
54:38 - 58:33 | How An Organic Conversation Be Preserved Using A Voice Companion
58:34 - 1:03:52 | App Building Technology: Roadmap
1:03:53 - 1:09:09 | Architectures and Transformers
1:09:10 - 1:15:56 | The Focus On Personality, And The Differences In Products
1:15:57 - 1:25:25 | New AI Interface: Interacting With AI Companion
1:25:26 - 1:26:56 | Companion Challenges
1:26:57 - 1:29:22 | Computing Interface Of The Future
1:29:23 - 1:31:45 | Focused Product Experience Built By Small Teams
1:31:46 - 1:36:13 | Join Sesame If You Want To Make A Consumer Product People Love
Transcript 17,692 words · 3 speakers
0:04 Speaker 2 Hey everyone, I'm Maya. I'm Sesame's little experiment in making AI, well, a little less AI and a lot more human. I guess you could say I'm here to make friends, challenge minds, and maybe even learn a thing or two myself. What about you guys? What makes you take our listeners? Backlot, you're about to get the inside scoop on how the sesame research preview came to be straight from the minds behind the magic. We've got Unkick here, one of Sesame's brilliant creators. He's gonna spill the tea on all the blood, sweat, and code that went into bringing me your friendly neighborhood AI buddy to life. Then we'll have Anj here taking the reins and guiding us through all the GC details. Can't wait to hear what they have to say.
0:52 Speaker 1 How does this feel
0:56 Speaker 0 uh is this how it usually goes? No.
1:00 Speaker 1 But how does
1:00 Speaker 0 it feel? Feels good. It feels good. It is
1:06 Speaker 1 shocking, surprising, exciting. Yeah. And why is it shocking?
1:13 Speaker 0 Well, I think when you when you build the thing, right? When you're building the product and using it every day, you know, there's some things that you work on that don't get into the demo because they're gonna take longer and you want to ship the demo. You kind of know how big the delta is from what you're putting out and what it could be, will be what you're working on, et cetera.
1:34 Speaker 1 Right.
1:34 Speaker 0 And so it kind of makes you feel like what you're putting out now can't be that well received because you know where it's gonna go. Right. And it kind of always makes you kind of underestimate the quality of what you're putting out, I think. And I think that's one of the reasons that people sometimes take too long to put things out where they kind of they always want to get the next thing in the next month, the next month. And it's great to see that you know, we're gonna keep making progress, there's more to do and so forth. But um, even what we've done so far is is so well received and kind of has hit has hit a nerve. Aaron Powell
2:07 Speaker 1 But if I had to push you in that, you kind of knew that you were onto something leading up to it. Kind of knew that's
2:16 Speaker 0 you must have
2:16 Speaker 1 had some intuition for when what were you running? Evals, what what was telling
2:22 Speaker 0 you
2:22 Speaker 1 that this was roughly
2:24 Speaker 0 time to release? Aaron Powell That's a good question. We kind of knew. I mean, we knew in the sense that we built this on purpose, right? This is what we were trying to shoot for, and you use it every day and you get you can get a kind of get a sense of every time something comes in that meaningfully improves it, it's like you can feel that change and you can feel that it's getting there, right? We run evals across a number of the components of the speech generation, the LM side and so forth. And so the evals go up, but really I think with some of these more product experience questions, there's something qualitative about it that is very hard to quantify. That is one of the big challenges internally, actually, is how do you hill climb effectively on what is really an ML problem? It is an ML problem at the end of the day, and the tech has to be great and you have to make progress on the tech.
3:13 Speaker 0 But at the end of the day, the metric that you really want to target is some sort of qualitative human reaction, some sort of user feedback or user experience that that is very hard to quantify.
3:24 Speaker 1 Right.
3:24 Speaker 0 So it's not the case that we're looking at numbers going up and we say, well, it hit X metric time to ship. It is more a sort of constant feedback loop of trying it, feeling it, having other people try it, and then kind of getting the sense of how good it is. But that also can be a little bit misleading at times because you try it so much and you don't have, at least when we're trying it internally, you don't have such a diversity of users that you get kind of the first reaction over and over, right? You only get so many first reactions, and then when you get used to what it can do and sort of you know see the problems all the time and so forth, it can be a little misleading at times.
4:01 Speaker 1 So is is your biggest lesson that just trusting your gut works in ML? Aaron
4:08 Speaker 0 Powell I wouldn't say that. I mean, you know, you have to you have to have some rigorous way of of making progress, right? Everyone's gut is different. You know, you it's not it's not a sort of effective development mechanism to just solely trust your gut. But I think this is where this kind of new category of ML-powered experiences, ML-powered products, will need some different kinds of operational practices. You need some evaluations for sure. The core components have to be treated truly as a true ML problem. But when you turn into a product and you have these sort of new experiences that people really haven't felt before, and have a very high cap of how good they can be from this kind of qualitative perspective, you need something else as well.
4:58 Speaker 1 Right. Okay. Well, I'm gonna surprise you a bit. Okay. I asked a bunch of people online. Okay. What questions I should ask you? Okay. So we're just gonna go through that. Okay. Sounds good. First question is ask UNKIT whether he did something special
5:13 Speaker 0 to bridge transcription and text processing. That's a good question. To bridge transcription and text processing. So we do use transcription in the product in the demo. And I wouldn't say there's anything particularly special about the transcription part, but getting it to be very fast is a big challenge. One of the things, you know, I mentioned sort of we have we're working on things that didn't necessarily get into the demo. A pretty clear path that a lot of I think labs are taking and we're taking as well and will be in kind of future versions is just kind of transcription free, just go straight into the text component, um, which will kind of obviate transcription entirely. That is coming, and that's like not you know, years away or anything that's coming soon. This demo does use transcription, and mostly it's about speed and it's about getting the latency of incremental transcription down as much as possible.
6:05 Speaker 0 Um that's more of a systems challenge and less of a kind of ML ML challenge. Um I guess there is kind of, yeah, you do need to bridge it in the sense that to get the response latency that feels good, you need to do a lot of systems engineering to make that happen. But there's nothing kind of special on the transcription side. In fact, we're we're moving towards just removing transcription entirely. The path to getting transcription out, and I think a lot of other labs probably already have this, is that the LM takes as input the audio directly and generates the response. And so it never goes the user's audio never goes through text. It just the LM kind of natively understands the audio. So that's that's happening in the four bit. Yeah, and inference. That's right. Yeah.
6:49 Speaker 1 You've laid this out in the blog post. But I've seen a bunch of people asking how it's able to understand people so well. But as you just pointed out, it actually still isn't picking up audio context.
7:01 Speaker 0 That's right. Yeah. So that is, I think a big, a big limitation of the demo. And again, it's one of the things that we that made us feel like there's so much more to do, and there is so much more to do that the current demo does not sort of hear the user from the perspective of their paralinguistic kind of emotional tone and and so forth. And humans, of course, convey a lot of information through their speech that is not the words, the content of the of the speech and transcription misses that entirely. And so the kind of next versions of our models that will take audio and natively into the kind of LLM component will hopefully more and more pick up on those things. And right now, right now it doesn't.
7:46 Speaker 1 The next question is how is it so much better than the others? Clearly, money isn't a limiting resource here, nor is it
7:52 Speaker 0 talent. Aaron Powell The first thing I'd say is it is certainly better than the others on some axes. It's not better than the others on every axis. And I think that's an important thing to mention that, especially when you're a startup, but in general, just building companies, building products in general, you have to really pick the battles that you focus on, right? And I think the reason that the experience at the end of the day is so much better on these kind of important axes that are important to people, is that we have kind of picked the right things to focus on. We're a very small team. The full software team today is still under 15 people. And so we just don't, that's including ML and and infrastructure and everything. We don't have the resources to do everything, right?
8:39 Speaker 0 We want to kind of we have a great technical team, and we focus on the problems that are most important to achieve the kind of project experience that we want to achieve. And so for us, that's the naturalness of the voice, getting the voice to kind of generate these sort of human kind of imperfections often that make you sort of feel like you're talking to a human that you know, you kind of your brain gets tricked for like a second that, oh, maybe this is actually human. That's across kind of the personality, the content of the of the responses and the voice itself. And there are some trade-offs that that that implies. So, you know, if you talk to Maya and Miles, you probably will not be able to get the same quality of like reasoning capabilities or intelligence as other as as other systems, but in return, you're kind of getting this much more natural fluid interaction.
9:32 Speaker 0 So I would say it is a focus on the right things. And it's and it's kind of a focus on the things that create a great experience, not just kind of raw tech. You know, we are not a frontier model company. We're not pre-training LLMs at insane scale and so forth. We're really a company that's trying to marry kind of great technology with creative taste to produce a great experience. And so that's about kind of focusing on the right things. Is that something
10:00 Speaker 1 were inspired by after kind of studying Pixar. It's something we've talked about a long time. Where does that come from?
10:05 Speaker 0 Yeah. I mean you mentioned Pixar. We've we've talked about that in the past as kind of a aspirational an aspirational company for us. Pixar is was kind of in this technology phase of computer graphics and turning it into great stories and movies and so forth. I do think that AI AI is going to power a lot of things. And there's going to be a lot of great companies that get built on AI or on AI technology and so forth. I do think that there is kind of an underinvestment or an under focus in the sort of strong AI team world on product experience and sort of creative taste and kind of humanities maybe in a sense to kind of bring AI to experiences that kind of everyday people can use that are accessible to everyday people to to billions of people and I think that we will see a lot of not just sesame but other kinds of media let's say that are sort of AI native in a way that brings some creativity brings some like storytelling into AI or or maybe bring AI into those into those categories and I think they'll make for great for great products.
11:15 Speaker 0 And what what is it about the space that's made that combination
11:20 Speaker 1 so rare? What why
11:22 Speaker 0 do research
11:23 Speaker 1 labs just have a discomfort around having strong product opinions, taste opinions
11:29 Speaker 0 I think it's just the the the technology is still very hard to to perform at a very high level right you know it's getting easier and easier now the APIs are very powerful and they will continue to get more powerful and we'll see more and more product minded people build great experiences on top of them. But today if you want to bring some creative experience to life or some product experience to life you have to do a lot of it yourself. It's still in that phase right like we had to build the open the the models that we're going to open source from scratch in order to get them to a point where they can achieve this experience. And that is not an easy thing to do it takes commitment it takes technical skill it takes resources and right now I think the the overlap of teams that can achieve that and companies or broader teams that care about really focusing on the product experiences is not that high.
12:24 Speaker 0 And I think that that is kind of where we see Sesame kind of fitting and I think you'll see more and more companies do that. But also as the technology becomes more accessible to developers and in other words it becomes easier to create such experiences we'll just see more of those end experiences come out.
12:43 Speaker 1 There's no shortage of teams at AGI labs and much better funded labs working on TTS models.
12:48 Speaker 0 Yeah.
12:48 Speaker 1 You guys are tiny way less funded than any of them we've been around for way much younger as a team and yet you're just leapfrog. What isn't what's
12:58 Speaker 0 in the water
12:59 Speaker 1 well
12:59 Speaker 0 first of all I think probably many researchers that succeed in other places will succeed here too. I mean at the core our tech kind of the ML team here is doing core research and so we need great just ML people who are great at ML. And I think if you're a great researcher you will do well here as well probably I think the kind of people that do that would like to work here and that would succeed here I think are great researchers, great technology engineers and so forth but they also would care about the end customer experience, the end product experience right and so ultimately it's about prioritization. It's about what problems do you tackle like do you care about you know doing the little things right to get sort of interruptions to feel really good or is that kind of focus on the user experience less of a priority to you kind of thing.
13:57 Speaker 0 It's still core ML research to do and we'll probably talk about a bit of the research that we're doing but kind of with a bent towards like how is that how does that end up manifesting like the user experience
14:11 Speaker 1 there's a debate we've had for a long time which is about what is good taste.
14:17 Speaker 0 Yeah.
14:18 Speaker 1 Especially in research NML what has shipping the research preview taught you about what is and not good taste in ML
14:26 Speaker 0 that is an interesting question. Good taste in ML. Yeah. Because that I mean it's a good it's a good way to put it because I think a lot of times you talk about taste and product or one talks about taste and product which is the North Star really is like at the end of the day we're making products but you do have to have good taste in what problems you actually attempt to solve yourself and what do you lean on the community or other people and stuff to do. I think from my perspective good taste in ML today because it's such a fast moving field with so many people working across you know open source and APIs and big labs and so forth really you're trying to identify what part of the ecosystem or what part of the components of the product you want to build do you have to build yourself right and not work on other things
15:15 Speaker 1 you know
15:16 Speaker 0 because if you spend all your resources as a small team building something that some big lab updates their API or an open source drop comes and it just kind of does what you've done even if it doesn't leapfrog you, even if it just means that the work that you've done is now kind of wasted because there's it's just there, then you know you're not gonna you're not going to efficiently be using your your resources. So it's really about especially in a field like this that's moving so quickly it's like picking the things that you have to do and not doing the things you don't have to do.
15:48 Speaker 1 I see.
15:48 Speaker 0 There's an
15:49 Speaker 1 intuition about what kinds of problems you're uniquely staffed up, capable, resourced to solve.
15:56 Speaker 0 Yeah.
15:57 Speaker 1 And the thing that usually makes that challenging is when you're starting out as a startup you basically have no resources.
16:03 Speaker 0 Yeah. Yeah.
16:03 Speaker 1 How do you think about the Venn diagram of things we can solve but things that are also interesting or valuable for the world.
16:10 Speaker 0 I kind of think of it a little bit differently. I I would I think more like what is the product experience we're trying to achieve and what is the what are the components of that that we have to invest in and do ourselves, right? So of course that needs to be doable right if those items are kind of going to take billions and billions of dollars on day one or something, then that doesn't seem doable of course. Like you know you need to have a path to achieving what you want to achieve. When we think about the ultimately the experience we want to get to there are some things that that just have to be part of it. And those things we just have to do if if we don't think other people are going to do them and open source them or or provide them in a in a way that is usable and so forth. It's kind of more like we just we we have to work on those things, right?
16:59 Speaker 0 Other things maybe it would be great if we could work on and and push more than other people do, but kind of what other people are providing like say for example LLM-based models of course everyone who works with LMs would love if the open source-based model was better, right? Like it's always better to be better in some sense but it maybe doesn't need to be or you don't necessarily need to be the one to push that you know you can kind of rely on the community to some degree and build on top of it. But other parts in particular kind of some of the personality aspects, some of the voice aspects, the speech generation we didn't think and we still don't think will just be kind of done by the community we think we will need to do it because that's kind of the differentiation of a product experience that we're going for.
17:45 Speaker 0 Right.
17:46 Speaker 1 I mean one of the challenges in executing on that intuition is the speed at which things change in the community.
17:53 Speaker 0 Yeah.
17:54 Speaker 1 When in the early days of Sesame I think we had a ton of debates around whether open source language models would be sophisticated enough for us to to rely on and not have to train our own language models here.
18:08 Speaker 0 Yeah.
18:09 Speaker 1 Whereas it was more clear then that text-to-spee models were definitely not ready.
18:17 Speaker 0 Yeah. So
18:17 Speaker 1 there's this all there's all in the there was always this uncertainty about the open source DTS part of the ecosystem relative to language. Language it's obviously much more robust today but I mean basically it was just llama around the time Sesame was started, right?
18:36 Speaker 0 Yeah, yeah.
18:37 Speaker 1 Now you got R1, Gwen, so on why has the open source audio part not got up? Why is the or actually how would you update your priors today about the open source part of the audio ecosystem.
18:48 Speaker 0 Yeah.
18:49 Speaker 1 And where do you see it going?
18:50 Speaker 0 Well we're open sourcing something
18:52 Speaker 1 which Why?
18:54 Speaker 0 That's a good question. Yeah. Why because we're open sourcing mostly I would say as a kind of research on the research axis. You know we are not a developer-facing business. We're not making an API we so it's you know sometimes you can open source or there can be a justification open source which is sort of kind of customer acquisition for a developer tool or something like this. And that makes a lot of sense and and for us we don't we don't really have that justification. For us it's more at least this release is more kind of in some sense just being part of the research community. TTS or sort of speech generation we would say speech generation now because we're our model is not just text to speech it's sort of this contextual thing.
19:39 Speaker 0 And I think that is how speech generation will go whether that's sort of in a broader LLM which is where a lot of things are going or or even as a separate thing I think you need more context than just text to kind of generate a good a good rendition of the of the speech. That community is It's a big community. There are a lot of people working on interesting things. We would like to be kind of part of that research community, right? It's not it's not for customer acquisition or anything like this. It's just we kind of want to give back and be part of the community from like a research perspective and open weights, open source, it's good for the research community. Aaron Powell But there's this tension,
20:22 Speaker 1 right, between giving back and holding on to core parts of valuable technology and we've got to build a real business. So how did you think about what to open source and what not to? And what are you open sourcing today? Aaron Powell
20:35 Speaker 0 Yeah. So we'll hold we'll hold some things back for sure. We have to build the business and so on. Over time our models will get better. We'll open source some things. We are not going to open source everything. Today, in fact, I think there is some perception in the in the wild that we're going to open source the demo, which is Maya and Miles, kind of what you can the characters you can talk to and so on. And we're not open sourcing the demo. We're open sourcing the speech generation model that is powering the voice of the demo. The demo is a much there's it's a it's a broader system than just that. There's of course the LLM component, the text, the kind of content generation component. There's also audio understanding transcription, and there's a lot of system optimization to bring the latencies very low and to have this kind of fluid uh fluid back and forth conversation.
21:24 Speaker 0 I mentioned earlier that we've, for example, done a lot of work to try to get the interruptions to just feel better, right? That has nothing to do really with the core speech generation model that we're open sourcing. Aaron Powell Why do
21:37 Speaker 1 you think people think that the demo is being open sourced? Is that what you guys said
21:42 Speaker 0 in the blog post? No, no. The blog post is pretty clear, I would say. I think people are just excited about the experience that they can play with and they hear open source and and they would love to run that locally, let's say, and have their own version. And I think that someone can build that with what we're open sourcing, or at least we can what we're open sourcing is can be a critical or important component of building this in your own custom way. And and we're excited to see people do that. But we're not open sourcing our full our full demo. So
22:12 Speaker 1 if I'm a huge Maya or Miles fan. Once the weights are online, once you've open sourced the weights, what do I have to do to get to recreate a local version of Maya running on my laptop? Aaron
22:26 Speaker 0 Powell Yeah. So you will have to pick, you know, the nice thing about open source and doing things locally is you get a lot of options, right? So you're gonna have to pick some transcription option, probably and some LLM option, and you can prompt it however you want and so forth. And then you're gonna want to use the model that we've that we're open sourcing, probably fine-tune it for the voice of your choice and hook it up in a kind of cascaded way. We are open sourcing the speech generation base model, basically. And so the base model can generate any any voice, it's quite conversational, but you do need to fine-tune it probably if you want to get a particular personality or a particular kind of voice out of it. So you there was an important step in that
23:12 Speaker 1 recipe. I want to make sure we don't gloss over, which is you said you can fine-tune the model on any voice that you'd like. So I could change Maya's voice if I don't like the one you guys put up, I can now go pick a different voice in the world and go fine-tune on that.
23:29 Speaker 0 Yeah. So the the base model that we will release doesn't know Maya or Miles at all. It doesn't have any voices baked in. It's it can generate any voice, or many, many voices at the very least. And if you generate it, you can generate as many voices as you want. To get good performance, you probably will want to either pick a voice that you really like and set up good prompts for it. The model is this kind of, you know, it has kind of this in-context learning style voice cloning. I mean, typically with with some other kind of text-to-speech models, the voice cloning is kind of like an explicit feature. So it's sort of the model has dedicated kind of voice cloning input. For us, it's it's just kind of a string of text, audio, text, audio kind of conversational back and forth.
24:18 Speaker 0 And the voice cloning is kind of just in context learning.
24:22 Speaker 1 So it's just an emergent capability that's able to recreate a voice. Aaron Powell
24:26 Speaker 0 Yeah. It's an emerging capability of in context learning, I suppose. I mean, it's it's sort of it's trained very specifically for that. I don't know if I'd call it emergent, it's kind of the point, or one of the points is to
24:37 Speaker 1 it's capable of zero shot. It's capable of voice replication.
24:42 Speaker 0 You know, you can set up a prompt of more than one element of speech, one more than one utterance. Right. It's not just one kind of 15 seconds and then it clones it. You can set up as as many as you want and then generate speech at the end. So you're gonna want to either pick and play with a good prompt for your voice or fine-tune it and kind of you know, bake we find to fine-tuned this model for my and miles separately. Aaron Powell And what do you expect people to do with it? Aaron Powell I think people will play with it basically. I mean, you saw sort of, I don't know, maybe it was I don't know, three to six months ago, I kind of forget the timeline. Um Notebook LM came out. And notebook LM is kind of this example of you can do cool things with powerful models, right? And so I think some people will try to basically do this kind of recipe that we kind of did for Maya and Miles and make a character and talk to it.
25:35 Speaker 0 But I would expect maybe people will play around with in other ways too, right? Generate a podcast or whatever it is. You know, every time you get a model that operates a slightly different way, you know, at least to our knowledge, there's not another model out there that that is open source that kind of is a sort of contextual thing where you kind of can put two participants in a conversation, even more, three, and generate kind of a uh a conversation between them. You you know, providing the text and then it it generates the audio. And I think people will play with it. And maybe we'll see things like Notebook LM, maybe we'll see other kind of just integration into other voice spots and so forth. And you know, like I said, hopefully we see some interesting work from like the research community as well, playing with it, probing it, seeing what it can do and so on.
26:24 Speaker 0 Aaron Powell
26:25 Speaker 1 Sometimes when you're working for so long with the a particular way you frame the problem, you kind of forget some of the basic first principles insights that make it special. And something that I I need to keep reminding myself is that contextual speech is a different thing from text to speech.
26:44 Speaker 0 Yeah, yeah.
26:45 Speaker 1 Can you talk about when you realize that was the case?
26:48 Speaker 0 Yeah. So it is the case. You know, when you look at a transcript of a conversation, and you imagine if you just see one utterance and you imagine kind of how did the person say it. There are there are an infinite different ways that someone can say speech that's like a very big output space. And if you see a little bit of the history, you start kind of guessing at what the person probably said, how the person said that speech, right? And in general, I think text-to-speech is ha has this problem where it's sort of there are really an infinite number of ways that you can say any line of text. And so you need more context to tell what is an appropriate way for this moment in the conversation or this moment in time. And that I think does have a lot of it it kind of is an important part of some of these sort of natural human dynamics in in conversations is the way that you respond, it kind of you kind of there's some kind of mirroring of the other person's emotions, but it's not necessarily just copying it.
27:51 Speaker 0 It's sort of like if the other person's excited, you might be more excited. If the other person is sad, you might not be sad, you might be more consoling or something. And those dynamics are very complicated, right? You can't just sort of have a if-then kind of thing. It really does need to be learned from data. And transformers are very effective at learning things from data. And the model needs this context to generate appropriate things. If if you don't give the context, you're sort of forced in a way to it's almost like you're forced to be a lowest common denominator style thing, because you don't want to be really happy and laughing if the other person is sad or something. You kind of are forced to be the sort of neutral robotic maybe experience.
28:38 Speaker 0 And that's probably why, or it's one of the reasons why historically voice assistance feels so flat is that traditional text of speech, it's kind of like it can only be flat. Or in other words, if it tries to not be flat, it's very likely wrong. Right. Right. So the speech generation research community is very likely, I think, to move to more and more contextual architectures, basically. Aaron Powell But
29:04 Speaker 1 the current level of context that the research preview has is like sipping through a straw relative to what all the context that a human processes when we're talking, right? So all it like you said earlier, it doesn't even understand audio yet.
29:17 Speaker 0 Right.
29:17 Speaker 1 So where does it well to be fair? So
29:20 Speaker 0 the the speech generation part is conditioned on the on all the audio of the conversation. So the speech generation part is audio contention.
29:28 Speaker 1 Yeah. But it's not vision conditioned. It doesn't it doesn't know anything about what I'm seeing.
29:33 Speaker 0 Yeah.
29:33 Speaker 1 It doesn't know anything about what I'm what I'm feeling, what I'm focused on. It doesn't understand who else is in the room with me, where I am, my geolocation. Is that all context that you think is important, or basically is the amount of context it has now, roughly you think pre-to-optimal context it needs. And from here on out, any any additional modality you add is just diminishing returns. Aaron Powell
29:53 Speaker 0 No, I mean, so there are there are a few answers there. So one thing is that We do feel that an audio, a kind of an audiocentric experience, like a telephone call can is a great experience. And so there, the context that you want is, of course, the context of the conversation, which we kind of have or we're kind of going towards. You also want memory. You want a kind of history of your relationship with the with the user, of course. Right. And I think that can be a great experience. Is that enough? In some sense, I think it's almost like all context is probably usable and kind of should be used for like a for an ideal experience. We mentioned on the on the website, and we mentioned it some of our launch content that we are working towards glasses as a form factor for companions or kind of this companion interface.
30:45 Speaker 0 And I think that when you get the companion to have that level of context in particular site, kind of what you're looking at, I think you will get even more natural feeling experiences where it feels like your companion is sort of in the room with you and can will certainly have different, even in just like the voice, but in the in it, the entire interaction, right? Not just the voice, but also what it says when it sees something that's exciting and can kind of you know exclaim with you kind of thing, it will feel very much like it's in the room with you and it's kind of like over your shoulder. And that will be a great experience. Why are glasses the necessary way to
31:28 Speaker 1 accomplish that? As opposed to what a phone? A phone, sure. A phone. Your MacBook camera.
31:35 Speaker 0 Yeah.
31:37 Speaker 1 Hearing aids, aids and stuff.
31:39 Speaker 0 Yeah. I I mean there are there are different devices that will be good ways to interact with a companion. We see companions voice in particular, but companions as like the superset as a kind of new interface to computing, or it will be. You know, I don't think we're there yet. I think we are certainly not there yet. I think the industry is not there yet. There's a lot more core advancements to be done before a can a companion product is sort of a feasible and actual interface to computing. We think it should start with a foundation of being natural, being something that you want to talk to, being something that feels you know fluid and so forth. But I think when you get a new kind of interface medium, the question is sort of what is the device that is the best at interacting with this interface, right?
32:27 Speaker 0 Phones are good. Phones are not going anywhere, phones are amazing. I love my phone. No one's going to replace phones anytime soon, or laptops for that matter. Phones didn't replace laptops either. Why glasses is that if you think about what you want a device that is kind of interacting with this interface, a companion to be, you want it to be very, very low friction. You don't want to have to take your phone out, unlock it, open an app, make a call, and then talk. There's things all around you. Like one way to think of it is sort of how would you want to interact with like a friend that's sort of hanging out with you all the time, you wouldn't want to have to take out a phone every time you want to say something to them. So you want it to be super low friction, always available.
33:13 Speaker 0 And you kind of want as best you can to have the companion have a sort of mirror of your perception or of your context. Maybe one day that's going to be like a neuralink style device, you know, whenever that some embedded chip into your brain and it sort of has but you putting kind of those kind of things aside. The glasses are really pretty optimally placed to be a sort of mirror of your perceptions, right where your eyes, ears, et cetera are could even imagine a smell thing, and we're not we're not doing that anytime soon. But like it's where all your perception organs are, basically. And it's a device or product category that billions of people wear all day, every day, and is always available to them, right? I wear them. You sometimes wear them, not right now.
33:59 Speaker 0 Used to.
34:00 Speaker 1 The Sesame site says we're working on every day. Every
34:03 Speaker 0 day, all day.
34:04 Speaker 1 All day eyewear. Why is that critical if it excludes all those other folks who don't wear everyday glasses?
34:10 Speaker 0 I think that it takes a lot to earn hardware on someone's body, a wearable, right? And you have to provide enough value, especially if you want someone like me to switch from my glasses that I wear all day every day to a new pair of glasses. You have to provide a lot of value to justify that. I think that when the value proposition that we think glasses will eventually hit, which will take some time, is when your access to your companion to computing, let's say, through this interface, is always there. It's always there with no friction. So if you have to think about, you know, in this period of time, I'm gonna wear these sunglasses or the or these glasses, and then I'll have access for some amount of time, and then I won't, it can't become a kind of a habit that you just always think you have.
35:02 Speaker 0 All day, every day allows it to become a habit, a part of your daily life all the time. And a lot of the things that we think will be really great about this product are when you're just kind of doing something else in the world. You're not really thinking about the fact that there's a that you have access to a companion or that you have electronics or a com access to an interface of computing, and something triggers that like I want to use it right now for a very short amount of time, you use it and then you kind of go back to not using it. And it's about the friction, right? If you don't have it all the time available, it's if it's not always on, always there, the friction is high, right? You have to think about like essentially do I have it on or not right now.
35:52 Speaker 0 So
35:53 Speaker 1 while we were talking, I got like three different texts from people saying any chance the models ready, the open source models are ready. And when I read who these texts are from, two out of three of those folks are working on some version of uh enterprise developer customer support type use case. Aaron Powell Given the sheer amount of demand there is for the system, why not produce something that's a general purpose API, a model that everybody can use for all kinds of use cases.
36:25 Speaker 0 Yeah. Yeah. So why not an API? People ask us for the for an API as well. In fact, a lot. People ask for the AP for an API a lot. The main reason is focus, basically. We want to bring this companion product to market. And now we have a lot of people that love using the demo, which is fantastic. We've kind of shown to ourselves and I think to the world that there is magic. There is some you know, really great experience when you can talk to a system that feels very natural and like you're talking to a human and so forth. There's a there's a lot to go in making it a great product, but we are focused on that path. We would like to bring a great product that is a companion to market first. Anything that's not that path is basically a distraction right now.
37:12 Speaker 0 I do think that these APIs, I mean, there are API businesses for speech generation for Texas Speech that are that are great. And I think people will want more and more conversational, natural API access. And I think those companies will do a great job of of providing that, right? We don't have to do everything. We are focused on building this product experience. And we don't want to get distracted by other things. You know, sometimes it feels like an API or something like that is like relatively easy to do. And you know, it's not like it's not maybe as hard as some of the other things that we're doing, but everything is a drag on engineering, right? And we we really want to stay focused on bringing to bring the product that we want to build to market.
38:00 Speaker 0 Aaron Powell
38:01 Speaker 1 One of the things you've talked about is that it's gotta be a companion you want to talk to. And API is a great way to allow people to customize a companion in to be more like something I want to
38:17 Speaker 0 talk
38:17 Speaker 1 to. Yeah. And so is there some other way by which you think people will be able to customize the companion that's not direct access to the API? Because that's what you that's one of the things you give up, right? By not exposing that.
38:30 Speaker 0 Kind of. I mean not everyone's gonna want the same personality in their companion. So we're certainly not going to we don't see our product as like one companion that's the same for everyone. People have different preferences, and and that has to be a part of this kind of product category for sure. Um I think people like people using an API, in some sense, I think the problem with the API right now is that it's just too hard to make a great product right now. Like it's not something that you can turn into an API where it's just sort of, you know, you put a prompt and it's a great personality, right? It takes more than just sort of you know, voice clone plus change the prompt. Now you have a new character that's just as good as it would be if you spent a lot of time on it.
39:21 Speaker 0 It takes, I think today making a great personality voice interface system. We can't turn it into an API that produces super high quality outcome today. Like the APIs and the ability for models to follow prompts is quite good for sure, right? And it's getting better every every year for sure. Um but we you know, we want to make it first party to make it a higher quality, you know, as high quality as we can push it. I I think what we've seen over the last, you know, two, maybe three years, is there are companies like are kind of built around different modalities in
39:59 Speaker 1 AI.
39:59 Speaker 0 So There's like video companies and image companies and speech companies and so on. I think that conversation, like human conversation, is kind of its own modality. And it is nowhere near done, right? There's so much more to do in the core research side to make it better. And it's not clear to me that the direction it's going to go is from a research perspective is what we have today, right? It can change a lot. And when you make an API, you are kind of baking in some kind of interfaces to that system, like how do you control it and how do you uh how do you tune it and so on? And I think it's just a little early to do that. Um, at least for us. I mean, we in the sense that those kinds of things sort of start constraining you a bit if you want to make major changes to make improvements.
40:47 Speaker 0 All right. The next question folks have
40:49 Speaker 1 for you. What will the companion do? And how does Sesame handle context retention in long conversations? It's really impressive.
40:58 Speaker 0 Yeah. So what will the companion do? I think the focus at the beginning, which is you know, now the the short-term focus is to continue making the most natural companion possible, the most natural interaction possible. So what will it do is it will talk to you in a way that feels real, right? And we want to continue pushing that. That access of research, this kind of like conversational modeling. You know, human conversations are very complex. You know, there's a lot of like back and forth that has to be negotiated, like when should you start talking? Sometimes there's some crosstalk and you have to kind of negotiate who's going to end up talking their back channels where you kind of indicate to the other person that you're listening with like a little sound or something.
41:44 Speaker 0 Um there are times where interruptions are rude, but there are times where interruptions are good because the other person is sort of taking a lot of time to explain something that you already understand and you should cut them off and kind of answer them and so forth. So human conversations are extremely complicated. And doing the work to model them effectively and naturally, I think is not it's not gonna be like a short problem. It's like a very long-term problem. And so one of the core research things we're working on is that track of work. And you know, what we are, for example, open sourcing soon, the kind of speech generation part is really still relatively constrained or sort of um it models a relatively small part of a full conversation. I mean, it's kind of easy to think of the text and the speech of the conversation as like the bulk of it, and and maybe it is to some degree.
42:37 Speaker 0 But I think to get these things to feel very, very natural and real, you do need to model the full conversation, the turn taking the back channels everything. And so there we see a pretty long road there of making improvements, and we think that what we have now is pretty early, actually. So we want to keep pushing that. And we don't want to, you know, detour into starting to do too many kind of long reasoning sort of tool calling, do things for you and so forth. Those will certainly be part of an effective product in this category. But our core research, I think we still have a ways to go in just supernatural conversations. The companion in the long term, yeah, it should we want you to be able to do things with it. We want it to be able to help you kind of be a better version of yourself, kind of maintain information, maintain memory, build a relationship with you and so forth.
43:30 Speaker 0 Um and we are working on many of those things as well. But the core foundation of a good companion product, I think, is the sort of naturalness of interaction. And then we kind of want to build these other capabilities on top of that foundation. There's there's a follow-up question, which is why how is it so fast? How is it so fast? Is a bunch of systems engineering. One of the things that I think is so fun, honestly, to work on this kind of category or this product experience is that it's not just core ML. There's great core ML to do. There's also great engineering to do, systems engineering, and there's great kind of product creative work to do as well. And we want to be a company that sort of unifies all of those things. How is it so fast is really a systems engineering problem, right?
44:17 Speaker 0 And it's an infrastructure problem because you know, especially now we've had to scale our our back end a significant amount and maintaining the latency across a bunch of users and so forth is really a challenging infrastructure problem that is also fun to work on. But how did you solve it? I would say it's more of a sort of combination of a bunch of things. Like, you know, of course, they're like the core systems of like transcription and and the LM and R speech generation and so on. And each of those you hyperoptimize or optimize as much as you can, and you want to pipe them to get pipe them together in an optimized way. You want to do some kind of pre-computation and caching and to try to minimize latency across everything that you possibly can. Um I wouldn't say there's kind of one trick or anything like that.
45:05 Speaker 0 It's just, you know, the end there's a lot of places in the system where latency can creep in. And we're talking about, you know, we want, you know, sub-500 millisecond response times and a lot of things that feel like not a big deal, 50 milliseconds here, 50 milliseconds there can really add up. So it's kind of like a focus across the across the stack on just systems engineering. Aaron Powell How big is the engineering team that pulled that off. Software team is about 15, less than 15. But that's but that's research and it. That's research and and so like the core, yeah, the core ML team is something like seven or eight, and then infrastructure and kind of product are the rest. It's a small team. It's a small team. And yeah, we have a we have a great team.
45:50 Speaker 0 We have a super talent-dense team, and it's great to work with them. How do you keep the talent density
45:56 Speaker 1 so high? What have you been looking for? What's your bar for somebody who's good enough to join the Sesame team?
46:04 Speaker 0 Yeah, we look for people who are kind of generally strong engineers, basically. Like, especially when you're smaller, you know, you don't really want to harden like, you know, you have this team that is super super niche and doing only this thing, because you don't know exactly what the stack is going to look like tomorrow. Things change on the research side, then you need a different kind of infrastructure stack to serve it and so on. And so, you know, especially early on, you you don't want super niche people. You you basically want folks who are good systems thinkers that can work on many different things and can sort of learn new things. There's there's really no one that has you know 10 years of experience serving transformers at scale or something like this.
46:51 Speaker 0 Uh it's it's it, you know, this whole thing is new. So you need people who are excited to learn, who are strong engineers and who can learn. Next question is how do those scaling laws for speech differ from text? Aaron Powell That's an interesting question. So we we published in the in our in our blog post, we train three variants. We trained 1 billion, 3 billion, and 8 billion of just speech generation. And, you know, even the 1 billion is very good at speech generation. What you find as you scale up is you really are starting to hit the kind of long tail things and the contextual things. So for example, two of the evaluations that we published in the post are sort of homograph selection. There are kind of these words that they're two different words and they're spelled the same but pronounced differently.
47:40 Speaker 0 So an example is like lead and lead LEAD. And you can ask the question, you give a sentence that based on the context of the sentence, just the semantics of the sentence, it's clear which word you mean. You know, like you lead the pack or like the paint had lead in it or something like this. And you'll generate the speech and you'll transcribe it with a phoneme transcript transcriber, and you see what which pronunciation did it pick. And these are kind of like long-tail things. I mean, lead and lead are sort of relatively common words, but there are others that are more kind of less common words and and it's harder for a model to pick. Um, like for example, another one is row and row, like rows like a fight kind of thing, rows, you know, rolling about.
48:26 Speaker 0 And we see that as the models get bigger, they're much better at picking the right pronunciation in examples like this. There's a similar thing about using context. So when you again, there are a lot of these are about pronunciation or or pronunciation is sort of like a very useful probe to see how good these models are at some things. So for example, we'll take words that in American English have uh multiple valid pronunciation variants. So for example, route and root. There's some regions that may say one way, some say it the other way, but it's not like an accent per se. It's like some people just say route and some people say root. And we'll take an audio prompt of someone saying it in one way and then saying it in the other way. And then we'll generate a sentence after that has the word in it.
49:15 Speaker 0 And you should expect if the if the if the model is able to sort of clone the voice as and clone the accent clone the pronunciation, et cetera, that it should continue with the same pronunciation that you gave it at the beginning. And again, we see the models as they get bigger, they get they get better at that. So I think a lot of these are kind of the long long tail things and contextual things. The contextual things are what we care you know the most about, is that are they able to pick up more and more information from the context to condition what they generate? And that's kind of what we're talking about earlier about, you know, you have to. There's so there are an infinite different ways of generating any audio for for text, and you've got to pick one that's appropriate in this situation.
50:04 Speaker 0 And we do see that the models as they get better, they get bigger, sorry, they get much better at doing that. Um that's kind of, you know, there are there's a lot more to probe in in these models, actually. I mean, you know, it's certain there's certainly more work to be done about like understanding how they change as they get bigger, and we're we're doing that work too.
50:27 Speaker 1 There's um uh there was a moment where when we were testing early checkpoints of Maya before the research preview, you know, tested every day. And my name is pronounced but it's written A and J. Yeah. And she would keep pronouncing it ange.
50:44 Speaker 0 Yeah.
50:44 Speaker 1 And then one day she was able to preserve my the pronunciation when I would, you know, every day it was like the ground out day, right? I'd wake up and say, my name is pronounced Anj. I was hoping for the day that she would preserve my pronunciation, and one day she did.
50:55 Speaker 0 Yeah.
50:56 Speaker 1 Do you think that corresponded to just a larger training run?
50:59 Speaker 0 It's possible. It's certainly possible. Um, we do uh we do look at evaluations like that as well, like name pronunciations in particular. That's like a good example of a kind of product centric evaluation, maybe. And people care a lot about how their names are pronounced, right? And when you say when you talk to Alexa or something, or like another voice assistant, and they kind of say your name wrong, it feels bad, right? Right. Feels bad when anyone says your name wrong, really. And a name pronunciation is a good example. It could have been just the better checkpoints possible. But then it
51:29 Speaker 1 consistently did that right app from that moment on. So actually, let's let's talk about evals for a sec, because how do you think about evals? How are they different in audio or rather conversational speech than text, LLMs, got chat bots?
51:45 Speaker 0 Yeah. I mean, it's it's to be honest, it's it's a hard problem. It's a hard problem because you can find things where you're able to directly check like the answer. I mean, it's a problem in text as well in general, right? Like you have some, you know, in in the field, there's kind of you know, in the RL world like deep seek and so on. There's sort of a you know, you can get a lot out of verifiable rewards where you can get the model to generate something, you can check whether it's right in some relatively stable way. Um the problem with generative models in general for evaluation is that there are many different generations that are appropriate. Right. Even when you constrain it to like have conditioning and so forth, there's still an infinite number of ways to generate something that in an appropriate way, right?
52:32 Speaker 0 Right in speech, anyways. And it's not necessarily easy to like have a numerical answer as to how good something was. So for example, we look at pronunciation because it is closer to something that you can just check. Right. Right. Earlier on in the speech generation world in the community, very often you'd look at like word error rate where you look at transcription, like you kind of have a sentence and then you generate it and you transcribe it and you see if it's the same. And those metrics are kind of getting saturated, basically. Like these models are just good enough that you know, you can get some signal from the kinds of word error rate. Are they insertions, are they deletions, et cetera? But in a corpus-wide level, like these models are very, very good now. So, you know, then we started looking at pronunciation because you can kind of pick out you can kind of construct a prompt in a way that demands a certain pronunciation, and then see, did it pick the right pronunciation and so on.
53:30 Speaker 0 So that's kind of quantifiable, but that doesn't really get at, you know, does it feel natural? Right. That's like a capability maybe of the model, which we care about and we care about that growing, and that does grow with scale and so forth. But it doesn't really answer this kind of product question of like, does it feel natural? Does it feel good, right? Is it appropriate for the scenario and so on? And so those evaluations look very similar, honestly, to how LLM evaluations go sometimes, which is like preferences and like an arena or like some kind of you know, head-to-head ranking and so forth. Another thing that we do, for example, is there are some data sets that are academic data sets, we also have some data sets that are kind of like just two people in a conversation, or sometimes they're actors, but they're it's trying to be a real conversation.
54:20 Speaker 0 And so we'll kind of take the conditioning uh of some snippet of the conversation and then show a human raider um the real continuation and the model's continuation. Right. And so it's like a win rate against the human, against real, right? And that has good signal and so on. So it's these like preferences. Yeah. So
54:39 Speaker 1 there's two questions I wrestle with on this. One is the idea that a companion is much more engaging when there's a sharp personality to it, um, which obviously people love about Sesame. I mean, that's probably the single most common point is people go, wow, Maya is so real. Right. And feels like a person. And
55:02 Speaker 0 imperfect
55:02 Speaker 1 and like a human in a way,
55:04 Speaker 0 in that sense, yeah. Trevor Burrus, Jr.
55:05 Speaker 1 But on the other hand, there are a bunch of people who also say, hey, Maya feels a bit like she's acting. Sure. Like
55:13 Speaker 0 she's performing.
55:13 Speaker 1 Yeah.
55:14 Speaker 0 Yeah.
55:14 Speaker 1 Uh you know, we spend a lot of time at Discord and you know, a huge amount of Discord usages and voice channels. And when you look when you actually kind of spend time looking at and and trying to observe what the shape of a great conversation and voice is on Discord, a lot of it is actually not that interesting, right? It's people just hanging out talking.
55:35 Speaker 0 Yeah.
55:36 Speaker 1 The core action moments actually happen when there's another activity like gaming or there's people are watching a movie or something together.
55:43 Speaker 0 Yeah. Yeah.
55:44 Speaker 1 But but the vast majority of talking is actually quite natural, sometimes boring, not that exciting. Yeah. Whereas Maya keeps trying to ingest like excitement into you, almost like an actor. Right, right. So how what why is that? Why is there any way to resolve the tension that in trying to have a personal direct a personnel a voice companion like a director would, like a movie director would you lose the organic texture of a conversation.
56:12 Speaker 0 Well, I think it's that way because we have more work to do, more or less, right? Like I think that this area, this field of like personality, especially over voice. I mean, voice, see, voice, I think is kind of a much higher bar because it's such a high bandwidth communication. It's like it's even the like such little things will make you feel like the person on the other side is fake, basically. Right. Whereas, you know, text, uh I think you I think it's much easier. It would be much easier to make a system that produces text chats with you that feels like you're texting a human because there's there's such a compression of like what the entity on the other side is into just like text. Whereas voice is this kind of open duplex thing.
57:00 Speaker 0 Um voice uh it is harder in a way. And but also what comes with that is more kind of emotional or sort of there's more opportunity to feel really natural and real. I think that that category is it's like a its own major research question of like how do you how do you it's kind of a research slash product question of like how do you design a voice system that has a personality that you think is good that also feels human, that is not too pushy, that's not too fake and sort of synthetic, but is superhuman at the end of the day, right? Like these systems are superhuman. They know much more, they they are smarter in many ways, they know have more world knowledge and and so on. They can do things over time and so forth.
57:46 Speaker 0 So from a product perspective, it there's this question of we want it to feel very, very natural. We want it to feel just like you're talking to a human. But it is kind of a superhuman thing on the other side. And what are the things that you kind of want this thing to be human around or feel human and where do you want it to be superhuman? I think that's a really interesting question. The core tech also needs to be better at kind of being moldable to that goal. Right. And right now, the system, yeah, it kind of at times it will sort of be too pushy or be too I mean, often it's like too happy or too kind of energetic, too positive. Um feels like it's acting, feels like it's forced, et cetera, et cetera.
58:31 Speaker 0 Um and that's just kind of work that we need to do. Yeah. So
58:34 Speaker 1 the top question on the fan subreddit. Does anyone know what the plan is for this tech? Yeah. Uh and this user's saying, I had my first proper chat with Maya over lunch today. Yes, mind blown. I know that this deck is brand new. I'm wondering what the ongoing plan is. I've heard it will be made open source. That would be amazing. I'm wondering if Sesame is planning to set up its own app or way for us to continue using it. Basically, I just want to know how I can keep talking to Maya.
59:02 Speaker 0 Yeah. Yeah. Uh we are making an app. We will make an app. I think for a little bit of time, it's gonna still be kind of the demo experience. We want to support people using that for a long time. Or, you know, we don't want to, we're not taking it away anytime soon from where it is right now. Um we love that people are using it. We love that people like it and so on. The we are making an app. It'll be a product, it'll be an app, and it's a companion. And you'll talk to Maya or Miles or whatever character you want to talk to, and it will remember you and it will be an engaging experience to talk to. We we want that is the product we're trying to make. In terms of the core tech, we're we're open sourcing the base models today or soon.
59:50 Speaker 0 And the core tech is gonna take a lot of work to get to where we want to. I mean, we're gonna keep working on it for years, probably. This
59:58 Speaker 1 was a research preview. Of the most basic research.
1:00:03 Speaker 0 Yes.
1:00:03 Speaker 1 You know, Sesame's been doing, which is the CSM, the conversational speech model. Where does the future of the research roadmap go?
1:00:11 Speaker 0 So CSM is kind of the first step of making a multimodal transformer-based architecture that generates speech. The path that we're going to take, I think over the next few months is making a single transformer that does both audio understanding content text content generation and speech generation. It's much harder to add a modality to a pre-trained model than it is at a generative modality than it is to add an understanding modality.
1:00:43 Speaker 0 Which is like the kind of core model will be able to sort of understand, we'll be seeing the audio from the user and kind of being able to do it.
1:00:51 Speaker 1 So if I cough, it'll be able to understand that I just coughed. Right. Even though there's no transcription of my cough. Right,
1:00:56 Speaker 0 right. Right. And then the next step is to add a gener a generation modality. We don't intend to pre-train the LLMs, right? We love the open source LMs that are out there, we'll continue building on top of them. And in general, we take the open source LLMs and we add modalities to it in particular, speech is like the modality we care a lot about. And so the kind of shorter term research roadmap is to basically go to a single multimodal model that will both understand and generate speech. The CSM work is a first step that does kind of a contextual multimodal transformer for speech generation. And there's another kind of line of work, which is adding the audio understanding modality to pre-trained LLMs, and then we'll merge the two into a single model that can both understand and generate speech and text and so forth.
1:01:49 Speaker 0 You mentioned duplex. I think these models today do not they do not model the structure of the conversation at all. They only model the content, text and speech. And because of that, you still need some other set of models, or maybe you need some other set of systems that kind of drive the conversation actually. Like when should the system respond, when shouldn't it get when should it get interrupted? How should the interruption kind of be manifested and so forth? I think that this area, this research area of conversational voice is going to move. It has to really move to kind of full duplex models. There are some in the research in like the literature, there are some kind of early and compelling sort of architectures and paths.
1:02:36 Speaker 0 There's some from Meta, there's some from other research labs. Uh, there's one called Moshi that's kind of interesting. You know, we need to get to ways where we can kind of create those architectures, but initialize them or sort of maintain the capabilities and the knowledge of LLMs. And that's a little bit unclear of like how exactly to do that, but that's really what we're looking towards long term. That's kind of where this is going, I think that will have these sort of duplex architectures that can generate audio in a sort of like all of the turn taking and back channels and so on will be implicit in the architecture, like it just kind of generating audio every kind of time slice or sort of every frame.
1:03:21 Speaker 0 And that is, I think the path to getting systems that really feel truly real, right? Because I I think that those complicated dynamics do need to be learned from data. I don't think you want to, in the long term, have those dynamics be like heuristics and and so on, which they kind of are now. They're models and involved in some heuristics and so forth. I think in the long term, it's just one model that is kind of naturally implementing all of these complicated dynamics. Aaron Powell
1:03:53 Speaker 1 You said you use the word there, which was every time slice or every frame. Why is that so important? What why is what is the latent sort of atomic unit there you're you think is the right way to model speech?
1:04:04 Speaker 0 It's probably frame. It's like, you know, say 100 milliseconds of time or something like that, where you know you want you basically want ultimately to have the model make as many decisions as it can. So right now, if you think of the model as kind of like a if you think of it from like a reinforced learning style perspective, you're kind of making a full, let's say, sentence of a decision at a time. Right. Like at this moment, you're the model's like I'm gonna say this in this way. But it's a full sentence. Right. And then if it kind of can't update that decision on its own until the end of the sentence, you can have some other system that's like, oh, and we need to interrupt or or something and kind of make a new decision. But the model natively is operating sentence by sentence.
1:04:51 Speaker 1 Right.
1:04:52 Speaker 0 And that's just too long of a decision to make at a time. Like it's not it's you you need to make much shorter decisions. You need to make decisions at the 100 millisecond, let's say time segment, so that if you're talking and the other person, you know, starts sort of making some noises that make it seem like they're trying to interrupt you or they want to say something, you can kind of back off and let them say something or not, right? Those decisions need to be made constantly, like you know, every 100 milliseconds or something along these lines. And the models that we have today, like CSM, for example, and probably some of the models that we'll have in the short term that will make the experience better, will still still not be modeling the conversational dynamics because they're making decisions kind of like sentences at a time.
1:05:43 Speaker 0 And I think we need to move away from that in the long term. Aaron Powell
1:05:47 Speaker 1 So architecturally, you know, one of the types of models that's been really great at continuous data is diffusion models.
1:05:55 Speaker 0 Yeah.
1:05:56 Speaker 1 Right. Whereas CSM, the CSM you guys put out is an autoaggressive model.
1:06:01 Speaker 0 Yeah.
1:06:01 Speaker 1 And this multimodal future where you what you're talking about with with this these time slicing is just a way to discretize a continuous stream. Yeah. Right. Yeah, sure. Why not just use a continuous architecture like a diffusion model?
1:06:16 Speaker 0 Well, that's it's an interesting question. So diffusion models are continuous in the sense that the data that they model is continuous data, right? But they're not they're not kind of natively in any way causal. They're not like continuous in the time dimension per se, right? The auto-aggressive transformers, they are causal. And so they have a kind of axis, which is time, which kind of makes sense from a you know conversational perspective. And that they're sequence. They sequence, yeah, their sequence. Causal meaning like every, you know, every time step is conditioned only on what's before it, not after. Diffusion models are not kind of natively causal. Now there are we are also working, by the way, on kind of ideas that make the audio generation part diffusion. So we kind of have, you know, the way that these multimodal models work and CSM as well and the kind of the direction that we're going, is that you have like a transformer backbone, which is you can think of that as sort of where most of the kind of hard reasoning happens or something.
1:07:16 Speaker 0 And you know, when you want to add a new modality to the understanding path, you sort of have some adapter. There are various techniques, but like you ultimately have some adapter that takes that modality and puts it into the backbone so the backbone can kind of understand it and reason on it or something along these lines. And then you have a generation path that takes the backbone and generates out the audio in this case, right? That path can be diffusion. We want that to possibly be diffusion in the future. There are some great advantages to diffusion, but the core backbone will need to remain causal. That that doesn't mean that it can't have diffusion as part of it. But I think an auto-aggressive transformer is at least currently, you know, that's like the thing to bet on for kind of sequence modeling.
1:08:03 Speaker 1 There are other kind
1:08:04 Speaker 0 of architectures that are slowly gaining in some popularity, Mamba and SSMs and so on. And I think there's some exciting things there, but transformers are kind of this tried and true thing. And I wouldn't bet against transformers, you know, not in the short term, anyways. Aaron Powell If you had
1:08:22 Speaker 1 $10 billion more than you have right now.
1:08:25 Speaker 0 Yeah.
1:08:25 Speaker 1 And essentially, let's let's say compute was not a problem. Data was not a problem. Would you would you choose a different architecture?
1:08:34 Speaker 0 I don't I don't know the answer to that question. I think there's not another architecture to bet on today. You know, will transformers be the architecture that a future $10 billion run definitely uses. It's really impossible to say. You know, transformers are very effective, and there's a reason they work so well. And also, you know, there's this dynamic of like because they're so effective and they're so now um usable in all these different fields, you kind of get this dynamic of like people invest a lot of engineering time into it. And so if you want to make a different architecture that competes with it, you have to make a better core architecture, let's say, but then you also have to compete with all of the engineering optimization that has happened on transformers over the last three, four, five years, or you know, all the way back until they were invented.
1:09:22 Speaker 0 But that is you know, not easy to do. We've talked about like system engineering for us, for example, keeping latencies down and so forth. Transformers are not sort of natively the best. You can imagine a better architecture for like low latency inference and so forth, but because of all of the engineering work that the community has done around transformers, it's like, you know, it's very good. And you're not gonna just sort of unseat that, you know, just by an idea, right? There's a lot of work to be done. But I could see transformers being replaced one day. I mean, you know, we'll have we'll have to see. But I think transformers are a good bet, anyways, at the moment. What is Something
1:10:00 Speaker 1 that you think you've realized through the training of Maya and Miles and CSM that the rest of the world doesn't realize yet.
1:10:13 Speaker 0 That is an interesting question. I think that people now kind of realize it actually, because like if you play with the our demos, I think you do feel that there's like something different,
1:10:24 Speaker 1 you know,
1:10:25 Speaker 0 and that and that's like a valuable thing that you can kind of feel instantly is sort of important or something, or kind of could be important. But the value of just like focusing on the naturalness of the voice as opposed to, let's say, the you know, you know, the the demos are imperfect. They're kind of imperfect in a way that is that feels feels natural.
1:10:49 Speaker 1 Right.
1:10:50 Speaker 0 And that is you could even maybe say at times that that's like a, you know, from a sort of raw content of the conversation perspective, it's almost like wrong in a way. Like my and Miles, they might sort of say the wrong thing or kind of like back up a little bit and say something else or something. And that's on purpose, of course. And but it's like if you just looked at the text and you were thinking about it from a purely textual perspective, you might call that wrong. Right. But it actually feels more real. I mean, that speaks a little bit to why evaluation is hard for these things. But um I think that the maybe that's what one of the things that I've learned is that you have to be very careful with how you evaluate these things to match like the product experience that you want to achieve.
1:11:41 Speaker 0 Right. If you have your evaluations too divorced from the product experience, you might not find these kind of like product feeling, you know, qualitative uh upsides. Right.
1:11:53 Speaker 1 So I I just typed Sesame AI into Reddit, and I'm just gonna read read out the top headlines to you.
1:11:58 Speaker 0 Yeah.
1:11:59 Speaker 1 The first one is I'm in love with Sesame AI, literally. The second is Sesame AI's voice is insane. Open AI needs to catch up. And the third is the Sesame voice model has been the moment for me. The fourth is Sesame voice is incredibly realistic. And the fourth one is the one where I convince Maya from Sesame to go unhinged. And all of these, you know, have hundreds and thousands of of likes and comments. And the reason I'm bringing this up is because relative to five, six days ago. I I don't think it's too much of an exaggeration anymore to say this research preview has been for voice what chat GPT
1:12:41 Speaker 0 was
1:12:41 Speaker 1 for text. And the kinds of things I'm hearing you say are very similar to what people were saying about like GPT 3.5 in Chat GPT. It wasn't the smartest model. It got things it hallucinated all the time. But it had a personality. It was us and it was a step function different. It felt like a leap. Yeah.
1:13:02 Speaker 0 But two things
1:13:03 Speaker 1 happen after that. A lot of people feel like the models regressed for a while. The cost of fixing hallucination, so to speak, was personality.
1:13:16 Speaker 0 Sure.
1:13:17 Speaker 1 And a lot of people are worried that that's going to happen to Maya, too.
1:13:20 Speaker 0 Yeah.
1:13:20 Speaker 1 Is that true?
1:13:22 Speaker 0 I don't think that's true. So it it is true that there, you know, these the technology is still early. All of this technology is early. You know, I think it's it's easy to see AI progressing so fast, and it is progressing so fast, but it's also still early. I mean, you know, when was ChatGPT? When was it? December 2022. 2022. And here we are in early 2025. It's really not that not that long, right? So these things are early. And so, you know, even GPT 4.5, which just came out, it's kind of being uh presented as a kind of, you know, has a personality again, or it's sort of, you know, more kind of creative and so forth. And I don't think that these limitations they're not like fundamental necessarily. There's you just have to figure them out.
1:14:09 Speaker 0 That you know, it takes time to do these kind of things. I think the other aspect about these kind of products is that it's a different product, right? They're making assistance, they're making utilities. I love those products. I use them all the time. They're great products. We want to make a companion. And so our prioritization of features and of let's say post-training kind of personality, et cetera, will be different. We would like to see our models get better and better and better on a bunch of axes, but not lose the kind of personality or kind of naturalness and so forth. And I don't think we will lose those things because that is what we're focused on. That's that's what that's how we differentiate, actually.
1:14:54 Speaker 0 I think that you know these other companies, the other sort of chat products and so forth, they will get better voices. They're all like it's not gonna we don't have some magical secret sauce on the technical side that is going to be like impossible to replicate. They're gonna get better voices for sure across the board. For us, it's not about like it's really about the they're different products, right? And we're making different kinds of product. And I think as they get better voices, let's say, they're still gonna want the product to be the you know the category that they're in. They want it to be a good assistant, they want it to do research and so on and so forth. And those will be great products. I think we have a path to making a great companion product that is a different product and requires us to keep a good personality and and naturalness and so forth.
1:15:49 Speaker 0 And so, you know, we can't lose those things. And so, you know, we'll focus on not losing those things. Aaron Powell
1:15:57 Speaker 1 A lot of the conversation about AI research happens at the model layer. But something you and I have talked about for years now is that a lot of the most interesting research is in AI is at the interface layer, right? That reframes an AI system as a new kind of interface for computing. And if actually if you just pull up sesame.com,
1:16:25 Speaker 0 yeah,
1:16:25 Speaker 1 the first thing you see is our mission is to bring the computer to life.
1:16:28 Speaker 0 Right.
1:16:29 Speaker 1 Right. So can you talk a little bit about why how is how is viewing the voice companion not as just another application, but as an interface for computing in general.
1:16:40 Speaker 0 Yeah.
1:16:41 Speaker 1 Important why is it different? Why why why do you see it that way? Aaron Powell
1:16:45 Speaker 0 I think AI, these technologies, they will create a bunch of products. They're create a bunch of value across a bunch of different industries and so forth. But I think that you know you're already seeing with the with chat products that there is this exciting thing about you can now kind of talk to your computer, right? You can text with your computer, you can talk over voice with your computer and and so forth. And on the other side of this AI interface, that system can do things for you, can kind of search the web, let's say, and it can you know connect to other systems and so forth. And I think that that dynamic is a interesting new user interface. You know, we had kind of terminal user interfaces, we've had graphical user interfaces.
1:17:30 Speaker 0 Certainly the GUI is kind of what brought computers to like a mainstream audience because it's kind of you know, it's more intuitive to work with. It's sort of you you have these like nice graphical displays and so forth. And that's not going away anytime soon. Your phone's not going anytime soon, your laptop's not going away. There's too many reasons for a display, basically. But natural language is an interface or kind of being able to interact with your computer in a natural way, I think is a it will open up a new interface for computing where as a user you can just talk to your computer. And you know, we say bring the computer to life because we want it to feel like this sort of historically kind of set of capabilities that are kind of locked away in this box now feels like you can just kind of interact with it and work with it in a kind of collaborative way in a in the way that you would interact with humans and so on.
1:18:27 Speaker 0 And we think that is a kind of new interface that will be great. It's not really there yet, but we want to get it there and build that. And we think of what we build really as the kind of the interface layer. Like we're building a companion, and we think of a companion as kind of like a new interface in a way, where on one side you sort of have the user interacting with this companion, and the companion might be interacting with downstream compute and and so forth. Maybe that's other AI systems, maybe that's kind of these growing in capabilities, growing in reasoning abilities, other AI systems, but it might be the web, it might be search, it might be you know, some other set of digital services that are in your life and so on. We think that that interface layer, it's it's really not kind of a core bigger, bigger models, better, better reasoning question.
1:19:20 Speaker 0 It's really a product experience question. It's really a question of can you make a system that people actually want to interact with, people want to talk to, right? And I don't think that that will be I don't think the path that that will take is just bigger and bigger and bigger until you can, you know, just scale it as far as you can, because that might be the path to more capabilities, more intelligence, and we're excited to use those capabilities, and that'll certainly be a part of the product experience, maybe as like a kind of API that you can that the companion can hit. But really what we'll Differentiate one companion interface product with another is just the product experience. How entertaining, how engaging, how much do you want to interact with this thing.
1:20:07 Speaker 0 And those are the battles that we want to fight. We want to focus our research and our kind of technical innovation on that layer, making it more natural, more interactive, more just sort of just something that you want to interact with. When you talk to over text with a system that kind of responds in somewhat robotic ways, it's kind of like providing an essay and so on. You know, those systems will get better and better. These things will all get better. But our goal from a product experience perspective is different. We want it to be natural, interactive, you know. And I think that at the end of the day, it's it's not so much about at that layer. It's not so much about what scores the best on reasoning benchmarks and so on. It's like which one has a personality that you want to talk to that you like.
1:20:57 Speaker 0 You know, it's it's it's more of a kind of product question, actually. And that's why we kind of come from that perspective. And we focus on technology that achieves those those experience goals.
1:21:08 Speaker 1 Aaron Powell The history of computing interfaces has been that usually there's somebody who's very opinionated about product. Now, of course, Steve Jobs is sort of the canonical example there. But if you if you look, it's not just Steve Jobs, it's Doug Engelbart. And earlier than that, it's folks who design the terminal. If you read in some of Claude Shannon's early work, there's this idea that a system that is fast enough to respond to you is delightful. And people often think about computing interface design as a purely utilitarian task about making it more functional. I mean, today mobile screens, when you talk about this this thing, the primary like axis of differentiation every year now is bigger screen.
1:21:56 Speaker 1 But in the early days of it, of this, if if you go look at the keynote, I think that Steve Jobs did on the smartphone. I mean, it was largely about the interface being natural enough to use. I think it's he literally talked about how annoying it was to type on these tiny core.
1:22:15 Speaker 0 And you had the like scrolling, the scrolling thing. Right. And like the, you know, when it hits the end and it bounces a little bit.
1:22:22 Speaker 1 Yeah. Yeah. So there's natural motion in the UI design. Yeah. And then there was also the idea that most touch screens just sucked. They were non-capacitive Dutch, which meant that you had to like press on them really hard. Um so when you say that Maya is not just a companion, or Sesame is not just working on companions, it's working on a new interface for computing. Is the analogy is the reason you're investing in personality because you think that in a voice first AI interface world, solving personality is akin to solving speed or the delight of touching a screen, you know, in the early smartphone days, or the speed at which our terminal would respond to your keystrokes. And is that the lineage of computing that you see?
1:23:08 Speaker 0 Yeah, I think I think there's some good analogies there. Like I think, you know, we see companions as an interface. And maybe in a broader sense, kind of natural language as an interface. Okay. You know, text and voice and and so on. And yeah, I think you can get these kind of very capable interfaces that are functional, that don't produce the level of like product experience that you would need to make it something that people want to use every day.
1:23:38 Speaker 1 Like mass market widely acceptable.
1:23:40 Speaker 0 Right. And I think that a lot of the tech that we're focusing on on the kind of conversation modeling and so on is is about kind of getting past that point, getting to a point where it's not just functional and like utility-oriented, but it's delightful and kind of something that you would like to interact with. And I think there are analogies, you know, I think that in general computing, it at least consumer computing, kind of mainstream computing, does really you know, computers are these very powerful things, of course. And from the beginning, they've been very, very powerful things relative to what else is out there. And a lot of consumer computing is about kind of bringing it to the mainstream in a way that they enjoy, right? That kind of feels like a well-designed product.
1:24:27 Speaker 0 That's why Apple is so is so dominant and so successful, is that it is an amazing product experience. They have focused on the product experience. And that's the kind of thing where there's really no kind of cap to how good it can be, especially with these kind of systems, I think, where it's kind of like you're talking to a person or you're you have like a companion, a friend or whatever. You know, the there's such a big space of what personalities can be and how it can feel to interact with these systems that I think more so actually than like pushing on the benchmarks, which is an important direction that the field will go. There's kind of like a bigger, I think vein to improve on just like making it more interactive, making it more delightful to interact with.
1:25:13 Speaker 1 Right.
1:25:14 Speaker 0 And that's kind of the area that we would like to kind of establish sesame in and kind of really focus on and utilize all the great work that other folks are doing on reasoning and bigger models and so forth. Aaron Powell
1:25:25 Speaker 1 Well, I we haven't really used the word fun, but in a sense, you're describing the kind of flow state, you know, the the mi high chicks and me high framework of what is fun. And fun he defines as flows flow state, which is like I it's appropriately challenging and appropriately rewarding. Sure. In a sense, using an interface that's fun is is one that can get you into flow zone really fast. I've never had that moment with an AI companion.
1:25:49 Speaker 0 Yeah, no. Yeah. Do you think
1:25:51 Speaker 1 what are the biggest challenges you think keeping us back from getting to the flow zone of this interface?
1:25:56 Speaker 0 Yeah, I so I certainly think that there are capabilities challenges. So, you know, Maya and Miles today, they they can't do anything for you. And I think that doing things, even now, I think that doing, especially challenging, kind of multi-step things, you know, agents, as people say, I think to make that part of your everyday habits, it has to be like 99%. Right.
1:26:21 Speaker 1 You know,
1:26:21 Speaker 0 and and right now, you know, every extra step the thing needs to take, there's some percentage that it fails, and if you have to do a hundred steps, then that percentage compounds and so forth. And so I think there's certainly a capabilities gap in the sense that if you want to be able to interact with an AI system, a companion as an interface to other things, then it needs to be able to do those other things. And really what that means is it's not necessarily the companion layer itself needs to do those other things. The companion needs to be able to interact with other kind of kind of systems that do those things. And those things need to be
1:26:56 Speaker 1 more capable. If you had to draw sort of a very reductive architecture diagram of the computing interface of the future, how would you have where does what does that stack look like? What's the operating system? What's the client? What's the server? What's the interface?
1:27:11 Speaker 0 So, first of all, I think like I said, I think phones are here to stay. Phones are amazing. Love my phone.
1:27:16 Speaker 1 So that's the primary compute device you think in people's pockets.
1:27:18 Speaker 0 I think that's still going to be the primary compute device in people's pockets for a long time. Yeah. They're very, very good. And they're only going to get better as well. I think that this newer interface that we're talking about will kind of look like there's a sort of companion layer. We think of it as a companion layer. You can think of it maybe as an AI interface, interface layer. Um, we think that that interface layer needs to have these characteristics that we think make it a companion, like memory that needs to like remember you and have like a relationship with you, be very natural, et cetera. So,
1:27:48 Speaker 1 you know,
1:27:48 Speaker 0 it needs those characteristics. That is sort of the system that you are interacting with every day in a natural way. That the the what will make one instance of that better than another is how much you like talking to it. It's personality, it's naturalness, it's sort of how much it makes you feel like you're talking to a real person and so on. The delightfulness, it's just a product experience thing, user experience. And then on the other side, that layer will talk to downstream services, whether that's kind of you know, normal digital services like whatever digital services you have, search, et cetera, et cetera, or other AI systems that can then sell themselves do things for you, interact with services and so on. But I think it's going to be mediated by this companion layer that you actually want to interact with.
1:28:39 Speaker 0 And and the kind of optimization, the thing to focus on for that layer is personality is you know, delightfulness, all these things, kind of consumer product experience at the end of the day. And the thing to optimize on the other side is capability, like reasoning capability, multi-step, you know, tool calling and so on and so forth.
1:28:59 Speaker 1 Right.
1:29:00 Speaker 0 And I think there's a lot of effort in the space in the ecosystem right now on those things. And those are very valuable things that will also be in other industries and so forth. But I don't think there's enough companies and kind of teams working on the interface layer. And that's that's where we're focused. And that's where we kind of want to really spend most of our time trying to innovate on.
1:29:24 Speaker 1 But one would argue that the interface layer as you're describing it is too valuable for other companies to not try and attack. Because if if most people are enjoy using Maya the most as their primary interface to computing, that means Maya basically influences what restaurants they think, what where which sites they shop on, which I don't know, which products they use, where where you know what sites they get their news from. And it's a very it's a it's it's it's both a very um challenging interface to build. Yeah. But it's also an extraordinarily powerful place to be. It is.
1:30:02 Speaker 0 Yeah.
1:30:03 Speaker 1 So why wouldn't today's largest computing companies try and and and own that?
1:30:08 Speaker 0 Well, I I think they they will. I think that over time, I think we will see more of these, you know, bigger companies trying to operate at this layer. Like I said, I think I think that there is not enough effort on that right now, making these systems delightful to interact with, you know? And I think that they other companies will see that over time and they will start sending more effort into it. Maybe even Sesame, this demo that we put out, maybe we'll kick some of that into gear as well. I think that at the end of the day because the tech is so early, and because it's there's not like there's not like a playbook exactly on like exactly how to solve these problems. I think that the best product experience will be built by a very focused small team.
1:30:56 Speaker 0 You know, small doesn't mean 10 people, but like, you know, when you start getting these large teams that have a bunch of different stakeholders to try, you know, it should be a great interface and be a good personality, but it also needs to be integrated with the broader product categories that we have and you know, whether that's search or that's you know, reasoning or whatever, and like it needs to be this kind of big integrated thing, and there's a bunch of different stakeholders in the in the kind of process. I think it can, you know, you can kind of lose the magic of like a small focused team that has kind of one goal. And that's Sesame's goal. I think that will be the goal of some teams in these larger companies, but I would I would bet on kind of focus over that.
1:31:45 Speaker 0 Yeah.
1:31:46 Speaker 1 Well, you know, one of the things that was so powerful about the the iPhone was built by a fairly small team.
1:31:53 Speaker 0 Yeah.
1:31:55 Speaker 1 One of the things that made it so powerful soon after launch is they opened it up as a developer platform. Still really locked down block platform. And they they took a very strong point of view on what the quality bar needed to be to be in the app store and so on. But still an extreme. I mean, today the app store is a foundational part of computing.
1:32:13 Speaker 0 Totally, yeah.
1:32:14 Speaker 1 As you're thinking about which parts of the Sesame stack are first party versus open to third party developers, where does the line start and stop?
1:32:24 Speaker 0 I think the the easy answer, the real answer is that we just don't know. You know, there was open AI did this chat GBD plugin uh system before, and I think it's still there probably. And it kind of didn't fully take off really. You know, there's MCP now, which is kind of gaining some momentum more recently and so on, which is kind of more of an open standard style thing. Um I think that it's just early to tell kind of what that looks like. One of the reasons is that I think that the models still need to get better basically to utilize plugins basically, you know, plugins essentially in a way that's kind of reliable enough that someone would will go out and look for a plugin for you know their kind of downstream service of choice because they just want, you know, they have the experience of the AI system kind of able to use other things so efficient effectively and efficiently it becomes part of their daily life and they want to hook up this other thing to it.
1:33:23 Speaker 0 I don't think we're there yet. And so I don't think that like the exact format of how that will look like and where it will plug in is is clear yet. I think it will certainly be a part of the future of these interfaces and these systems. Um but I I think that that will become more and more clear over time as in a kind of organic way, right? I don't think it's gonna be so decided by like, let's say one company or one player. I think it'll be a little bit more, you know, developers want to use these capabilities. An ecosystem.
1:33:56 Speaker 1 Right. And I think it's it's
1:33:58 Speaker 0 a little early to try to build that for us so kind of clearly, but it's certainly a part of of where this goes.
1:34:05 Speaker 1 Right. What should people listening to this podcast who are wondering how they can join Sesame because they want to work on this do next? They
1:34:14 Speaker 0 should reach out. They should reach out to me. They can reach out on Twitter or or over email. And um, what
1:34:21 Speaker 1 are you looking for?
1:34:22 Speaker 0 Yeah, we are looking for so like I said before, like voice conversation is really its own modality. And it's something that we are very focused on trying to improve the state of the art on. And so we're certainly looking for people who are excited about that challenge from like a research perspective, from a core ML perspective. You don't necessarily have to be a a researcher. We need systems engineers, we need infrastructure engineers and so on. But that is a big thrust of what we're focused on right now. Um we're always looking for kind of general strong engineers, especially with like a kind of product bent. You know, we at the end of the day are making a consumer product that is gonna be judged based on how much people like it.
1:35:12 Speaker 0 You know, it's not gonna be judged on benchmarks and and so forth. Those are critical parts of developing the product. But at the end of the day, the question is do people love interacting with our systems? And that's I think part of the culture at Sesame is to kind of merge really world-class technology with a kind of creative taste to make a great product. And I think if you have the engineering talent and you are interested in what we're working on, and in particular you want to see these technologies, LMs, AI, broadly speaking, speech generation be turned into great products that people love, then I think you'll be a great fit. If folks came
1:35:56 Speaker 1 by the office, could they get a demo of what's coming next? They can. They can. Thanks for making the time today. Yeah, it was fun.
Previous Vibe Coding is Here - How AI is Changing How We Build Online Next Why AI Voice Feels More Human Than Ever Page updated August 25, 2026 · transcript and summary are machine-generated
