Personalized AI Language Education — with Andrew Hsu, Speak

Speak (https://speak.com) may not be very well known to native English speakers, but they have come from a slow start in 2016 to emerge as one of the favorite partners of OpenAI, with their Startup Fund leading and joining their Series B and C as one of the new AI-native unicorns, noting that “Speak has the potential to revolutionize not just language learning, but education broadly”. Today we speak with Speak’s CTO, Andrew Hsu, on the journey of building the “3rd generation” of language learning software (with Rosetta Stone being Gen 1, and Duolingo being Gen 2). Speak’s premise is that speech and language models can now do what was previously only possible with human tutors—provide fluent, responsive, and adaptive instruction—and this belief has shaped its product and company strategy since its early days. https://www.linkedin.com/in/adhsu/ https://speak.com One of the most interesting strategic decisions discussed in the episode is Speak’s early focus on South Korea. While counterintuitive for a San Francisco-based startup, the decision was influenced by a combination of market opportunity and founder proximity via a Korean first employee. South Korea’s intense demand for English fluency and a highly competitive education market made it a proving ground for a deeply AI-native product. By succeeding in a market saturated with human-based education solutions, Speak validated its model and built strong product-market fit before expanding to other Asian markets and eventually, globally. The arrival of Whisper and GPT-based LLMs in 2022 marked a turning point for Speak. Suddenly, capabilities that were once theoretical—real-time feedback, semantic understanding, conversational memory—became technically feasible. Speak didn’t pivot, but rather evolved into its second phase: from a supplemental practice tool to a full-featured language tutor. This transition required significant engineering work, including building custom ASR models, managing latency, and integrating real-time APIs for interactive lessons. It also unlocked the possibility of developing voice-first, immersive roleplay experiences and a roadmap to real-time conversational fluency. To scale globally and support many languages, Speak is investing heavily in AI-generated curriculum and content. Instead of manually scripting all lessons, they are building agents and pipelines that can scaffold curriculum, generate lesson content, and adapt pedagogically to the learner. This ties into one of Speak’s most ambitious goals: creating a knowledge graph that captures what a learner knows and can do in a target language, and then adapting the course path accordingly. This level-adjusting tutor model aims to personalize learning at scale and could eventually be applied beyond language learning to any educational domain. Finally, the conversation touches on the broader implications of AI-powered education and the slow real-world adoption of transformative AI technologies. Despite the capabilities of GPT-4 and others, most people’s daily lives haven’t changed dramatically. Speak sees itself as part of the generation of startups that will translate AI’s raw power into tangible consumer value. The company is also a testament to long-term conviction—founded in 2016, it weathered years of slow growth before AI caught up to its vision. Now, with over $50M ARR, a growing B2B arm, and plans to expand across languages and learning domains, Speak represents what AI-native education could look like in the next decade. Full Video Episode Timestamps 00:00 Introductions & Thiel Fellowship Origins 02:13 Genesis of Speak: Early Vision & Market Focus 03:44 Building the Product: Iterations and Lessons Learned 10:59 AI’s Role in Language Learning 13:49 Scaling Globally & B2B Expansion 16:30 Why Korea? Localizing for Success 19:08 Content Creation, The Speak Method, and Engineering Culture 23:31 The Impact of Whisper and LLM Advances 29:08 AI-Generated Content & Measuring Fluency 35:30 Personalization, Dialects, and Pronunciation 39:38 Immersive Learning, Multimodality, and Real-Time Voice 50:02 Engineering Challenges & Company Culture 53:20 Beyond Languages: B2B, Knowledge Graphs, and Broader Learning 57:32 Fun Stories, Lessons, and Reflections 1:02:03 Final Thoughts: The Future of AI Learning & Slow Takeoff This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Alessio Fanelli?

  Hey everyone, welcome to the Late in Space Podcast. This is Alessio, Partner and CTO at Decibel, and I'm joined by my co-host Wix, founder of Small AI.

• 0:12Speaker 01

  Hello, hello. We're back in the studio with Andrew Sugu of Speak. Welcome.

• 0:15Speaker 02

  Thank you for having me.

• 0:16Speaker 01

  I have to start this. I didn't prep you on this at all. But you were a Thiel fellow in 2011?

• 0:22Speaker 02

  First class.

• 0:23Speaker 01

  First class?

• 0:23Speaker 02

  Yeah.

• 0:24Speaker 01

  Is that the one with SBF?

• 0:27Speaker 02

  No, he was, I think, several years later, actually.

• 0:29Speaker 01

  Okay. What was it like? Just talk about the...

• 0:32Speaker 02

  That's a good question. Haven't been asked that one in a while. It was a really crazy idea at the time and very controversial. And I think the first few years of the fellowship were definitely, let's just find 20 people that are 20 and give them $100,000 to drop out of college. And it could be, it was no holds barred. You could do anything. You could be doing some crazy research idea, a startup, anything. And I actually met my current co-founder at Speak. He was in the second year of the fellowship and made many, like, very close friends from the first few years. But I mean, for me, it was life-changing. I had a very unusual path where I was actually, I did finish college, unfortunately.

• 1:17Speaker 02

  I was in grad school at the time because I went to school really early.

• 1:19Speaker 01

  Yeah, I was like, aren't you too old? You know, Thiel likes I'm young.

• 1:22Speaker 02

  I was...

• 1:25Speaker 02

  I was 19 at the time and in grad school. It was a very accelerated path, but I think like, I knew at the time that I was going to leave grad school and do startups anyway. And the timing lined up really well.

• 1:38Speaker 01

  Yeah, yeah.

• 1:38Speaker 02

  Yeah.

• 1:39Speaker 01

  Vitalik, I think.

• 1:40Speaker 02

  He was also in a later year.

• 1:42Speaker 01

  Ah, damn. Okay, anyway.

• 1:44Speaker 02

  But the first two years, I mean, there are some crazy successes. You know, Dylan from Figma. I mean, yeah, like a lot of people.

• 1:52Speaker 01

  Awesome. Well, you know, feel free to bring in those stories as and when. Because obviously, only you know, like, those kinds of people. You are now CTO of co-founder of Speak. I would say, from a very early stage, like, one of the most successful and prominent OpenAI partners that like, anyone would know is like... Doing doing well and like teaching English to Koreans is like your your your rough remit at the time. How did that all come about?

• 2:17Speaker 02

  It's funny that you say that because despite our current sort of revenue scale and objectively, I think, how successful we are, we've always operated in a market, at least initially, on the other side of the world and been much, much more popular in the sort of eastern world and a bunch of Asian markets and relatively unknown in the West. So it hasn't really felt like we've had that sort of awareness until, you know, the past few years, really. But brief story is that my co-founder and I, back in 2016, were fascinated by the promise of AI. And we spent a year sabbatical, basically, learning everything we could. We talked to

• 3:00Speaker 02

  Karpathi back then, actually, when he was like just finishing grad school, and did a lot of sort of self-study research. And we were just so convinced, I think fundamentally, that speech models were going like this, language models were going like this. And in the five to 10-year span, they would become superhuman. And we were utterly convinced of this future. And we saw that the way people learn things and specifically learn languages, which was a very sort of human-based thing, if you really care about fluency, that would completely change. And we'd be able to build language shooters that were pure software, pure AI. So that was kind of the genesis story of Speak. It took much,

• 3:42Speaker 02

  much longer than we expected to build a great product and find good PMF. The first few years were very painful. And I think without this really compelling vision of the future, we would have quit. We actually like never pivoted. Last year, we brought the entire company to Taipei. We do this company trip every year. And we played our original YC application video on screen. And it was really funny, because the things we were saying in that video were the exact same things that I still say today about the long-term vision and what we're building towards. So that was really cool to see.

• 4:16Speaker 01

  Can you summarize the long-term vision again?

• 4:18Speaker 02

  It was that as speech models and language models become superhuman, that would let us create an AI language tutor that would help you become fluent faster than any human could. And I think we're like eighty to ninety percent of the tech is here now.

• 4:35Alessio Fanelli?

  And you have this big focus on like speaking, obviously it's in the name of the company.

• 4:38Speaker 02

  Yeah, that's right.

• 4:39Alessio Fanelli?

  And I think the speech models were maybe a little delayed compared to the text models. Did you ever think about, okay, maybe speech is just not going to work for this use case? Or like what were kind of like the valleys of, you know, discomfort? And then what were maybe some of the pivotal releases and models that you were like, okay, it's gonna work, it might take a little longer, but it's gonna work.

• 4:58Speaker 02

  So we've always done custom speech stuff. The first act of the company, if you will, was before LMS, right? Before twenty twenty two, when Whisper came out, when ChatGPT came out. In the years before that, you know, like roughly two to three years, is when we feel like we found PMF in South Korea and then started growing still only in that market, still only teaching English. And we developed custom speech recognition models and users were speaking into the app all day. So we had a ton of this non-native English speaker data. And we would use that to fine tune models, understand our users better. We still do that today. And it's important for us for the core recording loop in many of our lessons that it's extremely fast. So we're very latency sensitive.

• 5:42Speaker 02

  There's many other sort of product surfaces within the app today that are more L M powered, where it's more open-ended, real tutoring, where we actually give you feedback on what you said in the semantics and so on. So that stuff is more like Whisper powered, more L M powered. But we've always had like a very fast core ASR loop that's been fully custom.

• 6:02Alessio Fanelli?

  I just on boarded to the app earlier today. I like other apps, there's kind of like this tutor conversation that you do for onboarding. I'm guessing that is mostly L L M based and then you're kind of judging the person response. So I selected Spanish and the conversation was in Spanish via text to start. And then from there, started to create lessons for me.

• 6:19Speaker 02

  Yeah.

• 6:20Alessio Fanelli?

  Was that all unlocked from L L M's where now you can kind of have these conversations and then bring people into the speech flow?

• 6:25Speaker 02

  Yeah, so before that, so we call that magic onboarding. And it was a new thing we built that was. More conversational. We wanted it to feel more like you were talking with a tutor and they were sort of learning things about you and we would use that later to personalize the experience. Before that, we had like a much more traditional app onboarding. There's still a lot of open questions, interesting questions around what is the proper onboarding UX? Because a lot of people start using Speak and they're not in a situation where they can actually speak aloud. So we have like, you know, fallback outlets and so on, but it's something we're like super actively experimenting with.

• 6:58Speaker 01

  Is there a structured output behind that? You know, anything that you found implementing magic onboarding? I think people always want to improve onboarding. What's the uplift or was there one? We still don't know yet.

• 7:09Speaker 02

  The interesting thing is that in general, because it's speaking based, which is a much higher barrier than just like tapping a multiple choice button. What we see is that install to signup rate is a decent amount lower, but trial start rate is higher. It's still something that we have an active experiment that that's running and we're trying to be super agile about testing many different sort of like formats of this. I don't think I have like the final answer yet.

• 7:36Speaker 01

  Yeah.

• 7:37Speaker 02

  But I think the intent, the really like vision that we're going for here is that as soon as you download the app from the App Store, maybe you see it in an ad. The first thing that the first interaction, when you have like a fresh open of the app, should feel pretty futuristic. It should feel like, okay, this is like the new AI native, next gen way of learning a language to fluency. And that's kind of been always our ambition. Like, we want to build something that wasn't possible before without LM and like AI technology.

• 8:09Speaker 01

  Yeah, I think I wanted to go back on the onboarding soon, but there's a general idea of like when you replace a form with a voice bot, that you need to have some kind of state machine behind the hook, the thing to drive. Like, what else don't I know about you? Let me proactively ask that. And I'm just wondering if you had any insights there or is it literally just a state machine?

• 8:31Speaker 02

  We tried both, actually.

• 8:32Speaker 01

  Yeah.

• 8:32Speaker 02

  Right now, I think probably what you saw is a state machine, but I think that.

• 8:37Speaker 01

  Just the AGI.

• 8:38Speaker 02

  Yeah, right. I think that things should move in a direction where it's much more of a natural conversation. There is a general sense of a goal in the prompt that you can specify. And part of the hard thing here is all the guardrails, right? Like, once you start talking about what you had for breakfast yesterday, right? And trying to be antagonistic to the system, then things start really going off the rails. So for a bunch of these experiences, we're pretty careful about the fallbacks and we have a lot of evals around that. But I think where it should end up is just feeling like you have a quick three to five minute conversation with your tutor and then it knows a lot about you. And then you create your account, et cetera.

• 9:19Alessio Fanelli?

  And you create memories?

• 9:21Speaker 02

  Yeah, so we store what you're saying. We summarize. In the experience, the way it works is the tutor will ask you some sort of question like, what are your goals around learning English or the language? And then we will basically use a separate L M prompt to summarize. So it's not the full transcript for what you said that you see. It's more of like an abstracted, okay, here's what you care about. And we think that's a better product experience.

• 9:46Alessio Fanelli?

  What were some of the other key tenants on the product? Obviously, language learning is like one of those consumer markets where dozens of companies always trying to get started and you get these old companies like Babbel and you got Duolingo. So speaking, the act of speaking was like a big part of it. I think this memory stuff is great. I think if you tried some of the other apps, it's like they always start to re-ask you the same things that you got wrong before, but you're not really learning. Is there anything else that is maybe not as obvious from the outside in the design of the app and the product that is like you think is really different?

• 10:18Speaker 02

  I would say from a macro level, this is actually a pretty new product category, AI-powered language learning. And all of these apps that you mentioned, Duolingo, Babbel, et cetera, they're more of like the Gen two of language learning. So like if you think about Gen one was Rosetta Stone, if you remember, right? CD-ROMs, in your ports. And then Gen two was basically mobile. So you have these very casual, massively popular mobile apps like Duolingo that I think the comp there is probably closer to a mobile game. Something that feels productive, something that's very engaging, very gamified. And Duolingo is very leaning into it, the gamification. And they've done an amazing job of that, to be clear. Yeah, they might be the world's best people at it.

• 11:00Speaker 02

  Yeah. And our view is that LLMs and AI now enable Gen three of language learning, which is something that is very AI-native, very focused on functional fluency, which is why we do all these role plays and let you practice Spanish by talking to your Uber driver. We don't teach vocabulary and grammar. We teach sentence patterns and we try to get you to just repeat and drill and drill and drill, almost like you're in a gym, until it's automatic. Because that's what speaking is, right? Like, has to be spontaneous and automatic. In terms of the other aspects of the design, though, we went through many, many iterations over the first few years of starting the company. This is kind of what I was mentioning about, it was like really painful in the first four or five years.

• 11:40Speaker 02

  And in fact, the current version of the Speak app is not the first thing that we launched. We had something that we call internally the Red app, which was like a red app icon, still a similar logo. And it was more around packs of content instead of courses, where you could sort of choose any topic that you wanted to learn. It was for many different languages for learning. It was essentially like not a very directed experience and it didn't really work. It was free. It was a very basic thing. But we, in 2018, tore everything down and realized that we had to really fully change what we were doing. And that's when we decided to focus on South Korea specifically, on teaching English.

• 12:25Speaker 02

  We built a bunch of new lesson types and we created our courses so that the experience was much more on rails. We realized people don't want to choose, they're already using some of their motivation on a daily basis just to open the app. They don't want to make another choice after that, right? Just tell me what to do, right? Like, you know, give me a big button and then I can tap it and just start a video lesson or whatever. We also, pretty critically, I think, abandoned the free version and just went straight premium. And we kind of sidestepped the motivation question that way. Because we knew that there were a ton of users that really wanted to learn English and were already really motivated. So we wanted to basically filter for these users. So, you know, like, it was, I wouldn't say there was one silver bullet.

• 13:08Speaker 02

  It was kind of the combination of many learnings over three or four years. And then that started really growing in South Korea. And from there, I guess, like, phase two was really 2022, when LLMs came out and Whisper came out. And that allowed us to go from this more supplemental speaking practice tool to more full-featured language tutoring, where we could use LLMs, like 3.5 Turbo back then, to give you direct feedback on your wording and on, you know, like, that was kind of a weird thing to say. A native speaker would say it this way, or use a different word, whatever.

• 13:45Speaker 01

  I always do a poor job of doing this, but can we get some headline numbers, like, just to get a sense of scale? Because I think maybe some audiences don't know, where are you at now in terms of your reach?

• 13:55Speaker 02

  So we're now the biggest English app in South Korea.

• 13:58Speaker 01

  Yeah.

• 13:58Speaker 02

  We do billboards, big celebrity campaigns, that sort of scale, like, we're, you know, very popular there. I think, like, 6% of the Korean population has tried us. We're, you know, like, well on the way in a bunch of other Asian markets, like Japan, Taiwan. So the Asian markets are currently our mainstay. We also teach English in 40 more countries. We're coming to the US as well, launching, I mean, we have Spanish, French, Live, and several more languages are coming this year. That's a huge focus of the company right now. In terms of revenue scale, well over 50 million ARR. It's a pretty simple business model. It's like, mostly consumer.

• 14:37Speaker 01

  Subscription, yeah.

• 14:37Speaker 02

  The B2B stuff is super, super exciting, and that's also growing really fast. And I think it'll be a really meaningful part of the business.

• 14:44Speaker 01

  When did you start B2B?

• 14:46Speaker 02

  About a year ago. It would, it kind of, it was like, very much a side bet slash experiment at first, and then it just started working. And

• 14:53Speaker 01

  Of course it's gonna work.

• 14:55Speaker 02

  Yeah. And now it's like, okay. You know, this is part of the future, right? This is a real thing. Yeah, so that's exciting.

• 15:01Speaker 01

  What's the B to B raise between learning language and like real time AI translation? At Google, they're like one of those like Google Beam things for like, you know, for conferencing, they do real time translation like.

• 15:12Speaker 02

  Yeah, yeah, yeah. So people always ask this, right? There there was like, what happens when the Babbel fish comes, right? When the real time translation comes.

• 15:20Speaker 01

  And Babbel fish is the Hitchhiker's Guide.

• 15:22Speaker 02

  Right, yes, exactly. The counter example that I always have that I think is quite illustrative is in German, the verb is at the end of the sentence, right? So if you're trying to do real time translation from German to English as an example, you can't actually make any progress on the English until you hear the whole German sentence and you know what the verb is at the end, right? So like the minimum latency there is is the full sentence. And that that's like an example of the technical blocker for like why it'll never be truly, truly perfect. But also, I think besides that, if you talk to all of our users in Asia, they don't want a translator. The reason that they are trying to learn English is to make themselves a better person

• 16:06Speaker 02

  to connect with other people. Like, they want to be able to look you in the eye and speak English, speak the same language as you, right? So it's actually like a very different thing. I think what will end up happening is that we will build a real time translation feature into speak and have it integrated into the learning experience.

• 16:22Speaker 01

  And also, like, there's always that human side, right? Like, I'm dating a Romanian woman.

• 16:26Speaker 02

  Yeah.

• 16:27Speaker 01

  His wife is trying to learn Italian, like, there's always that.

• 16:30Speaker 02

  Yeah, absolutely.

• 16:31Speaker 01

  Going to keep happening. I want to double click on Korea.

• 16:34Speaker 02

  Yeah.

• 16:34Speaker 01

  I think it's like a very insightful, smart decision. Maybe people only know Korea through K-pop. But actually, I think a lot of Americans learn Korean because of K-pop. That's a that's a side thing. But like, you could have done Taiwan, you could have done China. I saw, I remember starting a documentary about how China was crazy about English or mad about English. You know, it was the title of the documentary. Was it obvious? Were you sure when you went into Korea or was it just a test?

• 16:59Speaker 02

  We visited a bunch of Asian countries when we were thinking about how do we relaunch things, how do we focus in? And we almost chose Taiwan, actually.

• 17:07Speaker 01

  Yeah.

• 17:07Speaker 02

  But I think it was a little bit serendipitous. So our first employee is Korean and was my co-founder's college roommate, actually. When my co-founder visited Seoul to check out the market, he asked. SJ to come along as essentially a translator and to like, you know, facilitate. And I think that just went really well. And it was just very obvious from being on the ground in the market that Korea is pretty obsessed with learning English. And there is every human based solution possible, right? You know, like, English academies, classes, skyscrapers full of classrooms, stuff like that. And

• 17:51Speaker 02

  our logic was basically, if we can really make headway and win this market, that is chock full of these human competitor products and all these people that fundamentally care about fluency, then we probably have something pretty real and strong PMF that we could win other markets with. So that was the original logic. And, you know, so far, it's been working.

• 18:13Speaker 01

  It's retroactively obvious, which is the best kind of obvious. But like, it's so counterintuitive that you would be the team to do this and not a Korean team, right?

• 18:21Speaker 02

  Yeah.

• 18:22Speaker 01

  Where they would be, they would know because they had personal experiences. Like, I started in Korea and I learned English. Here's how you do it.

• 18:28Speaker 02

  Yeah. In hindsight, it's super weird, right? Like, we were definitely, you know, sitting in an office here in San Francisco, operating with users in a market all the way on the other side of the world. It would not have worked without Sunjae. I have to give him a lot of credit here because we paid a lot of attention to the specific wording of button text in the app and locals, you know, like localized strings. We had a lot of reports from users pretty early on that they were shocked that it was an American company. Like, they thought, dude, right? Because you can always tell. There's always some weird wording or whatever, but there wasn't in speak. And I think that probably had like a large sort of non-tangible effect.

• 19:07Speaker 01

  Yeah. Focus, attention to detail. Tech stack. You, this was 2018. What were you rolling? You just did ASR and there's no LLM. So BERT, but maybe?

• 19:18Speaker 02

  We actually had really no LM component of it. So all of the content, oh, yeah. Another thing we did that I forgot to mention was, we decided we needed to fully own all the content. So the way that we teach. All in-house, all sort of thought from first principles. We build this thing called the SPEAK method, which is basically like a pedagogical philosophy around teaching sentence patterns that you drill and then sort of combine into higher order patterns. And all of that was in-house with, you know, our content team and our teachers. And we build a lot of internal tooling to make this possible. There is just a lot of operational overhead, I would say.

• 20:03Speaker 02

  This is something we've struggled with to scale to many more languages. And that's like a big research effort within the company right now. We're building a mobile product, right? My co-founder and I have always just loved apps and been big iPhone users. So we cared a lot about the app being native, feeling great, being high performance. The DNA of the company was always consumer. Frankly, my co-founder and I had never worked in a real company. I dropped out of grad school, had a few failed startups and then eventually started SPEAK. And he had never worked in a real company either. He just startups in the past. So we didn't know anything about enterprise. Enterprise workflows or like what sort of software real companies used. So I think frankly, consumer was the only path.

• 20:48Speaker 02

  I don't think we could have done anything else. We just didn't know enough.

• 20:53Speaker 02

  And I think that has served us well, though, in terms of just really caring about the craft of it and wanting to build something that felt not 90 to 95%, but 95 to 100% in terms of polish.

• 21:07Alessio Fanelli?

  Was it hard to build an engineering team that did that at the time? Because ML engineering is very academia driven back then. And then you have like the more consumer stuff that it's maybe more nascent. And it's mobile.

• 21:20Speaker 02

  I'm now realizing that our story is very weird. So

• 21:24Speaker 01

  You only just realized?

• 21:26Speaker 02

  In addition to the market on the other side of the world, our first iOS engineer that we hired through a YC referral was in Slovenia. If you don't know where Slovenia is, look it up on Google Maps, but it's, you know, it's like a pretty obscure little country.

• 21:40Speaker 01

  A little nowhere.

• 21:41Speaker 02

  Yeah. And then we needed to hire a backend engineer and one of his best friends was a great backend engineer and we hired him. And then this happened four more times, all in the same city. And then we were like, okay, we should probably just open a physical office. So for

• 21:55Speaker 01

  Slovenia?

• 21:55Speaker 02

  Yes. So for several years, we had an engineering office in Slovenia.

• 22:00Speaker 01

  What?

• 22:01Speaker 02

  And then a few people here in San Francisco, and we still do. Now we have 90% of our core product development team in San Francisco here. Office and FIDAI, we're really only hiring here. But for the first like several years that, you know, that was like another very interesting sort of cultural aspect of the company, I guess.

• 22:21Speaker 01

  I think a lot of early stage founders have to do that. That's the only people they can afford or whatever.

• 22:27Speaker 02

  Yeah.

• 22:28Speaker 01

  What are your tips that make that remote stage work?

• 22:31Speaker 02

  For us, it wasn't really a price thing.

• 22:34Speaker 01

  Yeah.

• 22:35Speaker 02

  I think legitimately thought he was the best person that we interviewed. And then it just kind of happened that way when you roll it out.

• 22:43Speaker 01

  Yeah, it's not a price. It's more about remote work, right? Like distributed team.

• 22:46Speaker 02

  Yeah.

• 22:46Speaker 01

  Early stage, like a lot of people say like, no, you have to move everyone to SF or your startup will die.

• 22:51Speaker 02

  Yeah. I don't think that we were good at remote work. I don't think that my personality or my co-founder's personality is inherently very good at async, just to be perfectly frank. I actually think that almost like in spite of it, we made it work. It was a little bit brute force. Like I would just sync with them every single day. Right? And there was pain because the time zone overlaps. Like it was like exactly the most inconvenient.

• 23:19Speaker 01

  Yep.

• 23:19Speaker 02

  But I think for several years, we did that. We got really good at the cadence of it. I think they were excellent engineers as well. So. It worked out, but if I had to do it over again, I probably wouldn't do it. It's hard to say, yeah.

• 23:32Speaker 01

  Shall we move to phase two on the LLM side? That's when OpenAI started opening up and when did they invest?

• 23:38Speaker 02

  This was 2022, so that was also when Whisper dropped. And Whisper was a really exciting moment for us. It was actually since we started the company and made that prediction of, okay, in five or 10 years, speech models, language models will become superhuman level. Whisper was really that magic moment for us where we were like, oh shit, I think what we predicted is here. And I pretty distinctly remember this moment in the office when we got access to the model and we were testing it on an audio clip of like a very beginner English learner in Korea saying something. And it was, if you close your eyes as a human, you'd have no idea what they were saying.

• 24:22Speaker 02

  There were four of us in the room, we all closed our eyes and none of us had any idea and the model got it right. So I mean, superhuman. I think that was the moment that we had been waiting on. And at the same time, LLMs were on the ascendancy. ChatGPT would come out, I think on Thanksgiving of 2022 and 3.5 Turbo came out. And I think like, we kind of realized very quickly that all the pieces were clicking now, right? Like, we have what we need at our fingertips now to go from something that was listen and repeat, where the user would see something on screen, hear a reference of the teacher saying the thing and then they would just repeat the thing, right? It was like very simple.

• 25:04Speaker 02

  Still a great product, by the way, you know, still grew to like several million AR in South Korea. So clearly there was like a big market need for that.

• 25:12Speaker 01

  Pre Whisper?

• 25:13Speaker 02

  Yes.

• 25:13Speaker 01

  Wow.

• 25:14Speaker 02

  This is from like 2019 through 2022.

• 25:16Speaker 01

  Yeah, that's the grind. You needed to hang in there.

• 25:19Speaker 02

  Yeah. And again, I think that if we, there were many moments when things weren't working from 2017 through 2019. We were looking in the mirror and we were like, why are we doing this? This is, this is crazy. But I think we were so convinced about the vision. We just like couldn't believe that the vision would not come true. So we stuck with it. So fast forward to 2022, the pieces started coming together. We realized that we could start building something that felt more like a language tutor. That could give you feedback, that could start explaining to you why you did something wrong. And that was act two of Speak, True English Tutor.

• 25:59Alessio Fanelli?

  This is something that a lot of founders struggle with today. It's like, I'm kind of building something, hoping that the models get better later.

• 26:07Speaker 02

  Yeah.

• 26:07Alessio Fanelli?

  How did you feel once the models got better? Did you feel like, okay, I am ahead of the curve because I built all this history of building product and like doing all this work? Or did you almost feel like, okay, we spent all this money and time building these models and now we're just gonna use Whisper?

• 26:21Speaker 02

  It was purely positive for us. We still kept using our custom ASR system because it was streaming real time, really fast, really well fine tuned. Whisper wasn't streaming, so it was a different use case. We used it for the more spontaneous stuff. And I think in almost every way, we were just really excited because pretty directly, as the frontier of model intelligence improved, it would just unlock things on our roadmap that were locked before, if that makes sense. And we still really operate in that mode today where we take a model and then we try to think about, okay, how do we saturate model capability by building product on top of it? And then it happens again, right?

• 27:05Speaker 02

  And then we build and saturate the model capability again. I think that's a really cool paradigm to like, you know, think about. But all the LM stuff basically allowed us to build a tutor for English. And we still didn't have like real time voice, for example, right? But the barriers are coming down now. Obviously, it's a really hot topic. We're actively building out a real time voice platform that we can build a lot of more verticalized, specific lesson experiences on top of that I'm super, super excited about. I don't think they're going to replace our current lessons. They're going to be more immersive, just a different thing, probably for more advanced learners.

• 27:41Alessio Fanelli?

  Still language learning, though, not broadening out from language?

• 27:44Speaker 02

  Yeah, so I think that language learning is interesting because. It is so universal. Ninety nine percent of people you know have certainly tried to learn a language. And it's so hard, right? Becoming fluent just has a huge failure rate. And it's something people are willing to pay for. So I think that has been just like a pretty amazing beachhead for us. And I think we'll be doing language learning for a long time. There's a huge, huge, huge company to be built here, but our even longer term ambition is really this idea that even beyond language, we think AI will reinvent how people learn anything, right? It already has for me, right? I use ChatGPT to learn things every 10 minutes. And I think I'm just naturally like a very curious person. So whenever

• 28:29Speaker 02

  I'm thinking about something, I want to know more about it and then I'll naturally go to ChatGPT. And then I'll learn about it. It's unlocked this like entirely new dimension of learning. And I'm spending way more time learning as well as an adult, which is really cool. And I want to bring that in a more sort of structured systematic way to everyone. So I think that that's like the vision beyond language.

• 28:51Speaker 01

  I'm curious to sort of double click on to just the tech side. We talked a little bit about the content that you that you own and develop in house. And we talked a little bit about the onboarding memory. I assume that you have conversational memory as you as you go, right? And any other major pieces of the puzzle that really unlocked it for you?

• 29:08Speaker 02

  So there's a few things I can talk about. I think one thing is in order to go from teaching English to teaching a bunch more languages, we needed to really figure out more direct AI content generation. That was a pretty right, because it's hard to scale like our little studio in LA where we shoot a lot of the video lessons. All of the scripts were written manually before by our content team. But we want like a hundred X more content, right? And 10 X more languages. Eventually a hundred X more language pairs, which is how we think about it. It's like, what's your native language? And then what language are you learning? And really, the only way to do that is to make it more AI generated.

• 29:53Speaker 02

  And, you know, very much like a AI native company, we want to be on a frontier here. We want To keep a small team and to have as much leverage as possible through these types of tools. So that's a big, active area where we're building out, I think, using, you know, people overuse the word agent. But we have a tutor agent, we have a curriculum writing agent, we have a giant LM based pipeline that creates curriculum, scaffolds it in the right way, writes the lessons themselves. That's a big, active area that will basically help us to scale to a lot more markets and a lot more languages. So that's like one big thing. Another big thing is we care a lot about fluency, obviously.

• 30:37Speaker 02

  Specifically, we want to be able to quantify how fluent you are. So if you're learning Spanish, it's like, okay, what, what does it mean to be fluent, right?

• 30:46Speaker 01

  There's some real world test for that.

• 30:47Speaker 02

  We care about real world fluency, your ability to go to Mexico City and go to a street taco stand and actually order. Right? That's very functional fluency in one aspect. You might be really good at that, but be completely unable to like, talk about your family. Right? So the frontier of fluency is very jagged, but we're very pragmatic and we care a lot about meeting user goals and helping them become fluent at what they care about. And we're thinking a lot about, okay, how do you quantify that? How do you actually store a knowledge graph of everything you know about Spanish? In terms of the vocabulary you know, or you don't know this, you know,

• 31:27Speaker 02

  the patterns that you know, or you don't know, the mistakes you made using Speak over the last month that are clustered.

• 31:34Speaker 01

  You said the magic word of knowledge graphs. Is that live? Is that experimental?

• 31:40Speaker 02

  There are aspects of it that are live and it's, it's a very sort of multi-dimensional system where we think of it as there are many aspects of fluency, right? There's many sub-scores and we have a few of them that are currently live and we're actively developing other aspects of it. And then all those will fold up into a more holistic fluency score. The idea is that eventually, once we have a complete enough picture, everything will fold up into a number that we call the Speak score. That is a very sort of holistic measure of just like, how good are you at Spanish? Right? And obviously 54.

• 32:18Speaker 02

  It's kind of meaningless by itself, but it does give you a general sense, right? Like being at fifty four versus being at five is very different, right? And I think everyone can kind of intuitively understand that.

• 32:30Alessio Fanelli?

  It's surprising. Like, I would have grounded it more in real world. Like, we will get you to pass this exam that is a standard that is like the ESL standard or whatever.

• 32:39Speaker 02

  So the way that we think about that is we don't really teach for the test. I think it's possible in the future that we'll do a test prep product. But in general, we care about real world proficiency in various functional situations. So the way that we think about it is if you're at this level, then these are the things you can do. Right? So it is exactly that.

• 32:59Alessio Fanelli?

  We have that a lot in Italy. I grew up in Italy, so English is my second language. And there's a lot of people that pass a lot of tests and like get high grades in all the classes. And then they travel to the US and the UK. And it's like hard to speak because they don't, you know, I feel like the hard part is like being in the conversation, you know? It's I think like when I started, my brain and reading was like much higher than that conversation. Which like doesn't really help you if you're like traveling somewhere.

• 33:24Speaker 02

  That's me for Chinese because my my parents spoke Mandarin to me growing up. So I can understand like a non trivial amount, but I'm very bad at speaking.

• 33:32Alessio Fanelli?

  Hmm.

• 33:32Speaker 02

  Yeah.

• 33:33Alessio Fanelli?

  I heard there's a good language learning products.

• 33:37Alessio Fanelli?

  I have one question on the course generation.

• 33:39Speaker 02

  Yeah.

• 33:40Alessio Fanelli?

  How do you eval that product? Like when you're asking the AI to generate courses, how do you figure out the courses are going to be good?

• 33:47Speaker 02

  Rely very heavily on our content team. And we are trying to build out an eval suite. It's really hard. There's the illustrative example here is that as we try to hire and train new content writers on our content team. It's so nuanced. There's many different aspects of training them in the speak method and how to write the right types of lessons and articulating.

• 34:15Speaker 02

  Why this form of lesson, which is subtly different from this other form of lesson is better. Right? So we try as hard as we can to articulate that. So I think like forming like a sense of evals, using model gridded evals like that. That's one piece of it. And I also think like in the future, a really good curriculum or lesson writer agent will probably be like reinforcement fine tuned on a lot of our internal data as well. That's something we're experimenting with, but it's still pretty early.

• 34:46Alessio Fanelli?

  This seems like a great example of like, you know, AI removing jobs, which is like, oh, you're creating the courses with AI. You don't have a person, but it's actually like, instead of one person creating two courses, like reviewing 50 courses. Yeah. I generate, that's kind of how you're seeing the content team.

• 35:00Speaker 02

  The way that we see it, really not just for, for our content members, but also I think it's perfectly applicable to engineering is that it's, it's leverage. It just allows you to do a hundred X in the same amount of time. We still need human review of the syllabus, the curriculum, the specific lines, et cetera. But the hope is that this will allow us to launch a hundred X more courses.

• 35:22Speaker 01

  A lot of language is colloquial. I think that the way that you put it on one of our episodes one time was the Italian that is taught in school is not the Italian that Italian speak.

• 35:32Speaker 02

  Yeah.

• 35:33Speaker 01

  How much of that do you adjust for informal versus formal?

• 35:37Speaker 02

  Entirely. That's one of our fundamental tenets, which is that we don't teach textbook English or textbook language. Like we try very hard to.

• 35:47Speaker 01

  Teach Gen Z slang.

• 35:49Speaker 02

  We don't go quite that far, but.

• 35:51Speaker 01

  Slay kings.

• 35:52Speaker 02

  We, we tried to teach very casual conversational language that is actually what real people use. And like you said, that's usually very, very different. Like if you pick up like a typical English textbook in Korea, it's all really. Traditional and weird formulations and it's not how people actually speak.

• 36:12Speaker 01

  Yeah.

• 36:12Alessio Fanelli?

  I know you're gonna release Italian soon, so I can give you a hand on that. I don't the US, there's not that many dialects, there's like accents, but like most of the language is like, like the words that people use are similar. Because I know, for example, Spanish is like, you know, Spanish spoken in Argentina is like very different than Spanish spoken in Mexico. How do you kind of adjust for that? Or maybe you don't.

• 36:33Speaker 02

  But so I would say that, for example, currently we teach American English, standard American English. We don't really teach other accents or other dialects. For now, given how small we are, we just have to be pragmatic and teach in the direction that most people want and most of our users know. So we've made those decisions like on the content team side for American Spanish, every language that we're teaching. But I do expect that in the future, we're going to get a lot more sharply differentiated. Like if you want to learn British English, then we'll teach you British English, we'll teach you how to pronounce it, et cetera. I think all of that feels like something that superhuman language, you know, tutors should be able to do.

• 37:12Speaker 01

  I just think it'd be very funny if all the Koreans had like a very distinct southern accent. It'd be, it'd be great to make that happen.

• 37:19Speaker 02

  Yeah.

• 37:20Speaker 01

  I do think about this because, you know, obviously there's a moving of the goalposts. Like now that we have this, now we want the next thing.

• 37:27Speaker 02

  Yeah.

• 37:28Speaker 01

  And obviously, people who are, English is a second language, always have an accent. Like I haven't, like, a lot of people think I don't have an accent, but if you know any Singaporeans, you know, I'm a Singaporean. How much accent training is important, right? Like, I think, like, actually, that does help a lot with, for people.

• 37:44Speaker 02

  Yeah.

• 37:44Speaker 01

  And you cannot tokenize accents yet.

• 37:47Speaker 02

  Yes, that's right. So I have two main thoughts on this. I think the first one is that communication and your ability to speak spontaneously and get a concept across, an idea across, is almost fully orthogonal to pronunciation. You can be really bad at pronunciation, but still communicate effectively. So a lot of the current core product experience is about just speak as much as possible, make mistakes, don't worry about.

• 38:18Speaker 02

  Screwing something up on the accent or the pronunciation side.

• 38:21Speaker 01

  Yeah.

• 38:21Speaker 02

  The important thing is that you literally move your mouth and you make the sounds. Right. And it turns out there's like a really key psychological barrier there. Where people are just not willing to do this in front of a human, even if it's a human that is a teacher that you're paying. Right. So a lot of the core message of our marketing campaigns in many of our like biggest markets is along lines of. But you can make mistakes in this private space with speak. And I think psychologically, that's extremely powerful. And then you can go and get it right more confidently in the real world after you practice with speak. Now, having said that, people do care about their pronunciation and their accent, right? So we, we have for English only right now, a pronunciation coach.

• 39:04Speaker 02

  That is basically like a fine tuned version of Wave to VAC, which is a meta model. But we basically fine tune it on a bunch of our own phonetic transcripts, like fine tuned data. It works pretty well. It's currently for single words. We're going to expand it to full sentences to more languages, et cetera. But I think that just if you look at like the pure market opportunity. Our sense is that we really want to push people to just speak very freely as much as possible. You know, just get that volume up.

• 39:39Speaker 01

  Yeah, yeah. In terms of immersing language learning in the real world. One of the more interesting approaches that people keep trying is to have, let's say, like a Chrome extension or something on top of a page. I think Toucan was doing this.

• 39:52Speaker 02

  There's a bunch of those. Yeah.

• 39:53Speaker 01

  Yeah. And then there was another one I saw recently, which is like, watch a YouTube video and it'll transcribe for you, but randomly mask out.

• 40:02Speaker 02

  I saw that too. Yeah, yeah, yeah.

• 40:03Speaker 01

  That was like a show hacker news.

• 40:05Speaker 02

  Yeah.

• 40:05Speaker 01

  Do those work?

• 40:06Speaker 02

  There's kind of the question of is, you know, is that the right product, right?

• 40:09Speaker 01

  Yeah. So basically the difference is your content or real world content, right? Obviously you want real world content.

• 40:15Speaker 02

  I think that for work, right? So for speak for business, for the, for the B two B product. Another part of the vision is really like, what should? A superhuman language tutor be able to do. It should probably be able to handle kids as well as a Samsung employee that wants to transfer to the US office and wants to use Speak for work. Right? So our view there is that it's the same product. It's a different distribution mechanism, right? Consumer versus B two B. And I think that we will eventually build something like a Mac app. Maybe it'll be integrated with the browser in some way. We're not really sure yet, but obviously in order to apply it to your day to day,

• 40:58Speaker 02

  there needs to be some way to hook into your actual sort of work documents, whatever. That's a whole can of worms. We are actively thinking about it, but I think my sense is that it's not clear to me that any of these products have really taken off. And I think that there's many other approaches that are possible. I don't have the answer, but like another example, very hypothetical future world is maybe OpenAI, you know, the new Johnny I thing will come out with some hardware that will be listening to you all day. And then we can, you know, give you some sort of like very deep analysis that is integrated with the Speak app at the end of the day or like, you know, the end of the week, whatever. I don't know.

• 41:40Speaker 01

  Okay, remember time since you brought that up? I'm sure you don't, I should, they haven't told you anything, but what?

• 41:44Speaker 02

  I don't know anything.

• 41:44Speaker 01

  What's it gonna be?

• 41:45Speaker 02

  I don't know anything.

• 41:46Speaker 01

  It's like a very, like, this is the number one topic in all the parties I go to now.

• 41:49Speaker 02

  Really?

• 41:49Speaker 01

  Yeah.

• 41:50Speaker 02

  What's the most compelling idea you've heard?

• 41:52Speaker 01

  Okay, so there's, there's people that say Joni hates wearables.

• 41:55Speaker 02

  Yeah, I've heard that too.

• 41:56Speaker 01

  And I'm like, if it's not wearable, then you've just made a second phone. In that case, just make a phone.

• 42:02Alessio Fanelli?

  Yeah, I thought they said it was gonna, I mean, didn't Sam say that it was gonna, he wanted to do a phone in the past?

• 42:08Speaker 02

  That was in the far, that was like in the far past.

• 42:10Speaker 01

  He says a lot of things.

• 42:11Speaker 02

  He'll say a lot of things.

• 42:13Speaker 01

  Yes. Okay, anyway, I think wearable makes sense. I think the, the, the race is to capture context.

• 42:18Speaker 02

  I mean, I, I have a wearable.

• 42:20Speaker 01

  Yeah, I, I have, we have a wearable here too.

• 42:22Speaker 02

  Yeah.

• 42:22Speaker 01

  Transcribe everything.

• 42:23Speaker 02

  That transcribes everything?

• 42:24Speaker 01

  Yeah.

• 42:24Speaker 02

  That's cool.

• 42:25Speaker 01

  Yeah, it's a previous episode of ours with.

• 42:26Speaker 02

  Oh, cool.

• 42:27Speaker 01

  I can hook you up if you want.

• 42:29Speaker 02

  Yeah.

• 42:29Speaker 01

  But yeah, I think like, it's something that a lot of people are interested, obviously, because it's a huge bet by them and. Yeah, curious. Okay, you mentioned video. I just wanted to double click on that a little bit. I'm sure engagement very high for video because people love to watch video. I thought that speak would be one of those places where like you just kind of leave it in your pocket. You walk, you take, take a walk, learn to speak. Probably that's not true.

• 42:52Speaker 02

  What we've done so far is part of the course experience is a teacher video. We've tested other more audio forward types as well. We found that, of course, like you said, video is very engaging. But at the same time, we have a lot of users that do want to be able to walk around with the phone locked in their pocket. So doing something that is more like voice mode with optional visuals, I think, is really good. I think there's huge opportunity for a better way to learn things like listening comprehension. So I took German in grad school for two years and I thought I was getting somewhere.

• 43:36Speaker 02

  But anytime I listen to a native German speaker, it's so fast. It's, it's completely on a different level. And I think you can imagine a plethora of really cool experiences that feel kind of like you're listening to a podcast. But it's all AI generated, it's fully controllable, it's integrated with the app. You know, there's something there for sure. Yeah.

• 43:57Alessio Fanelli?

  Don't want to do AI podcasts, man. We're cooked.

• 44:02Speaker 01

  It's okay. We'll, we'll document the, the own ending. I, I mean, I think when that happens, we just end the show. Like, why not, Nick?

• 44:08Speaker 02

  To zoom out a little bit, in the pretty near future, multimodal models will cross the threshold where they will be able to generate images a lot faster than they currently are. Maybe somewhat close to real time even, right? And audio at the same time. Text at the same time. And you can imagine just like a very powerful multimodal tutor that can kind of do it all at once where there's an audio track. And then if the teacher is teaching you something with the right timing, it chooses, okay, at this point, I'm about to introduce a new concept. So I'm going to show the word on screen so that the user can see how it's spelled. Right? There's a lot there. You can do generative UI, a lot of nuance there where it's easy to do it badly, but

• 44:53Speaker 02

  to do it well requires a fair amount of reasoning and mental modeling of what the user knows.

• 44:57Speaker 01

  Yeah.

• 44:58Speaker 02

  Which feeds into what you need to show at what time. So that's probably going to have to be like a pretty parallel set of systems.

• 45:05Alessio Fanelli?

  Have you spent any time looking at this like, you know, like VO3 where you do video plus audio at the same time on how you can tweak the audio part versus the video part? Because I can imagine you might work on a video part and then you want to change the audio generation model. I don't actually know how the model works inside on like how much you can.

• 45:24Speaker 02

  We haven't really looked at the video stuff much. We basically think that we're very bandwidth constrained. Right? So we're just scaling and trying to hire as fast as possible like everyone else is. And as a result, we're really focusing on just like the most in-reach, highest impact things. I do think that the barriers are coming down very fast for all of this sort of stuff. I'm just so excited about multimodality and where things are going here because imagine if you're learning Spanish, being able to look at an image that the model generates for you and then doing Q and A on it. Right? Like a beach scene and then the model will ask you like, how many people are running on the beach? And then you have to sort of respond in the target language that you're learning.

• 46:09Speaker 02

  Very traditional language learning exercise, but you can imagine it being fully generative, which is really cool.

• 46:14Speaker 01

  Awesome.

• 46:14Speaker 02

  Lots of stuff like that.

• 46:15Speaker 01

  The engineering me worries about inference costs, but I think you can just kind of soup that under the rug.

• 46:21Speaker 01

  See if it works first and then you can worry about costs.

• 46:24Speaker 02

  Yes.

• 46:24Speaker 01

  You mentioned our real time voice platform. Uh, I just want to give you the platform to platform to talk more about that. Just like you mentioned, for example, that you're a very heavy user of the real time API from OpenAI. And you build a bunch of tooling around it.

• 46:39Speaker 02

  Yeah, so we last year had early access to the real time API. And there's a very obvious sort of use case for language learning. I think one common theme that has just been pretty awesome since L M's came out is that language learning as an application is just a really good fit for L M's, all these model types in almost every way, which has been just really great for a speak specifically. For real time, I think the audio piece promises to really like infuse almost every surface in the app. You can imagine this is the primary way that you talk to your tutor, right? And an additional complication is that it needs to be multilingual and there needs to be code switching. So that's a pretty frontier problem right now, right?

• 47:24Speaker 02

  So like, I should be able, if I'm learning Spanish, to speak both English and Spanish and vice versa from the model. That's a pretty hard TTS problem today. It's actually like only a few models are able to speak two languages in the same sentence and then and then pronounce them properly. Sorry.

• 47:39Speaker 01

  You can't have a router model, like a tiny little router model, guess which language first and then route.

• 47:45Speaker 02

  Well, the problem is that there is, you could have a subword in in a single sentence. You a different language.

• 47:53Speaker 01

  Yeah.

• 47:54Speaker 02

  So you can't just concatenate either.

• 47:55Speaker 01

  Yeah.

• 47:56Speaker 02

  Because it won't sound right.

• 47:57Speaker 01

  Yeah.

• 47:57Speaker 02

  Right? It won't sound natural. That's not how humans do it. So this needs to be like a, like a very native controllable audio, you know, function. Yeah. But we are in the process of building a variety of experiences on top of the real time API. I want to clarify that actually nothing is in production yet. Mostly for price reasons, frankly. The pricing model of the real time API makes more sense for something like a customer support agent, where you're very directly replacing somebody that you would pay hourly otherwise. And that's how you're seeing the price model for a lot of these initial agents work out. For us, we want our users to be able to do these real time role plays and have these conversations for many hours a day, right? If they want.

• 48:40Speaker 02

  Getting cost under control is definitely a pretty key consideration right now. But we are pretty close. Maybe actually even by the time that this episode is released, we'll have something live. But we have what I think is a really cool application of the real time API, which is basically a new instructional lesson where it's the model actually teaching you something, like a new language concept. And it's intended to sort of augment slash play the same role as our current video lessons, which are the instructional lesson type. And it's interactive, obviously. At certain points in the three to five minute lesson, you're interacting with the real time API. It's semi on guard rails.

• 49:21Speaker 02

  There was a lot of scaffolding we needed to build to basically, number one, switch between the interactive and non-interactive portions of this lesson properly. If that makes sense, right? There's some portions where you're just listening or looking. And then some portions where you're actively in a short conversation. And we kind of swap back and forth. And we have like a bunch of sort of custom architecture and info around that. And then there's also making the cost make sense, or at least like semi make sense. And then there's a bunch of web RTC infrastructure. We're at sort of, you know, not huge, but non-trivial scale either. So we definitely just, it'll cost us millions of dollars if we do something wrong.

• 50:04Speaker 01

  Yeah, yeah. Do you do inference in Korea? Because of the latency and all that.

• 50:08Speaker 02

  It's something that we have been increasingly paying attention to for all the realtime pads. Like, I would say, two or three years ago, when realtime stuff was still quite nascent, users didn't really care as much, but I think now the standards have risen, right? Like latency has to be low. Everyone cares.

• 50:28Speaker 01

  Do you have a hard latency budget for responses? Or do you just kind of work it out? So, for example, right, like, you have a knowledge graph that you're accessing, you have content that you're retrieving, there's a lot of stuff there. And then, like, you know, maybe you're using a reasoning model, probably not, but like, that all eats into the budget.

• 50:46Speaker 02

  I will say that, from the, like, realtime engineering side, everyone talks about, okay, submit user request to get agent audio response, like, first bytes, right, first audio bytes, what's that latency? And then, we try to get that as low as possible. I would argue that's actually like a vanity metric. Because what you don't take into account is how the VAD works. How do you do turn detection to detect when the user has finished speaking? Right? Because that can easily add like another second if you do it badly. And nobody talks about that for some reason, right? Like, what you need to measure is actually when does the user stop talking to when does the model first audio come? And usually, that number is much larger. That is a very domain-specific problem.

• 51:31Speaker 02

  You can use like the semantic VAD on realtime API for regular English conversation. And that will basically classify at every token how likely it is that you're done speaking as a sort of normal conversational English speaker, like in this conversation. That's fine, but it doesn't work at all for language learners, right? If I am trying to respond in a language that I'm learning, I'm going to be hesitating halfway through for 10 seconds. For more, right? So it needs to be fully custom, probably. It's something that we're also actively working on. But that is actually like the dominating factor in perceived latency.

• 52:10Speaker 01

  Coding, do you use Cursor, WinSurf and other autonomous agents?

• 52:16Speaker 02

  It's kind of all of the above.

• 52:17Speaker 01

  Yeah.

• 52:17Speaker 02

  So I think like as the CTO, I view it as part of my responsibility to really set expectations, push everyone on the team, show them what's possible. We've been trying everything.

• 52:31Speaker 01

  Yeah.

• 52:32Speaker 02

  And I think we tried to basically set the expectation that the frontier is moving so fast. It's deeply non-intuitive. If you've tried coding tools six months ago and they weren't that great, especially if it's not TypeScript or Python, right?

• 52:49Speaker 01

  It's a more collapsed of the most popular languages. Like that's all it is.

• 52:52Speaker 02

  We tried to set a culture in the engineering team where usage of these tools as much as possible and as a default path is the expectation. And in hiring, we are now explicitly asking about this a lot, thinking about what are the types of people that are going to be better higher agency at trying these types of tools. It's so important.

• 53:16Speaker 01

  Before we zoom out, anything we missed about Speak? That you really want to highlight or something that people underrated about it?

• 53:23Speaker 02

  One thing that I've always been really excited about is that I feel like a lot of the foundational pieces that we're building around Knowledge Graph, for example, a lot of these concepts should be applicable to not just learning language, but also other things in the future. We're already starting to see the very beginnings of this on the B2B side, where a lot of it is more like management skills and hospitality skills, communication skills, more like true L and D for enterprise, less like core pure English proficiency. So I think you're, you know, that's like obviously immediate neighborhood. But you can imagine many academic subjects, math, biology, et cetera. You know, this could also work for, uh, super excited about that.

• 54:08Speaker 01

  If I knew my employer was giving me a language tool, but then he was evaluating me on my management skills while learning a language. I might use it less, just, you know.

• 54:17Speaker 02

  Fair, very fair.

• 54:18Speaker 01

  You wanna, you wanna, you wanna separate that out.

• 54:20Speaker 02

  Yeah, very fair.

• 54:21Speaker 01

  I agree overall that the knowledge graph problem is very important. We have a whole track on it for, for the conference. And I think that the amount of data can be so high. And actually, like, you want to, you want to generate relevant triplets. I assume you use the normal subjects, predicate, objects type.

• 54:38Speaker 02

  It's a bit more custom than that because it's a bit more domain specific around the way that we conceptualize the vocabulary you know, and the sentence patterns and so on. So it's, it's more specifically around like language learning concepts, if you will.

• 54:50Speaker 01

  Yeah. But what I think we can extract from SPEAK, or as it is generalized as a framework, is what I think calling sort of like the Bloom two segment problem type thing, like the level adjusting tutor. Like, where are you at? Let me adjust my thing to where you're at, and then I'll push you up to the next level. And I think the knowledge graph is, is a part of it, but I don't know if that's all of it. I've never seen a working example.

• 55:12Speaker 02

  We are approaching that problem from a few different angles. I think part of it is knowledge graph. Part of it is being very careful in how we structure the curriculum. So that you're placed at the right level, so that the learning path itself, which has a foundational backbone, because beginner to intermediate English learners actually like all need to know a bunch of similar concepts. It isn't really until you get intermediate and more advanced where that starts to like more sharply diverge and A zero through B one, I would say. There's a pretty well-defined like sort of linear path, actually. A lot of the deep thinking that we've done around how do we structure the pedagogy

• 55:55Speaker 02

  is also super useful in terms of just like matching people to the right level. And then you can take this backbone and then basically modify it based on the knowledge graph, on your system's knowledge of what the user is like bad at versus good at.

• 56:11Speaker 01

  I think a lot of startups, or especially ed tech, like that is the core engine. Like, you know, once you do that, you can kind of teach anything.

• 56:19Speaker 02

  Totally, yeah.

• 56:20Alessio Fanelli?

  We have a few more broader fun questions.

• 56:23Speaker 01

  Yeah.

• 56:23Alessio Fanelli?

  Um, so speak.com.

• 56:25Speaker 01

  Yeah.

• 56:25Alessio Fanelli?

  Great domain. I looked it up, voice.com, got bought for 30 million.

• 56:30Speaker 01

  Oh my god. When?

• 56:31Alessio Fanelli?

  2019.

• 56:33Speaker 01

  Okay.

• 56:33Alessio Fanelli?

  So I don't know if you want to share how much you paid for it, but I think it was a lot less. I figure it would be a lot less, but I'm curious if.

• 56:40Speaker 01

  My estimate was a hundred K, but.

• 56:42Speaker 02

  It was it was more than that.

• 56:43Speaker 01

  More than that. Wow.

• 56:44Speaker 02

  Okay, I'm not gonna say anymore about the numbers.

• 56:45Alessio Fanelli?

  So what's the, yeah, what was the start? Was it easy? Was it? Did you use a broker? Like, we had, you know, their mesh shop from Hotspot, uh, who sold chat.com to OpenAI and he has a lot of very.

• 56:55Speaker 02

  That was like a hundred million deal or something, right?

• 56:57Alessio Fanelli?

  That was, that was very big.

• 56:59Speaker 02

  Oh wait, no, that was AI.com.

• 57:00Speaker 01

  Uh, chat.com.

• 57:01Speaker 02

  A chat, okay.

• 57:01Speaker 01

  Yeah.

• 57:02Speaker 02

  We bought it several years ago. It felt very expensive for us at the time. It was a little bit of a crazy move, but I think we were very convinced that we needed a super strong consumer brand that was scalable globally. And that was just always our ambition. Like, we want to be the way the next billion people learn languages. And we need speak.com.

• 57:26Speaker 02

  So we don't regret it.

• 57:27Speaker 01

  It's it's such a such a great word. Makes for great swag. Very nice decision. You had a couple other fun, fun questions. Any fun Korean celebrity stories? Cause you work with so many influencers.

• 57:38Speaker 02

  We have a bunch baking right now, but I think it was, you know, some something more generally that has just been so fun on the journey. So we we would visit Seoul every year.

• 57:49Speaker 01

  Yeah.

• 57:49Speaker 02

  And seeing speak go from nothing to the first time we saw somebody on the street using speak. To now our main teacher in the app is like a mini celebrity. People come up to her on the street as she's just walking around Seoul and recognize her from the app, which is really cool. Now we do a lot of advertising. We do billboards, TV commercials. We work with big influencers and so on. So just like seeing the scale of that has me kind of like in awe. It's like really cool. Uh, just to see something that used to be nothing.

• 58:23Speaker 01

  Now I wanted you to name drop like Blackpink or I don't know.

• 58:27Speaker 02

  Look, there's some stuff baking right now.

• 58:28Alessio Fanelli?

  Yeah, okay, alright. We talked about the TIO fellowship.

• 58:31Speaker 02

  Yeah.

• 58:32Alessio Fanelli?

  On your LinkedIn, you're gonna have this hole between 2012 and 2016.

• 58:35Speaker 02

  Yeah.

• 58:35Alessio Fanelli?

  You talked about you did some startups. Any of them that you wanna share? Like, ideas that you worked on that you thought were cool?

• 58:41Speaker 02

  Maybe it was just early, but, you know.

• 58:42Alessio Fanelli?

  Well, yeah, what people should revisit.

• 58:45Speaker 02

  I've always been interested in learning and education. One of the other field startups that I did in that time was, it feels silly to even talk about this because it amounted to nothing, but it was called Bloom. You know, the Bloom two-singer problem. It was actually like named after that. And we were trying to, to basically build like a better adult learning platform and have really cool, interactive JavaScript widgets for various concepts that you could learn. Didn't find PMF. I was young and didn't really know anything about business at the time either. But I think that the common thread through actually like everything that I've been interested in since leaving grad school has been,

• 59:28Speaker 02

  how do we build software, build tools that help people learn things more effectively and better and faster? And now I feel very lucky to be in this position because obviously AI is the ultimate version of that. Right? And it's been completely transformative for me personally because I just get a lot of just inherent fun and pleasure out of being able to like think of a concept and then, oh, now I can talk to this omniscient LLM that can tell me more about it. And I'm really good at asking the right follow-up questions that I want to know. So that's been completely transformative for me.

• 1:00:02Alessio Fanelli?

  Do you get a lot of like people using Speak for therapy? Like, you know, because it's not meant to be that, but since you have inference, they will use it.

• 1:00:13Speaker 02

  In 2023, when we first launched our AI role plays using GPT-4, back then, people were way more concerned about safety. Right? And obviously the models now are much better. I have refusals and the line is sharper between what's appropriate and not. But we did see a lot of our first users. Start to put in pretty questionable custom scenarios.

• 1:00:38Speaker 02

  You probably guess.

• 1:00:38Speaker 01

  Yeah.

• 1:00:39Speaker 02

  And, you know, like, this was something we expected, but I think seeing the logs in person is like very different.

• 1:00:46Speaker 01

  Got it.

• 1:00:47Speaker 02

  Some shocking stuff in there.

• 1:00:48Speaker 01

  Last couple of questions, one on Andre, you talked to him in your machine learning journey.

• 1:00:53Speaker 02

  That was a long time ago, yeah.

• 1:00:54Speaker 01

  He's also working on EdTech now.

• 1:00:56Speaker 02

  Yeah.

• 1:00:56Speaker 01

  I don't know if you've ever had conversations with him.

• 1:00:58Speaker 02

  No, I haven't.

• 1:00:59Speaker 01

  He's also interested in language learning, by the way, you know?

• 1:01:01Speaker 02

  One thing that I think we didn't really realize early on, or like fully internalize at least, was just like how deep the market is.

• 1:01:09Speaker 01

  Say more.

• 1:01:10Speaker 02

  It was so universal where we really struggled to do some of the basic startup stuff around define your like ideal customer profile and like, you know, segment your users because our users were everyone like we had parents using it with their kids. We had really old people using it. We had people using it for work. So that was kind of like mind boggling.

• 1:01:31Speaker 01

  You still did customer segmentation or are you saying it doesn't matter?

• 1:01:34Speaker 02

  I'm saying it was hard to do. Like, we tried and we have a sweet spot in Korea. It's like 25 to 45, more professional, more white collar, but it's very, it's like, like a very long tail on either side. Yeah, I think it's, you know, it's a huge market and I think it's a very special moment in time right now where it's obvious that a lot of the tech is here. I think it's really good for humanity if we make a lot of progress here. So I'm really excited for his company too.

• 1:02:03Alessio Fanelli?

  We started asking about the TIO fellowship, so maybe we can wrap with one of TIO's favorite questions, which is, what's something you believe in today that most people will not agree with you on?

• 1:02:13Speaker 02

  I think that people, if you recall, expected the world to kind of explode when GPT-4 came out. And, you know, like, everything would change. And I think if you like go to another state outside of the Bay Area, probably even in California outside of the Bay Area, and then you ask somebody how much their life has materially changed, it's like pretty close to zero. Real world inertia is enormous. Obviously, AI is probably the most transformative technology we've ever built. But I think in a very real sense, the world hasn't changed that much either. And that's a really weird thing, right? So I think we need more builders. We need more people building applications.

• 1:02:55Speaker 02

  It's weird to me that speak is actually like not that many net new consumer AI native applications at scale. Like, there should be way more. I would love for there to be way more. Consumer is hard.

• 1:03:05Speaker 01

  Yeah, I'm intimidated, but like, you know.

• 1:03:08Speaker 02

  It was just like, there was never any alternative for us.

• 1:03:10Speaker 01

  Yeah.

• 1:03:10Speaker 02

  Like I said before.

• 1:03:11Speaker 01

  You didn't have a choice. But also, you're very smart. But also, maybe you have some growth hack things that you can advise people on that, like that people could could learn. But yeah, I agree. I think like the general take actually is this is what we want, which is slow takeoff, short timeline.

• 1:03:26Speaker 02

  That's fair.

• 1:03:27Speaker 01

  Right? This is the two by two that everyone always talks about in AI safety. You're seeing slow takeoff and like, maybe don't complain. We make, we have a heads up because.

• 1:03:35Speaker 02

  Or, you know, Dario's right and like, half of us lose our jobs in the next two years. Yeah. It's it's it's so hard to predict.

• 1:03:42Speaker 01

  Yeah.

• 1:03:44Speaker 02

  Sometimes I get AI anxiety. And then I just.

• 1:03:48Speaker 01

  You get anxiety?

• 1:03:49Speaker 02

  Yeah.

• 1:03:50Speaker 01

  Okay.

• 1:03:50Speaker 02

  Then I just focus on our users.

• 1:03:52Speaker 01

  It's a perfect place to wrap. Thank you so much for taking the time.

• 1:03:55Speaker 02

  Yeah, thank you both so much. This is great.