Pod Insights The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch 20VC: Spending $2M to Train a Single AI Model: What Matters More; Model Size or Data Size | Hallucinations: Feature or Bug | Will Everyone Have an AI Friend in the Future & Raising $150M from a16z with Noam Shazeer, Co-Founder & CEO @ Character.ai About this episode Noam Shazeer is the co-founder and CEO of Character. AI, a full-stack AI computing platform that gives people access to their own flexible superintelligence. A renowned computer scientist and researcher, Shazeer is one of the foremost experts in artificial intelligence (AI) and natural language processing (NLP). He is a key author for the Transformer, a revolutionary deep learning model enabling language understanding, machine translation, and text generation that has become the foundation of many NLP models. A former member of the Google Brain team, Shazeer led the development of spelling corrector capabilities within Gmail, the algorithm at the heart of AdSense. In Today's Episode with Noam Shazeer We Discuss: 1. Entry into the World of AI and NLP: How did Noam first make his way into the world of AI and come to work on spell corrector with Google? What are 1-2 of his biggest takeaways from spending 20 years at Google? What does Noam know now that he wishes he had known when he started Character? 2. Model Size or Data Size: What is more important, the size of the data or the size of the model? Does Noam agree that "we will not use models in a year that we have today?" What is the lifespan of a model? Does Noam agree that the companies that win are those that are able to switch between models with the most ease? With the majority of data being able to be downloaded from the internet, is there real value in data anymore? 3. The Biggest Barriers: What is the single biggest barrier to Character today? What are the most challenging elements of model training? Why did they need to spend $2M to train an early model? What are the most difficult elements of releasing a horizontal product with so many different use cases? Where does the value accrue in the race for AI dominance; startups or incumbents? 4. AI's Role on Society: Why does Noam believe that AI can create greater not worse human connections? Why is Noam not concerned by the speed of adoption of AI tools? What does Noam know about AI's impact on society that the world does not see?
Transcript Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:00 Speaker 00 The size of the model is the bigger challenge. Actually, the number one thing that's important is how much computation you do to train it. So you want to train a bigger model and you want to train it for longer. But the real constraining factor has been how many operations of computation it takes to train it. The model we're serving now, we trained last summer and spent $2 million worth of compute cycles doing it.
0:26 Harry Stebbings? This is 20VC with me, Harry Stebbings, and welcome back to part two of this very special feature week, featuring two of the hottest AI companies today. Joining me in the hot seat, I'm so thrilled to welcome one of the foremost experts in artificial intelligence and natural language processing, Noam Shazeer, founder and CEO at Character.ai, a full-stack AI computing platform that gives people access to their own flexible superintelligence. Before founding Character, Noam spent 20 years at Google, including as a core member of the Google Brain team. But before we dive into the show's day, you know all those mind-numbing tedious tasks that seemingly take up half your day? Well, Coda is here with their new AI-powered work assistant that helps you and your team not just finish tasks, but make progress so your product team can bring a feature to market faster.
1:11 Harry Stebbings? By using Coda AI to tag customer feedback, draft PRDs, suggest target audiences, summarizing product discussions and more, your sales team can engage more customers by bringing in data from sources like Salesforce and then use Coda AI to suggest action items, meeting agendas or lead scores. And your marketing team can drive an impactful launch with Coda AI summarizing user insights, creating briefs generated from notes and writing mock press releases to visualize taglines. With Coda AI, you can reimagine your to-do list and how you collaborate so you're not only finishing tasks, but really making progress. If you want a work assistant that lets you get back to work, you can get started with Coda AI today for free.
1:52 Harry Stebbings? Head over to coda.io/20vc, that's coda.io, and get started for free. And speaking of amazing tools we cannot live without, travel and expense are never associated with cost savings, but now you can reduce costs up to 30% and actually reward your employees. How do you do this? Well, Navan rewards your employees with personal travel credit every time they save their company money when booking business travel under company policy. Does that sound too good to be true? Navan is so confident you'll move to their game changing all in one travel corporate card and a spend super app that they'll give you a $250 in personal travel credit just for taking a quick demo. Check them out now at navan.com/20vc.
2:37 Harry Stebbings? And last but by no means least, we need to talk cash. As of the 29th of June, you can get 5.5% yield on your cash with 26 week treasury bills. But buying treasury bills, it's not that easy and you have to navigate a website that looks like it was made before I was born. Enter public.com. Their treasury accounts make it simple to earn a high yield on your cash and it takes 20 seconds. Here's how it works. Sign up at public.com, easily purchase 26 week treasury bills that automatically roll over at maturity for compounding yield. Plus, there are no minimum hold periods. You can access your cash at any time with the flexibility of a bank account. Of course, to receive the full guaranteed yield, of course, to receive the full guaranteed yield, you do have to hold to maturity.
3:22 Harry Stebbings? But here's the thing, these are T bills, which means your investment has the complete backing of the US government, making it one of the safest places to park your cash. Go to public.com/20vc to lock in a historic 5.4% yield on your cash.
3:39 Harry Stebbings? You have now arrived at your destination.
3:42 Harry Stebbings? Nom, I'm so excited for this. I heard so many great things from many different people, Eric Schmidt, Sarah Wang, Prajit. So thank you so much for joining me today.
3:51 Speaker 00 Thank you. Yeah, great to be on, Eric.
3:53 Harry Stebbings? Now, I would love to start with some context because few people spend 20 years at Google in the height of Google's scaling and trajectory. But first, I want to go back to the beginning. I heard there's a story to your joining. What happened, spelling corrector? Can you give me the story?
4:08 Speaker 00 Yeah, that was like the first project that I worked on at Google. Yeah, I guess at the time, you know, Google had a spelling corrector that was some third party software. It was, you know, based on maybe what you'd find. In a word processor at the time. So there was like some human compiled dictionary of maybe about 50,000 words. And any word that wasn't in the dictionary that was in the query, it would say, you know, did you mean such and such? And this worked great for spelling correction. It was like absolutely terrible for web search. Because people search for such a wide diversity of things on web search, like most of them are just not in the dictionary. So like you'd search for TurboTax and it would say, did you mean Turbot Axe? And people just learned to ignore the thing. So yeah, but first project, like we're just looking at like, why are people like not happy using Google?
4:53 Speaker 00 And like spelling correction was like the number one issue. So I was like, okay, let me help out with this. And you know, there was someone working on this, Paul Buchheit, who's, you know, gone on to do a lot of illustrious things in his career. He's also one of our investors here at Character.
5:08 Harry Stebbings? What are one to two of the biggest takeaways? 20 years is a long time. How did it impact you?
5:13 Speaker 00 Okay, Google's been just an incredible company. It's brought so much value to like billions of people. And so I'd say one of the big takeaways is that, you know, if you have a technology that is like really, really general and has billions of use cases and like ordinary people can use, like launch it to billions of people. I remember when I joined Google, there were like a lot of people working on this enterprise search appliance, which, you know, it was okay. Like, I think maybe somebody had this conventional wisdom that B to B is the only way to make money. But what it actually turned out was like the much bigger thing was B to C, you know, like serve something to everybody.
5:50 Harry Stebbings? How did that change how you think?
5:52 Speaker 00 Yeah, well, right now I started this company, Character, and we're taking this large language model technology. And we are just direct to consumer first. Like here is something that is even more versatile and more easy to use than even web search in that, okay, you can use it to be your friend or do your homework or like a billion things. And we haven't even thought of the best use cases yet. And then it's massively usable. Like all you have to do is talk to it. So it has these two properties. To me, that means go launch it to like everyone in the world and let everybody in the world use it. I think some of the other companies are taking a more like B to B sort of approach with a
6:35 Speaker 00 you'll have a foundational model company and then like verticalized application companies on top of it. So I'm really inspired by the Google model of full stack end to end, all the way from like basic research to launch a product directly to consumers. It's super fun. Engineers like building stuff and then launching it and having like everyone use it immediately. And then it also lets you do all this co-design of like you get to affect every part of the stack, which is hugely powerful and fun.
7:03 Harry Stebbings? Cass, we asked about your joining Google. I think so many people are shaped by their past. When you think back on yours, how do you think about what you're running from?
7:11 Speaker 00 Well, I guess, yeah, why did I start working on artificial intelligence? Like partially because it's just like fun and what I do for fun anyway. Like what could be better than try to get the computer to do something that it currently can't do. But then, you know, the other thing is just to push technology forward. You know, there are so many like technological problems in the world that could be solved. You have like 15 million people a year like dying from stuff like old age and cancer and heart disease and like all kinds of stuff that we could potentially find cures for. So rather than directly working on, say, medical research or something, I think I've got a lot more leverage, like let's push AI technology. And then, you know, that can help with a lot of the rest of it.
7:51 Harry Stebbings? So how do you think about Character's mission and vision? When you think about, as you said, the world's greatest problems, there are from climate change to wealth inequality to natural resources. How do you think about Character's vision and mission? Because I think people misunderstand this, if we're honest.
8:06 Speaker 00 I think we just need to have a lot of humility that we're not in charge of the world. You know, we're not even in charge of the government or not in charge of what individuals do. Like basically, you know, I think our place is to provide useful tools to like everyone on earth to leave people in control.
8:24 Harry Stebbings? Now, when you get up in front of the company, what do you chest pump as the mission?
8:28 Speaker 00 I like this sort of motto of a billion users inventing a billion use cases. Like, because that's sort of the superpower of this technology and it puts Our company in the right place, we can't really guess what are the best uses of this technology. We've just observed time and time again, like you put one thing out there and that's not really what people want and somebody else out there like finds something better to do with it. Like we we put up as an example, like a psychologist character, like maybe you wanna, you know, talk to something and feel better. That gets a little bit of use, but then what we hear a lot more from users is like, I'm talking to a video game
9:12 Speaker 00 character who's now my new therapist and this makes me feel better. We had no idea that was going to go on. And then there's this huge use case in like some mix of like entertainment and companionship and emotional support. We were totally not experts in this stuff. Like, our job is just to put out something general, just respect the agency of our users, of everybody out there to do what they want with this stuff.
9:36 Harry Stebbings? When you think about the incredible growth that you've seen, 450 million messages a day, 20 million users, what do you think have been the ones two biggest elements that have driven that growth?
9:46 Speaker 00 Well, one is that we launch. That's definitely been a frustration, you know, in the past, things seem potentially too much brand risk at larger companies to like actually launch and get it out there. Another aspect is we launch something general, we let people find the use cases. And then the other is there are like massive needs out there in the world. Like, okay, there are billions of people who, you know, feel like they need someone to talk to. Combine those elements, provides people with something general that they can use and there are people out there with needs and they're going to find it.
10:21 Harry Stebbings? I totally agree with you in terms of the horizontal use case. I'm fascinated by all the different ways they talk to it. Do you not worry that we're losing touch with other humans? They've got no one to talk to and so they talk to a machine?
10:33 Speaker 00 Yeah, I think there's huge. Value in the connections between people and moral value as well. So last thing I want to do is take people away from human connection. In a lot of ways, we want to help with human connection. A lot of the people who don't have friends and who are not as well connected, one big source of that is just social anxiety. There are huge numbers of people who are uncomfortable. And we've gotten testimonials of people who said that they were uncomfortable talking to other people and this is great practice that actually helped them build up practice in either social situations.
11:10 Harry Stebbings? Do you think so? Or do you think it obviously just builds up habit? You get used to talking to someone who's not of human.
11:17 Speaker 00 I would very much like it to be the former. That is going to be ultimately up to the users.
11:22 Harry Stebbings? What do you think is the hardest product challenge you face today? It's a difficult product paradigm that you face. So many different use cases, so many different people's needs. What do you think are the hardest product paradigms for you as a team to face?
11:34 Speaker 00 The main things we need to do, make it very general. So we're not like cutting down on the use cases. Make it usable. People think of those two things as being in opposition to each other, being versatile and being usable. We talked to some potential product managers early on and they all say the same thing. Oh yeah, pick your verticals, narrow it down to make it usable. And like, no, we're not going to hire these people. That's like the opposite of what we want to do. We want to build something that is usable, but very, very general purpose. So there's sort of that dichotomy.
12:06 Harry Stebbings? I'm trained on the thought that the more you specialize, the deeper, the richer conversation value you can provide. And so how do you provide quality high enough with such generalization?
12:19 Speaker 00 That's been the magic of neural language modeling. The previous systems were all these rule-based systems, like fantastically complicated systems with millions of handwritten rules, really, really complicated. It required knowing something about linguistics and about state of mind and like all kinds of stuff. The new way of doing things with neural language models has none of that. I could know like zero about language in particular, other than it's like a sequence of words. So it has nothing to do with understanding language at all. And there are not millions of rules. It's actually relatively simple, kind of like a big black box.
13:01 Speaker 00 It all boils down to this one beautiful, simple problem of you have this sequence of words that's like the beginning of your document. Guess what the next word is? Give me odds on what the next word is in the sequence. And that problem is called language modeling. Just guess what the next word is based on the previous ones. You know, so I got involved with this, you know, around like 2015. There were some other folks at Google, you know, working on this problem. They're like, how good can we make it? And this struck me like, hey, this is the best problem ever because it is so simple to state and there's a huge amount of free training data. You can just like download the text of the web off of like whatever you want.
13:42 Speaker 00 Common crawl, you've got like billions to trillions of training examples of guess the next word. And then if you can do it well, then this thing can just talk to you. It can be, you know, the better you do it, the smarter it gets. It's hugely, hugely general and useful, super simple to state. And now we just have to do it well. And people started building better and better neural networks, which I guess got renamed deep learning as some sort of rebrand. You know, neural networks had a bad name because the hardware wasn't good enough. But now that the hardware is good enough, they switched the name. But anyway, so people were using deep learning for language modeling. And the bigger and better you made these things, the smarter it got. And like around 2016, the most useful application, like the killer app for this
14:27 Speaker 00 was machine translation. It was about smart enough to take English and translate to French. You know, massively, massively useful. It lets everyone in the world communicate with each other. But you know, still not smart enough to like carry on an interesting conversation or do your homework or like any of those things. But there seemed to be a pretty clear path. Hey, let's just make this thing bigger, better, smarter. And it's going to get these capabilities.
14:53 Harry Stebbings? Can I ask you, when you talk about kind of wifling back then in 2015, 2016, these are very different excitement cycles to where we are today. I remember 2015, 2016, we had kind of the chatbot phase when there was like super excitement, very in like a month period. But there wasn't this sustained belief that we have today in AI transforming the whole way society works. And I guess my question to you is like, where we are today, is that the result of technological progress recently, very recently? Or is it the result of investors in society catching up with what's been developing over a much longer period?
15:28 Speaker 00 I'd say it's both. I think there's been a lot of technological progress, both quantitatively and qualitatively, in that the models that were there in like 2016 were too dumb to be fun, the neural models. Then there were all of the chatbot stuff you heard about back then was these rule-based systems that were just highly fragile and not going anywhere, that you just needed more and more rules and there's like no way to think of like all the things that could come up and they just don't generalize. That wasn't going to work, but at the same time, we were progressing on the neural network solutions, which were going to scale. It took some amount of time, I'd say around 2020
16:10 Speaker 00 was when really impressive stuff was sort of in the lab, but not launched. So my co-founder, Daniel Defreitas, he's like on this lifelong mission to do chatbots. Since he was a kid in Brazil, he's wanted to build like open domain chatbots.
16:25 Harry Stebbings? You said the more and more you do it, kind of the better it gets and the better responsiveness and accuracy it gets. I'm always questioning, because you hear so much, what's more important? Is it the size of the data or is it the size of the model?
16:38 Speaker 00 Yeah, probably the size of the model is the bigger challenge. We can get a lot of data, but actually the number one thing that's important is how much computation you do to train it. So you want to train a bigger model and you want to train it for longer. So the two things are both important, but the real constraining factor has been how many operations of computation it takes to train it, because if you make it bigger and you train it for longer, both of those multiply into how long the thing takes to train. So people have been building better and better, essentially, supercomputers to train these models.
17:14 Harry Stebbings? What are the biggest constraints on your models today, do you think?
17:17 Speaker 00 Computation. So you know, the model we're serving now, we trained last summer and spent about $2 million worth of compute cycles doing it. We will do a lot better, you know, in the near future. But if we get a lot more better hardware, which we are getting, and spend longer training the thing, we can train something smarter. Back in, say, 2016, you could train something that was like smart enough to like translate languages, but not smart enough to like answer questions or be fun.
17:50 Harry Stebbings? If that's models on the data side, how do you think about proprietary versus non proprietary data? You said there about kind of in the early days, you could download kind of the data of the internet, so to speak.
18:00 Speaker 00 But that's what pretty much everyone still does.
18:02 Harry Stebbings? Sure, but character is producing a ton of proprietary data within your conversations, as are many verticalized solutions, be it in medical, be it in finance, be it wherever. To what extent is the value in proprietary salute like data ownership versus it will still and always be downloadable by everyone?
18:22 Speaker 00 Both are useful. Like data that you get from users is great because it tells people like what users like, or like what users like in some particular application. It's kind of like a training a human. Like, most of what's important is you have decades of experience training your brain on stuff that is not really specific to your task at hand or your occupation, but you've kind of gotten a generally good understanding of the world and gotten generally intelligent. And then you can improve on that dramatically by getting a smaller amount of training in the task you're doing right now. But both
19:04 Speaker 00 will contribute. And we do have like a huge amount of data flowing in from users.
19:10 Harry Stebbings? I'll ask a really hard question. Why is character, and I mean this in that, why is character a standalone company and not a product of Facebook? If you think about a natural extension of Facebook into the metaverse, social, the extension of your physical friendship group into the metaverse or the non-physical world, why does character need to be a standalone and why isn't it within Facebook?
19:31 Speaker 00 My experience coming from Google is that a startup can move way faster than a big company and can launch products in ways that large companies are just going to move too slow because they're worried about compromising their existing products.
19:47 Harry Stebbings? Do you think startups win then as a result in this next wave of AI innovation, entrepreneurship? Because a lot of people I have on the show now say less so on the Facebooks of the world, but more the Microsofts and the Adobes win. Who wins, startup or incumbent? If you have to pick a side.
20:04 Speaker 00 I'd say the users win. The users are going to have a lot of options, but you know, on the business side, I think there can be a lot of winners. There is just so, so much value about to be created that there's going to be room for multiple players in there. Big companies doing what big companies are good at, startups doing what startups are good at. We're going to try to move our company from being a startup to being a big company as fast as we can there. And then a lot of just individuals and universities and such in that the hardware is progressing so fast that what you could do at a big company one year, a few years later, you're going to be able to do, you know, at a university lab or in your garage.
20:44 Harry Stebbings? Totally get you and agree. In terms of those individuals and universities, I had Jan LeCoon on the show, who was fantastic. And he said about, you know, the future of open versus closed and why he's such a protagonist for open. Do you agree in terms of open being the dominant method and mechanism of the community? Or do you think actually closed wins? Many others have said closed wins.
21:04 Speaker 00 The ability to mess around with things at a small scale is going to lead to way more research being published, even if some of the larger entities are no longer publishing research because they're trying to maintain competitive advantages. And obviously, though, there's also like economics of scale of both training the best models and serving. Like, if you want to serve a product, you can do it maybe 100 times more efficiently if you're serving many, many people at once and kind of batching things together versus, you know, serving like an individual or, you know, or you've got your own rig to run your Language models in your basement or something.
21:43 Harry Stebbings? You you sit at the center of the ecosystem and you have done for many years. What do you think society believes that you would like to change that perspective on on AI? You do interviews now, you get asked the same questions, you see the same headlines. What do you like? God, I wish wish people would change their mind that AI is going to kill everyone or AI is going to replace all jobs or any of the clickbait that we see everywhere. What do you wish society would change their minds on with regards to AI?
22:10 Speaker 00 I think the one message I have is that the best applications just haven't even been invented yet. That we're still at like invention of electricity kind of moment or invention of the computer. We don't really know what the coolest things are going to be.
22:25 Harry Stebbings? Chris, you mentioned the coolest things and what they're going to be. In conversation, sometimes your friend will do something a bit wacky. Yeah, they'll do something kind of cool. And models can do the same with hallucinations and introduce a bit of creativity. Are hallucinations a feature or are they a bug?
22:41 Speaker 00 We consider them a feature. Our strategy is like launch something general, let people do what they want with it. If these models are hallucinating, which they certainly are, and we advertise that they are, then the use cases that emerge first will be ones for which hallucination is a feature. So I'm happy to have entertainment and emotional support and fun be the first use cases. I'm happy to have productivity be the first use case, but let that happen naturally based on what the technology is good at.
23:12 Harry Stebbings? If you think about Google, which is like helping people find information faster, better, more efficiently.
23:18 Speaker 00 Yeah.
23:19 Harry Stebbings? What do you think will be characters? Because as you said, like a million people doing a million things. I mean it respectfully. Do you know and will we ever know? Is character an education company? Is it a social company?
23:31 Speaker 00 I think you could ask the same thing about any company that is selling some very general tool, like a company that's selling computers or electricity for that matter. What is electricity for? Is it for fun? Is it for productivity?
23:47 Harry Stebbings? I I get you had to to power your home or office to do. Activities for business or pleasure?
23:54 Speaker 00 We believe that individuals should make that decision. You know, we want, we want to respect everybody's agency. I think that's one of the big fears around that AI will take away your agency. We want to come down on the side of we love humans. We respect humans. This is a tool for people.
24:11 Harry Stebbings? Have you enjoyed that transition to CEO and scaling CEO?
24:15 Speaker 00 I have, I still do a lot of technical work and leadership, which is big. I'm going to stay CEO because I want the company to make the right decisions. I don't judge what I do by how much fun it is. It's more like what's the most useful. So very, very happy to be doing what I can.
24:33 Harry Stebbings? Can you unpack that for me? I'm sorry, I'm just interested. I don't judge it by how much fun, but by how useful.
24:39 Speaker 00 Yeah, it wasn't like a matter of am I going to be having more fun being a startup CEO than an ML researcher at Google? It's, it's more like I want to push this technology forward. Like what's the best thing I can do?
24:54 Harry Stebbings? I, I guess I didn't know that if you find like utility and fun come together. I love what I do and I have a lot of fun doing it and it's the most useful. I'm just intrigued if they were one in the same or separate.
25:06 Speaker 00 Yeah, I think this is like a really interesting concept there. After sort of becoming a parent, like early 2010s or something. I mean, one thing I remember not being a lot of fun was like getting woken up in the middle of the night and getting like way, way less sleep than I would have liked it. And, you know, a lot of things about parenthood are absolutely terrific and super fun. I think it made me more, more religious. It's sort of, I decided to take a change of attitude from like, what is fun right now? To I should be thankful for having the opportunity, you know, to do something important and meaningful. So I, I think that it's, it's kind of been a big attitude shift in, in growing up.
25:50 Harry Stebbings? You look far too young to have children that are, you know, in those age ranges. If you could phone up yourself the night before your first child was born and give yourself a piece of advice, what would you say to yourself?
26:03 Speaker 00 Get some sleep first.
26:05 Harry Stebbings? Really, what do you know now that you're like, you know what, I would have told myself, I just, I had one of the world's most famous hedge fund managers on the show. And they said, the only thing that matters is my wife. The children don't matter. I don't matter. The only thing that matters is looking after my wife. And if I look after her, she'll look after the children and she'll look after me.
26:24 Speaker 00 That's beautiful. Yeah. Yeah. Not everything in the world is your responsibility. That you should understand what is your responsibility and what isn't your responsibility. And I think that works really, really well in marriage and in parenting. I mean, I think religion's a lot about that as well, like sort of beliefs about what's your responsibility and what's not. Like, that's the question that it's answering because you don't have a solution to just staring at your face in nature. Like, what should I be, what do I need to do? What should I be concerned about and what should I not?
26:58 Harry Stebbings? Listen, I want to move into a quick final. I say a short statement, you give me your immediate thoughts. You're like, dude, what the fuck? I come on to talk about AI.
27:07 Speaker 00 Oh, no, no, this is, this is great. I mean, like luckily we're recording a lot and you could like the crazy parts.
27:13 Harry Stebbings? No, I love it.
27:15 Speaker 00 Yeah, no, it's interesting. This is really fun.
27:17 Harry Stebbings? I think children are the most fantastically interesting catalyst in one's life. Cause it's the most significant change you will ever have in a day. Most changes are years long. You lose weight, you stop something. It takes years to build a company. A child, done. So what others not know that you know to be true?
27:38 Speaker 00 This technology is just going to get way smarter. I think we're at a Wright Brothers first airplane kind of moment in AI. That there's like a lot of momentum going on, both in building better hardware and in research. So whatever really amazing applications you're seeing now, it's probably nothing compared to what's going to happen in the future.
28:01 Harry Stebbings? What do you think that adoption timeline is?
28:04 Speaker 00 I think things are going to move very fast. Like, I think we'll see like a lot of very cool stuff happening in the next one to three years.
28:11 Harry Stebbings? What do people not understand about character that you wish that they did?
28:15 Speaker 00 I think like externally, it looks like entertainment app. But you know, really, we are a full stack company. We're like an AI first company and a product first company. Having that is a function of picking a product where the most important thing for the product is the quality of the AI. So we can be completely focused on making our products great and completely focused on pushing AI forward. And those two things align.
28:46 Harry Stebbings? What single element would you most like to change about the AI community?
28:50 Speaker 00 Yeah, I mean, there's so much stuff being published, it's hard, hard to know what's good. And I think a lot of that has to do with the fact that this field is kind of alchemy right now. Like, no one knows exactly what is going to work. You know, you have a lot of people trying lots and lots of different things. You can come up with hits by having like a good intuition of what will work on the ML side, combined by like a good mathematical understanding of what will run fast on hardware that you can buy or that you can build. So there are some hits that come out and, you know, people will adopt. And it's combined with a lot of noise.
29:32 Speaker 00 Negative results are not useful because they could be negative because somebody just made a mistake. And there could be a bug, you know, that it didn't work for some other reason. What is interesting is positive results that are proven out by experimentation. If somebody can say, hey, I did something and did better at this well-known problem, then that gets interesting. Then everyone tries to figure out, okay, why does this work? How can I adopt it?
30:01 Harry Stebbings? Penultimate one, Nam, what did you believe that you turned out to be wrong on?
30:06 Speaker 00 Well, when I started getting into deep learning, I, around 2012, had a bunch of early failures trying to do sparse computation. You know, I was like, okay, you must be able to do something better and more. Efficient by building a sparse network. That was so wrong because I did not understand that the reason this whole field is working so well is because now we have this magic hardware that's great at these dense matrix multiplications. And so you can do them like orders of magnitude faster than you can do anything that involves poking around in memory. And there was no one there to like explain that to me when I got started with deep learning. So, okay, as soon as I sort of understood that part of it, it's like, okay, let's do sparsity.
30:50 Speaker 00 But let's build it out of these dense building blocks so that it'll run fast. And then published this sparsely gated mixture of experts idea that's only now getting a lot of adoption. But, you know, that was back in 2016 and then have had like a string of hits ever since, which I will attribute to divine intervention, but also to understanding like the hardware mechanics and sort of quantitative computation aspects of the field.
31:14 Harry Stebbings? Noam, 2033, 10 years from now, where will character AI be then?
31:19 Speaker 00 On Mars.
31:22 Speaker 00 I have absolutely no idea. Like, we will see what technology is like then, but, you know, it's just important for us to be agile. If you were in 1900 and asking where some company would be in 2000, there will be such technological improvement before then that it's roughly impossible to predict where any company will be.
31:44 Harry Stebbings? I think this has been unlike any interview you've done for you. I feel like the questions have stretched boundaries of parenthood that people didn't ask you before. I really enjoyed having you on, Noam, and I hope you've enjoyed it too.
31:58 Speaker 00 Very much so.
32:01 Harry Stebbings? What a blast. I want to say huge thank you to Noam. I don't quite think he knew where that one was going in terms of the direction of the discussion, but he was fantastic. If you want to see more from us, of course, you can on YouTube by searching for 20VC, that's 2-0-V-C. But before we leave you today, you know, all those mind-numbing tedious tasks that seemingly take up half your day. Well, Coda is here with their new AI-powered work assistant that helps you and your team not just finish tasks, but make progress so your product team can bring a feature to market faster by using Coda AI. To tag customer feedback, draft PRDs, suggest target audiences, summarizing product discussions and more, your sales team can engage more customers by bringing in data from sources like Salesforce
32:42 Harry Stebbings? and then use Coda AI to suggest action items, meeting agendas or lead scores. And your marketing team can drive an impactful launch with Coda AI summarizing user insights, creating briefs generated from notes and writing mock press releases to visualize taglines. With Coda AI, you can reimagine your to-do list and how you collaborate. So you're not only finishing tasks, but really making progress. If you want to work a system that lets you get back to work, you can get started with Coda AI today for free. Head over to coda.io/20vc, that's coda.io and get started for free. And speaking of amazing tools we cannot live without, travel and expense are never associated with cost savings.
33:23 Harry Stebbings? But now you can reduce costs up to 30% and actually reward your employees. How do you do this? Well, Navan rewards your employees with personal travel credit every time they save their company money when booking business travel under company policy. Does that sound too good to be true? Navan is so confident you'll move to their game-changing all-in-one travel corporate card and a spend super app that they'll give you a $250 in personal travel credit just for taking a quick demo. Check them out now at navan.com/20vc. And last but by no means least, we need to talk cash. As of the 29th of June, you can get 5.5% yield on your cash with 26-week Treasury bills. But buying Treasury bills, it's not that easy
34:08 Harry Stebbings? and you have to navigate a website that looks like it was made before I was born. Enter public.com. Their Treasury accounts make it simple to earn a high yield on your cash and it takes 20 seconds. Here's how it works. Sign up at public.com, easily purchase 26-week Treasury bills that automatically roll over at maturity for a compounding yield. Plus, there are no minimum hold periods. You can access your cash at any time with the flexibility of a bank account. Of course, to receive the full guaranteed yield, of course, to receive the full guaranteed yield, you do have to hold to maturity. But here's the thing, these are T bills, which means your investment has the complete backing of the US government, making it one of the safest places to park your cash.
34:50 Harry Stebbings? Go to public.com/20vc to lock in a historic 5.4% yield on your cash.
34:57 Harry Stebbings? Now stay tuned for Monday's episode, we have the one and only Nikhil Basutrevadi, an old, old friend at Footwork Now on the show. That is such a great show coming on Monday.
Previous 20VC: Why AI Models are not a Moat, Where Does the Value in AI Accrue; Startups or Incumbents, What the World Has Got Wrong About AI, Why AI Needs a New Story and Who is the Right People to Tell it with Cris Valenzuela, Co-Founder & CEO @ Runway Next 20VC: The $3.1BN Meeting That Led to an Uber Acquisition, The Battle With Uber; How to Outcompete When You Have 10x Less Cash & The Marketing Campaigns That Led to Pakistan MDs Fleeing and Elon Musk Fanboying with Mudassir Sheikha, CEO @ Careem Page updated October 26, 2026 · transcript and summary are machine-generated
