Al Engineering 101 with Chip Huyen (Nvidia, Stanford, Netflix)

Chip Huyen is a core developer on Nvidia’s Nemo platform, a former AI researcher at Netflix, and taught machine learning at Stanford. She’s a two-time founder and the author of two widely read books on AI, including AI Engineering, which has been the most-read book on the O’Reilly platform since its launch. Unlike many AI commentators, Chip has built multiple successful AI products and platforms and works directly with enterprises on their AI strategies, giving her unique visibility into what’s actually happening inside companies building AI products. *We discuss:* 1. What people think makes AI apps better vs. what actually makes AI apps better 2. What pre-training vs. post-training is, and why fine-tuning should be your last resort 3. How RLHF (reinforcement learning from human feedback) actually works 4. Why data quality matters more than which vector database you choose 5. Why high performers are seeing the most gains from AI coding tools 6. Why most AI problems are actually UX issues *Brought to you by:* Dscout—The UX platform to capture insights at every stage: from ideation to production: https://www.dscout.com/ Justworks—The all-in-one HR solution for managing your small business with confidence: https://www.justworks.com Persona—A global leader in digital identity verification: https://withpersona.com/lenny *Transcript:* https://www.lennysnewsletter.com/p/al-engineering-101-with-chip-huyen *My biggest takeaways (for paid newsletter subscribers):* https://www.lennysnewsletter.com/i/176081814/my-biggest-takeaways-from-this-conversation *Where to find Chip Huyen:* • X: https://x.com/chipro • LinkedIn: https://www.linkedin.com/in/chiphuyen/ • Website: https://huyenchip.com/ • Substack: https://substack.com/@chiphuyen *Where to find Lenny:* • Newsletter: https://www.lennysnewsletter.com • X: https://twitter.com/lennysan • LinkedIn: https://www.linkedin.com/in/lennyrachitsky/ *In this episode, we cover:* (00:00) Introduction to Chip Huyen (04:28) Chip’s viral LinkedIn post (07:05) Understanding AI training: pre-training vs. post-training (08:50) Language modeling explained (13:55) The importance of post-training (15:20) Reinforcement learning and human feedback (22:23) The importance of evals in AI development (31:55) Retrieval augmented generation (RAG) explained (38:50) Challenges in AI tool adoption (43:19) Challenges in measuring productivity (45:20) The three-bucket test (49:10) The future of engineering roles (55:31) ML Engineers vs. AI engineers (57:12) Looking forward: the impact of AI (01:05:48) Model capabilities vs. perceived performance (01:08:23) Lightning round and final thoughts *Referenced:* • Chip’s LinkedIn post on what actually improves AI apps: https://www.linkedin.com/posts/chiphuyen_aiapplications-aiengineering-activity-7358971409227792384-y0mf/ • Prediction and Entropy of Printed English: https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf • Why experts writing AI evals is creating the fastest-growing companies in history | Brendan Foody (CEO of Mercor): https://www.lennysnewsletter.com/p/experts-writing-ai-evals-brendan-foody •Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO): https://www.lennysnewsletter.com/p/inside-handshake-garrett-lord • First interview with Scale AI’s CEO: $14B Meta deal, what’s working in enterprise AI, and what frontier labs are building next | Jason Droege: https://www.lennysnewsletter.com/p/first-interview-with-scale-ais-ceo-jason-droege • Anthropic’s CPO on what comes next | Mike Krieger (co-founder of Instagram): https://www.lennysnewsletter.com/p/anthropics-cpo-heres-what-comes-next • Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course): https://www.lennysnewsletter.com/p/why-ai-evals-are-the-hottest-new-skill ... References continued at: https://www.lennysnewsletter.com/p/al-engineering-101-with-chip-huyen *Recommended books:* • The Complete Sherlock Holmes: https://www.amazon.com/Complete-Sherlock-Holmes-Volumes/dp/0553328255 • AI Engineering: Building Applications with Foundation Models: https://www.amazon.com/AI-Engineering-Building-Applications-Foundation/dp/1098166302 • The Selfish Gene: https://www.amazon.com/Selfish-Gene-Anniversary-Introduction/dp/0199291152 • From Third World to First: The Singapore Story: 1965-2000: https://www.amazon.com/Third-World-First-Singapore-1965-2000/dp/0060197765 _Production and marketing by https://penname.co/._ _For inquiries about sponsoring the podcast, email [email protected]._ Lenny may be an investor in the companies discussed.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  A question that got asked a lot and a lot is how do we keep up to date with the latest AI news? Why, why do you have to keep up to date with the latest AI news? If you talk to the users, you understand what they want, what they don't want, look into the feedbacks, then it can actually improve the application way, way, way more.

• 0:14Lenny Rachitsky?

  A lot of companies are building AI products. A lot of companies are not having a good time building AI products.

• 0:19Speaker 00

  We are in an ideal crisis. Now we have all these really cool tools to have to everything from scratch. You have your design, it can have your write code, you have your website. So in theory, we should see a lot more. But at the same time, people are like somehow stuck, they don't know what to build.

• 0:33Lenny Rachitsky?

  All this AI hype, the data is actually showing most companies try it, doesn't do a lot, they stop. What do you think is the gap here?

• 0:38Speaker 00

  It's really hard to measure productivity. So I do ask people to ask their managers, would you rather have give everyone on the team very expensive coding agent subscriptions? Or you get an extra headcount? Almost everyone, the managers would say headcount. But if you ask VP level or someone who manage a lot of teams, they would say one AI assistant. Because as managers, you are still growing. So for you, having one HR headcount is big. Whereas for executive, maybe you have more business metrics that you you care about. So you actually think about what actually drive productivity metrics for you.

• 1:11Lenny Rachitsky?

  Today, my guest is Chip Huyen. Unlike a lot of people who share insights into building great AI products and where things are heading. Chip has built multiple successful AI products, platforms, tools. Chip was a core developer on Nvidia's Nemo platform, an AI researcher at Netflix. She taught machine learning at Stanford. She's also a two-time founder and the author of two of the most popular books in the world of AI, including her most recent book called AI Engineering, which has been the most read book on the O'Reilly platform since its launch. She's also gotten to work with a lot of enterprises on their AI strategies. And so she gets to see what's actually happening on the ground inside a lot of different companies. In her conversation, Chip explains a lot of the basics, like what exactly does pre-training and post-training look like?

• 1:56Lenny Rachitsky?

  What is RAG? What is reinforcement learning? What is RLHF? We also get into everything she's learned about how to build great AI products, including what people think it takes and what it actually takes. We talk about the most common pitfalls that companies run into, where she's seeing. The most productivity gains and so much more. This episode is quite technical, more technical than most conversations I've had, and is meant for anyone looking for a more in-depth conversation about AI. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. And if you become an annual subscriber of my newsletter, you get a year free of 16 incredible products, including Devon, Lovable, Replit, Bolt, N8N, Linear, Superhuman, Descript, Whisperflow, Gamma, Perplexity, Warp, Granola, Magic Patterns, Ricast, JPRD, and Mobin. Head on over to lenny'snewsletter.com and click Product Pass.

• 2:42Lenny Rachitsky?

  With that, I bring you Chip, when, after a short word from our sponsors. This episode is brought to you by DScout. Design teams today are expected to move fast, but also to get it right. That's where DScout comes in. DScout is the all-in-one research platform built for modern product and design teams. Whether you're running usability tests, interviews, surveys, or in-the-wild fieldwork, DScout makes it easy to connect with real users and get real insights fast. You can even test your Figma prototypes directly inside the platform. No juggling tools, no chasing ghost participants. And with the industry's most trusted panel, plus AI-powered analysis, your team gets clarity and confidence to build better without slowing down.

• 3:26Lenny Rachitsky?

  So if you're ready to streamline your research, speed of decisions, and design with impact, head to DScout.com to learn more. That's D-S-C-O-U-T dot com, the answers you need to move confidently. Did you know that I have a whole team that helps me with my podcast and with my newsletter? I want everyone on that team to be super happy and thrive in their roles. Justworks knows that your employees are more than just your employees. They're your people. My team is spread out across Colorado, Australia, Nepal, West Africa, and San Francisco. My life would be so incredibly complicated to hire people internationally, to pay people on time and in their local currencies, and to answer their HR questions 24/7. But with Justworks, it's super easy.

• 4:08Lenny Rachitsky?

  Whether you're setting up your own automated payroll, offering premium benefits, or hiring internationally, Justworks offers simple software and 24/7 human support from small business experts for you and your people. They do your human resources right, so that you can do right by your people. Justworks, for your people.

• 4:30Lenny Rachitsky?

  Chip, thank you so much for being here and welcome to the podcast.

• 4:34Speaker 00

  Hi, Lenny. I've been a big fan of the podcast for a while, so I'm really excited to be here. Thank you for having me.

• 4:39Lenny Rachitsky?

  I want to start with this table slash chart that you shared on LinkedIn a while ago that went super viral. And I think it went super viral because it hit a nerve with a lot of people. And let me just read this and we'll show this on YouTube for people that are watching. So it's this very simple table you shared of what people think will improve AI apps and what actually improves AI apps. What people think will improve AI apps, staying up to date with latest AI news, adopting the newest agentic framework, agonizing over vector databases to use, constantly evaluating what model is smarter, fine tuning a model. And then you have what actually improves AI apps, talking to users, building more reliable platforms, preparing better data, optimizing end-to-end workflows, writing better prompts. Why do you think this hit such a nerve with people?

• 5:24Lenny Rachitsky?

  And just what, if you had to boil it down, what do you think is, what do you think people are missing about building successful AI apps?

• 5:28Speaker 00

  One question that got asked a lot and a lot is that, how do we keep up to date with the latest AI news? And I'm like, why, why do you need to keep up to date with the latest AI news? And I was very counterintuitive, but I guess there's so much news out there. A lot of people also ask me questions like, how do I choose between two different technologies? Like maybe like recently like MCP versus like agent to agents, right? Like protocol and it was like, which one is better or like this or that? And things are a serious question. If you ask them, it's like first, like if, how much of the improvements would you get? Like from like optimal solutions versus non-optimal solutions, right? And sometimes they were like, actually, it's not much, right?

• 6:10Speaker 00

  And I was like, okay, if it's not much improvements, then why do you want to spend so much time debating? So it means it doesn't make so much difference to your performance. And another question they ask is like, if you adopt a new technology, like how hard it could be to switch that out to another? And sometimes it will like, oh, I think it could be like a lot of work switching it out. And I was just like, hmm, let's say here's a new technology. It hasn't been tested by a lot of people and if you adopt it, it would be stuck with it forever. Like, do you actually want to adopt it, right? And maybe you want to think twice about. About like over commit to like a new technologies that hasn't been better tested.

• 6:49Lenny Rachitsky?

  I love your just broader advice is just simple, like talk to, to build successful apps, talk to users, build better data, write better prompts, optimize the user experience. Versus just like, what is the latest and greatest? What's the best model to use right now? What's happening in AI? Let me follow this thread of this idea of fine tuning and basically post-training. There's all these terms that people hear in AI, and I think this is going to be a really good opportunity for people to learn what we're actually talking about. Since you actually do these things, you build these things, you work with companies doing these things. And there's a few terms I want to sprinkle in through the conversation, but let's start with this one. What, what's the simplest way for someone to understand? What is this difference between pre-training and post-training? And then just how fine tuning fits into that, just what fine tuning actually is?

• 7:34Speaker 00

  Disclaimer, I don't have like full visibility into like on what like this big secretive like frontier labs are doing. Uh, but right from what I heard, right? So, so I think it's like one is um like supervised fine tuning, we have demonstration data. And you have like a bunch of like um experts, okay, here's a prompt, right? And here is uh what the answer should be like. And you, you, you just train it like on like to like stim- like uh simulate, um like emulate what the human expert could be like. And that's also like what a lot of people with like um so, so open source models are doing as they do it by distillation. So instead of having human experts to like write really good, starting a great answers to like prompts, they get like very popular, famous, good models.

• 8:18Speaker 00

  To like generate a response to it and like getting this trained smaller model to emulate. So, so it's sometimes you see people was just like, so that's because like some, I, I really appreciate open source community by the way. But like going from like have been with a trainer models that can emulate a existing good model. It's very different from like being actually trained of good models like an output for existing good models. So it's a big step there. So yeah, so like we have my supervised fine tuning. And another thing that's like very big, I'm not sure you have guests talking about it already. But like reinforcement learning is like everywhere.

• 8:52Lenny Rachitsky?

  Let's pause on that because I would definitely want to spend time on that. And that's such a cool topic that I want, that's merging more and more in my conversations. But just to even summarize the things you just shared, which I think is really, really important stuff. So the idea here is a model, essentially this algorithm, piece of code that someone writes. And say the frontier models are feeding it just like the entire internet of content. And basically it's trying to test itself on predicting in all of the, across all that data, the next word. Essentially, the token is a simpler way, is the correct way to think about it. But a simpler way to think about it is like the next word in a, in text. And as it gets it wrong, it adjusts these things called weights essentially. Just like, is that a simple way to think about it, even though that's, even that's just like very surface level?

• 9:35Speaker 00

  So I think of language modeling as a way of encoding statistical information about a language, right? So, so let's say that we both speak English. So we can't get a sense of like, what is most statistically likely? Like if I say my favorite color is, then you would say, okay, that should be another color. Like the word blue would be much more likely to appear than the word like, uh, I don't know, table, right? Because statistically, blue is more likely to come up to my favorite color is. So, so it's a sign get this, um, it's, it's, it's a way of encoding statistical information. So like when language modeling, when you train a large amount of data, like you see a lot of languages, a lot of domains.

• 10:13Speaker 00

  So it can tell like, okay, you guys say this sentence, then it user do the prompts and it would come like with the next, uh, most likely token. So by the way, it's not a new idea. I shouldn't have realized. So this idea comes very, very old, like from the 1951 papers.

• 10:30Speaker 00

  The English entropy, I think it's like called Shannon, it's a great paper. And I think I've read this, a story I really like, um, is from, did you read Sherlock Holmes, by the way?

• 10:38Lenny Rachitsky?

  Yeah, I read a few Sherlock Holmes books, yeah.

• 10:41Speaker 00

  Yeah. So, so this is a story of like when Sherlock Holmes was using this statistical information to like have shown a case. So he was getting, um, so this is this story, uh, there is. Somebody left a message with a lot of stick figures. So Shuro Home was like, okay, he knows that in English, the most common letter is E. Then the most common stick figure must be E, right? And then he goes, he stuck like that, it was really so, uh, so the, uh, the code. So I think that's language, so in a way, it's like simple language modeling, right? But instead of like at a word level, he does it as like to like character level. And token is something in between, right? A token is not quite a word, uh, but it's bigger than a character.

• 11:22Speaker 00

  So let's say, uh, we we say token because, uh, it helps us like read what helps reduce vocabulary. Because with character is like smallest amount of like vocabulary, right? And so so AMP has like twenty six character, but words can have like millions and millions, right? Uh, whereas, um, um, tokens, you can like be able to like get like the sweet spot maybe the two. So let's say that we have a, uh, the new word like, uh, uh, how to say like podcasting, right? Let's say it's a, it's a new word, but it can be visioned podcast and ing. So people understand, okay, podcast, you know, the meaning, you know, that ing is like, uh, like a verb, like gerund, whatever it is. So we we know the word like podcasting, so that's why the token comes in.

• 12:05Speaker 00

  But yeah, uh, that's like the pre-tuning is basically like encoding statistical informations of language to have you predict what is most likely. Um, I think that most likely is the most simple way of doing it. Uh, because it's more like building distributions of like, okay, so there's a next token could be like more like a ninety percent of the chat. It could be like a color, ninety, like ten percent of the time could be something else, right? So you basically to build, so language could like pick, like depending on your sampling strategy. Like, do you want it to always pick the most likely token? Or you wanted to pick something more creative, you know? So so so I think my sampling strategy, I think is something extremely important. Uh, so it can have your boost of performance in a, in a huge way and very, very underrated.

• 12:49Lenny Rachitsky?

  Okay, awesome. So essentially, uh, a model is just code with this whole, uh, set of weights. Essentially, the statistical model that has learned to predict what comes next after certain words and phrases.

• 13:02Speaker 00

  Yeah.

• 13:03Lenny Rachitsky?

  And then post-training and fine-tuning specifically is doing that same thing. So pre-training, you get like GPT five. Fine tuning is someone taking GPT five and doing the same sort of thing. Adjusting these weights a little bit for specific use cases on data that they find is necessary to do their very specific use case. Is that a simple way to think about it?

• 13:24Speaker 00

  Yeah, I think you play with it as like functions, right? So let's say just like you have a maybe has a functions of like, maybe Lenny's height is maybe like one X, like one X plus something like two X, like one and plus something is is a weight, right? So you change it until you fit the the the correct data, which is like my height and your height, right? So so you can take it the weight is just like a weight like they function. So you so so you like chain adjust the weight so they can fit the data, which is the training data.

• 13:54Lenny Rachitsky?

  Awesome. Okay, so so we're talking about pre-training, post-training, fine-tuning. Is there anything else here that's important to share about just like what this is exactly? What people need to understand about these parts of training?

• 14:06Speaker 00

  So the vast majority of time, we don't touch on like pre-training model like as users, we don't.

• 14:11Lenny Rachitsky?

  Right, it's already done for us.

• 14:13Speaker 00

  Yeah, so so I think my action is a bit of fun, like a process like when my friend's training models, I tried to play with their pre-training model and they're horrendous. They're like saying things just like, oh, my gosh, it's like, yeah, it's crazy. So so is this very interesting to look at like how much of like post-training can change the model behavior? Yeah, and I think that's where like a lot of time is that a lot of people are spending energy on nowadays. They functionally lab is on like post-training. Because pre-training, I think, so pre-training have been

• 14:46Speaker 00

  used to like increase the general capacity of of of a model capabilities of a model. And it depends on even is a lot of data and like model size like to increase to increase the model capabilities. And at some point, we are actually like have cut max out on the internet data, right? And people like text it up very much. I think a lot of people are doing like other data like audios and videos. And everyone's trying to think of like, what is the new source of data? But we're like post-training, but like method course of like this more of like everyone can have very similar pre-training data. It's like post-training is where they make a big difference nowadays.

• 15:21Lenny Rachitsky?

  This is a good segue to you talked about supervised learning versus unsupervised learning. I love getting into this, by the way, this is super interesting. So you're talking about labeled data. Basically supervised learning is AI learning on data that somebody has already labeled and told it here is correct versus incorrect. For example, this is spam versus not spam. This is a good short story. This is not a good short story. We've had the CEOs of a lot of these companies that do this for labs. Merckor and Scale, Handshake, there's Micro, there's a few others. So is that essentially what these companies are doing for labs? Giving them labeled data, high-quality data to drain on?

• 15:57Speaker 00

  It is in a way, but I think it's more like a product of big equations. So there are a lot more different components than that. So that's why we're talking about reinforcement learning. I'm not sure if your CEOs that you interview bring up like that term. So the idea is that you want people to like, so like, let's say you have a model, give the model like a prompt, right? And it produce an output, right? You want to buy, like, you want to reinforce, encourage the model to produce an output that is better, right? So like, how like, now it comes to like, how do we know that the answer is good or bad? Right? So easily people realize on like signals. So one way to get like a first one good or bad is like human feedback, right?

• 16:40Speaker 00

  It happens, you have two responses, you can, okay, this one's better than the other. And we do that is because like as humans, we tend to, it's very hard to give like concrete score. But it's easier to do comparisons, right? Like if you ask me, okay, give this song a score. I'm not a musician, like, and don't know like how hard it is. Like, it's like, yeah, I don't know, like, what, like, how 10, I'm going to 6, you know? And like, if you ask me again a month from now and I completely forgotten, it's okay, maybe now 7 or like 4. I don't know. But then if you ask me, okay, here are two songs, and which one would you prefer to play for the birthday party? I was like, okay, I can prefer this song. So like comparison is a lot easier. So yeah, so you have a humans, you have human feedback, and then you use this human feedback to trade a reward model.

• 17:23Speaker 00

  So you like tell like which, like, so and then if reward model help you like, okay, it's a model, I'll produce this response. It's a reward model can score. Is this good or bad? And you're charging bias to what like producing better model, the better responses. Another way is like you can instead of using a human, you can use like AI, right? Like, because the response say, yes, a good, good or bad, right? Or in terms of things, like people are very big on our day, it's like verifiable rewards, which is like natural. So basically they give it a math problems and then math solutions. Like, is a model output a solution? It's, you know, okay, it's a expected response should be in a 482. And it doesn't provide 482s, and it's, then it's wrong, right? And it's not a good response. Um. So so yeah, so like a lot of time people like using this um human labor like human um

• 18:07Speaker 00

  human laborers should like produce like ma- like um how to say expert questions and like the expected answers and in a way that like desire systems that like verifiable so that the the models can can be trained on, yeah.

• 18:21Lenny Rachitsky?

  Okay, I'm really glad you went there. This is essentially RLHF, reinforcement learning with human feedback, which is exactly what I wanted to also talk about, right?

• 18:29Speaker 00

  Yeah, so um I think it's like it's general it's like it's a way of learning. It's like training is conversational learning and whether it learned from human feedback or like AI feedback or like verifiable rewards. Uh I think I say you say it's just different way of like um clipping signals.

• 18:44Lenny Rachitsky?

  Awesome, yeah, that's uh I we had the CEO of Anthropic on the podcast and he talked about their version of RLHF, which is AI-driven reinforcement learning. I love the way you phrased it where you basically you want to help the model, you want to reinforce correct behavior and correct answers, and this is the method to do it, whether it's say an engineer seeing an output from a model being like, no, here's how I would code it differently and then training and it's training a different model that the original model works with to tell it, am I correct or not correct? Is that right?

• 19:15Speaker 00

  Yeah, I I think I think that's a way of of looking into it. And I think that's a space is so exciting nowadays because there are so many like domain experts task that the model like that model developers want model to do well on, right? Let's say you're like accountant, right? Like maybe you want to use a model to have accounting task, so you you I need a lot of like accounting data, like examples for my accountant, so you need to hire a lot of them to like do it. Or if you want to physics problem, I want to do um I don't know, like legal questions and stuff or like engineering questions or like somebody was telling me they want to do like uh using like coding for to solve scientific problems and not just like coding to build product, which is another different whole realm of things. And I also like using very specific toolings like uh

• 20:00Speaker 00

  Yeah, like I'm not sure what apps you use, but maybe like a 48 day app or like QuickBooks or like Google Excel, like they are very specific, like tool specific um expert expertise, so you want the model to what you learn. So they need a lot of like humans expert in this area to like create data to treat them. Um and it's a massive things, it's like people, because everyone wants a lot of data and like won't slap it like unlimited budget. Uh but uh whether I think this is also like a little bit of low key interesting economics. I'm not sure you've talked to to like the guests about, I thought it's very interesting if I think about, because it's very lopsided, right? Because like there only like a very small numbers of frontier labs, right?

• 20:43Speaker 00

  And they want a lot of data and there's like a massive amount of like startups or companies providing later data. So like you can see these companies, like this startup like doing data labeling, they have like maybe some like massive AR. But you ask them like, okay, so how many customers you have? And they could be like a very small numbers. I'm not sure, I'm not sure you you you, I saw you smiling.

• 21:03Lenny Rachitsky?

  Yeah, yeah, we chat, we chat about that.

• 21:05Speaker 00

  Yeah, so so I'm like a bit bit like, look me uneasy, right? Here we have like a companies growing like crazy, but it's like heavily dependent on like two or three companies. And at the same time, like if I if I was this company frontier labs, what could be the right economical things for me to do right now? I want a lot of startups. I want to have a lot of providers so I can pick and choose. And then this providers can also like to compete each other to lower the price and it's so dependent on me, it would sound to me regardless. So so I feel like, yeah, so so this economics, the whole economics is very interesting to me and I'm curious to see how it plays out.

• 21:41Lenny Rachitsky?

  What I'm hearing is you're you're bearish on the future of these data labeling companies. Because as you said, they don't have a lot of leverage over pricing because they have so few customers. And there's so many people getting into the space. So basically, even though there's some of the fastest growing companies in the world, you're feeling like there's there's a challenge up ahead.

• 21:59Speaker 00

  I'm not sure if I'm bearish on it. Um I think I'm curious. Because I think things have has a way of work out in ways I don't expect. So I think that maybe these companies, they have a lot of data, maybe they wouldn't be what you use that to like have some insights that helps them like stay ahead of the curve, uh you know, so so I don't know.

• 22:22Lenny Rachitsky?

  A very fair answer. Okay, while we're on this topic, I want to chat about evals, which is a very recurring topic in this podcast. This is the other piece of data content these companies share that AI labs really need. Can you just talk about what an eval is, the simplest way to understand it? And then how this helps models get smarter?

• 22:41Speaker 00

  So I think if people approach eval, I think there are like two very different problems. One is a app builder, right? And like, can I say have an app that do like, uh, maybe a chatbot? Very simple, and it was first thing that came to my mind. And I want you to know if chatbot is good or bad, right? So so I need to come a little way with like eval, chatbot. Another thing is, I think of this as a task-specific eval design. So let's say I'm a model developer and I want to make my model better at code writing, right? And I was like, okay, but how, how, how do I even measure code writing, right? So I would need someone to like, okay, understand code writing and think about like, what makes good story, like what makes a story good?

• 23:26Speaker 00

  And then design the whole data set and then criteria to evaluate code writing. So yeah, so so I think there's that, I think it's like more like eval design. That is very interesting, come up with criteria, come up with guideline, how to do it? And then also like train people, like how to do it effectively. So I guess, you know, guys, I think eval is really, really fun because it's extremely creative. I was looking at like different evals, people were building, I was like, wow. Like, it's just not dry at all. It's just like super, super, super fun.

• 24:01Lenny Rachitsky?

  We had a whole podcast on evals with Hamel and Shreya and and that's exactly what they talked about. It's just, it's actually really fun to create evals for, for companies especially. So let's still dig into that one a little bit more. There's this kind of debate online that I don't know how big of a deal this debate is, but it feels like people spend a lot of time thinking about this. This idea of do we need evals for AI products? Some of the best companies say they don't really do evals. They just go on vibes. They're just like, is this working well? Can I feel it or not? What's your take on just the importance of building evals and the skill of evals for app, AI apps, not the model companies?

• 24:38Speaker 00

  You know, how to be like? Absolutely perfect a thing to win. You just need to be like good enough and being consistent about it. Okay, this is not the philosophy I follow, but like I have worked with enough companies to see that play out. So when I say like white company don't need Eva, right, let's say you are like an executive, right? And you want to have a new use case. So here's a use case you you started out with built and it's like it works well, right? The customers are somewhat happy, you don't have the exact metric for it, but like. So the traffic keeps increasing, like people seem happy, people keep buying stuff, right? And now here's our engineer coming like, okay, we need Eva for it. And so and it's like, it was like, okay, how much effort do we need to go into Eva? And they were like, okay, uh, maybe like two engineers, as much as much.

• 25:21Speaker 00

  And they could maybe would improve that and was like, okay, so how much expected gain can I get from it? And the engineer will be like, oh, maybe you can improve it from like eighty percent to like eighty two percent, eighty five percent, right? And I was like, okay, but who will take this, that's two engineers and be able to launch a new feature. Then it could give me like so much more like improvement, right? So so I think it's like one of them is like Eva, sometime people think of Eva, it's like, okay, this is good enough, just don't touch it. Like if you do spend a lot of energy on Eva, I would like only incremental improvement where it spends the energy on like another use case. And maybe like it's good enough that you, because it's vibe check it, right? So so I I do things that's like, maybe it's like just a deep bit is about, um. I do things that's like a lot of time people just like get things to the to the place where it's like, okay, good enough.

• 26:06Speaker 00

  People run, but and then, but of course, it's like there's a lot of risk associated with it, because if we don't have a clear metric, uh, you have a good visibility to how the applications or models are performing, it might do something very dumb, or it can cause you like. I know some something like crazy can happen. So so, yeah, so, um, so so I do think Eva is very, very important if you have Eva operate a scale. And we're like. Failures can have catastrophic consequences. Then you do need to be very tyrannical about like what you put in front of the users. Understand different failure modes, like what could go wrong. And also maybe in this basement that like it's a feature, the product is a competitive advantage, right?

• 26:51Speaker 00

  You want to be the best at it. So you want to have like a very strong understanding of like where you are and like where you are with the competitors. But it's just something that's like more like a low key. Okay, it's like something that's like, okay, that's not the core or like it helps with the users. Then maybe you don't need to be so, so obsessed or like theoretical about it. It's okay, that's good enough for now. And if it fails, then it fails. Like, okay, I know it's like it's such a fight, but like, yeah. Yeah, I think it's all about the question of like return investment. I'm a big fan of eval. I love reading eval. And I say it's like, I understand why some people would choose to not focus on eval right away and choose like bringing on new functionalities instead.

• 27:31Lenny Rachitsky?

  Awesome. That is a really pragmatic answer. What I'm hearing is evals are great, very important, especially if you're operating at scale. But pick your battles. You don't need to write evals for every little feature. Something that Hamon and Shreya shared is that people need just like, I don't know, five or seven evals for the most important elements of their product. Is that, is that what you see or do you see a lot more in production that people build and need?

• 27:54Speaker 00

  I, I don't think of like just a fixed number on like the evals. Like what was a good eval, right? The good eval is to guide the product development. So, so like you see eval, because I think our big fan of eval is, is that it helps you uncover opportunities where the product is doing well. So I sometimes I've seen it very obvious. It was okay, we look at eval and we realize it's like, okay, it performed really poorly on this like specific segment of users. And then we're looking to it's like, okay, what, what, what's wrong with it? And it turns out it's like, we just like, don't have a good messaging to it. So like maybe we should like just focus on the things of reading poorly and can improve significantly. Yeah. So I kind of like the number of eval is really depends. Like we have seen product with like

• 28:39Speaker 00

  hundreds of different metrics, right? Like people are going crazy. This is because like that product is like general, right? Have different, they have like one eval for like, I don't know, like veracity, have like one eval for like user sensitive data. And like another is like is for length, but like has a number of like. Okay, let's just put a good example, completely simple, like deep, deep research. So, so you have the application, you have like views a model, she like do deep research for you, right? Like, okay, like have a prompt, let me say, okay, do me a com- comprehensive research on all Lenny's podcast. And help me like, sort of like, uh, propose like, uh, show me report on what type of topics he's interested in, what kind of videos get the most views, or like, what topics that he's missing on that he should be covering, right?

• 29:25Speaker 00

  Like, has a kind of like prompt, then how do you evaluate the, the result, right? I don't think there's like one, like, metrics that would help, maybe it's just like, maybe you have like a hundred, I think if somebody has a benchmark and is a get like a hundred expert, like write a bunch of prompts and they go through like all the, all the answers on AI and I do it. And it's like, it's extremely costly and slow, right? But if I might have something else, for example, like, uh, one way I was thinking about it, um, I was talking to a friend about it, and, and one way is just like, how to produce the result of the, of the summary, right? At first, you need to go to do like gather informations, and to gather informations, you need to do a lot of search queries.

• 30:05Speaker 00

  Uh, you like gathers, craft the search results, and then from the search results, you like, uh, aggregate, and then maybe say, okay, I'm still missing on this. You have to go another route, or like another round, and it's the end of the summary. So every step of the way, you need evaluations, right? You don't need to the end to end. So maybe if it was a search query, it might first think about like, okay, now I write five search queries. And I look into like, how good are these search queries? Like, do they like, are they like similar to each other? Because in this five search queries that are very similar, like, okay, Lenny podcast, Lenny, Lenny podcast, uh, last month, Lenny podcast like two months ago, right? It's not, it's not very, very exciting, but like if the query is a podcast, like the, the keywords are like more, um, more diverse, right?

• 30:47Speaker 00

  And then I look at the results of the, of the search query, and they say you enter the search query like, Lenny podcast data labeling. And then they come up with like ten pages, uh, ten results, and then you come up with like, oh, Lenny podcast on, uh, I don't know, mug, um, I don't know, like, uh, frontier labs and have like ten results. And my look was a different webpage, like, how much of them overlapping? Like, are we, are we doing both like the breadth, like getting a lot of, uh, page, and so like, do we have depth? And also like have relevance, because if we come up with a search queries that completely irrelevant to the, to, to the original prompt? So I feel like every aspect of it, it would need a way of evaluating, right? So I don't think it's just like. How many eval should I get like?

• 31:29Speaker 00

  How many eval should do I need to get a good coverage, a high confidence? In my application's performance? And also to help me understand like where it is not performing well, so that I can fix it.

• 31:42Lenny Rachitsky?

  Awesome, and I'm hearing also just especially for the very core use case, like the most common path people take in your product is where you want to focus.

• 31:52Speaker 00

  Yeah, so yeah.

• 31:54Lenny Rachitsky?

  Okay, let me, there's one more term I want to cover and I want to go to somewhat different direction. RAG, people see this term a lot, R A G. What does it mean?

• 32:04Speaker 00

  So RAG is then for retrieval augmented generations, in a so not a specific to chat G P T. So the idea is just like for a lot of questions, we need context to answer. So I think it came pretty, oh, I think it's from the paper twenty seventeen. So so someone was like, so they realized it's like for a bunch of like benchmark, when the question answering benchmarks. They realize it's like, okay, if we give the model informations about the questions, the next to answer can be much, much better. So what's it do was just try to retrieve information from Wikipedia. So for for question for topics, it's like retrieve that and then put into the context and like answer it does much better. So I feel like it sounds like a no brainer, right? I mean, like obviously, so so I think that's what RAG it as a simple sense is just like

• 32:49Speaker 00

  providing the model with a relevant context, so so that it can answer the questions. And and it's where like things get like a really more more interesting. Because traditionally, when it started out, RAG is mostly like text. So so we we talk about like a lot of way of like how to prepare data, so that the model can retrieve effectively. Let's say it's like, not everything is a Wikipedia page, right? Like Wikipedia page is pretty contained and like you know, okay, everything about it is about a topic. Uh, but a lot of time have documents, like true me a lot, right? And like they have a weird way of like structures and documents. Let's say that you have documents about Lenny, uh, podcast, right? And in the in the future, in the beginning of documents, like from now on. Podcast wouldn't refer to Lenny's podcast, right?

• 33:34Speaker 00

  So let's say somebody in the future is like, okay, tell me about Lenny, right? Lenny's work. And because as a rest of the document does not have the term Lenny, you just don't know, uh, you might not reach through it. And the document is long enough that it's chunked into a different part. So like the second part happens, doesn't have the, the word many, so you cannot reach it. So you have to find a way to process data so that make sure it's like, it can retrieve the information that's relevant to the query, even though it might not immediate, like obvious that is related. So it will come up with like only thing if I think like contextual visual. Um, like, uh, giving a chunk of the data, uh, that relevant like maybe in a summary metadata so that it knows. Um, or like some people use like as a hypothetical questions is very interesting.

• 34:18Speaker 00

  Like for a given the chunk of like documents, I must share it a bunch of questions that the chunks can help answer. So that when I have a, a query is okay, does it match any of the like hypothetical questions? So I can, it can fetch it. So it's very interesting approach. Okay. So maybe before I go to the next thing, I just wanna say that's like data preparations for RAC is extremely important. And I would say just like in the, a lot of the companies that I have seen, so it's like the biggest performance in their RAC solutions coming from like data, data preparations, not agonizing over like what better databases to use. Uh, we got very database, uh, a quite is very important to care about, I think, like latency or like if you have like very specific access patterns, like read heavy or write heavy, of course, it's like it matters.

• 35:03Speaker 00

  But in term of like pure quality answers, right? I think the data preparation is just like hands out.

• 35:08Lenny Rachitsky?

  When you say data preparation, what's an example to make that real and concrete for us to understand?

• 35:13Speaker 00

  So, so like one way is just like, uh, um, just mentioned as in like, um, you have like chunks of data. So we could think about like how big of a chunk should be, right? Because, um, if it's like, so the thing about like, if the context you want to maximize, maybe you can, it's very simple example, right? Now you want to retrieve like a thousand words, right? So if a chunk of data is too long, um, then, so if, if a data chunk is long, then it's more likely to contain more relevant metadata, so you can retrieve more. But if it's too long, like then you have a thousand word and so chunk is like a thousand words to get a rich one chunk. So it's not very useful. But if it's too short, then you can can retrieve more relevant information. Like also it can retrieve a.

• 35:56Speaker 00

  Wider range of like documents and chunks. But the same time, each chunk is too small to contain relevant information. So we have like very nice like chunk design, like how big a chunk should be. Uh you add like contextual informations, like summary, metadata, hypothetical questions. Uh somebody was telling me this like um a very big performance they got is that from um rewriting their data in the question answering format. So like instead of having like, so they have a podcast, right? Instead of the chunking the podcast, you just like reframe, rewrite it into like, here's a question, here's answers um like and and produce a lot of them. It can use AI for that as well. So that's one example of data processing. A lot of example we I see is like for people helping like using AI to have like specific uh tool use and documentations, right?

• 36:41Speaker 00

  And a lot and we write documentation usually, a lot documents uh documentation today is written for human um reading. And AI reading is different because it's different because humans, we have like common sense and we kind of know what it is. Um so so one things are like uh human for human experts, they have the context that AI doesn't quite have. So somebody told me is that like what's a big change they have is like, let's say that um you have a you have a function, the document uh documentation for this, maybe this library. And this library say, okay, the output of this one is like maybe talking for like, I know, some crazy term, maybe like some uh temperature or something under graph. It should be like one, zero or minus one. And as a human expert, maybe you understand the scale and what one in the scale mean. But like for AI, it just really doesn't understand what it means.

• 37:26Speaker 00

  So so actually have like another annotation annotation la- layer for AI. It's like, okay, good temperatures, you've got one mint like that. It's not like it's a absolute temperature, it's my associated with the scale over there. So it's just saving that all this data processing to make it easier for AI to retrieve the relevant information to answer the questions.

• 37:45Lenny Rachitsky?

  This episode is brought to you by Persona, the verified identity platform, helping organizations onboard users, fight fraud and build trust. We talk a lot on this podcast about the amazing advances in AI. But this can be a double edged sword. For every wow moment, there are fraudsters using the same tech to wreak havoc, laundering money, taking over employee ID. Identities and impersonating businesses. Persona helps combat these threats with automated user, business and employee verification. Whether you're looking to catch candidate fraud, meet age restrictions, or keep your platform safe. Persona helps you verify users in a way that's tailored to your specific needs. Best of all, Persona makes it easy to know who you're dealing with without adding friction for good users.

• 38:28Lenny Rachitsky?

  This is why leading platforms like Etsy, LinkedIn, Square and Lyft trust Persona to secure their platform. Persona is also offering my listeners 500 free services per month for one full year. Just head to withpersona.com/lenny to get started. That's withpersona.com/lenny. Thanks again to Persona for sponsoring this episode. Awesome. Okay, so you've talked a bit about how you work with companies on these sorts of things, on their AI strategies, on their AI products, how they build, which tools they build, all these things. I want to spend a little time here because a lot of companies are building AI products. A lot of companies are not having a good time building AI products. Let me ask a few questions along these lines of what you've learned working with companies that are doing this well.

• 39:14Lenny Rachitsky?

  One is just, I guess, in terms of AI tool adoption and adoption in general within companies. There's always this talk recently of just like, all this AI hype, the data is actually showing most companies try it. Doesn't do a lot, they stop. And so there's all this just like, maybe this isn't going anywhere. So in terms of just adoption of tools and AI within companies, what are you seeing there?

• 39:32Speaker 00

  For gen AI in company, I think there are two types of gen AI toolings that have been, I have seen. Like ones is to like internal productivity, right? And I have coding tools, like chatbot, internal knowledge, like a lot of big enterprises have some kind of like a wrapper around like models. So like, but like with access, like maybe some different kind of a rack. So we should, I think we talk about that, okay, like text-based rack. I haven't talked about the agentic rack or like, I haven't seen like multimodal rack yet. But this like, yes, there's a whole very exciting area around that. Yeah. So like basically it should allow the employees to like access. Internal document, uh, somebody, someone will ask like, okay, um, I'm having a baby.

• 40:16Speaker 00

  What could be the maternal or paternal policy, right? Or like, am I having these operations? What the hell benefit like cover that? Or like, I want you like interview, I, I want to like refer my friend, what could be the process for that? So a lot of this like having chatbot, internal chatbot to help with internal operations. Um, and um, another things, um, another category is more like customer facing. Um, so, or like partner facing, um, so, so what a customer support chatbot is a big one. If a hotel chain, you might have like a booking chatbot, which is like somehow massive, like a lot of booking chatbot. Because I guess it's, it's, it's, I do have this theory of like a lot of applications. Uh, companies pursue because it can measure the concrete outcome.

• 41:00Speaker 00

  And I feel like booking or like sale chatbot is very clear, right? Like what's a conversion rate right now with a chatbot, uh, with human operators? And what could be conversion rate with a chatbot? And it's sometimes I think it's like very clear outcomes and companies are easier to buy into this, um, this solutions. So a lot of companies have that like customer, uh, facing chatbot. Uh, so yeah, so, so that is, um, another category of tool. Um, and I think that, um, I think for customers or external facing tools, um, because people are driven to, um, people are driven to choose applications with clear outcomes. So, so the questions of, uh, adopting them is really based on like whether they see the outcome or not.

• 41:45Speaker 00

  Of course, it's not perfect because sometimes, uh, the outcome can be bad, not because the idea or like the application's idea sell is bad. It's just because of the, I know, the, the, the process of building it is like not that great. Um, yeah, so, so it's tricky. For the internal adoptions of like tooling, so internal productivity, that's where it gets tricky. I would say like a lot of companies, uh, was a thing of a strategy, like I think of a strategy, if I usually have very, um, Have like two key aspect, right? It's like use cases and the second is talent. You might have a great data for great use cases, but you don't have talents and you cannot do it. So a lot of time in the beginning with Gen AI and it's still, and sometimes I'm really admire a lot of companies for that.

• 42:27Speaker 00

  It's just like, exactly, it was like, okay, we need our employees to be very Gen AI aware, like very AI literate, right? So what to do is just like, maybe like adopting a bunch of tools for, for the team to use. They have a lot of upskill, upscaling workshops, like they encourage learning. And then it's like a really, really good thing. And it's also like willing to spend a lot of money into like adopting like, um, giving people like chat G P T subscriptions, cursor subscriptions, uh, Claude code subscriptions, like to get the employees to like, to, to be more AI literate. Um, and that's the thing is like a lot of, this is exactly what comes to me, say, okay, we spent a ton of money on this tooling.

• 43:07Speaker 00

  But then we don't see, because you can see the usage, it was like, but people don't seem to use them as much. And what is the issue? So, so yeah, so I think that that is, um, that is tricky, yeah.

• 43:19Lenny Rachitsky?

  What do you think is the issue? Is it just they're not, they're like, they don't know how to use them? Like, what do you think is the gap here? Do you think we'll get to a place of just like, wow, work is completely different because of AI for a lot of companies?

• 43:32Speaker 00

  The main thing is like, it's really hard to measure productivity again. So I touch you a lot of people on their website, first of all, on my, on this sample is coding, right? A lot of companies not using coding agents, uh, or the coding, yes, I said the coding. Uh, and, um, I was asking, I was like, I was like, could you, do you think that like it helps with your productivity? And a lot of the questions are very handwavy, just like, so like, okay, so like, okay, I feel like it's been better, right? And I was like, okay, because we have more PRs, uh, we see more code and then immediate correct me, okay. But of course, code number live is not a good metric for that, right? So, so it's, it's really, really tricky.

• 44:11Speaker 00

  And it's something funny, uh, so, so, so I do ask, um, people to ask their managers, because I work with like either the VP level. So they do have like multiple teams under them. So I asked them like, okay, do you ask some managers, um, like, okay, would you rather? Have access, uh, could you rather have, give everyone on the team like very expensive coding agent um subscriptions? Or you get an extra headcount, right? Let's say it's like maybe like um and and almost everyone could say the managers could say headcount. But even as VP level or like someone who manage a lot of teams, they could say it's like they could want AI assist them, assist the tools.

• 44:50Speaker 00

  And the reason is that people say like, okay, because as managers, right, because you are still growing, like you're not as a level when you you manage hundreds of thousands of people. So for you, like having one HR headcount is like, is big. So you want that not for productivity reasons, but because you just want to have more people working for you. Whereas for executive, you care more about like the uh the maybe you have more like business metrics that you you care about. So so you actually think about it, what actually drive, drive productivity uh metrics for you. Uh so so yeah, so it's it's tricky. Um and I think that's like the question of like productivity, um it's not.

• 45:29Speaker 00

  I'm not sure it's like fundamentally is the some people more productive, but it's just like we don't have a good way of measuring productivity improvement. Uh another thing is also very widely, um and I think that people do tell me that they notice different buckets of of employees, like different reactions. To AI assist tools, like first of all, I I keep going back to coding because it's for it's big and it's like easier to my reasons about. Um so it's just like um I have different reports, like one team would tell me is that like um one of the people tell me, okay, amongst all his engineers. He thinks it's like senior engineers would get the most output, like would be more productive because it's like, okay, so that person very interesting.

• 46:11Speaker 00

  So so he actually divided his team to like three buckets, but he didn't tell them, obviously. He was like, okay, here's more like currently like a. Best performing, average performing and lowest performing. And then this is randomized trial. So like they give like half of each of each group like access to like. To like cursor. And then it was noticed like over time, it was like, okay, so something funny, like the the group. That get the biggest performance boost, like in his opinion, so it was very close in his team. There's a biggest boom boost like the senior engineer, so the highest performing, so the highest performing engineer get the biggest boost out of it. And then the second group is just like the the um. The average performing. So, so he, so his opinion is like, okay, the highest performing engineers is so normal proactive.

• 46:56Speaker 00

  They also know how to solve problems. So AI helps them solve problems better. Whereas the people who are having lowest performing, they only don't care much about work, right? So like this easier to just like go on autopilot, get it to like Jared, like that code and just like do it. And I just don't know how to do it. As another company, however, they tell me this like actually senior engineers are the one most resistant. They try using AI as this tooling because they said it's like, okay, but AI, because they are more opinionated. And they have very high standard. It was like, okay, but AI code, Jeddah code just sucks. So just like very, very resistant in using this. So I don't know. I, I haven't quite be able to reconcile like very different reports on that yet.

• 47:39Lenny Rachitsky?

  This is so interesting. So just to make sure I'm hearing what your, the story. So there's a company I work with that did a three bucket test with their engineering team where they created three sorts of groups. The highest performing engineers, mid performing engineers, lowest performing engineers. Uh, and gave some of them, so they gave some of them access to say cursor. Was it cursor or what did they give them access to? It was cursor?

• 48:03Speaker 00

  I think by that was cursor.

• 48:04Lenny Rachitsky?

  Okay, cool.

• 48:05Speaker 00

  Okay, I didn't work with them. This is more like a friend company.

• 48:08Lenny Rachitsky?

  Okay, it's a friend's company. So did they give like half of the higher performing engineers cursor and half not? Or how did they do the split?

• 48:14Speaker 00

  Yeah. So like they give like half of the entire company, but like half for each bucket. Yeah. And then they observe the difference in like productivity.

• 48:21Lenny Rachitsky?

  I see. So how did they even do that? They're just like, okay, you get cursor, you don't get cursor. Is that, how did they do that? That's so interesting.

• 48:27Speaker 00

  Yeah, I, I did again just the mechanics of it. Uh, but, but I was like, I respect you for doing a randomized trial.

• 48:32Lenny Rachitsky?

  That is so cool. Okay, wow. How large was this engineering team? Was it like hundreds of people?

• 48:38Speaker 00

  It's, it's not that large. It's about like maybe the, um. Thirty, twenty, forty, yeah.

• 48:43Lenny Rachitsky?

  Thirty to forty, okay.

• 48:44Speaker 00

  Yeah.

• 48:44Lenny Rachitsky?

  Wow, okay, so they found that the highest performing engineers had the most benefit from using AI tools, and then behind them was the middle tier engineers and the worst performers were the lowest performers.

• 48:58Speaker 00

  Yeah, but it's still not the same everywhere, um, like some companies, yeah, yeah, different.

• 49:03Lenny Rachitsky?

  Right, this other example he shared of just senior engineers in this one example are most resistant to changing the way they work, which I get. I, I do feel like the most valuable people right now, other than M L researchers, uh, uh, and AI researchers like yourself, uh, are senior engineers, because it feels like junior engineers are just like, so much of this is now done by AI, but it's, but an engineer that knows what they're doing, that understands how things work at a large scale with AI tools, just basically like infinite junior engineers doing their bidding, feels like an extremely valuable and powerful asset.

• 49:38Speaker 00

  Yeah, I definitely like really appreciate, as you see companies, like we appreciate engineers who are, um, have a good understanding of the whole systems and be able to have good problem solving skill, like thinking holistically instead of like locally, uh, locally. Um, or when our company have seen the way they work, as I told me, it's like work completely different now, and like, so they actually restructured engineering org, so that like they get more senior engineers to be more in the PR review, because they like to get like, uh, sort of writing guidelines on like what is a good engineering practices, um, what is a process would be like, or maybe like, okay, so they write like a lot of like processes, uh, on how to work well. And then they, um, and then they have more, more junior engineers just like produce code

• 50:22Speaker 00

  and, and like submit PR, but senior engineer more in the reviewing case. So I think it's, it might be prepared for the future. So another company actually told me something very similar, so they're kind of preparing for the future when they only need a very small. Group of like very, very strong engineers to like create like processes and like reviewing code to get into production, but I get like AI or like junior engineers to like produce code. But then the question becomes just like, how does one become a very strong?

• 50:53Lenny Rachitsky?

  Right, that's right, that's right.

• 50:54Speaker 00

  Yeah, I feel like, yeah. Yeah, so, so I don't know what's the process. I was thinking about like, yeah, um.

• 51:01Lenny Rachitsky?

  No one's thinking about it, it's just, it's a problem. We won't have any more in ten, twenty years, there'll be no more engineers, because no one's hiring junior engineers. Although I could make the case, junior engineers, people just getting into computer science right now are just native, AI native. And in theory, you could argue they will become really good, really fast if they're curious, aren't just, you know, de- delegating learning and thinking to AI, but learning how to actually, using it to learn how to code well and architect correctly. Like, you could argue they will be the most successful engineers in the future.

• 51:33Speaker 00

  I do think that what I mentioned said loading to architect, um, I think I, I grouped that in like system thinking. I do think it's a very important skill, uh, because I think AI can help automate a lot of like, um, destroyed skills. But like, knowing how to utilize these skills together to solve a problem is, is very, uh, is, is, is, is hard. So there's a, a webinar between, um, Neera and Sammy, who's my, one of my favorite professors. He was a chair of the curriculum at the CS department at Stanford, so he spent a lot of time thinking about CS educations, right? Like, what, what, what should students learn nowadays in the era of like AI coding? And then the other person is like Andrew Ng, which is of course is like a, a legend in the AI space.

• 52:17Speaker 00

  And Neera Sami, person like Sami, something very interesting is like, he said like, a lot of people think that CS is about coding, but it's not. Like coding is just a means to an end. Like CS is about system thinking, like using like coding to solve actual problem. And problem solving will never go away, because like what like AI can automate more stuff, the problems just get bigger. But as a process of understanding what caused the issue and like how to like design step-by-step solution to it will always be there. Um, so I think an example of, um. Of like, I actually have a lot of issues with like AI for like um in the way of like is debugging. So I'm not sure you use a lot of AI for coding, but like a a something I have noticed and also seen from my friends, it's like.

• 53:01Speaker 00

  It is pretty good when you have very clear, well-defined tasks, maybe write documentation, fix specific features, or like build an app from scratch, right? Like doesn't have to interact with a large existing codebase. But you add something like a little bit more complicated, uh maybe require interaction with a lot of components and stuff, it's usually like not that good. Um and and for example, I was using AI to like use um to deploy applications. Um and I was testing out a new uh hosting service I was not familiar with, it was okay, like usually they form me. So what the AI does give me is like confidence to try new tool. Like before with AI, I was like trying new tools, history, not documentation for the beginning, but AI was like, okay, just try it out and and learn.

• 53:42Speaker 00

  So I was testing out this new hosting service and it kept getting a bug, it was like very, very annoying. And I was like, okay, I ask a carco like fix it. And it keep giving me like, it keep changing the way, like maybe change the environment variable, fix the code, maybe I change from the function to this function, maybe change the la- language. Maybe it doesn't process JavaScript, well, I don't know, whatever. And it didn't work and it was like, okay, that's it. I'm just gonna read the document, document uh documentation myself and see what's wrong. And it turns out it's like I'm on another tier, like the feature I want did not, is not available in this tier, right? So I feel like, okay, so the issue with carco is just trying to focus on fixing things from a very, a different component, whereas the issue is from a different component.

• 54:27Speaker 00

  So I think I think of like, okay, be understanding how different components um work together and where the source of issue might come from. You need to, you need to give a holistic view of it. And it's made me think is like, okay, how do we teach AI, like system thinking, like like that, right? And I have all the human experts, like having like write like very much, build on a scaffold. Uh just like, okay, for this kind of problem, look into this, look into that, look into that and then stuff. So so I think that could be one way, but that's what made me think is like, how do we teach humans, like system thinking? Uh yeah, so so yeah, so I think it's very interesting um skill, I I do think it's very important.

• 55:03Lenny Rachitsky?

  That's exactly the same insight Brett Taylor shared on the podcast. He's the co-founder of Sierra. He created Google Maps. He was CEO of Salesforce, Quip, a few other things. And I asked him, just like, should people learn to code? And his point is exactly what you said, which is learning, taking computer science classes is not about learning Java and Python. It's learning how systems work and how code operates and how software works broadly, not just here's like a function to do a thing. One thing that I wanted to help people understand, you, you have this book called AI Engineering, which is essentially helping people understand this new genre of engineer. And you have this really simple way of thinking about the difference between an ML engineer and an AI engineer,

• 55:46Lenny Rachitsky?

  which has a really good corollary to product managers now of just like an AI product manager versus a non-AI product manager. The way you describe it and fill in what I'm missing is just ML engineers build models themselves. AI engineers use existing models to build products. Anything you want to add there?

• 56:05Speaker 00

  One thing I really dislike about writing books is that it has to define like this. And, and I think it's like no definitions will be perfect because there always be like edge cases. But yeah, in general, I think it's like, it's like gen AI as a service, like more as a service. Like when somebody build the models for you and the base model performance is a pretty strong. So, so it's like, it's enable people to just like, okay, now I want to integrate AI into my product. I don't need to learn what great and this and that, even though knowing that could really help. But, but yeah, it's like, it's makes the entry barrier really low for people who want to use AI to build product. And at the same time, AI capabilities are like so strong.

• 56:46Speaker 00

  It's like, it's also like increased like the possibilities, like the type of applications that AI can be used for. So I think like, yes, like both entry barriers, like super low and I said, demand for like A applications like a lot bigger. So I feel this is very, very exciting. It's opens up like a whole new ball of possibilities.

• 57:04Lenny Rachitsky?

  Yeah, it's like now you don't have the time, you don't even spend time building this AI brain. Now you can just use it to do stuff. Such a, such an unlock. Okay, maybe just a final question. You get to see a lot of what's working, what's not working, where things are heading. I'm curious just if you had to think about in the next two or three years, just where things are heading. What do you think, what do you think, how do you think building products will be different? How do you think companies working will be different if you had to think of? Maybe the biggest change we expect to see in the next few years in terms of how companies work?

• 57:40Speaker 00

  I think in a lot of organizations, they don't move that fast, right? But at the same time, they should move faster than I expected. Because again, I think it's like biased, like I don't work with dinosaur companies. I don't care. I think a lot of executives who come to me are like very forward looking. So maybe for me, I'm very biased towards the work like organizations is like move fast. So, so yeah, so I think one big change I see is like in organizational structure. I think there's a lot of value placed in like, so before, right? We have like a lot of disjointed team, like we have very clear like engineering team, product team. But then there's a question of like, who should write Eva, right? Like who should own the metrics?

• 58:25Speaker 00

  And it turns out it's like Eva is not a, it's not, it's not a separate problem. It's a system problem, right? Because you, you, you need to look into different components, so they interact each other. You need to use the behaviors, but you need to know what users care about so that you can, so that you can like write, write Eva, cause it's like reflect what users care about. So, so all of that, like you can sort it from like, you know, go to different component architectures, place guardrails and stuff. So it's just engineering, but understanding users is like what product, right? So, so because of like a lot of things and Eva is extremely important. So like the guy bring product team and like engineering team, even like marketing team, like user acquisition, like very close to each other. So, so yeah, since in a way, so people are structuring so there's more communications between like previously very distinct functions.

• 59:11Speaker 00

  Another thing is just like I also see as teams. Of course, I think about like what can be automated in the next few years and what work cannot be automated. And I seen that people already like shedding like, actually it's, it's a little bit like. Scary to think about it, but I also think it's like the team, the web topies, it's like, okay, this is between you and me, but we have to like got rid of these functions, right? Like for a lot of thing like previously outsourced, for example, like traditionally is a business outsourcing this core to them. And like can be done with like not um can be more um systemized uh um systematized. Um so so with that, you can actually use AI to automate a lot of that.

• 59:51Speaker 00

  And so like the separation, we're thinking more like what is the value of like junior engineers or senior engineers, how to restructure engineering org for that? Um so so yeah, so I do definitely think that um is one thing to success organization, people are just moving pieces around and like thinking about like use cases. Um whether you need to like spin out new use cases and who would lead the new effort and like, yeah, um that is one big uh change. Another things in top of like AI, I think this is um, I'm not sure how true this is. Um I guess I'm I'm also like on the camp of like thinking that is has merit.

• 1:00:30Speaker 00

  Is is a camp of like, okay, uh base models, we have probably like not quite max out, but we want, we are unlikely to see like really, really strong, like crazily strong model. So like, you remember like when we have like GPT, right? And GPT two, which is a big step up, like an automatic to like, like better than like GPT and then GPT three, which is like much much bigger and GPT four, much, much bigger. And I of course, I have GPT five, but like it's GPT five, like that scale of like much bigger, like a step jump compared to like the previous. I think it's a debate, right? So so I think that it's like we had disappointment, like the base model um performance improvement is not gonna be like

• 1:01:14Speaker 00

  mind blowing, and it was in the last three years. Uh so so I think it's like a lot of like improvements we're gonna see in the post-training phase, in the application building phase, uh and um. And yes, also I think this is where I feel I would see a lot of improvement there. I also very like interested in like multimodality. Um. So we've seen a lot of uh text base, uh but I think there's a lot of um audio, videos, use cases, uh that is very, very exciting. And I think audio is not quite as soul as what I think because I do work with like uh with with like a a couple of like voice startups. And when I talk to vo- think about voice, it's an entirely different list. Uh. So let's say you have chatbot, right?

• 1:01:59Speaker 00

  It will go from a text chatbot to voice chatbot. It's like the consoles are completely different. Because now with voice chatbot, right, we need to think about like latency. Because I think multiple steps, uh first like have like text, uh like like voice to text, text to text and text question into text answer and then like and then text to voice answer, right? So it have like multiple hubs. And like latency become very important. And there's a question like what does it make to sound natural? So for example, like people think of like um in in in AI and humans, like when humans talk to each other. Like if I say, if I say, you try to interrupt me and say, um chip, that's right. I would like pause and I try to hear you out, right? But sometimes I just even may just say, say some word, not like acknowledge when I, um hum, um hum.

• 1:02:44Speaker 00

  Then I shouldn't stop, I just continue. So the question of like phone's interruption, whether it's like I should, should I stop or not, like it's it's a big in what perceived as like natural conversations. And that's also regulations, right? It was like because like a lot of time people want to build AI chatbot, voice chatbots as sound like humans. Try to like trick users into thinking they're talking to humans. But it's of right, maybe potential regulations saying like, okay, you have to disclose to users when you talk, if the if the bot is talking to is human or um or AI. So so I think just like um, there's a whole space. I think it's not quite as soul as as you think, uh is it it? But it's all it's all not quite like an AI foundation model problem, right?

• 1:03:28Speaker 00

  Because like a human interruption detection is actually a classical machine in problem. Like you you you is is a different uh framing of like you can build classifier for that. Or or like the question of like latency, it actually is a massive engineering challenge, not an AI challenge. Of course, it can be an AI challenge because people are trying to build like voice to voice model. So instead of having like, having to first like transcribe the voice from me into text and then get a model to generate text answer and get another model to like To in front place to speak, you can send the voice to voice directly. So that is something we're working on, but it's like very hard. Yeah, so, so yeah, so like even audio, I think of it is like the easier than video, right? Because video have like both image and voice. It's already like pretty hard.

• 1:04:13Speaker 00

  So I think there's a lot of challenges in that space.

• 1:04:16Lenny Rachitsky?

  That was an awesome list of things. Let me mirror them back real quick. So what you're predicting in the next few years, things that will change in the way we work. And these actually resonate with so many conversations I've had on this podcast. So this is just kind of double the doubling down on where things are heading. One is the blurring of lines between different functions. Instead of just like and design engineering, everyone's gonna be doing a lot of different things now. Two is just more of work being automated with agents and all these AI tools. And just in theory, productivity going up. Third is a shifting from pre-training models to post-training, fine tuning and things like that. Because to your point, models, models maybe are slowing down in how smart they're getting. Although I'll point folks to the chat with the co-founder of Anthropic. He made a really good point here.

• 1:05:01Lenny Rachitsky?

  He's like, we're really bad at understanding what exponentials feel like. We're in the middle of that. And also models are being released more often. So the difference between them, we may not notice because they're just happening more often. Versus GPT three came out like a year, I don't know, before, after GPT two. So maybe true, maybe not. And then the fourth point you made is this idea of multimodal investing in multimodal experiences. I cannot wait for chat GPT voicemail to get better at interruption, like exactly what you're saying. I'm just like talking to it and then someone makes a little sound and it's like, okay. And then you have to, and then it's like, and then it stops talking. So annoying.

• 1:05:36Speaker 00

  I'm shocked that we don't have better voice assistant at home yet. I think I have been testing out a bunch honestly, I keep hoping, oh my God. Zach could be the one and then I don't know how many of them I just like had to get away because they're not that good.

• 1:05:48Lenny Rachitsky?

  I think it's coming. I hear it's coming. Anthropic's working with someone that I don't know if it's launched or not yet.

• 1:05:54Speaker 00

  Yeah, I'm sorry, I want to bring back what you mentioned about like the, your guest like from Anthropic mentioned about the performance improvement. I think this is a big change. I think like this difference between a model based capability. So it's not talking about like the pre-trained model, right? Versus the perceived performance. So, so let's say it's like, I'm actually thought about like, are you familiar with the term a test time compute?

• 1:06:19Lenny Rachitsky?

  Uh, I don't think so.

• 1:06:21Speaker 00

  Yeah, so um, so so the idea is it's like, okay, like, um, you have some a fixed amount of compute, right? So you're gonna spend a lot of compute on pre-training or training the model. Pre-training, and then have spent a lot of uh some compute on like five training. And the ratio like pre-training and post-training compute is like crazy, very different, different, different lab. Um, um, and also like, since then, he has to spend compute uh on like Jerry inference, when I have a train and fighting the model, and now you want to like serve to users, so I might type of questions or prompt, and it's like Jared like do inference, like, and that requires a compute. And I guess I feel about discussion of like, uh, should I spend more compute on like pre-training or fine tuning or inference, right? Because like inference and people found out as like test time compute, so like

• 1:07:06Speaker 00

  spending more compute on inference is like call like test time, like uh compute, uh, like as a strategy of like just allocating more resources, compute resource to Jared uh inference. When I shouldn't bring better performance, and how does that do it? Like, let's say, um, let's say you have a math questions, right? And maybe instead of just Jerry one answer, again, Jared like four different answers and say, okay, whichever is uh the best according to some standard. Uh, or like, okay, have four answers, and then maybe like three of them say forty two, and one of them said like twenty, and say, okay, three of them in the in in agreement, so the answer should be forty two. Right, so like just be watching Jared a bunch of it. Or another thing is like a lot of time like reasoning, uh, thinking, it's just like be watching like Jared more thinking tokens.

• 1:07:49Speaker 00

  I spend more time thinking before showing the final answers. Uh, it's like require more compute, but it's only give more uh more and more better performance. So so yeah, so so I think it's like from the user perspective, right? Like, when the model spend more time exploring different potential answers, thinking longer, it can give you much better final answers. But the base model itself does not change.

• 1:08:16Speaker 00

  Does it make sense?

• 1:08:17Lenny Rachitsky?

  Yes, that does, absolutely.

• 1:08:18Speaker 00

  Yeah.

• 1:08:19Lenny Rachitsky?

  Uh, that is a good corollary to uh, to Ben Mann's point.

• 1:08:23Speaker 00

  Yeah.

• 1:08:23Lenny Rachitsky?

  Chip, we covered a lot of ground. I've gone through everything I was hoping to learn and more. Before we get to our very exciting lightning round, is there anything else that you wanted to share? Anything else you want to leave listeners with?

• 1:08:34Speaker 00

  So I do work at a few companies that does this things of like, they want employees to like come up with ideas. So there's a big debate on like, what is a better way for a strategy of like, should it be top down or like bottom up? Right? Should like executive come up with like, one or two like killer use case and like everyone like allocate resource to that? Or like should you give engineers and PMs and like smart people like come up with ideas? And I think this is a mixture of both. So, so some companies, it was like, okay, we hire a bunch of smart people. Like, let's see like what they come up with. And they, they organize like modern hackathons or like in, internal challenge to get people to, to build product. And one thing that um, I noticed is like, a lot of people just like, don't know what to build.

• 1:09:15Speaker 00

  Uh, and it shocked me like, why, I feel like we are in some kind of like an idea crisis, right? Now we have all this really cool tools to have you like, do everything from scratch. I can have you like, design, it can have you like write code. You can have your website. So in theory, we should see a lot more. But at the same time, people are like somehow stuck, like they don't know what to build. And, and I think it's like, maybe it's a lot of had to do with like, maybe like um, society expectations. Because like, we have gone through uh, we have gone into this phase of like specializations, like people like uh, very highly uh, specialized. And people are supposed to do like, focus on one thing really well, instead of being a big picture. And we don't have a big picture of you, it's hard to come up with like ideas of what you build.

• 1:09:58Speaker 00

  So, so I know what like, uh, when I work with this company, I'll just hackathon, like we do work out like a, how come up with a guideline and like how to come up with ideas. And usually what we think of is like, okay, like one tip is like, go look from the last week, right? Like for a week, just like pay attention to what you're doing, what frustrates you. And when something frustrate you, think about like, is there anything we can do? Is there like, can it be done a different way? So it's not frustrating? And I can talk like people can swap, accept sub notebooks or teams. And if you see they come on frustrations, maybe it's just something you can think about is just to build something around that. So yeah, so I feel like, um, just like notice like how we work, uh, thinking of like ways to like constantly ask questions like how can this be better? And then I just build something to like address the frustrations.

• 1:10:43Speaker 00

  I think it's a good way to just like learn and adopt AI.

• 1:10:46Lenny Rachitsky?

  I think people have felt exactly what you're describing every time they open up one of these vibe coding tools, whether you could just describe anything you want. I'm like, I don't know, what do I want? And, and I love this very tactical piece of advice, just like what frustrates you, just pay attention to where you're frustrated. For example, I just built a very cool little vibe coded app. I was working on a newsletter post inside Google Docs and I, I pasted all these images into the Google Doc from screenshots and stuff. And, and then I forgot, oh yeah, you can't take images out of Google Docs. It's like this Hotel California experience where you can paste stuff into it. Very hard to get images back out. So I, I just went to all the vibe coded tools and just build an app that I can give you a Google Doc URL. And it let me download all the images automatically and it worked amazingly well and I made it really cute and I'll, I'll link to it in the show notes.

• 1:11:32Speaker 00

  Oh, I would love to see that. I, I do, I'm very bullish on like using AI, just create like micro tools, like just something that's like make your life a bit easier.

• 1:11:40Lenny Rachitsky?

  And a hundred, a hundred percent, I feel like that's one of the main ways people are using these tools, just like a little niche problem they have. With that, Chip, we've reached our very exciting lightning round. I've got five questions for you. Are you ready?

• 1:11:54Speaker 00

  Yeah, always. No, no, no, uh, depends on how hard the questions are.

• 1:11:58Lenny Rachitsky?

  They're very consistent across every guest, so I imagine you've heard them before. First question, what are two or three books that you find yourself recommending most to other people?

• 1:12:09Speaker 00

  Oof, I'm really terrified of like book recommendations because I feel like what books a person should read really depends on what they want and whether they're in life and whether they want to get you. Uh, but I guess several books I do think is like she really changed the way I think to see the world. So one thing is the fish gene, that's like to understand, uh, it actually changed, uh, it actually helped me with the question like whether I want to have kids or not. Uh, because it's like, uh, understanding more of like, um, yeah, a lot of our. Functions of where we operate is functions of our genes. And genes, once you do one thing, it's like to procreate. So, so yes, in a way, but I feel like the book also proposed another thing. It's like, so everyone wants to live forever, right?

• 1:12:52Speaker 00

  And maybe it's not like consciously, but subconsciously, we do, we do want that. And, and I said two ways, like, one is like by genes, like genes, one is just like wants to continue forever. But it's also the two ideas. Um, I think there's something going to mean, uh, it's like being able, if you have some ideas out there. And then it's like last for a long time, this is one way to like live on. I know it's like, it's a little bit like, um, abstract, but I thought it's very interesting. The other books I really, really like is like from like, um, the book from Singapore and, um, previous, um. I think he's as a, going as a father of Singapore, I know, like, uh, Lee Lee Kwang Yo. I'm not sure what's the title of it, but like he did, so he was the one who left Singapore from, uh, he's changed, uh, Singapore from a terrible country to a fortunate country within twenty five years.

• 1:13:36Speaker 00

  And I have never seen any country leaders that spent so much effort into like putting down his thought on like how to build a country, uh, like, like that. Um, yeah, and I say talk a lot of like public policy, uh, like how to like create policies that encourage people to do the right things that is good for the nations. And I saw talking about like, uh, foreign affairs, foreign policies, like the duration of like the country with other. So it's a really good book to think about. For me, it's like system thinking, but like it's a different kind of system, which a country, which a lot of us don't get a chance to like ever experiment in our life. So it's good to learn about that.

• 1:14:12Lenny Rachitsky?

  What was the name of that second book?

• 1:14:14Speaker 00

  Uh, it's called like From Third to First World Fashion. I think I have it somewhere here, yeah.

• 1:14:20Lenny Rachitsky?

  There it is, show and tell. That's, that's awesome. I definitely want to read that. That's a really good tip. Uh, I've heard a lot about just the impact he's had and I've seen all these videos on Twitter of just his really wise insights into how to build a thriving society. And clearly it worked.

• 1:14:35Speaker 00

  How does he have time to write this such a thick book? It's like insane.

• 1:14:39Lenny Rachitsky?

  That is, Claude, please summarize. I'm just joking. Uh, by the way, selfish gene, I also absolutely love that book. That is such a good choice. It's such an under the radar kind of book that really changed the way I see the world as well. Uh, so really good pick. Okay, next question. Do you have a favorite recent movie or TV show you really enjoyed?

• 1:14:56Speaker 00

  So I watched a lot of movie and TV shows as a research. Uh, because I, I working on my first novel and I recently, uh, sold it. So I'm interested like what makes it's a drama, it's not a science fiction or uh, anything that like tech people usually read. So it's, it's very like, I know it's a very, um, out of the left, out of left field and like very, um, so it's almost like reading, watching TV to see like what kind of stories become popular, trying to understand the trope and, and stuff like that. So I'm not sure if the audience will like.

• 1:15:28Lenny Rachitsky?

  Well, what's one, what's one that taught you something about writing?

• 1:15:31Speaker 00

  I think you like, uh, Yamsi Palace. It's a Chinese TV show.

• 1:15:37Lenny Rachitsky?

  Cool. Okay. I haven't heard that one on the podcast before. Okay, cool. Next question, do you have a life motto that you often think about, come back to when you're dealing with something hard? Whether it's in work or in life?

• 1:15:51Speaker 00

  This sounds very nihilist. I think society is like in the end, nothing really matters. Uh, usually I think of like in the grand scale, I think like in a billion years, nothing will, like no one will ever be there. I think, okay, someone will argue with me about that. So I go to think like, I saw my theory is like in a billion years, like none of us will ever exist. So like, whatever, like messy things, like crazy things we do or like how bad we do it. I mean, no one would be remember, wouldn't be there should remember it. And I think in a way it's like it's sound scary, but it's very liberating. Because this allows me, okay, let's just try things out, right? Like, why does it matter? And then I saw a story of like recently, um, so I have a some family member who passed away recently. And I was talking to my dad.

• 1:16:35Speaker 00

  Because I couldn't be home for that. I was asking me that like, okay, is there anything I can do to make the person like, oh, something like comfort, so like anything that you can get the person. And my dad was just like, what can he possibly want at this moment? Like, it's maybe real, like at the end of life, like there's nothing that can bring you like, like much you can bring you joy, just no like money, no product, nothing. And in a way it's making me feel like, okay, what really do I really care about at the end of the day? Um, so I guess it's like I think about it, it's just like, okay, maybe I fail it, maybe I don't get that contract, maybe those things like, but it's the end, that's the end of life. Like, I don't think that actually really matters. So in a way, it's just like, it's kind of liberating.

• 1:17:15Lenny Rachitsky?

  Uh, I know you said it might be nihilistic, but this is what Steve Jobs shared too. And in one of his most famous speeches, just, we will all die someday, so don't take things so seriously. And it is freeing, absolutely. It just makes you appreciate every moment, every day you have, just like, yeah, let's just do something hard and scary. Okay, final question. You talked about how you're writing a novel. Most people in tech have never written something creative and fiction. What's just like one thing you learned in the process about how to write better stories, better fiction?

• 1:17:46Speaker 00

  A lot of time when we read, we get tripped up by some small things. So I think like I want to do creative writing because I just want to go better writer. And it tells us like, maybe try my, like, a different audience could have me like, become better, like anticipating what this different type of audience would want to hear and like, what they care about. So it's the way for me to get a, so I think about writing, or even like any kind of like content creations, it's about like predicting the users' reactions, right?

• 1:18:14Lenny Rachitsky?

  The next token. Just kidding.

• 1:18:17Speaker 00

  Yeah, so, so like you do a podcast, it's like, okay, what kind of things that the users could find engaging, right? And, and I find it's like a little bit like, and a lot of companies, like you have like launch a product, you have a narrative coming out. So, okay, what kind, how do we position this product in a ways of like users would want, right? So I feel like I have done technical writing for a while and I felt like I have had some experience like trying to predict what engineers would want to hear, all I care about. But then I don't have an experience like this completely different type of audience. So that's what I want to do, like creative writing, writing a story. And that's why I was doing a lot of research and like watching, I mean, going research, especially enjoy it a lot, like watching a lot of dramas, I just see like what, what it will like.

• 1:19:00Speaker 00

  So, so one thing that I care about is just like, I think I learned is like, well, like emotional journey, this from my editor, right? So like when we write something, like we, we care about like how users would feel, like across the story. Like we want something in the beginning, right? We want something just like. We need to have a hook so that people continue reading. But we also don't want too much of like drama. Because we'll get like too tired, right? Like uh because like you're emotionally exhausted. Like because it's like you're being emotionally manipulated like a lot of time. So it care about emotional emotional journey. Maybe have like some some climax or like some something more chill or like maybe like. And so care about another thing that I didn't realize is like for me. For for technical writing, you entirely focus on the content. Like the argument is very impersonal, right?

• 1:19:45Speaker 00

  Like it it like first of all, like people like ML compilers. Like doesn't matter if they like the person telling them about compiler or not, right? Because it's just like objective, like like. But like for for novel, people care about like character likability. So so like in in the first version is my story and makes the characters like a little bit more like. Uh very uh very logical, very rational and just does everything just like very rationally. And then the feedback I got is I have a very good friend read it and he was he's an amazing person. He's a great person and he was like, Chip, I'll be honest with you, I hate that person. So it doesn't matter the story, it's just like the person is so unlikable to say he doesn't want your community. So in the second version, it makes that person the character more likeable.

• 1:20:27Speaker 00

  Like what how to make that character more likeable is that you put in some vulnerability. Like sometimes they okay, maybe it's a person like have setback because sometimes we can relate to it. So in a lot of ways, like it's very interesting. It's like a lot of it is like um. Yeah, a lot of it is is about like understand the emotional bits. Uh like how the users feel, not just about the story, but also about the characters.

• 1:20:50Lenny Rachitsky?

  That is so interesting. Wow, I learned a lot more there than I thought. That was awesome, really good example. Chip, two final questions. Where can folks find you online if they want to reach out and maybe work with you? Or uh maybe even just share the stuff that you offer if folks want to reach out? And then how can listeners be useful to you?

• 1:21:08Speaker 00

  I'm like I'm a master for media, LinkedIn, Twitter. I don't post a lot, but I keep telling myself that I should do more. Because I kind of like the conversation with uh with um with with readers. Uh so I'm actually about to start a a slight uh a a subs back. Um so I have like a placeholder for subs back right now and I'm thinking of doing it for more system thinking. Because I think it's a very interesting skill. Um, and so like thinking of doing a YouTube channel on book reviews and basically books that help you think better. So I think it's the first book I'm gonna review is probably like this book because it's like my favorite book growing up and I have been like keep on reading it. Uh, so yeah, so how can it be helpful? Like send me books that you like, uh, books

• 1:21:50Speaker 00

  that help you have changed the way you think or change you the way you do anything. So I would appreciate it.

• 1:22:00Lenny Rachitsky?

  Amazing. I'm I'm excited to read that book. Uh, Chip, thank you so much for being here.

• 1:22:05Speaker 00

  Thank you so much, Lenny, for having me.

• 1:22:07Lenny Rachitsky?

  Bye, everyone.

• 1:22:11Lenny Rachitsky?

  Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.