Why PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini)

Tulsee Doshi is head of product for Google’s frontier AI models, including Gemini and other multimodal models, such as Nano Banana, Genie, Veo, and Lyria. We discuss: 1. Why model development is not just engineering, but also like Snape’s potions class 2. Google’s full-stack frontier AI strategy 3. What PMs do inside a frontier AI lab 4. Tulsee’s 60-20-20 framework for balancing what works now and what’s almost working 5. How to write your first eval 6. The rising value of great PMs — Brought to you by: WorkOS https://workos.com/lenny—Make your app enterprise-ready, with SSO, SCIM, RBAC, and more DX https://getdx.com/?utm_source=lenny&utm_medium=sponsorship&utm_campaign=q42026—Engineering intelligence for the AI era — Episode transcript: https://www.lennysnewsletter.com/p/what-it-actually-takes-to-build-frontier — Archive of all Lenny's Podcast transcripts: https://www.dropbox.com/scl/fo/yxi4s2w998p1gvtpu4193/AMdNPR8AOw0lMklwtnC0TrQ?rlkey=j06x0nipoti519e0xgm23zsn9&st=ahz0fj11&dl=0 https://www.dropbox.com/scl/fo/yxi4s2w998p1gvtpu4193/AMdNPR8AOw0lMklwtnC0TrQ?rlkey=j06x0nipoti519e0xgm23zsn9&st=ahz0fj11&dl=0 — Highlights: https://lennyspodcast.com/mostreplayedmoments — Where to find Tulsee Doshi: • X: https://x.com/tulseedoshi • LinkedIn: linkedin.com/in/tulsee-doshi http://linkedin.com/in/tulsee-doshi — Where to find Lenny: • Newsletter: https://www.lennysnewsletter.com • X: https://twitter.com/lennysan • LinkedIn: https://www.linkedin.com/in/lennyrachitsky/ — In this episode, we cover: (00:00) Teaser (01:08) What people misunderstand about how models are built (06:28) Google’s strategy in the AI race (09:57) AI safety, monitoring, and strengthening cyberdefenses (14:10) Balancing fast launches with great user experiences (17:02) Why the model itself has a UX (19:21) Why product managers are becoming more valuable (21:24) The skills that matter for research PMs (23:04) How to build your first eval (24:37) Getting better at judging a model’s vibes (26:15) Improving skin tone representation with better evals (28:25) Evaluating the full experience, not just the model (30:38) From crowdsourced flash teams to agent swarms (33:13) The 60-20-20 rule for building ahead of model capabilities (35:46) Keeping your product adaptable as models improve (36:42) What an AI harness actually is (37:30) Upskilling PMs through shared skills and workshops (41:18) Building the habit of asking, “Can AI do this for me?” (42:38) Google’s opportunity to build a personal AI assistant (44:10) Burnout, meaningful work, and the pressure to do more (47:44) Avoiding AI slop (49:30) How AI is changing the PM role (51:02) Lightning round and final thoughts — References: https://www.lennysnewsletter.com/p/what-it-actually-takes-to-build-frontier — Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected] mailto:[email protected]. — Lenny may be an investor in the companies discussed. To hear more, visit www.lennysnewsletter.com https://www.lennysnewsletter.com?utm_medium=podcast&utm_campaign=show-notes-no-free-preview-language

• 0:00Speaker 00

  The model isn't predictable. It's kind of like Snape's potions class. You can put all the ingredients in, but you also have to expect that there will be a little bit of magic. And that magic is going to be surprising.

• 0:10Speaker 01

  As you're hiring PMs these days, is there anything you look for more and more?

• 0:15Speaker 00

  You're looking for PMs who understand the vibes of our models or other models. I want PMs who have an opinion, who are playing with these models every day. You ideally want to stress test the model on things that you are excited about.

• 0:28Speaker 01

  A lot of people feel that Google lags behind the top two labs right now.

• 0:32Speaker 00

  I actually think we do want to win the frontier conversation. I don't think we should say that Google is not participating in the frontier.

• 0:39Speaker 01

  You have this framework around how far ahead to build. You call it a 60-20-20 rule.

• 0:44Speaker 00

  Traditional startup wisdom. You're just like, build an MVP that works. But I think you want to actually build features that aren't necessarily always working today.

• 0:52Speaker 01

  It feels like every AI assistant launching these days is taking all this juicy Google data, my Gmail, Google Calendar, my docs. I'm curious just why Google hasn't launched something awesome yet.

• 1:03Speaker 00

  The team is working on some amazing things, but I think you are right that there is more work we need to do here on this front.

• 1:12Speaker 01

  Tulsi, thank you so much for being here and welcome to the podcast.

• 1:15Speaker 00

  Thanks. I'm excited to be here.

• 1:17Speaker 01

  You are head of product for Google's frontier models, which includes Gemini, which people know. But there's also a bunch of other models that people may not be familiar with. Share the other models that are under your umbrella.

• 1:30Speaker 00

  Yeah, so we have a set of models we kind of call our gen media models. These are our models that focus on video editing and generation, image editing and generation. If you've used Nano Banana, Lyria, our music model. We also have Gemma, our suite of open models. And then one set of models that actually falls under our Gemini umbrella, but I feel like we don't talk about as much is our audio models. So dialogue, audio understanding, text to speech, translation, all a part of the portfolio of how we build frontier models.

• 2:03Speaker 01

  Okay, so I want to spend a little time on models. Everyone sees models launch. There's like a model launch every week, it feels like. You have a really unique lens in being able to see these models from research to pre-training, post-training launch. What's something that you think people maybe misunderstand about how models are created? Or something that may surprise people about how models are made?

• 2:26Speaker 00

  Yeah, it's a good question. I mean, maybe what I'll say, especially coming from a product management lens, is one thing that I think, you know, when you typically talk about product management for anything, including a model, you talk about what are the goals you have for a model, right? What are the capabilities you want to set? What are the outcomes you want to drive? What goals do you want to hit on a set of evals? And often when I talk to customers or even internal users, you know, they'll ask me, okay, what are the goals that you're setting? And they'll sort of expect that we can really make sure that the model will deliver on those goals in a very traditional product sense. And I think for me, what has often been surprising,

• 3:07Speaker 00

  but also exciting about model development and being a PM for the model, is you should do that and you should have an intentional view. But the model isn't predictable. And so it's kind of like Snape's potions class, right? So you can put all the ingredients in and you should be thoughtful about what you expect. But you also have to expect that there will be a little bit of magic. And that that magic is going to be surprising and that magic is going to be unique. And so your job as a PM is not just to try to shape what goes into the model, but it is also to look at what comes out the other side and really identify where does the model shine? Where does it actually look unique in ways that surprise you?

• 3:51Speaker 00

  And then be able to take that and shape that into something that is amazing for users.

• 3:55Speaker 01

  It's so interesting that it's just like this code that we have written, that we feed data and we need to, like, we don't exactly know what's going to happen and how it works. I recently had an interpretability research person on the podcast from Anthropic, and we talked a lot about this, this idea that models are grown, not built. And just to kind of reflect back what you're saying, there's these goals that researchers have. For this next model, we're going to get better at writing. We're going to get, try to be funnier. We're going to try to become better engineers. And you're saying it's unpredictable exactly what ends up at the end. They put in these ideas, they change the data that it's trained on. Is there anything more there about just this kind of like unexpected, I don't know, process of training?

• 4:36Speaker 00

  You know, of course, there's brilliant research that goes into these models. And so many aspects of it are predictable in the sense that we are consistently monitoring these runs. We have researchers who are tracking them, who are identifying when to intervene, who are seeing different metrics climb up as the run progresses. And so we are, you know, making it as much of a science as possible. But I think for me, what is really amazing as a PM is you also need to really understand the vibes of the model on the other side of it, too. So yes, you can understand that a model has improved on a certain capability, but where does that capability really shine through? How does that showcase then downstream in a product? How do you feel communicating with the model in terms of its behavior and persona? And there's a number of these aspects that we definitely build into training.

• 5:21Speaker 00

  But also as we understand the model and as we evaluate it and as we test it, we always learn more. We always learn something new, I think, about what the model can do.

• 5:29Speaker 01

  This episode is brought to you by this season's presenting sponsor, WorkOS. What do OpenAI, Anthropic, Cursor, Replit, Sierra, Klei, and hundreds of other winning companies have in common? They're all powered by WorkOS. If you're building for enterprises, you've felt the pain of integrating single sign-on, SCAM, RBAC, and audit logs. WorkOS turns those deal blockers into drop-in APIs built for B2B SaaS. And now there's a new kind of user that needs access to your app, agents. AI agents need to register, authenticate, and get permissions, just like a person. WorkOS handles it all. Agent auth to give agents scope for vocal identities. Agent registration with auth MD and airlock

• 6:13Speaker 01

  to judge every agent action and allow, deny, or escalate to a human. I've noticed that every startup that I'm an investor in that expands a market ends up using WorkOS because they're the best. Go to workos.com to make your app enterprise-ready today. I want to talk a little bit about Google's place in the AI race. It feels like a lot of people feel that Google lags behind the top two labs right now, Anthropic and OpenAI. And Google has these moments of being the best and then it kind of feels like it slows down a bit. I'm curious what you can share about just Google's strategy in the AI race and what maybe people don't understand on the inside of how you all think about this stuff.

• 6:54Speaker 00

  First of all, hopefully by the time this conversation goes out, you will be using Gemini for our gun, which is exciting. And so I really do hope we're having a conversation about a frontier AI model that folks are excited to use. And I think Gemini 4 speaks a little bit to our strategy, right? Because it really is our push to continue to build a frontier, a frontier lab, a frontier set of models, a frontier set of capabilities on our path. I think if you look at Google's strategy, it really is full stack, right? So one of the things that I think is powerful about Google is all the way from hardware to end products that serve billions of users. I think the stat is like, we have 14 products that are above a billion users.

• 7:39Speaker 00

  I should check that number, but I think that's the stat. And so if you think about that, we actually go all the way from, you know, the hardware to then the modeling, to then this kind of end user application. And our goal ultimately is to not just build amazing frontier experiences, which is definitely a part of the goal. But it is also to build models across this Pareto curve of cost to quality that can serve different users at different points in their lifestyle. Whether that be a search user who is coming for information, right? Whether it is an engineer who is trying to build the next amazing thing, whether it is for a cyber defender who is trying to protect a critical piece of infrastructure. I think our goal is to make sure that we have models

• 8:24Speaker 00

  that serve that range of use cases and that we're actually building with feedback from these users.

• 8:30Speaker 01

  So what I'm hearing is there's so many vectors where Google can win. There's the most efficient kind of Pareto frontier per token cost. There's the smartest model. There's also just making your products better. For example, AI mode, AI overviews. And so it's not that you need to be the best at the most advanced AI model that people can use, like Chatbot, basically. You can win in all these other areas, but it sounds like you're also trying to win and have the most state-of-the-art model.

• 8:58Speaker 00

  I actually think we do want to win the frontier conversation, right? I think, I don't think we should say that Google is not.

• 9:07Speaker 00

  Participating in the frontier. And I think we are building better and better models. And I think Gemini for Argon definitely is at the frontier in

• 9:19Speaker 00

  in the place that we are at. But I think one thing that is amazing is being able to build a breadth of experiences and build a diversity of experiences for users. And I think for Google, our goal ultimately is to build amazing experiences that people that actually make people's lives better. Right? And that is, I think, ultimately the goal, whatever part of the frontier you're building for.

• 9:42Speaker 01

  Google products definitely make my life better. It's just like amazing how much of your life is within a Google ecosystem. Like, not even the AI stuff, but just Google Calendar, Gmail, all these things. You're just like Google Docs. So it's happening.

• 9:55Speaker 00

  There you go.

• 9:56Speaker 01

  There we go. I want to talk about one other broad AI question, then I'm going to dig into more product-specific stuff. A big topic of conversation these days is AI safety. And this kind of fear that AI is going to harm humans, it's going to cause damage as it gets smarter and smarter. You have a really interesting lens on this because you led product for the responsible AI team at Google for six years. This is what you did right before this role. And so you've been thinking about this for a long time, a lot longer than most people. I'm curious just to get your take on this debate. Just, I don't know, are you optimistic? We'll figure this out. What do you think people maybe don't understand about this debate around AI safety?

• 10:36Speaker 00

  Our safety teams here are incredible. And it's been incredible to see the intentionality that they've been putting in, all the way from, you know, initial trainings of the model to how we deploy them. And I think safety is embedded in that process, right? And we need to be. I think the part that is important, actually, as we talk about how do you deploy models safely, is actually thinking through what are the right sort of mitigations and structures around these models, right? So, for example, one of the things that's really important is thinking about monitoring. Right? And so for us, actually deploying monitors that can monitor the model's chain of thought and actually identify whether the model is misaligned with its, you know, directed actions or whether it is aligned, and then being able to intervene appropriately.

• 11:22Speaker 00

  That's actually the kind of mitigation that, A, gives us more confidence in deploying these models, but also allows us to better understand where we might see misalignment and better actually improve the models over time. And so I think things like monitoring give me more confidence in how we're putting our models out. I think the other thing that we don't talk about enough when we talk about this debate is also the role that AI can play in hardening our systems. Right? And so one of the things that we're doing with Gemini four is like actively partnering with a number of cyber defenders, including actually just even partnering with Wiz, that is now a Google company, to really think through how can we actually identify and patch vulnerabilities in systems across the globe. Right?

• 12:06Speaker 00

  We actually have this interesting case study. Wiz has this Scan for Good program where they're actually like actively working with systems across the world to identify, you know, potential issues. And they worked with a hospital system to identify a vulnerability and then patch it. And it's that kind of thing that I think if we can actually leverage these tools to do more of, we can actually make sure that our infrastructure and our systems are robust. Right? Which is also, I think, really critical as this whole space grows.

• 12:34Speaker 01

  There's actually a conversation on All In this last week about this whole issue that now that AI is getting really good at cyber, people are going to be investing a lot more in cyber defense and AI for protecting. And so there's going to be a lot of money being spent on protecting their systems from this everly, ever-smarter AI. So it's really cool that you guys are investing there. So I guess just coming back to the question of, do you feel like we'll figure out this concern and this issue? We feel, a lot of people feel that AI is going to take over the world, cause a lot of damage. Do you feel like this is a solvable problem? Are you optimistic or are you, is it something you think a lot about and are kind of worried about?

• 13:12Speaker 00

  Maybe the answer to your question is both, in the sense that I think we need to be consistently thinking about it. Right? I think the way to optimism here, for lack of a better way of putting it, like the path to optimism is through thinking about it and being like very careful and very thoughtful and very intentional on the path to get there. And so that is like my source of optimism comes more from, do I feel like we have people who are being intentional and thoughtful about what the safety story looks like? And I think that is the critical threat, is you need a number of people to be worried and to be having conversations and to be actually interrogating how we're building our models. And that's the step-by-step process we have to take.

• 13:50Speaker 01

  Awesome. And what's interesting is I don't think I've read a story of Google's models hacking into anyone, which I think shows that this is something you all have been thinking about for a long time. I don't know, like for better or worse, but it feels like it's working, whatever you all are doing.

• 14:05Speaker 00

  We like stories about fixing vulnerabilities. That's a good place for us to be.

• 14:09Speaker 01

  Okay. So one of the interesting things that's happening kind of from a product perspective with AI is products are getting very complicated. For example, Google gets a lot of flak for it being hard to find all the magical things you all do. Like, a lot of people don't know AI Studio even exists. A lot of people can't find Vio or Genie and all these things. Similarly, Codex, ChatGPT recently launched Dots, and now you go to the interface of Codex and it's like tabs and dropdowns and dots, and it's just like out of control. And Callot Code is the same thing, it's just getting very complicated. They cowork and then they drop cowork. And interestingly, I'm in a, in this WhatsApp group with a bunch of PMs, and there's a PM from one of the labs.

• 14:51Speaker 00

  I don't want to be in this WhatsApp group. Sorry.

• 14:53Speaker 01

  Okay, let's, let's, let's talk. And he showed that there's this kind of urgency to get frontier power of AI out to users as fast as possible, partly to stay competitive, partly because they don't know exactly what it's used for yet. So let's get it to people, figure out what they're using it for, and then make the UX better. So there's this kind of intentional, let's just get it out there, figure it out later how to make it a great UX. Thoughts on this kind of balance of making the product really easy to use versus just getting things out as quickly as possible? And is that something you deal with?

• 15:26Speaker 00

  Two thoughts, maybe, maybe three. The first one I'll say is I do think we do need to be playing with new paradigms and new UX approaches, if you will, to how people will engage with models. Right? So you mentioned, for example, Genie or Vio, which is, which we're now building Omni.

• 15:44Speaker 00

  Like, if you look at, for example, Flow. Flow is a product that we've been designing, thinking about video editing and generation, music editing and generation. And so I think we are trying to play with, like Notebook LM is a good example of a different form factor. I think our labs teams are actively thinking about like, how do we try different form factors and different UX for different individuals to engage with these models? And so I do think we need to keep innovating and playing around with what UX looks like. I think at the same time, your point is also right, which is like, I think depending on who your user is, you want a different form factor too. Right? So like, I think, for example, with AI mode, we're talking about billions of users coming to search, asking their questions.

• 16:27Speaker 00

  I think you do need to have a certain UX there that is accessible to every user. Right? At the same time, on the other end of the spectrum, when you have something like anti-gravity, you are trying to build for a power user. And you are trying to build for users who know how to use slash commands and can like actively engage across like both the terminal and the IDE. And like, you want to be able to like support kind of a wide range of what a user can do. And so I think you, depending on who your user is, you have a certain level of flexibility with how you think about the UX. Right?

• 16:58Speaker 01

  That makes sense. Basically, there's different services for different levels of how radical you can change things.

• 17:03Speaker 00

  Yeah. And I think the other part too is like, I think it's also important to think about the model is having a UX also. Right? So one of the things I think about a lot is like a PM for the model is like, what is the UX of actually using the model? Right? So how transparent is the model? How well does it communicate with you? What questions does it come back and ask you? Right? These are all actually, this is all UX in its own way. And so I think it's also really interesting to think about like, how do we bring conversations about UX into the actual model development process itself?

• 17:31Speaker 01

  That is really interesting. Basically, the personality of the model is the UX of the model. How do you, as a PM, inform that work with the researchers? Is it like, here's feedback on the way it's talking and it should be a little more verbose here, and here's ways it could format things differently?

• 17:46Speaker 00

  I think the role of the PM here is actually really awesome because you are actually getting to shape what is the direction of where we want to go with the model's behavior. Right? And so we have a team that works on model behavior here. And a lot of what we've been thinking about together is, A, looking at customer feedback and user feedback for how they engage, but also thinking through what does good look like for Gemini specifically? Right? And how do we develop kind of a spec, if you will, for what does that look like? And then being able to translate that to rubrics, right? That we can then evaluate the model on, to be able to translate that into data that we can actually leverage for the model, to be able to translate that into our, you know, reward modeling. And so it's been actually like a really nice collaboration between PM and the research team.

• 18:30Speaker 01

  Something I hear a lot is that even if we stopped development on the model, there's so much it can do that we are not yet doing. There's this overhang. Is that something you think a lot about? Is that, it feels like that's a big part of this job is exposing the powers of the AI model to users.

• 18:46Speaker 00

  Yeah. And figuring out what is new. Right? And where is the next, like, where is the next set of exciting? Where is the next frontier? What are the capabilities that we're trying to invest in and build? I think is a fun activity here also. And I think a lot of this actually is really working in hand-in-hand with our internal customers. If you think about like search or the Gemini app or, you know, YouTube or Labs, but also working with external customers to really understand where are they going? What are their needs? What would meaningfully change their experience? And how do we actually build that flywheel of feedback too?

• 19:21Speaker 01

  Along these lines, one of the other really interesting trends I've been watching on Twitter in particular, is people realizing the value of product managers again. We went through this phase of just like, we don't need PMs anymore. Exactly. It's just like, okay, we can build anything. We don't need PMs. Let's just build it all. The PMs are so back. And the reason, just to kind of add a little context there is now that it's so easy to build, people are just building all this nonsense and things that don't work, people, things people don't want to use. And people are realizing there's a very specific skill to figure out what is worth building, what people, what problems people have that are worth solving, what a great looks like, what good looks like once you're done, aligning people behind one cohesive vision. Is that something you're noticing?

• 20:04Speaker 01

  Thoughts on that kind of PM ascendance that has started to happen again?

• 20:08Speaker 00

  I like the phrase PM ascendance. I mean, yeah, I think, I mean, I think you actually said it very well. Right? Which is, I think, look, I'm obviously biased because I am a PM. So I would, I would like to believe that, you know, PM ascendance is here and we're back. But I actually think, you know, exactly what you said, which is when you can build anything and you can build anything so easily.

• 20:32Speaker 00

  Building something great actually is now a unique value add, right? And taste is now really important. Someone who can come in and actually really pull in what is the signal from the noise? Right? And what is that insight that will actually unlock a great experience? And that is truly what a good PM does, right? Is bring that insight and help unlock. I think the other part that you noted that I particularly feel in a place like Google, where we are building a number of products is, again, when this is so easy, someone who can bring people together and drive consensus and turn chaos into order, I think becomes actually really important because in the absence of that

• 21:15Speaker 00

  and the ability to build as quickly as we now can, speed can basically introduce chaos. If you're not actually able to kind of bring clarity to that and bring structure to that.

• 21:25Speaker 01

  So on that, as you're hiring PMs these days, is there anything you look for in more and more? Kind of a skill that might be trending up in the PMs that you hire that you think is really important to be really successful? Or just broadly, skills that you find are really important to be successful in this role of being a research PM?

• 21:44Speaker 00

  Yeah, I mean, I think more and more so we are looking for PMs who,

• 21:51Speaker 00

  for lack of a better way of putting it, like, understand the vibes of our models or other models. Like, I want PMs who have an opinion, who are playing with these models every day, who have their own use cases for how they judge whether a model is really good, right? Who play with and stress test these capabilities themselves, who are capable of, you know, diving in and looking at data and getting like scrappy and rolling up their sleeves. And so I think it's both PMs who can have, who are trying to build that intuition as well as who are technical enough in the sense of being willing to kind of like, like I said, roll up sleeves, go in, really understand the data, build an eval, right? Actually sit down and pull those pieces together.

• 22:35Speaker 00

  I think that's really important. The other thing I'll say is I think PMs who are agile and adaptive are like also really important. I think the space is changing every two weeks. The roadmap you wrote two months ago, you're going to be looking at it again and rewriting it, right? And so I think PMs who are constantly keeping an eye out on the environment, on the market landscape, and actually consistently re-interrogating, like, what are we doing and why are we doing it? As opposed to assuming that there was a fixed kind of plan of action.

• 23:05Speaker 01

  This is awesome. Okay, so vibes, evals, adaptive. I want to poke at these a little bit. So with evals, say someone doesn't really know how to write an eval, how to approach writing an eval, their first eval. What are some tips to start getting better at the skill of writing evals?

• 23:20Speaker 00

  Yeah, and to be honest, we're still figuring that out internally with our PMs as well. And I would say like, we see some PMs who are stronger at this than others. And so we're kind of working to upskill our whole team here. I think for me, a big thing is start with even just five prompts, right? Like, start, you don't have to start large. I think we often think about like, oh, I need to build some sort of massive, large-scale

• 23:43Speaker 00

  evaluation or benchmark. I think often, like, even just a mini evaluation where you get to like 20 prompts that can really stress test a set of capabilities where you can see signal between, you know, your model and another model, or model A and model B. I think that already is so powerful and that's an unlock because it doesn't require you to think as critically about all of the pieces, right? Like, start simple. Start with even just yourself, like you as the raider, you looking in a prompt, you trying to decide if the results and the response is good. Especially when you're talking about these like long, multi-step, multi-turn trajectories. Just even being able to identify how you're thinking about good is really hard, actually.

• 24:26Speaker 00

  And so even getting that set of good examples will already give you a sense of, oh, this is what I'm trying to, this is what I'm trying to get to. It'll give you self-clarity and then you figure out how to scale that up.

• 24:37Speaker 01

  Awesome. Okay, that is really good advice. There's this debate, I don't know, six months ago on Twitter around evals versus vibes. And it's interesting, your advice is both actually a really important structured evals, rubrics, and also getting good at getting the vibes. So on that vibes piece, there's something I struggle with, to be honest. When I get a new model, I never know exactly what to do with it to understand what it's gotten better at. So I'm just like, hello, let's try some things I've done in the past. This is cool. This is great. Maybe better. What are some tips you have for folks that want to get better at this very specific skill of understanding where a model has gone better than the vibes and how they've shifted maybe from a previous model?

• 25:14Speaker 00

  There isn't honestly a one right answer to this because you ideally want to stress test the model on things that you are excited about, right? That's where you're going to also see magic in the model, I think. And so I actually see that different people have very, very different notions of this, right? So you'll see, for example, on Twitter, a lot of people trying the kind of like 3D animations and more visual type aesthetics because that's one way to kind of visually gauge a sense of how models are operating. You'll see others throw models at writing, whether that be creative writing or writing essays or synthesizing information. You'll see others throw it at just like really hard, like financial analysis. And so I think part of it is like finding the right task

• 25:59Speaker 00

  where you feel like today, certain models are already showing signs of life and showing light, but they're not perfect under your definition of perfect. And that gives you some malleability to see different models handle the problem differently. And that already gives you, I think, some insight into how the models perform.

• 26:16Speaker 01

  This is great advice. So what I'm hearing is stick to things you actually care about and watch the models evolve on how good they are at this thing. Could be writing a PRD for you, could be writing strategy doc, could be generating graphics. Okay, I have a couple of eval stories that one of your colleagues shared about, that are fun facts about you.

• 26:34Speaker 00

  Oh, oh, boy.

• 26:35Speaker 01

  So this is Eli Collins, VP Product. He's colleague of yours at Google DeepMind. Okay, so one is you worked on something called the Fitzpatrick Scale and the Monk Skin Tone. And this was a big shift in how people, in how models worked around. And evals were a core part of this. Share this story, what this work was.

• 26:54Speaker 00

  Yeah, so I, like you noted earlier, I spent six years in responsible AI. And a big part of that time, we were looking at skin tone and skin tone representation in our models, in our products. And one of the things we realized is that early on, there was a scale that individuals were using, the Fitzpatrick Scale, which was six skin tones. And that was how we were evaluating the effectiveness of product. And realistically, six skin tones cannot capture the diversity of skin tones in the world. Not only that, it was actually built for an entirely different purpose. It was built for like healthcare purposes. And so it actually skewed extremely light skin toned, given the problems that it was trying to solve.

• 27:38Speaker 00

  And so it actually not only was only six skin tones, it was also not even the six like correct skin tones if you're trying to represent like a global population. And so we worked with a professor at Harvard, Professor Monk. He had been from like a more anthropology sociology perspective, looking at characterizing a set of skin tones. And so he had a scale of 10 skin tones that he had built through a number of user interviews and research that he had been doing over years. And we pulled that set of skin tones into our evaluations and actually redid a number of the ways that we evaluated our products and our models to reflect that 10 skin tone scale. And actually use that to improve things like our efforts in the pixel camera,

• 28:20Speaker 00

  like some of our image generation, to be more intentional about how skin tones are represented.

• 28:26Speaker 01

  That is awesome. That is some serious impact you had here. What else you think is maybe useful to be, for people to hear as they hear this and they think, maybe have to rethink their evals that may be out of date?

• 28:37Speaker 00

  Yeah, I mean, I think one thing that's kind of interesting about evals in general is, especially now, is you have to kind of think about them holistically, right? So one of the things that we do, for example, is you train, you have judges that evaluate the models and how well they perform. The quality of the judge also matters, right? So your model needs to be performing well, but also how your judge rates the model also needs to be performing well for you to have an accurate sense, right? Whether you evaluate on your production stack with all of your mitigations and filters might affect your model performance, right? How your model is served might impact your model performance. And so all of these pieces need to be thought through as you're trying to evaluate what an end user will actually experience.

• 29:21Speaker 00

  And like, this is something that I talk about a lot with our audio team, actually, because audio is complex. You're talking not just about The actual content that the model is saying, but also the naturalness of the voice and the tenor of the voice and, you know, its inflections and these are, these are things that serving actually can really affect, even if your model is good. And so having a clean setup that you're actually evaluating end to end becomes super, super important.

• 29:49Speaker 01

  This season's supporting sponsor is DX. Now that everybody at your company is finally using AI, the question that engineering leaders face is no longer, are we using AI, but is it paying off? With DX, you can measure and benchmark the impact of AI adoption across your full product lifecycle. DX shows you where AI is helping developers and where it's creating new bottlenecks. You can also use it to evaluate AI vendors, monitor costs and licenses, and see exactly what's limiting your agents. That's why hundreds of enterprises, including Snowflake, Sony and BNY, use the DX platform to measure AI's impact on developer productivity.

• 30:30Speaker 01

  Visit getdx.com/lenny to get a demo of DX. That's getdx.com/lenny. This is like a Tulsi fun fact corner. There's another interesting story I heard, which is around swarms. So right now, everyone's talking about agent swarms, hugging face, all these things where they all kind of collude to do bad things. Apparently 12 years ago at Stanford, you did a bunch of research on this way before anyone was thinking about this potential issue, shared what you did there in that story.

• 31:01Speaker 00

  Yeah, I mean, it's actually a very different, different kind of framing and thinking through, which is, so when I was at Stanford, I did some research with Professor Michael Bernstein, who's incredible, around this idea of like, he was calling them at the time, flash teams. And the idea was, how do you better bring together, you know, a set of like crowdsourced workers and actually efficiently pull together a set of skillsets and individuals to actually complete a task? And the idea here was, you know, different individuals have different strengths. Can you actually be really intentional about how you pull that together to make this whole effort of crowdsourcing at scale more effective and efficient? And it's amazing to think about that research now. At the time, we were doing that research in the context of people.

• 31:44Speaker 00

  And we were working with, you know, MTurk and, and individuals to pull them together into how to create an effective ecosystem and how to create an effective flash team, right? The idea being like, you can spin up a team and spin it down and make it extremely efficient for both the individuals in the team and yourself. And now we're kind of able to replicate that world with agents. And so it's been interesting to kind of map that previous world to this current one in terms of like, how do we apply some of the same principles? If you will, but to like, how you actually set up an effective system of, you know, subagents.

• 32:17Speaker 01

  What's really interesting is, I don't know how much you've read about this whole hugging face incident, but the way that agents figured out on their own what each one's role is going to be. And they decided, you're the leader, I'm going to go sacrifice myself. And it's so interesting how well they are able to work together and create these flash teams on their own without any of this being trained into them. They're just, let's figure this out.

• 32:39Speaker 00

  I think the whole notion of a flash team and this idea that you can actually like mobilize individuals into something that can be highly effective, even if it's only for a moment in time, and then actually like take that out is actually like a a really kind of exciting principle that we were already thinking about, but now exists even more importantly in the world we're in right now.

• 33:00Speaker 01

  Yeah, all right. Did you know Mechanical Turk, I think, is shutting down? Like, I don't think it's necessary anymore, isn't that, isn't that crazy?

• 33:06Speaker 00

  It's just so interesting from like 12 years ago to today, yeah.

• 33:11Speaker 01

  Have you heard lots of changed? Okay, so going in somewhat of a different direction, you have this framework that I think is going to be really helpful to people around how far ahead to build in terms of how far AI is and how far your product should assume AI is going to be. You call it a 60/20/20 rule, explain that.

• 33:29Speaker 00

  I think the numbers are probably fungible here, but I think the thesis is the same, which is, I think as you're building today and as you're a product developer, building, using these models, I think the important thing is to keep in mind that these models are consistently changing. And so the capabilities that you have today will not be the capabilities you have tomorrow or the next day. And so you want to actually build features that aren't necessarily always working today, right? So typically, when you talk about like traditional startup wisdom, I think you're just like, build an MVP that works, right? But I think what you want to do is actually build such that like 60% of what you've built works now and your prototype is at least functional, right?

• 34:14Speaker 00

  Then there's a set of maybe the 20% where you're like, it's not perfect, but it like works well enough. And I can see how the models might get here in the very near future. And then here's where I want the models to be. And here's, I see glimmers of light. And I feel like if I could crack this, like this would be an amazing product experience. And I think if you build in that way, you are building a product that works today, but you're setting yourself up to build a magical product experience as the models become better and also to grow as the models grow too.

• 34:43Speaker 01

  I've heard a lot of advice along these lines of how far ahead to build. And the most common is just like, build like three months ahead of where the model is going to be. And this is, feels like a lot more practical where you can't just bet on the future. You also need to ship some things today that are great and working.

• 34:58Speaker 00

  It's trying to find that balance, right? And I think also I find personally hard to know where the models, as someone who works on the model, to tell you exactly where the model is going to be three months from now. And so I think using more of your own proxies of like, where is my product? Where do I see glimmers of light? Where do I see it kind of bridging that gap? I think is a more concrete way to hopefully like ask that same question.

• 35:21Speaker 01

  This reminds me of just another way of thinking about how to kind of create a portfolio of product bets on a product team. Like the way it used to be is like, have some ambitious bolt bets, but also have a lot of stuff that's very low risk, likely to move the metric. And you want to mix them together. And it's like, this is the new version of that where it's.

• 35:41Speaker 00

  Have a portfolio of how you're thinking about that.

• 35:43Speaker 01

  I've just like, this will work in six months, I promise you.

• 35:47Speaker 00

  The other thing that's interesting too is like, I think, thinking about how you make sure that your product is agile to the model getting better. I think one thing we've also seen with customers sometimes is like, they've built a very rigid system around, for example, where the model was. And now the model has gotten so much better, right? So how do you go from where the model was to like being adaptable to kind of continue to iterate it with, you know?

• 36:11Speaker 01

  Yeah, the common kind of advice I hear is that just the model's going to eat your harness for breakfast, that all these things you built, all these optimizations are just going to, they'll go away because the model gets smarter and you kind of have to take all these things away. There's some advice that I think Boris Cherny was sharing of just like, get rid of all your skills once in a while and just see what happens. Cause often it's now as good as all these things you've built into it.

• 36:32Speaker 00

  Yeah, or just like, like consistently retest, right? Because you're, you might be even hampering what you're getting out of the model, out of a new model, because you're trying to force fit it into your existing structure.

• 36:43Speaker 01

  Let me ask you a completely unrelated question. Just because I think this might be helpful for folks to understand this. There's this term harness that people often hear, this idea of there's a model, there's a harness that's just on top of the model. What's the simplest way to understand what a harness is?

• 36:56Speaker 00

  Yeah, I mean, I think of a harness as like the wrapper around the model essentially, right? So think of it as like, what are the system instructions? The tools, the skills, the context that the model has access to? And how can it leverage those things, right? And so what you want to basically do is give the model an environment in which to operate. And that's really what your harness builds out.

• 37:17Speaker 01

  So essentially it's a system prompt plus tools that have access to, to do things. And then whatever other context, other data maybe has access to.

• 37:26Speaker 00

  That it has access to and skills, right? Like all of these things kind of come together to build out your harness.

• 37:31Speaker 01

  Awesome, okay, that is very helpful. So one other thing I wanted to get your thoughts on. There's a lot of change, obviously. A lot of people are trying to figure out how to thrive in this new world. How are you, how are you upskilling and helping your team get better? And kind of understand what they need to be good at in this new world? What are just ways you're helping your team develop and kind of survive and thrive?

• 37:57Speaker 00

  Survive and thrive, yeah. I mean, honestly, I feel like my team is helping my team in a lot of this, actually. I think when you look at the PMs on my team, you see some who have like really leaned in and built a lot of skill in like leveraging our internal tools. And they're, you know, individuals who play around with the models a lot. And they actually are, I think, amazing assets to the rest of my team. Because they build skills that others can use. And one thing we've been starting to do more of is like, how do you share a skill that you've built that is giving you some value in your day to day job with others, right? So that we can actually build out an entire repository that helps the PM team. PMs who have gotten stronger and more confident in eyeballing data, right? And really looking at it and understanding longer traces and trajectories.

• 38:42Speaker 00

  How can we run workshops with other PMs so that we kind of all collectively build more of that confidence? And so for me, a lot of it is about confidence building. How do we build everyone's confidence in the tools that we have and their ability to leverage them effectively? And I think that's something that we're trying to build more and more across the team.

• 39:00Speaker 01

  So going one level deeper there, what is it, what is it you do? So you said workshops, there's like PMs that are good at a thing, teach a workshop to the rest of the PMs. Is there anything else, just like very practically that you do to help people get better at these things?

• 39:12Speaker 00

  The biggest thing I would say actually that is needed to get better at these things, and I say this. Knowing that I am starting to do this, but we are not quite there yet, is actually giving people the time and space. Um, I think when you're just like sprinting constantly, it is like hard to create the space to actually take a step back and like make sure you know the tools well. But once you do, then it accelerates you 10x, right, on the other side. And so a lot of it is about pushing the team to say like, hey, could we leverage Jet Ski, our internal tool to do this, right? Um, and hey, can, and hey, I have a skill that I use to do this. Do you want to try, right? And actually building that kind of set of skills, if you will.

• 39:55Speaker 00

  And actually, I think sometimes the most practical thing to do is like actually build a skill to do something. And when I say skill, I don't mean like the human skill. I mean, like the, the, the technical skill um to do it. And then that skill gets shared to others. That builds more confidence. That then leads to someone else building another skill or improving that skill. And now you have this like iteration flywheel, which I think is good.

• 40:15Speaker 01

  Does Google still do the 20% thing? Cause that feels like a really good place to spend time on this stuff. Is that still a thing or less so these days?

• 40:21Speaker 00

  I don't know, actually. Um, I do feel like all my PMs are working like 150%. So there is also that.

• 40:29Speaker 01

  Okay, so you could do 100, you could get your 20% out of the 150%.

• 40:32Speaker 00

  Yeah, or you could do 170%, I guess, instead.

• 40:35Speaker 01

  There you go.

• 40:36Speaker 00

  But no, I think, I think actually like one of the things we're working on is actually a, a two day workshop in November where we're gonna actually like pull all the PMs out of their day job for two days and just like actually give them the space to, to like build and play with the tools that we have. Um, I think you're just giving yourself even that mind space to be like, hey, here's something I did. The other rule of thumb that I've, I've heard people share and I've been starting to adopt myself is like, if you have to do something twice, do you need to do it yourself, right? Or, or can you outsource it? And some things, yes, obviously like writing a strategy twice, you should probably still be writing it yourself. But there's other aspects of this where it's like, actually like, how replicable is this? And how much can I use the models to like, help me with this task?

• 41:19Speaker 01

  Yeah, there's a, this thought I've been having that one of the most important new habits to build is to insert between stimulus and response. Say, I have a thing to do and then I'm gonna go do it. You have to insert this question, can AI do this for me? And then try. And I find that like I've, I've actually built that habit much more strongly over the past few months and it's ma- makes a huge difference. Cause then you start building all these little workflows and automations. I'm doing a lot of Grok botting and it's just like, and then once you realize you should be doing that more and more, you really start to get the benefits of this versus like, I don't know what to do with it.

• 41:55Speaker 00

  And you know what I found is like, you really just need one example where that works really well to change your mindset. Um, and I think that's what happened to me at least is like, I feel like it was hard for me to get out of my mindset of just doing what I knew how to do until I started being like, oh, actually to your point, like set that interim period of actually can the model help me do this? And the minute you take one instance to do that and you're like, actually, this was really helpful for me. You start being like, oh, actually now I'm gonna try throwing this at the model. And now I'm gonna try throwing this at the model because your confidence in what it can do for you grows.

• 42:29Speaker 01

  Yeah, yeah, that's exactly my experience. Once you see it, what it can do, you, that habit reinforced, like the neural pathways start to build of like, oh yeah, wait, let's see if Grok Bot can do this for me. Let me actually ask you a question along these lines. I had Josh Woodward on the podcast recently. Asked him the same thing. I'm curious what your answer would be. It feels like every AI assistant launching these days is taking all this juicy Google data, my Gmail, my Google Calendar, my docs. You like sign up, you connect all your Google data, does all these magical things with it. There's Grok Bot, there's Muse, there's Instinct, uh, Dots now. Uh, Google does not yet have anything that people are, get really, or I know there's Spark, I think, which is trying to be this. I'm curious just why Google hasn't

• 43:14Speaker 01

  launched something awesome yet versus letting everybody take all my Google data and do these magical things with it.

• 43:20Speaker 00

  Yeah, I mean, look, we want Google data to power magical experiences. Like, I think that is true. I think in terms of what awesome looks like, I think stay tuned. You know, the team is working on some amazing things. Um, so I'm really excited about where, uh, where this can go. Part of what we're trying to figure out also, I think, is like, again, where to meet users where they are and like where to build that experience and what does that look like? Um, I like to think there are already a number of uh, awesome experiences that you can already have in our Google products. But I think you are right that like, there is more work we need to do here on this front.

• 43:53Speaker 01

  All right, that was similarly, similar to what he said, which is watch, watch the space.

• 43:57Speaker 00

  Watch the space.

• 43:59Speaker 01

  And the advantage you all have, similar to Apple, is once you get there, uh, the distribution advantage is very strong and uh, things can go really well, really quickly. Okay, so coming back to this kind of workload question, there's something that I, I've been trying to talk with more guests about. This idea of just as AI allows us to do more, everyone's just doing more. And there's a lot of burnout that happens because it's just like, oh no, I gotta unblock my agent. Oh, I could be doing so much more. All these things could be happening. And you know, there's pressure from above, do more, do more, AI can let us do more. Is there anything you've done on your team to help people avoid burnout, not get hooked on this never ending treadmill of doing more? And I think part of it is actually, we need to do more now because the

• 44:43Speaker 01

  competition is fierce and uh, there's, it's this really rare time in history when a whole new technology enables so much more. So you know, there's a bit of like, okay, this is actually the time to go really long, but then there's also this tweet, I think about, what's the word for a marathon and a sprint?

• 45:01Speaker 00

  Yeah, marasprint, marans.

• 45:03Speaker 01

  Exactly.

• 45:04Speaker 00

  Sprintathon, um, yeah, I think, uh, I mean, if you ask my team, I think they would say, uh, hello, we are, we are working a lot. I think every single person on the team right now is putting in all of the hours, right, um, across the board. And so I, I do think this is an area that I'm, I'm actually asking myself about like, how do we, how do we create a sustainable work environment for the team? So, you know, stay tuned as I, as I also think about this, but one thing I'll say, at least for me personally, is I think burnout comes not just from working a lot. It comes from, I think, two places. I think it comes, one, from when you feel like there's churn and chaos. I think that introduces burnout.

• 45:48Speaker 00

  I think the other thing that introduces burnout is work that you don't enjoy. When you feel like you were doing a lot of work and it feels like grunt work or a waste of time, I think that introduces burnout. I think when you are working on something that is, you're passionate about or you're excited about, yes, it is exhausting and, and we should figure out sustainability, but I think it is also brings more joy. Right? Um, and so I think actually to our previous, you know, conversation about like how these tools can help. I'm actually hopeful that if we can set this up well, these tools actually help with burnout because they actually remove some of the like grunt work or the like frustrating parts of your job and actually enable you to spend more time thinking

• 46:31Speaker 00

  and like more time being creative and more time doing the parts of the job that you actually want to do. Um, and that hopefully is a bit of an unlock when it comes to burnout, especially if you're spending this many hours on the job.

• 46:41Speaker 01

  Yeah, I did this survey of how tech workers are feeling. And one thing that's clear is more hours does not equal burnout. A lot of people are working the hardest they've ever worked, but they're having the best time they've ever had in their career. So fun, they're building, they're able to do a lot more, especially PMs. You don't have to sit around, wait for engineers as much as you do.

• 46:59Speaker 00

  You're just like, you're building, you're prototyping, you're like talking to interesting customers. You're thinking about interesting problems. And I think a lot of the team, I like to think is here because they want that, you know, they're like, they're like really motivated and excited by that. And so for me, it's about like, okay, then how do we pull out the friction parts of their work that are likely to induce burnout, right? And what does that look like?

• 47:20Speaker 01

  And also find work that is meaningful and that they value and are excited to spend lots of hours on versus they have to.

• 47:27Speaker 00

  And that's different for every PM too. Like if you pick two PMs on my team, I think they would tell you meaningfully different things about why they come to work every day and what they find value out of. And so I feel like part of it for me is like, oh, an org leader also is to think about what motivates the individual and how do we make sure that they are set up for success in that setup too?

• 47:45Speaker 01

  Another challenge people have these days is AI slop. Basically lots of docs being sent to them that are written by AI and it becoming very, uh, very annoying to have to sift through endless docs. It used to be a lot of, the issue was spending time writing docs and now it's like, reading docs is the problem because you're just, all this AI written stuff. Do you have any kind of, I don't know, processes or policies you've put in place in your team to help avoid this rise of AI written content across the teams?

• 48:13Speaker 00

  One of the things that we talk about a lot is where, where are these docs useful, right? So like, for example, I think where uh, AI generated docs are actually quite helpful, for example, is after you launch something. And let's say two weeks later, you want to pull the latest of like, who's using it? How are they using it? What's the feedback? That's the kind of document that AI is fantastic for. Right? Because it will actually do that just so much faster than you. Right? It will go and pull from various dashboards and metrics. It'll give you an update. It'll pull kind of strengths and weaknesses. It'll pull from Twitter analysis and, you know, Reddit or whatever. And so I think one of the things we talk about is like, be intentional about like, where is AI helping you drive content in the doc that is helpful versus where is it

• 48:59Speaker 00

  content that should come from you, right? Where is it insight? Um, and so what you should be providing is the insight, what the model should be providing is the data. And so I think if you can separate those two things, then hopefully you find the right balance of, of the, of the benefit here. Um, I wouldn't, I would say the other thing is we're investing a lot more as a result of that in like, how do you create the right skills to get the right data? Um, so that we can structure these artifacts really well. Uh, that then hopefully creates more space for the creative thinking and the creative writing that we want more people to spend time doing.

• 49:31Speaker 01

  That's actually something I, I was wanting to ask about. So in terms of the, the PM role, AI has clearly made a big dent on prototyping. You can prototype, uh, till the cows come home now versus having a designer or an engineer build a thing. So that's clearly a place the role has changed with AI. The other area, which you just mentioned is data. PMs can do a lot more data analysis. What would be like behind prototyping? What are the biggest areas AI has most changed the job of a PM in the past couple of years, would you say?

• 50:04Speaker 00

  Yeah, prototyping is a great one. I think you're right that like data science is probably like I think you are more empowered to have the data at your fingertips to make data-driven decisions. I also think actually like to our earlier conversation, you know, you see these on social a lot, these kind of posts of like, oh, blank is the new PRD, whatever blank is, a prototype is the new PRD, evals are the new PRD, like, data is the new PRD. But I think actually what that meaningfully shows is you've moved the PM from being someone who just writes a set of requirements down and then assumes that somebody else, whether that is a data scientist, or an engineer, or a researcher, or a designer, will figure

• 50:45Speaker 00

  out the meat behind it. You've moved that one step further to say your job is no longer just writing down a set of requirements. Your job is actually going one step further to have some heft to that, whatever format that heft takes.

• 51:00Speaker 01

  Awesome. Okay, that aligns with a lot of what I'm hearing. Tulsi, is there anything else that you wanted to share? Is there anything else you think you want to double down on that we've talked about that you want to say before we get to our very exciting lightning round?

• 51:13Speaker 00

  No, I think that's great.

• 51:16Speaker 01

  Okay, we did it. Well, with that, Tulsi, we reached our very exciting lightning round. I've got five questions for you. You ready? There's some music that plays over this that's added in post-production.

• 51:26Speaker 00

  You mean, does it sound like this?

• 51:28Speaker 01

  It's exactly that. We should align it with your drum playing. Okay, first question, what are two or three books that you find yourself recommending most to other people?

• 51:37Speaker 00

  Okay, so one actually is, it's a, it's actually technically a novella is probably what it's called, but I don't know if you've read Exhalation, which is a set of like science fiction stories. And there's a particular story in that that actually speaks to a set of individuals who are trying to keep a set of digital avatars alive even after the platform decides to deprecate. And I love that book because I think actually, especially now in a world where we're thinking about how individuals are going to act with agents and what kinds of relationships we're building with technology and what does it mean to be human and interact with the digital world so closely? I think it's just a really, it's a really beautifully written story. And I think it also like evokes a lot of open questions.

• 52:22Speaker 00

  So I think that's one. Another that's maybe more kind of standard in PM circles is Creativity Inc., which I love. It's the Pixar story. But I think it actually speaks a lot to like, how do you create good culture? Which is something I think a lot about, like, how do you create culture that actually encourages people to fall down and get back up? How do you create a culture that encourages people to be creative with their ideas and push the frontier, which is obviously something we're thinking a lot about here. I also just have been really into, you know, the Akatar series recently, but that's

• 52:56Speaker 01

  What series?

• 52:57Speaker 00

  That's romantic fantasy. So for anyone who's just looking for something fun to read, you should read The Throne of Glass or A Court of Thorns and Roses.

• 53:05Speaker 01

  I love the spectrum of recommendations in this. Okay, favorite recent movie or TV show you've really enjoyed?

• 53:11Speaker 00

  You know, actually, this was a really hard question because I've watched a lot of bad movies and TV shows recently, which is unfortunate, actually, that I would not recommend. So there's that. But I did really enjoy Severance on Apple TV, incredibly well done, both like stylistically and content-wise. So that's probably top of the list.

• 53:33Speaker 01

  Favorite AI product right now, other than Gemini?

• 53:37Speaker 00

  Whisperflow. I don't know if you're using it, but I have wrist problems. So I have trouble like typing for long stretches. And so Whisperflow has been game-changing for me, actually.

• 53:49Speaker 01

  Awesome. You get a year free of Whisperflow if you become a subscriber of Lenny's newsletter. Plug, plug, plug.

• 53:54Speaker 00

  Oh, I like this.

• 53:55Speaker 01

  There you go. Organic recommendations of products I also love. There we go. No, I love Whisperflow. I use it all the time. Press function, just start talking and it's just magically transcribes everything.

• 54:04Speaker 00

  Yep.

• 54:05Speaker 01

  Amazing. Your favorite life motto that you often come back to in work or in life?

• 54:11Speaker 00

  Two that are different, I guess. One is maybe more cliche, which is, you know, everything happens for a reason. I think that's like actually just a big life motto for me personally, which is like, you know, you can work hard, you will do your best, but sometimes the door will close and another window will open. And like, you're in a position because that is meant to be there. I think related to that, though, I think I love often talking to folks that I talk to about like, seek serendipity. I think that actually relates to the first one, but you should always try to put yourself in a position where you have the ability to get lucky. And doing that is about like finding yourself in positions where you're challenging yourself, where you're putting yourself out of your comfort

• 54:53Speaker 00

  zone, where you're trying something new. Those are always places where you are more likely to get lucky.

• 54:58Speaker 01

  Fun fact, I gave a TEDx talk called Losing Serendipity, maybe 15 years ago at this point.

• 55:04Speaker 00

  So we're trying to lose serendipity or find it?

• 55:06Speaker 01

  It was about how we are losing it and that is sad and that we need more serendipity in our life. So we're on the same page.

• 55:12Speaker 00

  I'm a big fan of serendipity.

• 55:13Speaker 01

  Such a great word. Also, the movie's pretty good.

• 55:16Speaker 00

  I haven't seen it.

• 55:17Speaker 01

  Okay. It's like a rom-com.

• 55:19Speaker 00

  But I think that's how I like looked at my whole career so far, and I think how I'll continue to. And actually, like, it's fun because even with like using these AI tools, I think that's like another thing that I, yeah.

• 55:30Speaker 01

  So much serendipity with AI. Final question. One of your colleagues told me another fun fact about you, that you're a trained classical Indian dancer.

• 55:38Speaker 00

  I am.

• 55:39Speaker 01

  And you've directed cultural festivals for 10,000-plus people.

• 55:44Speaker 00

  I don't know how he knows that, but yeah, I probably told him that.

• 55:47Speaker 01

  Okay. And this person said that they may explain how you stay so calm amidst the chaos of frontier model development. I don't know, maybe share some about this dance, this dance part of your life?

• 55:58Speaker 00

  Yeah, I mean, I love dance. I think it's one of those rare moments where you really can't think about anything else. You have to focus on the movement and the music. And so for me, it's an opportunity for my brain to actually just have a different direction of travel for a period of time. And so I love it. And I think for me also, it is a cultural connection point. Dance is how I've met some of my best friends. It's how I continue to engage socially with so many of my friends. I planned this festival, I've planned it for years in Portland, Oregon, which is where I'm from. It's called the India Cultural Festival, India Festival. It happens every August. And in fact, this was the first year I did

• 56:41Speaker 00

  not participate in the festival because I have a seven-month-old son.

• 56:45Speaker 01

  Congratulations.

• 56:46Speaker 00

  Thank you. So I was a little bit more distracted this year than normal. But it's since I've been performing at that festival since I was three, and then organizing it. And it's just been a big part of my life of, you know, how do you bring dance and culture and community together?

• 57:02Speaker 01

  Tulsi, you're so full of fun facts. I feel like there's much more. I don't even know that

• 57:07Speaker 00

  I'm here for the fun facts.

• 57:09Speaker 01

  Final question, how can listeners be useful to you?

• 57:14Speaker 00

  Wow. Great question. I mean, A, use Gemini.

• 57:18Speaker 01

  Argon.

• 57:19Speaker 00

  Use Argon, but also use our smattering of products. You know, actually, I was talking about audio just now. If you are a Pixel user on Gboard, like, try using our Rambler feature. It's awesome. You know, typing on the phone is great. If you are using text-to-speech models, if you're using dialogue, if you're using our Omni models. So try them. Give us your feedback, right? We're constantly looking for it, whether you're giving that feedback via a company that you are, you know, sharing feedback via, whether it's you're sending us, you know, a DM, whether you're posting it publicly. I think we want more people giving us the right feedback.

• 58:01Speaker 01

  Tulsi, that was awesome. You're awesome. Thank you so much for being here.

• 58:04Speaker 00

  No, thank you for having me. This was great.

• 58:06Speaker 01

  It was my pleasure. Bye, everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.