Behind the product: NotebookLM | Raiza Martin (Senior Product Manager, AI @ Google Labs)

Raiza Martin is a senior product manager for AI at Google Labs, where she leads the team behind NotebookLM, an AI-powered research tool that includes a delightful podcast-on-demand feature called “Audio Overviews.” NotebookLM started as a 20% project and has grown into a product that’s spreading across social media and has a Discord server with over 60,000 users. Raiza previously worked on AI Test Kitchen and has a background in startups, payments, and ads. In our conversation, we discuss: • The origin story of NotebookLM • The future road map for NotebookLM • How Google Labs operates differently from the rest of Google • The development of the “Audio Overviews” feature • Key metrics and growth of NotebookLM • Stories about collaborating with author Steven Johnson • Navigating potential misuse of AI technology — Brought to you by: • Explo — Embed customer-facing analytics in your product: https://explo.co/lenny • Sprig⁠⁠ — Build products for people, not data points: https://sprig.com/getstarted?utm_source=lenny&utm_medium=podcast • Sidebar — Accelerate your career by surrounding yourself with extraordinary peers: https://www.sidebar.com/lenny?utm_source=lennys&utm_medium=newsletter&utm_campaign=waitlist&utm_content=3 Find the transcript at: https://www.lennysnewsletter.com/p/googles-notebooklm-raiza-martin Where to find Raiza Martin: • X: https://x.com/raiza_abubakar • LinkedIn: https://www.linkedin.com/in/whatsaraiza/ Where to find Lenny: • Newsletter: https://www.lennysnewsletter.com • X: https://twitter.com/lennysan • LinkedIn: https://www.linkedin.com/in/lennyrachitsky/ In this episode, we cover: (00:00) Introduction to NotebookLM (05:43) The genesis of NotebookLM (08:08) Innovative features and use cases (18:52) Building a startup culture within Google (24:28) Expanding user demographics (27:30) The product roadmap (32:18) Other use cases (36:11) Collaborating with Steven Johnson (42:49) Ensuring ethical AI (46:06) Future directions and user engagement Referenced: • NotebookLM: https://notebooklm.google/ • AI Test Kitchen: https://aitestkitchen.withgoogle.com/ • Steven Johnson on X: https://x.com/stevenbjohnson • Gemini Pro: https://deepmind.google/technologies/gemini/pro/ • Usama Bin Shafqat on LinkedIn: https://www.linkedin.com/in/ushafqat/ • NotebookLM’s Discord server: https://discord.com/invite/Az2N7BwV7r?sjid=10560123093044576704-NA • Josh Woodward on LinkedIn: https://www.linkedin.com/in/joshwoodward/ • Andrej Karpathy’s website: https://karpathy.ai/ • Histories of Mysteries on Spotify: https://open.spotify.com/show/3K4LRyMCP44kBbiOziwJjb • NotebookLM — Analyzing Poop & Fart written 1,000 times — Creating meaning from the meaningless: https://www.youtube.com/watch?v=ftg7UC3CGjc • Reddit thread about the fake chicken.pdf: https://www.reddit.com/r/notebooklm/comments/1fwz5eh/the_google_podcast_duo_get_almost_9_minutes_out/ • Steven Johnson’s tweet about his 8K quotes: https://x.com/stevenbjohnson/status/1840844268065456456 • Readwise: https://readwise.io/ • How We Got to Now: https://www.pbs.org/show/how-we-got-now/ • “A. I. Is Mastering Language. Should We Trust What It Says?”: https://www.nytimes.com/2022/04/15/magazine/ai-language.html • AI Podcast Hosts Discover They’re AI, Not Human — NotebookLM: https://www.youtube.com/watch?v=AR4dRtzFvxM Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected]. Lenny may be an investor in the companies discussed.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Speaker 00

  Hey, everyone. We're here on Lenny's Podcast.

• 0:05Speaker 01

  It's great to be here. I'm a longtime listener.

• 0:09Speaker 00

  So awesome. Really is. We're the hosts of a different show of Deep Dive.

• 0:12Speaker 01

  And we just, we just wanted to say thanks, a huge, huge thank you to everyone, everyone who's been listening.

• 0:18Speaker 00

  Yeah, seriously. It's been incredible. Just incredible. And thank you to Lenny for having us.

• 0:23Speaker 01

  Blown away, really, by the response and all the shows you've all had us make on Notebook LM. Even that poop fart one, remember that?

• 0:30Speaker 00

  Oh, yeah, that was something. Learned a lot on that one.

• 0:33Speaker 01

  Definitely a learning experience for everyone, I think.

• 0:36Speaker 00

  But we're learning right alongside you.

• 0:38Speaker 01

  Exactly. Learning and growing.

• 0:40Speaker 00

  And we're glad you're along for the ride.

• 0:42Speaker 01

  So, yeah, keep listening.

• 0:43Speaker 00

  Keep listening and stay curious. We promise to keep diving deep and bringing you even more in the future.

• 0:52Speaker 01

  Stay curious.

• 0:54Lenny Rachitsky?

  If you are confused about what you just heard, don't worry, it'll all make sense very soon. Today, my guest is Raiza Martin. Raiza is Product Lead for a product called Notebook LM, one of the most delightful and inspiring new AI products out there, incubated within Google Labs. And this product is where the intro you just heard came from. In our conversation, Raiza shares how Notebook LM came to be, how it got so good, the technology that was necessary to make it possible, how the team works internally, how it's incubated specifically within Google Labs and out of the team's 20% time, plus a bunch of really fun and crazy use cases that she's seen, and a glimpse into where the product is going long term. This was such a fun and timely conversation, and I'm excited to spread the love for Notebook LM.

• 1:39Lenny Rachitsky?

  If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. It's the best way to avoid missing future episodes, and it helps the podcast tremendously. With that, I bring you Raiza Martin. Raiza, welcome to the podcast.

• 1:54Speaker 03

  Hi, Lenny. Thanks for having me.

• 1:56Lenny Rachitsky?

  What the heck did we just listen to? What was that?

• 1:59Speaker 03

  So that was an audio overview from Notebook LM, where you upload a source, any source, and it will generate an AI-generated audio for you.

• 2:10Lenny Rachitsky?

  Okay, so for folks that don't know anything about Notebook LM, it's basically been blowing up on Twitter, on LinkedIn. I think it's blowing a lot of people's minds. It's sparking a lot of imagination of what could happen in AI and what potential we have with the stuff that's happening. And I wanted to bring you on to talk about the history of this product, where it's going, how it became so great, and all these things. And so, thanks for doing this. I know this kind of came on short notice.

• 2:36Speaker 03

  Yeah, I mean, I was excited to do it, in particular because, you know, I'm a big fan, big listener of Lenny's, read the newsletter, I love it. So really happy to be here.

• 2:46Lenny Rachitsky?

  Awesome. Clearly, these hosts are too, which I love.

• 2:49Speaker 03

  Yeah.

• 2:50Lenny Rachitsky?

  By the way, I love the slight awkwardness at the end of their conversation. If folks rewind, they could hear it again, because it's very relatable. I sometimes have trouble ending a podcast conversation. And they're like, "Okay, let's say a couple more things," and then, "Okay, we're done."

• 3:04Speaker 03

  It's funny, because I've listened to so many of these, and they do have catchphrases. They say at the end. And "Stay curious" is one of my favorite ones.

• 3:14Lenny Rachitsky?

  Well, let me just ask, is that something you all told them to do? Or is that like an emergent property?

• 3:19Speaker 03

  So in this case, this is based on what they think is the most appropriate thing to say at the end.

• 3:25Lenny Rachitsky?

  Okay, interesting. Okay. This episode is brought to you by Xplo, a game changer for customer-facing analytics and data reporting. Are your users craving more dashboards, reports, and analytics within your product? Are you tired of trying to build it yourself? As a product leader, you probably have these requests in your roadmap, but the struggle to prioritize them is real. Building analytics from scratch can be time-consuming, expensive, and a really challenging process. Enter Xplo. Xplo is a fully white-labeled embedded analytics solution designed entirely with your user in mind. Getting started is easy. Xplo connects to any relational database or warehouse, and with its low-code functionality, you can build and style dashboards in minutes.

• 4:10Lenny Rachitsky?

  Once you're ready, simply embed the dashboard or report into your application with a tiny code snippet. The best part? Your end users can use Xplo's AI features for their own report and dashboard generation, eliminating customer data requests for your support team. Build and embed a fully white-labeled analytics experience in days. Try for free at xpl.co/leni. That's E-X-P-L-O dot C-O slash leni. This episode is brought to you by Sprig. What if product teams knew exactly what to build to reach their goals? From increasing conversion to boosting engagement, these challenges require a deep understanding of your users, something that you can't get

• 4:54Lenny Rachitsky?

  from product analytics alone. Meet Sprig, a product experience platform that generates AI-powered opportunities to continuously improve your product at scale. First, Sprig captures your product experience in real time through heat maps, replays, surveys, and feedback studies. Then, Sprig's industry-leading AI instantly analyzes all of your product experience data to generate real-time insights. Sprig AI goes even further with actionable product recommendations to drive revenue, retention, and user satisfaction. Join product teams at Figma and at Notion by uncovering AI-powered product opportunities at scale. Visit sprig.com/leni to book a demo and get a $75 gift card.

• 5:39Lenny Rachitsky?

  That's S-P-R-I-G dot com slash leni. I want to start with just kind of the history of this product. I think for a lot of people, it just like came out of nowhere. It's like, what the? Wow, where did this come from? This is blowing your mind. Rarely is anything actually like that. Rarely is it just like, we built it over a week and then it's taken over the internet. Talk about just the history of Notebook LM and the team behind it.

• 6:01Speaker 03

  Yeah, yeah, thank you. Actually, Notebook started as a 20% project, as things kind of do at Google, as they used to. And it's funny because I wouldn't even call it a 20% because it was way more than that. So I'll start from my perspective, where I was leading AI Test Kitchen, which was one of the first AI things we launched at Google. And this was, I think, maybe two IOs ago? Or maybe three now, actually. I think it was in '22. And I was leading that project and I remember there was a smaller project inside of labs called Talk to Small Corpus. And there was the idea that, you know, you could take a piece of content and use an L L M to interact with it.

• 6:45Speaker 03

  We were like, okay, there's a nugget there. It sounds interesting. How do, yeah, how do we keep revving on this, right, to make it actually really useful? And uh, it was just myself, an engineer, and eventually Stephen Johnson joined. And it was, it was really funny because the only person who was actually like really full time on this was the engineer working on the technology. And everybody else was kind of just like coming together, like really chiming in, being like, hey, this is super interesting. How do we make this better? So it really, it really started as a, as a 20% and then it kind of just blew up from there.

• 7:19Lenny Rachitsky?

  That's insane that it was just you, an engineer, and then Stephen Johnson, I want to talk about his role, that it's like, one engineer can build something like this. And I imagine it's more now that it's starting to take off. Or what's the team looking like these days roughly?

• 7:33Speaker 03

  We've had a lot more folks join in the last month. And a lot of that is really in anticipation of the future roadmap. But up until this point, I'd say we didn't even have 10 engineers. When, when we launched it, actually, when we announced Project Tailwind, I think we only had three. We had three engineers, it was myself, we had a designer, uh, we had Stephen, and that was it. And so even between the period when we did the, the first I O last year to as recently as like last month. You know, we've only had about eight engineers or so.

• 8:09Lenny Rachitsky?

  Let's talk about uh, audio overview, specifically this kind of podcast feature. I think audio overview is the technical name, right? Is that how you describe it? Okay. They call it a deep dive, right? That's like their name for the podcast.

• 8:18Speaker 03

  They do. Yeah, that's how it.

• 8:20Lenny Rachitsky?

  Uh, how did that especially specifically come about? When did that become a thing? What was the beginnings of that?

• 8:25Speaker 03

  We announced, we actually did a preview of this in I O uh, this year, so in May. And the way that it worked out is, you know, Notebook had already been launched. We had a bunch of people using it. It was this source grounded chat interface. And we were looking at, basically, what are the the new models coming out of Google, right? We had Gemini, 1.5 coming out. We had all of sort of these new technologies that are basically upgrades on fundamentally the same thing. And we were like, what are some interesting ways that we could, you know, power Notebook LAM or make it even better? And one of the things that we started playing around with, there's actually a different team inside of Google,

• 9:06Speaker 03

  inside of Labs specifically, uh, they were like, hey, we've got these like really powerful audio models. Right? Like, what's, what's a really good application of this? And that's when we started riffing and we were like, hey, what if you could, basically what you see today, give it a little bit of information. You don't have to do a lot. You could just enter a URL, upload, you know, your resume. And then it will completely generate something for you that could be unexpected, could be a surprise.

• 9:34Lenny Rachitsky?

  Okay, so it started, basically, there was like a technology that we think could be really interesting. Was there a problem you were trying to solve with this team, with this initiative? It was within Google Labs, which I know is a little bit probably less like we need to drive business outcomes. What was kind, what was the original thesis or problem you're going after?

• 9:51Speaker 03

  And so this is really funny. I talk about this quite a lot, uh, especially when I'm talking to other PMs. I feel like the way that I've always built products is I start from a problem. And I think about the solutions, you know, to solve it, solve it for people in a meaningful way. And in Labs in particular, we start with the technology. And it's actually a very interesting place to start. You're like, okay, what are the practical applications to this? How do I go about finding the answer? And I think we've seen different approaches to this, right? Where you just put the tool out there. You study how people are using it. And I think that's like a fine approach. But we tried to have a little bit more of a hypothesis about what, what is the shape of this thing? To actually get the most learning out of it. So for the audio feature in particular, the nugget there was really,

• 10:36Speaker 03

  we had this ability for you to interact with text. But people were like, the output is still always text, right? And I'm actually a big fan of voice modality in general. So I use a lot of voice input and voice output. I noticed when I was experimenting with it early on, it changed a lot of things for me. It's like, oh, it changes the way that I interact with the technology and changes the way I feel about the technology. It sort of affects the way that I'm even thinking in real time throughout this process. And so we thought about, what is a good way to introduce this to people? Where they could sort of easily get the value of it and have a little bit of fun. So I think, you know, for me, it's like fun is like such a big part of it too, which is like, how do we make this cool? Uh, and so hopefully, hopefully we've done that here.

• 11:22Lenny Rachitsky?

  To me, it feels very similar to the chat GPT moment where the technology already existed, like they had that same GPT model out for I don't know how long. But just that new medium and new way of interacting with it changed the way people imagined, just they saw the power immediately. And I feel like this is such a good example of that happening again, where just the tech was there and it's just this medium that you developed that really inspired people to like, oh, wow, I had no idea. AI was getting this good and LMS were getting this good. Any thoughts on that?

• 11:55Speaker 03

  Yeah, I think, I think that that totally strikes at the heart of what we're trying to do as well. Which is, you know, a lot of technology, I feel like you have to shape it and bring it closer to people. And I think it's like such an interesting thing to iterate on to be like, what is the shape? Right? Like soon, soon you'll get there. Like if you keep going at it, you'll eventually, I think, land on something that like when people look at it, they're like, wow, I get it. Right? And that's, that's always like what we're hunting for. But I think to your point, technology has been there. Right? Like, I mean, it feels so crazy to say that because even with like the, the advent of like the chatbots, right? Of LLMs as we know them today, and then to introduce this sort of new technology on top of it.

• 12:39Speaker 03

  That we've only been doing this for like two years. But already, right, we're exploring so many different ways to use it.

• 12:46Lenny Rachitsky?

  Yeah, it's such an intuitive but clever idea of making something a podcast. Let's take a tangent into the technology behind this thing. So there's kind of two questions I'm thinking about. One is just what technology needed to exist for this to be possible? And then two is just what did you do to make this so good? What did you, how did you train this model to get to create a podcast that's this good? So we could start with the first and then get into the second.

• 13:12Speaker 03

  Yeah, I mean, I, that's such a great question. I mean, the truly like the Gemini models are so powerful. And we use Gemini 1.5 Pro as sort of the base for Notebook LM. And then we have a powerful voice model on top of it, an audio model on top of it. But I think the real secret sauce to what makes this really good is something we've built, which is a content studio. And you kind of see a clue of this inside Notebook LM, where when you open up Notebook Guide, Notebook Guide is designed to take an opinionated approach to the content that you've given it. And what it's really trying to do is be as helpful as possible. It gives you a summary, gives you buttons where if you click on it, it'll write something for you. And the audio is a big part of that.

• 13:55Speaker 03

  And so this was very interesting for us because we had to think about what shape should that audio be in, right? Like the deep dive is the first format we've thought about. And we actually have a super talented engineer on our team. His name's Osama, and he's the real craftsman behind thinking through content studio and thinking about what makes something really relatable to people in a way that's engaging, that's interesting. So content studio is the real magic here.

• 14:23Lenny Rachitsky?

  So when you say content studio, what does that mean? It's a, what, yeah, what does that actually look like? What is, what is a content studio?

• 14:29Speaker 03

  I can't share too much about how Content Studio works, but you can imagine it's the same thing that powers all of Notebook LM, which is there's different ways that you could interact with your data, right? You could like Q and A inside of Notebook LM, but then there are places where you just one push a button and it creates something new. So Content Studio is what's

• 14:47Lenny Rachitsky?

  Okay, so this is the interface you're you're you use to interact with Notebook LM and the different prompts and the recommendations of how to ask questions and okay, got it. And then specifically in terms of the the way these hosts interact and talk like, if you listen to these podcasts, it's like they laugh and they're like hmm, and they interrupt each other a little bit and they get surprised and they have these like inflections that are really amazing. Is what did you have to do to make them sound this good? I know you said you can't share it tons, but just anything there you can share.

• 15:19Speaker 03

  I mean, that's that's the model. That's the the audio model that we use. Alongside the Content Studio is designed to bring out the best of that model. And I think when we listened to a lot of like the early uh attempts that we had, it wasn't nearly this good. And so we had to do a lot of listening to try to figure out, how do we actually get the model to behave in this way? And that's where I think the the magic happens.

• 15:46Lenny Rachitsky?

  That's amazing. Did you do all just listen to a bunch of podcast episodes and like hear something that really? Yeah, okay, shaking your head.

• 15:52Speaker 03

  I think I think the funniest thing was that um so my husband doesn't know, right? He doesn't know what I'm doing. Most of the time I try not to talk about work at home. And so I would listen to these audio overviews over and over at home. And you know, I get tired of wearing my headphones, so I'm just listening to them all out. And he's like, what is this thing you are listening to? This is like an ever running podcast about nothing.

• 16:17Speaker 03

  And it was just super funny. I had to be like, oh, it's just like something I have to do for work. So it was pretty funny. He actually couldn't tell that they were AI.

• 16:25Lenny Rachitsky?

  That's, I love that. I, uh, so one of the ways I've been playing around with it a bunch. One of the ways I used it is my mom wrote this autobiography, like a short autobiography of her life. And we have the PDF of it. So I fed it into the, into the notebook and then generated the podcast and sent it to her. Like, what the heck?

• 16:41Speaker 03

  That's amazing. What did she think?

• 16:43Lenny Rachitsky?

  She was blown away. She's showing all her friends. She's like, I don't know what's going on here. And, and I love that it creates a study guide and that's one of the options in the studio. And so I sent her the study guide and she's like, okay, at a Russia Shana dinner, we're going to be reviewing the study guide about my biography. You're going to talk about this?

• 17:02Speaker 03

  That's very funny. That's super cute. I did it for my dad, too.

• 17:06Lenny Rachitsky?

  What did you do?

• 17:07Speaker 03

  I actually took my dad's bio. So he worked for a, at a hospital. And he has a bio, you know, for the doctors that they have there. And I put his bio and I generated an audio overview for him. And it's so funny because my, my parents are both in the medical field. I don't think they fully know what I do. I think this is the first time they were like, oh, that's super interesting. That's what you do.

• 17:32Lenny Rachitsky?

  So maybe along these lines, what are some other surprising, hilarious use cases you've seen for especially the audio deep dives?

• 17:40Speaker 03

  I mean, I love, I love the resume ones. I think the resume ones are so delightful. And actually, really adjacent to that is Google has these Q3 check-ins or the quarterly check-ins where we have to write performance reviews for ourselves. And the amount of Googlers, yeah, the amount of Googlers that ping me that I don't even know. And they're like, wow, this was such a boost to my confidence. Just to upload my own, you know, my own check-in notes and then have it generate an audio overview. It's like people are going into meetings feeling really good about themselves.

• 18:15Lenny Rachitsky?

  Because they heard these hosts get really excited, right? About their, their quarter and

• 18:19Speaker 03

  My work, yeah.

• 18:20Lenny Rachitsky?

  Yeah. Uh, and when you say resume, so people upload the resume and then it describes you in a really positive way, right?

• 18:26Speaker 03

  Yeah, yeah. And I, and I thought that was so interesting. I was like, what was the use case? I think if I had to guess, somebody probably was like, hey, I want to try this new thing. Clicked on upload Google Doc and maybe, you know, the last thing you had there was like your resume or something. But imagine you click that and I think there's something really special about you. Don't know what's going to happen. Right? You just click generate and then you have this like this, these two people just hyping you up.

• 18:52Lenny Rachitsky?

  Okay, I want to talk a bit more about the how you made this happen within Google. So as an outsider, this does not feel like a Google product and the way you all are operating feels very startup. Like you're tweeting daily about what's going on. I heard there's a Discord server, folks that are like, really? Is that right?

• 19:09Speaker 03

  Yeah, there is. It's, it's pretty cool. We have about 60,000 people in it.

• 19:13Lenny Rachitsky?

  Okay, why? Okay, so yeah, so there's this Discord server. It's just you're shipping constantly. The product is just like very delightful. And Google makes delightful products, but this feels like a different realm of delight. How are you able to do this within Google? Is this like a model for how you think Google Teams will want to operate? More lessons there?

• 19:33Speaker 03

  I mean, it's such a good question. I think, you know, there's a question of how it came to be, how we're able to operate in this way. And then the question of, hey, is this, you know, how other Google Teams want to operate? And I think that, you know, to answer the first part, I'll say that it's interesting because, uh, you know, Google Labs is only about three years old. It's fairly new. And when I joined, there was nobody here.

• 20:01Speaker 03

  There's no one. And the reason why I joined was because my old boss went and actually started it. And so this is Josh Woodward, who's the VP of Google Labs. And it's really funny because I was like, it's just like a funny anecdote. I remember I had no idea where Google Labs was, but I like my old boss so much that I was like, I'll just work on whatever he's working on, whatever his new idea is, I'll just do it. And I remember when I joined, I was like, what's the mission? What are we here for? And he said, it's AI. We're going to ship AI products and we're going to build businesses out of them. Okay, sounds good. Sounds good. It's like now I have to, now I have to study. I have to do a lot of studying because I came from payments, came from ads before that.

• 20:42Speaker 03

  And so for me, it was, it was a little bit of a mental shift, but before that I actually worked only at startups. And so I felt like, hey, maybe this is my chance to do zero to one again. And I was really jazzed about that. And I remember talking with Josh about this in the early days of like, hey, if we really want to do zero to one, we have to do things very differently. And so as a result, I think that's why, you know, Notebook LM is able to operate differently is because in labs, we have the environment to do it. We have the environment to move fast. We have far, far, far fewer processes, maybe even to a fault. Sometimes we'll go to meetings that are literally, you know, the product managers, the engineers, the designers all together. And we'll just crank on the mocks and the PRDs at the same time.

• 21:26Speaker 03

  And Eng is basically already doing implementation as we're meeting. And it, you know, at Google, it's just not how things are traditionally done. Right? Especially, you know, coming from the previous organizations that I was at, like, just takes a lot of time to do each of these things.

• 21:42Lenny Rachitsky?

  So what I'm hearing, this is great because a lot of companies are trying to create teams like this. We're going to have a team off to the side, going to make, work on crazy futuristic stuff. Rarely do those work out. Even at Google, there's been a lot of this in the past and rarely do things actually work out. And so there's a lot of lessons to learn from how this is working. So a few things I've noticing as you're describing how this works. One is just very clear expectations from like a very senior leader. Here's a team, we're going to do things differently. Here's things we're not going to do. We're not going to go through regular approval processes. We're not, we're going to be public and build in public. We're not going to have like a goal necessarily. We're going to work on cool technology and see what happens. It's also you had a very small team. It feels like that was really key. One engineer and then a PM and then Steven Johnson.

• 22:26Speaker 03

  True, that's true. And I think, you know, one, one big thing too, is we got to try new things. Even, you know, from the beginning, I was like, hey, I want to have a Discord. Right? Like if we were building outside of Google, for sure, we would have one of these. And in true sort of Google fashion, everybody was like, what is, what is Discord? Okay, you know, great, do it. But what is that again?

• 22:47Lenny Rachitsky?

  Right. Why not Google Meet?

• 22:49Speaker 03

  And that, somebody did ask me, they were like, why not a Google Meet? Why not a Google group? Why not this? And I was like, I don't even know how to use these things, you know, outside. So I was like, the server, the server is the way to go. And I remember when we stood up the server, actually, one of the things I was most afraid of is what if nobody joins? Like, what if no people come in and want to talk to us about the thing that we built? And so that was, that was in the early days, you know, very exciting to, to look back and remember day zero and now have 60,000 people in it.

• 23:19Lenny Rachitsky?

  Let me tell you about a product called Sidebar. The most successful people that I know surround themselves with incredible peers. When you have a trusted group of peers, you can discuss challenges you're having, get career advice, and just gut check how you're thinking about your work, your career, and your life. This gives you more than a leg up. It gives you a leap forward. Having a group of trusted and amazing peers was key to my career growth, and this is the Sidebar ethos. But it's hard to build this trusted group of peers on your own. Sidebar is a platform for senior tech professionals, director to C-level, to advance in their career. Members are matched into peer groups to lean on for unbiased opinions, diverse perspectives, and raw feedback.

• 24:01Lenny Rachitsky?

  Guided by world-class programming and facilitation, all running on Sidebar's technology, Sidebar enables you to get focused, tactical feedback at every step of your career journey. If you're a listener to this podcast, you're already committed to growth. Sidebar is the missing piece to catalyze your career. 93% of members say Sidebar helped them achieve a significant positive change in their career. Check them out at sidebar.com/lenny.

• 24:29Lenny Rachitsky?

  Okay, so there's 60,000 in this Discord server. Any other traction numbers you can share? It feels like it's just going really well. I know there's probably a lot of sensitive information. Anything you can share about just like how well things are going?

• 24:38Speaker 03

  Couple of things. Well, I'll tell you three things. I think the first is, I can't share the exact numbers, but you know, for a product that's only been out for about a year, I think the rate at which our retention has gone up, across, you know, across like the typical measures, right? Daily retention, weekly retention, monthly retention, that I think has been very, very positive for us in terms of like, even when we talk to stakeholders, we say, "Hey, we've got something here." The second thing I'll say is even seeing how the demographics have changed. Where I think in the beginning, we saw educators, learners really loved the product, and they were a lot of our demographic.

• 25:20Speaker 03

  It's really now a big mix of, I would say educators and learners are still a big part of it, but really professionals, right? Huge interest from professionals that are like, "Wow, I want to use this at work." And really funny, I won't, I won't mention the company, but I actually had a call with a company recently where they were like, "Hey, we thought we'd schedule a meeting with you because we found out that a bunch of people in our company are using this tool with our Gmails, with their Gmail account." They're like, they're not supposed to do that. So we just want to make it official that they can use this at work. You know, with their official work emails. I thought that was pretty cool. And then the third thing I'll say is, uh, the number of businesses that we've seen using Notebook LM just has, it's astronomical. It's just like crazy.

• 26:03Speaker 03

  Um, I think we went, I don't want to say the exact numbers, but it is now at a point where I have to hire a business development person. Because I was like, hey, I have to ship products and I'm actually taking my customer calls every day now. Very, very exciting.

• 26:21Lenny Rachitsky?

  So I could see the path to monetization clearly. Enterprise feature, single sign on and SOC two and all these things. So along those lines, what does success look like for this team and for you? I know Matt initially is probably just like build interesting things, see what happens. Is there now a clear sense of like, here's what this team should be achieving in the future?

• 26:39Speaker 03

  When I joined, my mandate was to build a business. And I was like, all right, if we break down the steps here, along the way, I have to build something interesting first.

• 26:48Speaker 03

  I feel like, okay, we hit, we hit the first thing. We've built something interesting. Now we go figure out the business. Uh, and I think, you know, that, that comes naturally to Google in terms of thinking about, you know, distribution, how to monetize it, how to commercialize it. There's different pathways, whether it's cloud or workspace or even the consumer route. And so for me, I think these are the things that we want to be really thoughtful about because the machinery is already there. And for me personally, it's really about, there's something super exciting here. We need to deepen that, right? Like while in parallel, we should be thinking about commercialization. Let's deepen this user experience that so many people have really sort of latched on to.

• 27:30Lenny Rachitsky?

  So let's follow this thread and talk about just where you think the product goes broadly and especially the audio overviews. What's on the roadmap near term? And then where do you imagine this in the future? What's kind of the big vision?

• 27:41Speaker 03

  Yeah, yeah. I have a really ugly slide. I wish I could show you. I made it two years ago and it was lime green. I don't know, you know, what I was thinking. I think I was trying to go for like ungoogly was the vibe. It's like, hey, you're gonna do different things this time. And the reason why that slide is so important to me is because it has my, my vision. Like the thing that I, I wanted to do from the get go, which was, I imagined that in the future, you could have an AI editor service, right? Fully remixable, any input, any output.

• 28:15Speaker 03

  Right? And that that to me is sort of like a really powerful sort of core nugget where if you imagine you could take anything, whether it's video, audio, your emails, your LinkedIn, your Twitter, right? The world of things that we care about. You have an AI interface that allows you to shape it and say, look, out of these things, make me a blog post. Out of these things, make me a tutorial video. Out of these things, make chatbot. Right? I think there is like something interesting here where from for most of the things that I'm hearing people say they want to do, it's usually that. It's like take something and make it into something new. And so we're going to pursue that. Um, but even more tactically, something I'm really interested in is thinking about, hey, how do we bring this to mobile?

• 28:58Speaker 03

  Right? The mobile app to me is such a big gap in terms of the experience today, which I think is like understandable considering where we're at in like the product development cycle. But I think that's the next horizon is, hey, how different is the mobile experience and how how can we make it interesting?

• 29:13Lenny Rachitsky?

  Hmm. What I'm imagining as I talk, I participate in this podcast conversation with these hosts uh as a mobile experience.

• 29:20Speaker 03

  Yeah, yeah. I mean, that's that's one of the things that we're definitely, we're definitely experimenting with different forms. And when we demoed it at I O, you could interrupt it. But we're trying to be really thoughtful about, hey, what does that interrupt actually look like? Like, what do people want when they do this? And um, you know, I I was thinking about the next set of improvements we're going to ship. And the first thing that I thought was let's ship a bunch of knobs. Right? Like, to me, it was like what I was hearing people wanted. They want knobs. They want sliders. They want text boxes. But then when I looked at the mocks, I was like, oh, great. Like, this doesn't feel magical. It almost doesn't feel like the same thing that we've shipped so far. And so for me, I'm actually taking a little bit of time to think about how do we make

• 30:04Speaker 03

  even that thing and control experience much more magical and delightful?

• 30:10Lenny Rachitsky?

  Interesting. So yeah, the knobs I imagine are just like, go deeper, uh, be happier, get less serious, more serious. Cause yeah, cause right now it's like a one shot. Here's my doc. Here's the one podcast episode you get. It's the only version you can have.

• 30:24Speaker 03

  Yeah, yeah. And I, you know, I think it's funny because I was like, you know, and the knobs seemed really apparent. And I was like, is that it? Is that that what people want?

• 30:32Lenny Rachitsky?

  I love that. I love that approach. I'm excited to see where you go with it. What I especially love about this vision you're describing. It's super resonates with this insight I've had where so I've had a newsletter. And then I had just a podcast that's just audio and then I added video. And I I realized that there's just people that I just want to watch stuff. And there's people that I just want to listen to stuff. And then there's people I just want to read. I don't want this podcast to shut up. I need I want to read it. And basically what you're describing is here, well, here's information and we can deliver it to you in any medium you like. Could be blog posts, could be maybe a tweet, could be a podcast, could be a newsletter, blog post.

• 31:11Speaker 03

  Yeah, I mean, that's exactly it. And I think even for myself, it depends on my mood, right? If I'm going on a walk, yeah, I want audio. But if I'm at work, yeah, most of the time text is good. And it's kind of interesting to think about today. The formats are not as malleable. I have to accept it in the format, you know, that you give it to me. But if I had sort of that power myself to choose, hey, you know, thanks for this hundred page doc. I'm going to go ahead and turn it into an audio overview. Actually, it it sort of shifts, I think, the dynamic between the person and the knowledge that's given to them. And there are there are actually many, many times that I've been given a hundred page doc and I just didn't read it. So I wonder, I mean, just frankly speaking, it's one of my, one of my funny stories is actually when I first joined labs.

• 31:56Speaker 03

  Josh did give me a 50 page doc. He was like, here's, you know, what I think about this, this, this, it was like basically his vision. And instead of reading it, I just grilled him. I was just like Q and A him like a chatbot. And Josh was like, it's in the doc, right? But the chat is easier, Josh.

• 32:15Lenny Rachitsky?

  That's so funny. Chad bodied him. Let's talk a little bit more about different use cases for this, just so people potentially get inspired. So it feels like the original use case was a scientific paper and it creates a podcast about it. So you don't have to read this whole thing. Is that was that one of the original use cases or not?

• 32:32Speaker 03

  Um, I think it's like one of the common use cases.

• 32:35Lenny Rachitsky?

  Okay.

• 32:35Speaker 03

  I think that's one of the ones where it's like. I I think it's kind of interesting because it's like everybody wants to catch up on the latest on AI. We want to try to keep up to date on sort of the published papers. But most of the time, it's like reading the paper can take time, it's dense, complex, you have to break down the concepts in it. And I think that is a is a highly sort of extensible use case. I would say like the number one, the number one use case is actually a lot of students taking their study materials and wanting to turn it into a an audio guide.

• 33:08Lenny Rachitsky?

  So two recent use cases that are hilarious slash amazing, just to point to people in the show notes, just like they came to mind as you were talking. One is Andrej Karpathy, who's a leading AI thinker and big fan of which you're building. He keeps tweeting about how much he loves it all. He created this whole podcast series on history of mysteries, I think it's called History of Mysteries.

• 33:27Speaker 03

  That's right. It's Histories of Mysteries.

• 33:30Lenny Rachitsky?

  Where he basically took like the Wikipedia stories of of all these mysteries of history and then turned it into like a 10-episode podcast you could listen to on Spotify.

• 33:39Speaker 03

  Yeah.

• 33:41Lenny Rachitsky?

  Amazing.

• 33:42Speaker 03

  That that was amazing. Actually, when I saw it, I was like, wow, this is this by itself is like a great product. And I I think like the original potato podcast that that we had created was actually, I think it was because of the Wikipedia article of the day was about potatoes. So there is, there is something there. It's like, hey, if you want to learn something new today, maybe you could just listen to it on your drive to work or something.

• 34:04Lenny Rachitsky?

  Yeah, it's so good. Okay, the other one is the most funny is someone just uploaded the words poop and fart. Repeated, right, for a long time. And the host just came up with a really a kind of insightful analysis of of it.

• 34:20Speaker 03

  Yeah, I I actually, when I saw that one, I was about to go to bed. And I was like, all right, if I listen to this now, and this is a negative one.

• 34:32Speaker 03

  I'm going to have to get my laptop and work because I can't, I can't sleep, right? If I, if I feel like there's something here that's like, we need to address right now. Then I was like, okay, I'm just gonna do it. So I listened to it and I think, I, I listened to a lot of these and they always make me feel some kind of way. I laugh a little bit. There was, there's one where it's like they make scary voices. That one was cool too. But this one, this one was one of the ones where I genuinely was just really delighted at the output. Because it was so delightful. I, I don't know if you listened to the whole thing, but there's a segment where um, one of the speakers says, hey, it's kind of like, you know, you lean into the bizarre or something.

• 35:12Speaker 03

  And he, and they're describing walking past a, a shop that was full of rubber ducks wearing uh, costumes. I'm like, oh yeah, it's, you know, it's silly, but you just want to lean into it. So I walked into the store and I was like, oh, it's a great analogy, actually. For sort of a document that just says poop and fart, like, there's something interesting here.

• 35:32Lenny Rachitsky?

  It's so good. And just they like their analysis of structure and patterns and like, I love that they have to come up with a 10 minute conversation around anything you give them. Like, it feels a little mean.

• 35:42Speaker 03

  Yeah, yeah. It's, it's true. I, I don't know if you've seen the chicken one, but it's pretty similar. Um, somebody uploaded, I think it's on threads. Uh, it's a PDF that looks a lot like a research paper, but it just says chicken. Whole document says chicken. I, I would give it a listen. There's a funny segment where they're like, get this. It's a, is a paper, a research paper that has more chicken in it than KFC. Oh, it's good. It's good. It's so good.

• 36:11Lenny Rachitsky?

  Oh my god. Okay, I could keep going about this all day, but let's, let me, let me come back to the team that's working on this and let's talk about Stephen Johnson's role. So he's basically your peer. You two essentially lead this with the engineer, I imagine. Just like, what's his role? How does this work at this point? What is the team looking, look like?

• 36:29Speaker 03

  Uh, it's so funny. I'm, I'm so fond of Stephen and I, I tease him a lot that um, whenever people ping me and they want to have a meeting and I don't want to go to the meeting, I say, oh, I'm not the PM, it's Stephen Johnson. Well, that's a good guy. And he was like, so that's why all these people have been picking me. Um, but you know, kidding aside, uh, Steven, Steven joined. I was like, wow, this is super interesting. This has never happened to me before. There's this very accomplished distinguished person that I really respect. Love his books, love his writing. And now he's going to come work with me. In a capacity where I was like, I'm just not sure what he's going to do here.

• 37:10Speaker 03

  But the, the thing about Steven is he is the most curious, most respectful. And really just full of ideas kind of person. And so when Steven joined, um, a lot of what was really interesting to me is I would watch how he worked. The way that he thought about language, the way that he thought about information, the way that he thinks about knowledge and sharing that with others. Because Steven's books are like really incredible. It's like part mystery, super science, right? It's just very cool. And I was watching the way that he works and, you know, he does all of this research. And I thought, maybe this is, this is the nugget, right? Maybe I watch Steven and I look at the way that he does these things.

• 37:54Speaker 03

  And I look at how much time it takes for him to do it. And then I make it my own metric to crunch that down, right? To bring that expertise to every day people who are not Steven Johnson. And so, so I, I, I learned a lot just from watching Steven of like his craft and thinking about, okay, how do I make people really, really good? At de-densifying information. Is that something we all do every day, right? In our own little ways, maybe not in like a Steven Johnson way. But from the get-go, I told him this. I was like, Steven, I think you're the product. I think it's you and I'm going to follow you around. I'm going to watch everything that you do. And we're going to try to figure out how we use technology to build it.

• 38:35Speaker 03

  And he always laughs about this and it's really funny because he does have some very interesting workflows. And I'm like, I do not see any person ever doing anything like this. Like he, he always talks about his Readwise with like his 8,000 quotes or something. I'm like, that's extreme. That's crazy. And you know, I have like Post-its that like, sometimes they're crumpled in my pocket. Like this is, this is like the average normie flow. And what I learned from Steven is like, hey, there's something powerful about doing that, right? Like thinking about this person who has like a super flow and trying to bring that, bring that to people. And so Steven is also just like such a great partner with ideas. Like I always tell him ideas and I'm like, hey, crazy idea of the day.

• 39:17Speaker 03

  And he'll just banter with you and he'll be like, well, what about this? How can people do this? And so it feels, it feels really incredible to have him on the team.

• 39:25Lenny Rachitsky?

  Is there any lessons here for how you think you'll want to build product in the future? Or how teams might find their own Steven for building their own product? Or is this kind of a unique case, you think?

• 39:36Speaker 03

  I mean, I think that it's just crazy to, to think that you could sort of invite, you know, someone like Steven and become part of your team and have him sit with you every day and have him answer your questions about how he does things, right? But I'll say that the, the broader lesson there for me, it's like something we've tried to live by every day, is really the how do you get, you know, users or people and like really sit with them for meaningful periods of time? Because I think that has been so crucial for me, not even just for Steven, but even with students. Just follow students around, watch them do homework, watch them study, talk to them about how they feel when they study. I think just being able to do that at a really regular and sort of intentional capacity

• 40:21Speaker 03

  makes a huge difference to like the product insights you'll come up with.

• 40:25Lenny Rachitsky?

  Can you just give like a brief bio of Steven for folks that are like, who are you even talking about? I probably should have done that before I asked this question, but just like, what's like the, the TLDR of who Steven Johnson is?

• 40:36Speaker 03

  Yes. Oh, I think Steven is.

• 40:40Speaker 03

  Aside from one of the smartest people you'll ever meet, he has written fourteen books. He's a New York Times bestselling author and speaker. He has a show on PBS. I can't remember what it's called, but it's on YouTube. It is like really incredible. And he is a, he's a journalist. Actually, funny story about Steven is when I was about to join Labs, I really wanted to join Labs because I wanted to join Josh in whatever he was building. Josh told me, hey, you know, to give you an idea of what we're doing here, he sent me a couple of, of things to read. And one of them actually ended up being Steven's article that he wrote. Um, the one about how AI is mastering language. Should we trust what it says? The New York Times piece.

• 41:21Speaker 03

  And I remember reading it and being like, yes, this is the thing. I'm gonna work on it. And so before Steven had even joined, this was actually the piece that I was like, I want to go do that. I'm gonna go do that in Labs. And then he ended up joining and it was just the craziest thing.

• 41:36Lenny Rachitsky?

  That's so interesting. I imagine many PMs would be like, I don't need someone else on my team that isn't building or coding or product managing or designing. Just this other chef in my kitchen. Like, no. And I love this actually worked out really well. Like, it feels like he's a perfect combination of very smart, future-thinking, insightful. Plus, he's almost a model for what you want people to be able to work like.

• 41:57Speaker 03

  Yes, although I will say, like, to be very fair, I mean, Steven and I also disagree on a lot of different things. We've clashed a bunch and I think this is where I feel really grateful to have had the opportunity to work with him and to really grow with him in that way. And I used to make fun of him. I was like, have you ever had a coworker before, Steven? Because I think you've been an author forever. And, and it's really, it's really funny because he's so, he's so down-to-earth and he's so humble that even with our disagreements, we always reach. Like, at the end, we're always aligned, right? Even if, even if we don't agree about something, we're aligned on the next step. And that's really powerful, I think particularly for, for product professionals, right?

• 42:41Speaker 03

  For PMs where it's like, I don't want to disagree at the end and then not have an outcome on top.

• 42:47Lenny Rachitsky?

  I love that. Okay, I have just a few more questions. I know you have to get back to work to building this thing. One is, there is this hilarious moment of, uh, where the hosts basically realize they're like, we are AI people and I'm, I'm scared what, what's happening. I tried to call my wife and she didn't answer. Talk about that and I want to extend to just like red teaming this thing. How do you like red team this thing and make sure it's not doing things that are bad for the world, bad for Google, bad for the product?

• 43:21Speaker 03

  Yeah, yeah, it's such a such a good question. I think I I heard that it was over the weekend. I think it was on a Saturday or Sunday. And I remember hearing it and thinking, oh, my goodness. You know, this is actually one of those moments where you feel like we're at a fork in the road. And I I hadn't read any of the comments. I just heard the audio. I think I saw it on Reddit first and then I saw it blowing up on Twitter. And I remember thinking, what is the attitude of like the world now? How do we feel about this type of audio? It was really the first thing that I thought. And so I actually spent most of that morning reading the comments, reading on Twitter. And I was thinking about, you know, what is what is the right thing to do, right?

• 44:05Speaker 03

  Like, should we say something? Should we not say something? And I I basically just went with what I thought was the right thing to do, which is I think people are getting to experience this technology for the first time ever. And of course, they're going to try to do things that are maybe something that we didn't think about. We didn't think they would do or we didn't think that they could do, right? With like the jailbreaks and stuff. But I think that's such a natural part of of human curiosity. And I think like there was a moment where it's like, oh, do we like pull this back? Right? Because this is like, oh, is this like a safe thing? But because the response, I think, was that people were like, oh, it's saying that because of the sources.

• 44:46Speaker 03

  Like, this is no like AI realized they were alive or something. It's actually that it was in the show notes. They were supposed to act this out. And so I felt really much more confident that like, oh, people get it. Right? Like, that's exactly what this is. Somebody uploaded some notes and said, hey, this is the end of the show. So, you know, make sure that you you act accordingly. And that for me was enough for me to be able to say like, okay, I'm going to address this publicly and say something on Twitter and make sure that people know that like, hey, I've seen it. And I get it. Like, this is like what people do with it. In terms of red teaming at Google, though, what I will say is we have, you know, this is one of the things that we care about the most. We have huge teams that work on red teaming it.

• 45:29Speaker 03

  We test for about as many areas as you can think of that we imagine uh we need to do in order to make it safe. And I think certainly, I think we'll run into situations where, okay, we didn't think about that. Or like, we haven't tested that to the fullest extent, and we just add it to the test cases. I think in in if there was ever like a scenario where we're like, oh, this feels pretty unsafe, we would pull it back. But I think hopefully, you know, we've built it in such a way where we don't have to do it.

• 45:57Lenny Rachitsky?

  Awesome. Yeah, I love like people are just looking for ways to make it do stuff that is feels wrong. When it's just like, hey, we're just summarizing a thing for you in a really delightful way. Where do things go from here? We talked about it a bit. You're gonna maybe it sounds like maybe there's a mobile app in the works. There's maybe non-nobby knobs that you're working on to give people more power. Uh, for folks that are just excited about using this, where where's it going? Who are you aiming to build for? Just what should people know about the future of BookLM?

• 46:26Speaker 03

  If there's like one thing I could say is that we are continuing to learn from users every day. So please keep using it. Please keep sharing your feedback, whether it's on X or on Discord. Like, I'm there every day. Even if I don't respond, I've read everything.

• 46:44Speaker 03

  And and I say this in in the most non-creepy way possible, but I say this because we are just very passionate about trying to build the the right thing, you know, the best thing for everybody. Kind of going back to sort of, hey, who are you building for? I think that there's a lot of fun use cases, but there's a lot of actually very useful sort of game-changing workflows in here. And so we'll continue to build in that direction. I think particularly for educators, for learners, for professionals, right? Knowledge workers, as we call them. I think there's a lot there that we'll be able to execute against in the very short term.

• 47:17Lenny Rachitsky?

  Amazing. Okay, two final questions. If folks want to give you feedback, reach out. Uh, what's the best way to do that? Is it DM you? Is it is there an email, something else?

• 47:27Speaker 03

  Yeah, I mean Discord, join our Discord. Try me in. Yeah, we're there. You can at me or yeah, X also. It's pretty good. I just I just came back to it and kind of feel like I don't really know how it works now. So just a warning in case you DM me, I might just not know how it works.

• 47:42Lenny Rachitsky?

  On Twitter, you mean specifically?

• 47:44Speaker 03

  On Twitter, on Twitter.

• 47:45Lenny Rachitsky?

  Okay. And then final question, how can listeners be useful to you?

• 47:49Speaker 03

  I mean, please keep trying it. Try it. Share your feedback. Whether you find it useful, not useful, you think it's annoying, tell me. These are the things that I've heard, you know, for the last year that's helped me to make it better. And so if you're inclined, please go ahead, try it. Notebook LM dot google.com.

• 48:05Lenny Rachitsky?

  You're building an amazing product that is extremely delightful. It's one of the most interesting things on the internet today, which is a very high bar. And you've done it. And I'm really excited to see where it goes. I'm going to keep playing with it. I've been using it for my newsletter in a really fun way. And so, great work. And keep it up. And we'll see where things go. Thanks for, thanks for coming on. I know you have a lot of work to do.

• 48:25Speaker 03

  Oh, that's amazing. Thank you, Lenny. Thanks for having me. And, you know, thanks so much for your curiosity about it.

• 48:31Lenny Rachitsky?

  Stay curious. Bye, everyone.

• 48:35Lenny Rachitsky?

  Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.