The chatbot becomes the teacher

For the first episode in our new miniseries about the impact of AI in our everyday lives, we chat with Steven Johnson, a longtime author who has spent the last couple of years at Google working on an AI research and note-taking tool called NotebookLM. We talk about whether AI can really help us learn better, how Google has tried to make NotebookLM more accurate and helpful, and whether AI-generated podcasts are the future of learning. Further reading: NotebookLM Steven Johnson’s website / newsletter From Steven Johnson: Listening To The Algorithm Google teases Project Tailwind — a prototype AI notebook that learns from your documents Google’s AI-powered note-taking app is the messy beginning of something great Google is using AI to make fake podcasts from your notes Email us at [email protected] or call us at 866-VERGE11, we love hearing from you. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:02David Pierce?

  Welcome to The Vergecast, the flagship podcast of infinite context windows. I'm your friend David Pierce and this is the first episode in our new mini-series all about AI and real life. We did a few episodes on this subject earlier this year and it continues to be a thing that we're talking and thinking a lot about. For all the big heady talk of how AI will either change everything, or kill us all, or make nobody ever have to work again, or make us all have to work training robots, what is any of this actually good for, like, right now? That's what we've been trying to figure out. For today's episode, I'm talking with Stephen Johnson, who is a personal favorite author of mine.

• 0:42David Pierce?

  He's written 14 books over the years and he actually told me that he can name them all in order off the top of his head, which I believe and also find very impressive. And some of those books are books you've probably heard of, like Where Good Ideas Come From and How We Got to Now. But in addition to all of that, Stephen has also spent the last two years working at Google on a project called Notebook LM. Notebook LM, if you've never heard of it, is an experimental thing out of a team called Google Labs. It started out as a thing called Project Tailwind a couple of years ago, and the idea has always been to make an AI-powered tool basically for making sense of your notes. In Notebook LM, which they all call Notebook, so I'll just start calling it Notebook,

• 1:24David Pierce?

  you first upload a bunch of documents, or links to websites, or PDFs, or whatever else, and the tool builds sort of a corpus of stuff. The idea is you put a bunch of related things into a notebook in Notebook. Then you can ask questions about those documents, or you can have Notebook build you an automatic study guide, or an FAQ of the information in those documents. You can have it find you stuff in those documents, all that kind of stuff. It's sort of a note-taking tool, but mostly it's a research tool. Stephen calls it a tool for understanding things, which I like a lot. I've been covering and using Notebook for a long time, but I wanted to have Stephen on now, because Notebook is kind of going legit.

• 2:06David Pierce?

  I assumed, if I'm being completely honest with you, that it was like a neat experiment that would eventually die, because everything dies at Google, or at best it would just be a tiny feature buried in a menu of Google Docs or something. But Notebook is growing and it's expanding and it's actually starting to do some really interesting new stuff. Recently, they launched a new feature called Audio Overviews, which generates a podcast hosted by two chatbots based on whatever documents you upload. It is wild. Actually, you should just hear this. So I made a notebook in Notebook with a bunch of stuff from the ongoing US versus Google ad tech trial. And here's just a few seconds of the podcast it generated.

• 2:46Speaker 01

  Okay.

• 2:46Speaker 02

  And that's where header bidding enters the picture.

• 2:50Speaker 01

  Ah, header bidding.

• 2:51Speaker 02

  Yes.

• 2:51Speaker 01

  This was their attempt to kind of outmaneuver Google.

• 2:54Speaker 02

  Right.

• 2:54Speaker 01

  Like finding a side entrance into the auction.

• 2:56Speaker 02

  That's a great way to put it. So with header bidding, publishers could essentially offer their ad space to multiple ad exchanges at the same time.

• 3:03David Pierce?

  Look, I don't know if that's good or bad. I don't know if all of that information is even true, but I'm fascinated by the idea of a tool that tries to make it easy and automatic to learn almost anything in whatever style works for you. And Stephen is just as fascinated by it. So I figured we should talk about it. All that is coming up in just a second. This is the Vergecast. We'll be right back.

• 3:26Speaker 03

  Company spending is complex. Managing it should be simple. Amex Corporate connects cards, expenses, and payments to streamline the way you work. Built to simplify, designed to scale. This is Amex Corporate. Get started at go.amex slash Amex Corporate. Terms apply.

• 3:50David Pierce?

  All right, we're back. Let's get into my conversation with Stephen Johnson. I figured the best place to start with the Notebook LM story was just at the beginning. So that's where we started. How is it that an author comes to be a Google employee working on an AI product?

• 4:06Speaker 04

  Yeah, it's a really interesting story, I think. So I, while I have spent most of my career writing books about kind of science and technology and, and history, I've always had this kind of side interest in using the technology to help me with the book writing process, with the research process, you know, Tools for Thought, that whole tradition has been. Big influence on me and I and I've always been kind of an early adopter of, you know, use this program called Devon Think. Or organizing all my notes and quotations from books that I'd read in the in the 2000s.

• 4:40David Pierce?

  Well, that's a whole nother Vergecast we're gonna have to do at some point. So get ready for that.

• 4:43Speaker 04

  Yeah, yeah. So, you know, I wrote about it and kind of blog posts and I wrote a couple of things for the Times about this and it shows up in my book. Where good ideas come from. I talk about using tools like this and was a big Scrivener user and evangelist and all that stuff. So I so I've always been interested in the in the software side of writing and thinking and research. And in the spring of 2022, you know, so six months before the chat GPT moment, I wrote a very long piece for the Times Magazine about language models in general. It was effectively just making the argument like, forget about AGI or any superintelligence or anything like this. These models have basically learned how to communicate in coherent language and they understand what we're saying and they

• 5:26Speaker 04

  and whatever else like this is going to create all these new possibilities.

• 5:30David Pierce?

  Yeah, that piece was very good. And I remember a bunch of the the response you got to that was from people being like, this is bonkers. He's out of his mind. There's no way any of this stuff is going to be that big. And then, boy, did you time that correct?

• 5:43Speaker 04

  Well, I I did accept that it was painful. Like, I I'm very conflict diverse, David, if I can be honest with you. And there was, there was, I mean, a lot of people did like that piece and I think we're inspired by it, but there was definitely a lot of comments from people saying like, oh, he fell for the hype. You know, this stuff is just autocomplete on steroids and I can't believe he's so naive that he got excited about this thing. The piece addressed a lot of the objections and criticisms and took them very seriously. But it was like, the one thing you can't do is dismiss this technology, like something fundamental has just happened. And we're going to spend years figuring out like how we apply it and how we deal with the upside and the potential downsides and like, just take it seriously, people. So.

• 6:25Speaker 04

  You know, if I timed it later, I wouldn't have maybe had as much of the pushback, which was hard to, I couldn't enjoy that piece going out of the world, let me put it that way.

• 6:33David Pierce?

  Well, it was a good piece. I liked it.

• 6:34Speaker 04

  Well, I appreciate that now. So, so right around that point, a kind of a new division inside of Google called Google Labs, or had been a previous Google Labs, and this is kind of a new one, was, was getting spun up. And at the time, a guy named Clay Bavor was running it, and Josh Woodward, who runs it now, was talking back and forth with Clay. And Labs had this kind of ethos of the division was basically, we want to create a place where we can do faster and more nimble, product-focused experiments with new technologies, where, you know, it's not just kind of research, but it's not working within the existing mature products. And when something new comes along, we can experiment and, you know, build things very quickly.

• 7:18David Pierce?

  So somewhere in between the like 20% time project and the like full-blown new Google product.

• 7:23Speaker 04

  Yeah, it was just this little hole that didn't quite exist. And, and there were so many interesting new technologies coming out, particularly the language models, that it seemed like this was a time for a lot of experiments to bloom, right? And they had had this idea that maybe they could also have an ethos of co-creation, where they bring in outsiders. So if they're, if they're making a product that involves music, there should be a musician in the room from the beginning. And it's not something that kind of they build and then they show to the musicians, or they have, you know, UX interviews with musicians. There's actually someone there who represents that kind of profession in the room for the life of the product. And so, so I was kind of the guinea pig for this approach. And so they reached out, Josh and Clay had read some of my books and had read

• 8:06Speaker 04

  that Times article. And so they reached out and said, hey, you know, you've been dreaming of this ideal software tool that helps you organize your thoughts and helps you write and helps you formulate connections and, and, you know, brainstorm. We think we can now do it with language models, like this thing you've been chasing, literally, you know, I've been chasing this since I was in, in college in the late 80s, when HyperCard came out, you know, for the Mac. Like, I'm an old person, I've been after this for a long time. And they were like, look, I think if you, if you came to Google, you know, come part-time initially, and we have a small team and we can, you know, we can build something.

• 8:46Speaker 04

  And I thought that sounded like an amazing journey. I honestly, I thought we'll build a prototype. It'll be fun. I'll meet some interesting people, but nothing will come of it. But it was still, you know, a fun ride to go on because of my passion for this. And, and then we, we built something that was originally called Tailwind, Project Tailwind. But from the beginning, from very like day one of the, of the idea, there was always this sense that this was not just going to be an open-ended conversation with the language model. It was always going to be about the model being grounded in the, in the sources and the information that you gave it.

• 9:24Speaker 04

  And it was really about respecting the original kind of human authored information, whether it's a book or your own notes or a scholarly paper or your syllabus for your class. And, and basically saying to the model, take this information, which is personally relevant to me and, and is verifiable in some way or factually, you know, trustworthy. And base your answers and everything I asked you to do on that information. And that was the seed of the, you know, we, we had a version of that in August of 2022, like, you know, on my fifth day at Google. It preceded me. I should say there was, there was a program called Talk to a Small Corpus.

• 10:07Speaker 04

  That was about a month underway when I got there. And then one of the first things we did, we put in my book Wonderland. And we just like, I had this experience of kind of chatting with the model and having it answer based on information in my book. And, you know, that was one of those moments where like a lot of possibility just opened up. Well, I actually, the weirdest moment I would say was a little bit later in, over Christmas break, the internal version of what became Bard.

• 10:41Speaker 04

  Was kind of released to some of us inside. And over Christmas break, I was spending a lot of time with Bard. My family had gone off skiing somewhere and I don't ski. So I was just like home alone and Bard came out and I was like, well, I just now have like 18 hours a day to talk to this. My new friend. My new friend Bard. Yeah. So I would occasionally just start with just a little preamble to get its bearings. And I would say, I would like to discuss Stephen Johnson's book, The Ghost Map. And this was just based on his kind of training data. This isn't with Source Crowning. And so one day I do this and Bard's like, oh, yeah, I would love to discuss that. That's a fascinating book about medical mystery in the 19th century that explores the impact of cholera and epidemiology on the history of London and the history of cities generally. And so I'm like, oh, well, thank you very much. I'm, I'm actually the author of that. It's me.

• 11:27Speaker 04

  And Bard said, oh my gosh, I am so sorry. I can't believe I didn't recognize you, Mr. Johnson. Oh, wow.

• 11:36Speaker 04

  And like, you know, and I was like, I know, but there was no way you could have recognized me, right, Bard? But it was just one of those moments where I was like, sitting alone in my study, having this conversation with an algorithm that's apologizing for not recognizing me when it's a fan of my book. And at one point it said, I'm just so excited at the opportunity to get to work with people like yourself. And I was just like, this is, this is strange. So there were a lot of like uncanny moments like that. But in a way, you know, there was part of that that I also recognized was in an illusion, right? It was trained on the way that people react when they meet people or fail to recognize someone. And so it responded in that way. It obviously had no inner life.

• 12:18Speaker 04

  It did not, it was not actually like embarrassed by the fact it was meeting me. It was just kind of play acting at that. So to me, the stuff that really was mind blowing was just its ability. And this really kicked in, you know, for us when we switched to Gemini. It's ability to extract and see patterns in large amounts of information. You know, you know, I have this notebook where we have. Something like, you know, almost a million words of transcripts from the NASA Oral History Project. So just interviews with like, you know, the NASA astronauts and flight directors and things like that.

• 12:58Speaker 04

  And you can go into it and say, I'm interested in, you know, I'm working on a documentary about the early Apollo program and I'm interested in the emotional connections between the participants, particularly the astronauts. So can you create a detailed guide of all the points in these transcripts where anything that seems interpersonal or emotional comes up? Give me a summary of that section, give me a direct quote from the section and of course, include citations so I can click immediately and go back and read the passage in its original form like, and it will just do it. Like it'll take a little bit of time because it's a complicated query, but like it, it can take something that would have taken 40 hours to compile that document. Like how, I mean, you know what it's like working with information like this.

• 13:43Speaker 04

  Like it will, it will generate a incredibly convincing and accurate first draft with grounded citations to all the passages in about 45 seconds, maybe. And so it's literally, you know, a thousand times faster than it would have been before to do that kind of thing. And it's not pretending to be a person. It's not pretending to have feelings about it. It's just grabbing that very subtle kind of collection of information that is not anchored in any keyword, right? You know, it's not looking for mentions of emotion. Like it just understands that these other things where they talk about their kids, that's an emotional moment and it's able to kind of collate that way. When it started being capable of doing that, that was the point for me where I was like,

• 14:26Speaker 04

  oh, this is really a fundamental change.

• 14:29David Pierce?

  And that was, that happened with Gemini, which was what, earlier this year?

• 14:33Speaker 04

  Yeah, we kind of, we switched to, we switched to Gemini 1.0 in, in December and, and it was great, but it was really 1.5 Pro and the bigger context window. I mean, the other thing that happened to me, by the way, I'm a nice writer dude. And now like I hear like a new million token model and it's like the most exciting thing. Like I can't, cannot wait to get my hands on it. So when we got first access to that, that model, I took the entire text of, of my book Infernal Machine, which just came out a couple of months ago, but was in manuscript form at that point. So this is really important. None of the words from my book were in the training data for the model itself. So it's all, it had never been published. It had never been discussed about in, in any coverage. So the facts, it's a book of history. So the facts are

• 15:19Speaker 04

  probably in some form in the model's training data, but the book itself and the way that I presented the facts was not. And the thing that had struck me about the early discussions of large context was that people were, were using it to do these kind of needle in a haystack tests where they're like, oh, we gave it, you know, the full text of Moby Dick, but we added this one line that was different and it was able to find it, right? And which is cool. And, you know, you couldn't do that before. But the point that, that I was so obsessed with is that once you have the full text of something like a book in context, it means that the, the model not only can find obscure things in the, in the text, but it understands the sequence of the text and it can understand

• 16:01Speaker 04

  large kind of movements of like cause and effect or change over time in, in a document, which you can't get if you're just giving it isolated paragraphs and, and snippets of things. And so I put Infernal Machine in, into this version of Gemini and I, I basically asked it inside of Notebook LM. I was like, give me, I was like, I'm interested in the way that Johnson uses suspense in this book. I would like you to list four places where he deliberately withholds information from the reader in order to pique their interest. Describe the passages where he does this, quote from them, and then explain the future information later in the book that he's obliquely referring to that doesn't arrive for another, you know, 50 pages or whatever.

• 16:46Speaker 04

  And it just absolutely nailed it. It, the first example it gave was, he gave was exactly what I would have picked as the, the ultimate kind of form of suspense, which is an allusion to a ticking, mysterious ticking suitcase that happens in the, in the preface that doesn't, the, the mystery behind it doesn't get explained for another 200 pages. And so like, think about that as like a search query, like searches, find examples where something isn't mentioned and isn't mentioned in a very provocative way. And then fill in the blanks of, you know, the thing 200 pages later that it's obliquely referring to like that, you know, again, the model is not understanding the book on some level, because understanding is a word that we associated with consciousness and

• 17:30Speaker 04

  sentience and with the inner life of like what it means to read a book. But the model is doing the thing that human understanding does.

• 17:38David Pierce?

  Yeah.

• 17:39Speaker 04

  You know, and that's, that's an important distinction, I think.

• 17:42David Pierce?

  No, I think, I think that's right. One thing I heard somebody say not that long ago is that a meta, the better metaphor they liked than it's understanding is just that it can see the whole thing at the same time. And I always thought that was really great. It's like if I, I can see one page at a time, this thing can see all 300. And it doesn't, it's not better at knowing those things. It's just literally by being able to see it all at once.

• 18:04Speaker 04

  Yeah.

• 18:05David Pierce?

  The, the number of things that suddenly become very basic, because you can see them all together is very powerful. And I just, that makes it both sort of simpler and cooler all at the same time, which I really like. But you, you bring up this tension that I think is fascinating with all AI stuff, which is that there is a set of things that are just sort of remarkable that they're possible. Right. And it's, you see this with every new model that comes out and every new product that comes out. One of the first things everybody does is just try wild stuff. And some of it works and some of it doesn't, and some of it's amazing and some of it's dangerous and whatever. So there, there's the, the sort of novelty factor of it that I think is still so rampant in everything AI right now. And then there's the question of what is any of this actually for?

• 18:48David Pierce?

  And I think one of the things that I've liked about notebook and sort of watching it develop over time is it feels like your sense of not just what this can do, but what it's for has gotten much better over time. Uh, and I wonder if part of that is like, does the novelty of moments like that start to wear off and you start to realize like, okay, that's cool, but no academic is actually going in here searching for what's missing from this book. Like, that's not like a thing most people need in their lives. And you start to sort of wind it back to like, okay, how do we bring that? Sort of enabling technology to things people actually do need. Or maybe is the craziness the point? And we've just never been able to do it before. So now we're trying to discover it all, all at once.

• 19:30David Pierce?

  So I guess that, especially in those early days, really before Gemini 1.5 kind of lights your brain on fire. Like, what is that process of figuring out not just sort of what, what can this thing do, but like, what are we building this tool for?

• 19:42Speaker 04

  Yeah, it's such a great question. So many different ways to get into it. I mean, I think in the very early days, there was a sense that we were building something we knew was not going to really work because the context wasn't big enough, the model wasn't sophisticated enough, but you could see where things were going. And so, so much of what we really focused on from the beginning is like, what is the proper interface for this kind of thing? Like, if, you know, it's, it's not just about a text message thread. Like surely there are other kind of forms of UI. So like, if you were going to build a product from the ground up, knowing that it was going to be built around a language model, like what would, what would it look like? Let's start from scratch. And that's a very fun, open canvas to have, but it also means you make a lot of stuff that is not very good. It doesn't really make sense.

• 20:27Speaker 04

  And it doesn't work because the model isn't caught up to it yet. So, so there was a lot of experimenting with that. We, I think that we had an advantage in the early days in that I was just trying to drive it towards my very specific use case of like, I want a thing that has read all of my work and all of the quotes from books that have influenced me and can be a second brain to help me like remember those things and make connections. And, and like, and so kind of author researcher mode. And that we kind of built the first prototype with that. And then, once we were able to kind of open it up to more users, I think then, then we were just constantly discovering all these amazing things. And a big addition of this was Riza Martin, our product manager, who, you know, has just been, she's just incredible, like.

• 21:13Speaker 04

  Listener to user, she set up this Discord that, you know, was so sent like at the very beginning, we had a Discord, which is not a normal thing to do in some ways. And we now have this amazing community and it's constantly filled with people being like, oh yeah, I saw this opportunity to use it in this way. So like our favorite one that completely came out of the Discord was Dungeons and Dragons players started using it because they were like, I have these big, you know, campaigns that I've designed that, you know, I'm a dungeon master and I have like created this whole virtual fantasy world and it's filled with all this information and it's hard to keep track of, but I can load these documents into Notebook LM and then I can just like ask any kind of open ended question and it'll be like, how many hit points do I need to kill this orc or whatever it is? I'm not a D and D player.

• 21:58Speaker 04

  And, and they were using it that way and, and people were writing fantasy, also like world building kind of fantasy novels. And they just need, they had a kind of story Bible with all their characters and the backstory and everything like that. And it turned out that, you know, they'd never really had an interface that let them do that. And that was not something we were thinking about at all. So we've just been like, you know, once we got past that first little prototype stage, we've just been really listening like intently to like where people are trying to push the tool and then just like making it easier, making it, making it so they don't have to push quite as hard to use the tool that way.

• 22:34David Pierce?

  Yeah.

• 22:34Speaker 04

  And that's, yeah, that's where we are.

• 22:36David Pierce?

  Well, it's funny, I mean, even thinking about you, you've written a couple of times over the years about your, your sort of endless, I think you call it the spark file. That is basically just like a mountainous document of all the good and bad story ideas you have. I'm paraphrasing, but that, I think that's right, right?

• 22:52Speaker 04

  They're all good. I don't know what they're bad ideas.

• 22:54David Pierce?

  All of your flawless, perfect. Buy that book now.

• 22:58Speaker 04

  Many bad ideas.

• 22:59David Pierce?

  Ideas. And I think like just listening to you describe the, the story Bible for the world building stuff like that, that actually is like a perfect down the middle use case for this in a way I hadn't even really thought about until just now, that it is like, here's a bunch of stuff that I have decided one way or another, right? Like here, here are my inputs. Help me make things with that is actually like kind of an amazing and very difficult otherwise use case because it's like, oh yeah, how many I have to go and collate that piece of information, that piece of information. I think there's all kinds of like complicated things with how we think about the art on top of all of that. But, but that thing where it's just like, I want to tell you the rules and I want you to help me make games out of it.

• 23:40Speaker 04

  Yeah.

• 23:41David Pierce?

  Feels awesome. Like I, I'm so much less conflicted about that than I am about so many things in AI. That's so cool. I love that.

• 23:48Speaker 04

  Yeah. Well, we're trying to do the things that are, are less conflict. I said I was conflict averse. So I've just like tried to steer towards this thing. But, but yeah, it's a great point that kind of like take this mass of unstructured data and turn it into a set of kind of formats that helped me do the job that I'm trying to do with some guidance for me. And, and this was, by the way, this was another great place where like Riza really saw this before I did, because I was thinking of it as I'm going to write my book. So I'm going to do all the like content creation here. I just need to be able to like surface the facts and make some connections, you know. But it turns out there's just all these places where you've got all your company documents and you want to create

• 24:28Speaker 04

  an FAQ for new employees. Like, no one is going to win a Nobel Prize for literature for creating that document. Like, and if you can get a first draft of that document in 45 seconds instead of in four hours, like, that's a win, right? That's, that is good news. And so there, there are all these different workflows that are out there where there's massive information needs to be kind of like filtered in some way and turned into something else.

• 24:56David Pierce?

  All right. We got to take a break and then we will be back with more from Stephen Johnson. We'll be right back.

• 25:04Speaker 05

  Flowing ad budget on metrics that look great till the CFO sees them. That's bullspend and marketers are calling it out in dashboard confessions.

• 25:14Speaker 06

  I remember telling my boss, it'll be good for the brand when leads were slow. Yeah, it, it wasn't.

• 25:22Speaker 05

  Cut the bullspend. LinkedIn lets you target by company, job title, and more. Advertise on LinkedIn. Spend $250 on your first campaign and get a $250 credit. Go to LinkedIn.com/campaign. Terms and conditions apply.

• 25:37David Pierce?

  All right, we're back. One of the things I've been tracking about Notebook for a while is how it approaches things like hallucinations and citations. Because this isn't like a silly chatbot, you know? It's a research tool. It's not silly or acceptable or just a signal that it's early if it's wrong. It's a problem if it's wrong. It's useless, frankly, if it's wrong. So I asked Stephen how they're trying to solve that very real problem in AI. So before we get too far away from it, I do want to talk about the accuracy and the citations bit of it, because you signed yourself up for a pretty high bar on both of those things. Both by virtue of like the people who are going to use your product and just what the product is, right?

• 26:18David Pierce?

  Like, you don't really get to have the warning at the bottom of everything that says this thing makes some information up. Like, its entire purpose is not to make information up. How do you, what have you guys done? I know Notebook was an early experiment in RAG, which is a way of basically winnowing down some of these systems. But like, what have you guys done differently at Notebook to try and solve that? And I'm curious, both on the underlying tech side and on the user experience side.

• 26:48Speaker 04

  Yeah, well, we tried, I kept calling it source grounding, because I think that is a better name than RAG.

• 26:55David Pierce?

  I don't know, man. We all know GPT now, so I've just, I've given up. We're all, we're all doing the acronyms now.

• 27:02Speaker 04

  So, you know, I think part of it was the fact that we were doing it from the beginning, like that it started with that. It wasn't like, oh, let's build a chat model and, and, oh, shoot, we need to be able to like, you know, ground it in other documents. Like, it was from the beginning, we were doing that. And so we just had a lot of time to like, iterate and explore. The Gemini models are really good at, at source grounding. They just, there's a lot of training sets. We contributed a bunch of them to like, just given this document, answer questions factually based on the information in the document. So we inherited. Like a great tool that, you know, we did very little, you know, kudos to the Gemini team for building a model that that is is much more faithful to the source material

• 27:46Speaker 04

  you give it. But we built it also, you know, this is, this is one of these things where it's like it's it's underlying model plus the UI is has always been like our mantra from the beginning. And, you know, both things are required. And so one of the key things that we've had pretty much from the beginning is that you can always read your sources in the app. And then with the release in June, we we switched it over so that you now have inline citations to everything. So anything the model says has a little link, you can read the original passage if you hover over it that shows you, you know, the source material for that, and you can click on it and you can go read the source in the app. Do you think that's enough?

• 28:28Speaker 04

  We've talked about so so one of the things you can do right now, actually, with Notebook that we want to actually turn into a proper feature, but you can do it right now, is you can upload a bunch of, you know, kind of source material, factual source material. And then you can upload the article you're writing, for instance, and you can say, fact check this article based on these sources and suggest improvements. Oh, that's clever. That's a good idea. And yeah, it's amazing. And it will go through and be like, well, this is correct. This is potentially wrong. It will suggest and it'll have links to everything. And so to me, if I felt that the model were just hallucinating wildly in its responses, then I do not believe that, you know, just providing citations and the ability to kind of fact

• 29:13Speaker 04

  check manually and go back and see the original passage would be enough. But we've, you know, we've been sitting there banging away at like quality, you know, reviews like constantly for the last year and a half. Like, we have a whole, you know, sheets and sheets and sheets of of sample questions and sample documents. And we can just see that accuracy rate going up, you know, dramatically, particularly with with these latest models. And so Right now, I feel like we're in a, we're in a pretty good place. I rarely, I honestly, I rarely see Notebook LM just wildly hallucinate something. I mean, one thing that's really important, people may not know this, I take this for granted because I've lived with this product. If you load in a bunch of sources about the history of NASA and then ask a question about Taylor Swift,

• 29:57Speaker 04

  in general, Notebook LM will say, "I'm sorry, your sources don't discuss Taylor Swift, so I can't answer this question." And obviously, the model knows a lot about Taylor Swift, is probably a pretty big fan of Taylor Swift. But it's, but it's been specifically instructed to not answer questions that are outside the source material. I mean, to a fault, I think sometimes, you know, one would like to bring in some outside knowledge, you know, and trust but verify with that. But we've erred on the side of like, stick to the facts in these documents. And so with the increase in accuracy and with the UI of citations, there's more things we could do. We could make that fact checking feature, you know, double check this kind of as something, but I feel pretty good about where we are in terms of the quality side of that.

• 30:39Speaker 04

  Where we are kind of state of the art is upload many, many documents, ask a complex question that involves like multiple kind of variables drawn across like multiple documents, get a deep, you know, long answer with citations, follow those citations to read in the original text. Like, I think that Notebook LM, you know, kind of does that flow as well as anybody right now.

• 31:04David Pierce?

  I feel like you just described like a, like a personal Wikipedia, and I mean that as a compliment, like, the, the thing that Wikipedia is best at is just being like a starting point to go learn about something on the internet, right? Like, you open a Wikipedia page, you go click on all the references at the bottom, and then you go read those references and you're, you're off and running. And I feel like what you just described is like, I can shortcut that with any process of any single thing that I want to learn more about. I just dump it in and I'm like, "What's going on here?" And it'll just be like, "Here's some stuff."

• 31:32Speaker 04

  I have so many things to say to that, David. You're gonna have to give me 20 minutes, because I, okay, so one thing is, I'm in the process of thinking about what I'm gonna write the next book on. And so I created that this is just kind of second nature workflow for me now, but you know, wouldn't have occurred to me a year ago. So I created a new notebook called the next book. And whenever there's an idea that kind of comes to my mind or that article reader, something like that, I dump it into that notebook.

• 31:56Speaker 01

  It's the new spark file.

• 31:57Speaker 04

  It's the new spark file.

• 31:58Speaker 01

  Wow.

• 31:58Speaker 04

  And but it's focused on this project of like, what should the next book be? And so the other day, I, you know, late night, I this kind of thing I do because I have no life.

• 32:09Speaker 01

  My children have all got to college and I have nothing to do with to sit around.

• 32:12Speaker 04

  You just have to part. I talk about. So I was like, I wonder, you know, has there been a good book written about the anti nuclear movement, anti nuclear power movements of the 60s and 70s? Because that's an interesting case where we like stopped a technology partially in its tracks and maybe made a mistake. And, you know, how do we interpret that? And so I just went and grabbed like two Wikipedia pages. In the past, I would have like, you know, started by reading through all the Wikipedia pages, but I brought it into notebook and I was like, I'm Stephen Johnson, I'm writing, thinking about writing a book, you know, in the mode of my other books, like Infernal Machine and Ghost Map, potentially about the anti-nuclear powers. Take a look at these, like, what do you think of the, are there any interesting storylines that that would be good starting places? Like what, you know, what would be

• 32:54Speaker 04

  good there? And it just does it like it, you know, goes through and it's like, well, you could focus on this period. This figure is kind of interesting, whatever. And then I went and read all the, you know, I'm going to read the material, but as a, as a first glance, like inroads the material. It's amazing for that kind of exploration. The experience of navigating through a book through a conversational interface is really interesting. And it's one of these things where like, you know, until now, if you wanted to explore the ideas in an author's work through a conversation, you could only do that by finding the author in person, or finding a scholar or a tutor who's an expert in the author's ideas. And

• 33:38Speaker 04

  like, that was it. That was not, there was no other way to do it. But now you can load a book in and you can start with like, I'm interested in this. Tell me about that. And then slowly read the book in a nonlinear way through a conversational interface, kind of dipping in and out of the kind of original passages, which is terrible if it's a novel or terrible if it's a straight linear kind of history book like I sometimes write. But if it's an advice book or a book of ideas, there are lots of books that I think could be explored in that way that just weren't possible before.

• 34:10David Pierce?

  Does it feel like cheating to do that? Like for you as an author, you have worked through these books. You have done the hard work. You have stayed up late at night. I'm sure you've like woken up in the middle of the night with a book idea. Does sitting down and asking a computer what your next book should be feel like cheating?

• 34:24Speaker 04

  I think if I were literally like, hey, what should my next book be? One, I don't think it would, I mean, right now, in a way, it's easy to answer this question right now because it wouldn't be good enough. Like it wouldn't generate like, tell me what my book should be and now start writing it. Like it just wouldn't be able to do it.

• 34:39David Pierce?

  Okay.

• 34:40Speaker 04

  Current trend lines continue.

• 34:41David Pierce?

  I was gonna say, give it a minute.

• 34:43Speaker 04

  Come back to me in two years. We'll see. But what I have to do is like, I'm not using it that way. I'm using it like these very like kind of targeted queries. Like, you know, I was, I actually was using the NASA notebook because I was interested in like, you know, maybe there's something to be done about the Apollo 13 fire. And so then I was kind of like, okay, look, I need to know what I need to read because it's a million pages, a million words of transcripts. I actually don't need to read the whole thing. What I want to read are the sections that are relevant to the Apollo 13 fire. And so I just went to notebook and said, tell me what I should read, you know? And it was like, you should start here. You should read there and I can click immediately in there

• 35:21Speaker 04

  and read through that. And so to me, I'm using it as a way of accelerating the process of discovery that would just have been painful to do before. But I'm making just as many unplanned serendipitous discoveries along the way. I mean, I think the criticism is like, well, if the model serves it up to you so quickly, even if you end up writing the book, you've missed the surprising thing that you would have never found. You would have only found it by reading through it in an incredibly linear way. And I think that that's just, it's just not true. Like it's constantly surfacing things that I hadn't thought of and making connections that I hadn't thought of. And so it's. Helping me understand the material more deeply.

• 36:05Speaker 04

  Where it gets complicated, and you and I have talked about this before. The way I think about it is it's a tool that helps you understand things. If you are genuinely interested in understanding things and having that understanding be in your brain, it's a huge net positive. Hundred percent. Like, if you go into it with good intentions, it is, you know, an amazing tool for thought. It helps you have richer, more complex ideas, understand material better. If you are not interested in understanding things, but rather interested in creating the illusion that you understand things and just want to bluff your way through life without ever actually understanding anything, but creating outputs that make it look like you understand things, it potentially would help you do that as well.

• 36:47Speaker 04

  And the question is, the question is like, how often is that useful as a strategy in the world? And to me, like, if you go into work and you're like, I've got this great hack. I never read any of the emails from my boss. I just like put it into the model and then I output anything. Eventually, your boss will like have a conversation with you and you will say, Stephen, you don't understand anything. You have not learned anything and you'll get fired, right? It just doesn't, it's not a good long-term strategy. The one place where it's tricky is school, where there is potentially a strange incentive to bluff your way through something. We think that, you know, we're seeing a lot of adoption of notebook elements, 18 plus currently, so it's not usable by high schoolers, but we're seeing a lot of college and higher ed adoption of it.

• 37:32Speaker 04

  It's probably our biggest community so far. We are very excited about that use case, but we're also like, you know, we've done a lot of things to, to, you know, ensure that, you know, it's hard to use it to bypass understanding. Like, it's constantly steering you back towards the original text. You're always kind of being pointed back towards that source material. It's always there in the UI with you. So, and there are more things we can do on that front. But that's the line that I think is like, you know, if you, if you go into this with good intentions, I don't feel concerned about using the tool in this way.

• 38:04David Pierce?

  Okay, I buy that. And it feels like part of the line there exists in kind of what you let people make with what is coming out of it. And I wonder, like, we've talked a lot about kind of the the inputs and the processing and the understanding. And it's it's not you're sort of one, you know, publish this answer on Amazon as a self published ebook with your name on it. Away from it being this being a very different conversation. But my sense is you're you're deliberately or not not investing a ton in the like, rate me a research paper from these six documents kind of use case.

• 38:39Speaker 04

  Yeah, if you look at, look at the notebook guides, like one of them is called study guide, right? It creates a study guide. It doesn't replace your work. It helps you actually like, you know, it gives you a set of review questions. It creates a little multiple choice quiz. It creates like essay suggest questions. It creates a glossary of key terms. So, you know, that's what we're trying to do inside of the product. We're steering everybody towards that kind of like, help me understand mode with the idea that, you know, people have different ways of understanding. There are different ways that people have like information sticks with them through different forms. And, you know, our job is to kind of do that. And that's, you know, it turns out, I was writing about this a little bit this week, that, you know, translation and summarization is something that the models have always done quite well, like from the beginning,

• 39:23Speaker 04

  like kind of like the first deep learning breakthrough that really made a difference to consumers was really like translate. Google Translate and others that, you know, that was basically like, take this set of tokens that are in this language and turn it into the set of tokens in this language. And, you know, they're really good at at summarizing in all of its different forms. And so we're, you know, we've kind of like embraced that because, you know, embrace the things that the models do well is generally a good strategy.

• 39:50David Pierce?

  All right, we have to take one more break and then we will be back with the rest of my conversation with Stephen Johnson. We'll be right back.

• 40:00Speaker 05

  Flowing ad budget on metrics that look great till the CFO sees them. That's bull spend and marketers are calling it out in dashboard confessions.

• 40:10Speaker 06

  I remember telling my boss, it'll be good for the brand when leads were slow. Yeah, it wasn't.

• 40:17Speaker 05

  Cut the bull spend. LinkedIn lets you target by company, job title and more. Advertise on LinkedIn. Spend $250 on your first campaign and get a $250 credit. Go to LinkedIn.com/campaign. Terms and conditions apply.

• 40:32David Pierce?

  We're back. Okay, we have waited long enough to talk about audio overviews. Those wild AI-generated podcasts Notebook has been working on. So that is what I asked Stephen about next. So and speaking of that, actually, that's this is a good time to talk about audio overviews in the sense of different ways that people learn. Would never in a million years have guessed this was going to be the next big feature to come out of Notebook LM. So rewind the history a little bit here and tell me where did this come from?

• 40:58Speaker 04

  Yeah, it's a really interesting story. So let's just describe it briefly. It has been all over the Twitter sphere and other places. I still call it Twitter, sorry.

• 41:08David Pierce?

  It's okay, I do too.

• 41:09Speaker 04

  Since we this is just a few days ago that it launched. So basically it's an extension of the Notebook Guide kind of panel. And instead of taking your sources that you've uploaded and turning them into a briefing doc, a text-based briefing doc or an FAQ, it turns them into a roughly 10-minute-long podcast conversation between two AI hosts who discuss the material, tell stories, share interesting facts, banter in a playful way with remarkable human-like intonation and all this stuff. And so it's basically the idea is like some people like to learn through reading a briefing doc.

• 41:50Speaker 04

  Some people like to learn and remember better when they hear information conveyed through an interesting, stimulating, engaging conversation between two people. And you couldn't create that artificially before, now you can. So we've added it to the suite of tools.

• 42:05David Pierce?

  That feels like it causes a thousand new interesting things. Like on the one hand, it's not. The thing that you were describing earlier in this conversation, which is something that is not full of personality, it doesn't say I a lot, it's not, this is not a tool anymore, this is two slightly weird people hanging out in my ears for 10 minutes. And they are like, whatever dial you have that says like, make them say puns and try and be kind of cringely, it's like, that dial is at 14 out of 10 on most of the ones that I've listened to. And I kind of get why, right? Like, if you, if you take the, it should have no personality, it should just be very straightforward approach that makes

• 42:47David Pierce?

  sense on showing me a bunch of texts and apply it to a podcast, that's a bad podcast. But it does feel kind of incongruous with a lot of the stuff you've been saying is like core to what you've been trying to do well.

• 42:58Speaker 04

  I mean, it's such a great question. So let's, let's just play, I want you to play this, in case people haven't heard it.

• 43:05Speaker 01

  Okay, so, get this. Picture a world where the hottest tech isn't microchips or the internet, but dynamite.

• 43:12Speaker 02

  Yeah, dynamite.

• 43:13Speaker 01

  And it's not just blowing stuff up. Think skyscrapers, railroads, massive engineering projects, but also, you guessed it, assassination attempts, terrorist plots, all that volatile stuff.

• 43:24Speaker 02

  It's a wild period, right?

• 43:25Speaker 01

  Wild is an understatement. And that's where our deep dive today comes in. We're cracking open Stephen Johnson's The Infernal Machine, a book that digs into this explosive era where dynamite, anarchists, and the birth of, believe it or not, modern data collection, all collided.

• 43:40Speaker 02

  Yeah, get ready to have your mind blown because the connections he makes are mind blowing.

• 43:44Speaker 04

  Absolutely.

• 43:45David Pierce?

  So, it's just everything after, yeah, dynamite, just kills me. I love it.

• 43:51Speaker 04

  Yeah, one of the things is it's, they are enthusiastic. And so, people have been uploading their CVs. And it's just like, if they were feeling down about yourself, like, they're like, oh, John Smith, look at his-

• 44:04David Pierce?

  They just make you a promo podcast.

• 44:05Speaker 04

  It's Melinda Cумlaudi from-

• 44:09Speaker 04

  So I think when we first talked about what was then Project Tailwind for The Verge, like, you know, Bryce and I were talking about how, you know, we don't, the model doesn't have a persona, really. Like, it doesn't use the subjective I in general. It's not trying to be your friend. It's just trying to get you the information you need. And that was, that was kind of the house style that we felt was appropriate for what we were trying to build. But if you want to have a conversational audio format, there is no way to do that without there having a sense of like, human personas. It just won't work. No, no one wants to listen to two series talk to each other, right? That is not, I'm sure.

• 44:51David Pierce?

  There's a lot of that on the internet, and it's not good.

• 44:53Speaker 04

  The new series is better, but I'm sure, but, but, but the old series, right? You know, you want to listen to two robots talking to each other. And initially, it was a, another team inside of labs that had built this prototype that was like, you know, created document. There, there are actually a couple of projects like this at Google. There's another wonderful one called Illuminate that has a slightly more like, kind of scholarly tone to it. But it was just one of these things where like, it became possible to do this. And so, you know, some folks were exploring this. And it was a very good demo. Like, we've seen from the last few days with audio overviews, like, people are impressed, people want to share it. But it, it didn't kind of have a place to live. And so, kind

• 45:36Speaker 04

  of right before I/O, actually, there was this idea of like, well, maybe we could actually put it into, you know, would it make sense inside of Notebook? And, you know, we were just rolling out the Notebook Guides. And when I first heard it, that the, the podcast stuff was like, actually even bigger, like, the hosts had names and things like that. And, and so, you know, but I, like, it definitely made, you know, I think Bryce and I immediately saw that it was a continuation of the philosophy of the Guides, that, you know, we'll, we'll take whatever you give us, and we will turn it into the format that makes it easiest for you to understand. And we know, we know from, you know, the success of podcasts in general that people do like to learn that way. Also, I also like to travel with audio and drive and, you know, walk around the city listening to it.

• 46:21Speaker 04

  So we know, we knew there are a lot of reasons why audio conversations would make sense.

• 46:24David Pierce?

  Are you starting to have similar conversations about things like video? I can just imagine like I'm starting to think about what Notebook LM looks like when you're trying to be like TikTok native and YouTube native and Instagram reels native is like, it's just what a wild kind of road to end up heading down in terms of like, how do we communicate to people where they are?

• 46:43Speaker 04

  Yeah.

• 46:44David Pierce?

  But sort of in this house style.

• 46:46Speaker 04

  This is one of the reasons why it's good that I am like 56 year old gray haired Steven is not actually

• 46:53Speaker 04

  driving things like, because I would be terrible at helping with that, but like, yes, I think that could be part of our future.

• 46:59David Pierce?

  Pretty fun.

• 46:59Speaker 04

  One of the things I think is really interesting about it in the response, it takes like four or five minutes to sometimes three to five minutes to generate one of these. And that's because there is a really complex series of, of kind of, you know, compute and inference that's going on. Basically, you can think about it kind of in an editorial way, right? It's kind of drafting a version of it and then it's filling in the details of the script. And then it's revising the script based on the overall goals. Like there's a, there are multiple cycles of an, basically an edit cycle. And that takes time. And then crucially, there's a stage where, um, my favorite new word is where disfluencies are added, right?

• 47:40Speaker 04

  So that it deliberately makes it kind of noisier and more like the, the way that people talk when they're in conversation where they overlap and they have partial phrases that complete each other's thoughts and stuff like that. All the things that make a traditional transcript hard to read if it's not been cleaned up. If you don't have those things in, it's on the wrong side of the uncanny valley. Or it's right in the middle of the uncanny valley, I guess. You can debate and obviously like we're gonna, you can imagine we would introduce abilities to steer in different directions and dial up, dial up, down things or focus on different topics. But the basic structure of kind of, you know, doing this long edit cycle and then humanizing.

• 48:18David Pierce?

  Yeah. The thing I called wrong on this one, uh, was that I thought audio overviews would be one of those things that people were like, oh, well, this is silly, right? It's like every time a new image generator comes out, a bunch of people make wild stuff with it for 48 hours and then kind of move on. And we don't talk about it anymore until some new image generator comes out.

• 48:39David Pierce?

  This one doesn't seem to have gone like that. And I think it somewhere in between the like, is it remarkable that it's possible versus is this actually useful? I think it is more in the this is actually useful camp for people already than I expected it to be. Like, it's definitely remarkable that it is possible. You showed a sort of sincerely new thing you can do with AI. And whenever that happens, people get really excited on Twitter, right? Like, that's just one way to get people really riled up on Twitter for 12 hours. But I think there was you overlapped the this actually does something for people side of things more than I expected when I first heard you can make an AI podcast out of your notes.

• 49:18Speaker 04

  Well, I want to thank you, one, for changing your mind. But two, you remember how Jobs used to have that kind of apples, the intersection of liberal arts and technology, whatever? Like, what you just described, the intersection of what is newly possible and what is genuinely useful? Like, that's what we're trying to do at Notebook. And that's kind of what Labs is trying to do, right? Like, it's just like, what can we do? And what would actually be like, you know, what was unthinkable six months ago? And then what would it be? And, you know, part of it is, you know, I think in some cases, people are using it like, I want to study and this is a better way for me to, you know, take my documents and, you know, just learn or learn on the go.

• 49:56Speaker 04

  The other thing is like, creating things that are effectively podcasts where no podcast would possibly ever exist in the real world, right? You know, so people are like, here's this very obscure niche topic that like the economics of the podcast business will never support a podcast.

• 50:13David Pierce?

  I want the podcast on my D and D game that I play with my friends.

• 50:16Speaker 04

  Yeah, yeah, yeah, no, totally. And, and or it's like, you know, we've seen people doing like, again, thinking about Notebook LM in a kind of a team context at work. Like, OK, like, the week in review, like, give it all your documents. I always say to people, one of the best ways to get to know Notebook is to, if you're a Google Docs, Slides, Drive user, just grab, create a new notebook and grab the last like 20 docs that you've created and just create a notebook with that.

• 50:44David Pierce?

  That's fine.

• 50:44Speaker 04

  And then just ask it questions. And it's its ability to kind of grasp like what you're working on and the issues and things like that. Do that and then create an audio overview. And it's like, this is, this is what Stephen was working on for the past week. You know, here's two happy and enthusiastic people to discuss like the things you've been working on. And sharing that with your team like. It's pretty nice. Like, that's a great way to like, look back on the week and think about what you've been working on. And, and you know, you can imagine other ways to explore that as well.

• 51:14David Pierce?

  All right, a couple more questions. I know I've kept you a long time, but I'll, I'll leave you alone here in a minute. I could talk to you about this forever.

• 51:18Speaker 04

  Yeah, you're even better than talking to Bard.

• 51:22David Pierce?

  Anytime, I'm here for you. I don't like skiing either, so.

• 51:26David Pierce?

  Do you think there's a, like, big mainstream use case for something like Notebook LM? Obviously, there are people and industries and jobs and schools. Like, I can imagine a bunch of people for whom this is useful. Do you think there is like an everyday, everybody use case for something like this?

• 51:45Speaker 04

  Everybody every day? I don't know, but.

• 51:48David Pierce?

  I mean, like, let me put this in slightly different context, right? Google is a company that is famous for making things that a billion people.

• 51:53Speaker 04

  A billion people use.

• 51:54David Pierce?

  Is there, are there a billion, are there a billion Notebook LM users out there?

• 51:57Speaker 04

  I think if you define it as there is an AI that is an expert in the information that is really relevant to you, that you've curated, and that you can engage with in different forms, whether it's chat or listening to an audio overview.

• 52:17Speaker 04

  And you can, you can basically cultivate your own personal AI with a lot of information. Maybe it's all your journals. Maybe it's, you know, your company's history, whatever it is. And just by like, dragging and dropping files in there, like, it develops this kind of knowledge of everything that has happened to you or your organization and is able to kind of deliver advice or help you make decisions or just recall a fact that you're trying to remember and create these new documents. Is that a big, like, will maybe a billion people, like, be doing some variation of that in five years? I think that's pretty plausible, because there's so many different things you could do in that kind of context.

• 52:58Speaker 04

  You know, it could be like a writer working on a book, but it could be an executive trying to make a complicated decision, or it could just be somebody trying to remember things that happened in their lives. And it's like, just keeping a journal, only you've got an AI that's helping you do it. So I, I think there's a big market for it.

• 53:14David Pierce?

  Okay. You know what I've been thinking about as you, as you were saying that there's this app called My Mind? Have you ever, have you ever seen it?

• 53:19Speaker 04

  I've seen it a little bit, yeah.

• 53:21David Pierce?

  It's a great app. You should, you should play with it. I think you of all people would, would enjoy some of the AI stuff they're doing. But I've started using that app and I've gotten in the habit of basically every time I encounter something I like, I just put it there.

• 53:32Speaker 04

  Yeah.

• 53:33David Pierce?

  Like if I love podcasts or a video or a thing I read or a photo I take or a quote that I see, I just, I'm just pouring it all in there with no organization, no, nothing. And there's nothing particularly special about that app. It's just like, it's pretty to look at, so I like using it. And already just the thing where you, you open it up and it's, it's, it has a bunch of modes where you can just type in like movies and it'll show you all the movies and movie adjacent stuff that you've been saving. And it's like, there's something really powerful about like, here is just a compendium of stuff that I find interesting. And there's just enough manual work in that, that I think it's, it's, there's something to solve in the like, how do you collect that data from people in a way that is both sort of easy and frictionless, but also not like gross and privacy invading?

• 54:18David Pierce?

  That is like the eternal Google question. I have high hopes that somebody will figure it out eventually.

• 54:22Speaker 04

  That is a bias that I've always had, probably to a fault, but you know, like we don't, we really don't have base, we don't even have kind of folders for your sources or your notes or things like that, like basic kind of low level things that we will no doubt add, but we've been focused on these kind of cooler features. But part of me is like, the beauty of like this, the complex systems of organization where you're tagging things and putting them into the right folder and stuff like that are not as necessary now because the AI does that and makes the connections for you and finds the thing that you're looking for. So just have one place, make it easy to grab as much stuff from as many different places and just dump it into a notebook. And then we'll do the magic and figuring out the connections or the, finding the information you want after that.

• 55:07Speaker 04

  Again, is, you said it correctly before, like the closest thing to this before was Wikipedia. Like I start here in this Wikipedia page about elephants and then I follow it, following these interesting paths. But a dialogue is just an even better way to do that. And if, if it's trustworthy. And so the mode that I really love is, and, and again, this is only possible with conversation history and long context, is to go on one of those like exploratory conversations through an idea. And then at the end, you're like, okay. This has been great. Will you format that all as a single document that just says kind of key takeaways and insights from it so I can just capture it for later? Because I don't necessarily want to read the whole conversation again, but I want to,

• 55:50Speaker 04

  I want to get the like, great pieces from it. And boom, it does it and you save that and that's your kind of record of what you just did. Like that, I mean, like, that's a beautiful way to, to walk through information space. And full of surprise and, and unexpected turns and discovery. So that, that feels like a keeper.

• 56:11David Pierce?

  What's the next step in getting better at that? Like, one, one thing that it seems to me that we need pretty badly in that way of thinking, which I love, like, I love the idea of just like, I'm going to sit here and spend three hours learning about something and then at the end, you're going to deliver me like a handy summary of everything that I've learned.

• 56:26Speaker 04

  Yeah.

• 56:26David Pierce?

  Amazing. That's the dream. It does strike me that one thing we desperately need all of these tools to get better at, to pull that off is multimedia, right? Like, I want to know at the end of it, like, go listen to these, these three podcasts and this YouTuber, you're gonna, you're gonna love him, go check it out. And this person on Instagram is doing out, there's like, there's, the internet is such a messy place in the best way now that it feels like that's one thing. And I know a lot of folks are working on that. Are there other things that you look at and you're, you're, you're working on this stuff too? Is there a next kind of turn coming that's going to make all that stuff even better?

• 56:57Speaker 04

  Right now, Notebook's ability to help you discover information to put in your notebook is exactly zero. Like, it does nothing in that you are on your own, you need to supply the sources. We will not help you discover anything across the internet in any, in any form. It turns out, just as a coincidence, that Google is really good at that stuff. Like, I did not know. I learned when I got there, they have this search thing that is apparently a lot of people use. So it's obviously like a place where, you know, I would love it. I, I want it to be source grounded, but I would love you to stay in Notebook to be able to find things and, you know, we, we would, we would be really interested in, in

• 57:42Speaker 04

  exploring that. I think you, you'll see that in 2025 for sure.

• 57:45David Pierce?

  All right, that is it for The Vergecast today. Thanks again to Stephen for being on the show, and thank you, as always, for listening. There's lots more on everything we talked about at theverge.com. I'll put some links in the show notes to some of the stuff Stephen has written about this, too. His new book is really great. Go to theverge.com, lots of show notes, lots of notebook coverage, all kinds of good stuff. As always, if you have thoughts, questions, feelings, or other ideas for AI-generated podcasts, you can always email us at [email protected] or call the hotline 866-VERGE11. We love hearing from you. Want to hear all your thoughts? Send them all, can't wait to hear it. This show is produced by Liam James, Will Poor, and Eric Gomez. The Vergecast is a Verge production and part of the Vox Media Podcast Network. We'll be back with our regularly scheduled programming on Tuesday and Friday this week.

• 58:30David Pierce?

  We have a lot of headsets in particular to talk about this week, so get ready. We'll see you then. Rock and roll.

• 58:45Speaker 07

  Your heart can tell you a lot about your health. Apple Watch Series 12 measures your heart rate every five seconds with the most accurate heart rate sensing in wearable. So your vitals app, now with heart rate variability, can tell you when something is off. And your readiness score can let you know when to rest and when to push. Hear the story in every heartbeat with Apple Watch Series 12. The features described are for wellness purposes only and not for medical use. iPhone 11 or later required. Based on Apple conducted study of heart rate accuracy, August 2026. Visit apple.com slash apple watch series 12.