Kevin Roose is AGI-pilled. Why aren't you?

My guest today is Kevin Roose, a journalist and author of the new book The AGI Chronicles. Kevin was a longtime New York Times reporter and host of the popular podcast Hard Fork, but he and friend of the show Casey Newton just left The Times to start the all-new tech podcast Machine Gods, which launched last week. Kevin and I talked about launching Machine Gods and what it’s like to go from mainstream media to being an independent creator. Of course, we also talked about his book, which is excellently reported dive into the companies racing to build machine intelligence. Read the full interview transcript on The Verge. Links: The AGI Chronicles | Kevin Roose Can AI replicate in your smart TV? | Machine Gods (YouTube) Introducing Machine Gods | Platformer Inside the biggest feud in artificial intelligence | The Atlantic The essay that started the AI race | Kevin Roose A conversation with Bing’s chatbot left me deeply unsettled | NYT Subscribe to The Verge to access the ad-free version of Decoder! Credits: Decoder is a production of The Verge and part of the Vox Media Podcast Network. Decoder’s producers are Greg Ott, Kate Cox, and Nick Statt. This episode was edited by Kabir Chopra. Our editorial director is Kevin McShane. The Decoder music is by Breakmaster Cylinder. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Support for the show comes from CrowdStrike. It's not a huge stretch to say that AI is the next major computing platform. Every platform shift changes the way we build software, and it changes security, too. CrowdStrike is defining cybersecurity in the AI era with AI Detection and Response, or AIDR, a solution for seeing, monitoring, and securing AI across the enterprise. Learn more about how leading companies are turning to CrowdStrike to secure AI and secure their business at crowdstrike.com/decoder.

• 0:34Speaker 01

  You're either all in on this model, or maybe you're building with another. You're either speed, or is it security? Or you're AWS AI. Don't pick a side. Pick them all. AWS AI is how.

• 0:50Speaker 02

  Great meeting. Who took notes? Anyone?

• 0:54Speaker 03

  With Monday Notetaker, nobody has to take notes again. It joins your call, transcribes it, and turns the whole conversation into a clear plan. Action items, owners, deadlines, all right there on your monday.com board, ready for the team. So the meeting ends, and the plan already exists. Meetings aren't just talk anymore. Go to monday.com, the AI workspace platform.

• 1:21Nilay Patel?

  Hello, and welcome to Decoder. I'm Neel Apatel, editor in chief of The Verge, and Decoder is my show about big ideas and other problems. My guest today is Kevin Roos, a journalist and author of the new book The AGI Chronicles, which details the recent and explosive transformation of the AI industry. Kevin was a longtime New York Times reporter and host of the popular podcast Hartford, but he and friend of the show Casey Newton just left The Times to start the all-new tech podcast Machine Gods, which launched last week. As you'll hear, Kevin and I have been friends for a long time, and we've often talked about his desire to enter the creator economy. So we talked about launching the show, what it's like to go from The Times to being independent. I wanted to get a sense of Kevin of what it was like to report this book, to try and understand these characters in this moment in time, and his idea of where it might all go next.

• 2:04Nilay Patel?

  Fair warning, this conversation gets pretty philosophical at the end, but I think you're gonna find it worthwhile. Before we get going, do me a favor and subscribe to Decoder on YouTube, where we put out new episodes every Monday and Thursday. Okay, journalist Kevin Roos on The AGI Chronicles, starting Machine Gods, and where AI goes from here. Here we go.

• 2:32Nilay Patel?

  Kevin Roos, you are the author of The AGI Chronicles and co-founder and co-host of the new podcast Machine Gods. Welcome to Decoder.

• 2:39Speaker 05

  Thanks so much for having me.

• 2:41Nilay Patel?

  I was listening to the first episode of Machine Gods, and you and Casey discussed the fact that technically you have the title of CEO.

• 2:48Speaker 05

  It's true. I won the coin flip, and we needed some, you know, when you start a company, as we have, you need to like list someone as the CEO, I guess, to know who to haul in front of Congress when our first scandal inevitably breaks. So that's gonna be me.

• 3:05Nilay Patel?

  It's funny, because you're a CEO in Decoder. That means I'm gonna have to ask you the Decoder questions.

• 3:10Speaker 05

  Okay, well, I've been a CEO for about five minutes, and most days I frankly forget I'm the CEO, but I'll...

• 3:15Nilay Patel?

  Honestly, that tracks with most of the CEOs on the show. They've all acted like they've only done it for five minutes. We'll come to it. I want to talk about the book, The AGI Chronicles. It is a meticulously reported book about the main frontier labs and the culture shift they've all undergone as AI has come into the forefront and become a much more commercial product in many ways. But let's just start with Machine Gods. You were a journalist at The New York Times for a long time. Obviously, I think the audience probably knows this, but you and I have known each other for a long time. I will say that since I have known you, your desire to be a YouTuber has been like top of mind. You're just like, how do I get over there? And I've always been like, run, like, don't do it. But you did it. You left The Times, you started your own podcast with our friend Casey Newton. It is the next generation of the very popular podcast Hartford, which was excellent.

• 4:00Nilay Patel?

  I'm excited for you, but what made you take the leap?

• 4:03Speaker 05

  Yeah, you have been trying to convince me out of this for as long as I've been trying to convince myself into it, and you have not succeeded, because here I am.

• 4:10Nilay Patel?

  I failed, utterly failed, yep.

• 4:12Speaker 05

  No, look, I think that for some very good reasons, I think media companies have long been a little bit wary of going all in on YouTube or any other video platform. And the ones that have gone all in have not necessarily come out of that with good feelings about it, because there are a lot of...

• 4:32Nilay Patel?

  Yeah, they pivot to video and they also into death routinely.

• 4:34Speaker 05

  Yeah, you pivot to video, or you just, you sort of waste a bunch of money doing things that don't pay for themselves. And so I think there are some like structural economic reasons for that, but I also think it's just like, there just aren't a lot of institutions that are equipped to like make that kind of a long-term bet. And so, you know, I've been like trying to YouTube pill people in media for many years, as you know. You've been trying to undo the YouTube pilling on your end, but I just think it's like clearly the most important media platform in the world right now. And it's a place where journalism has not historically had a central place.

• 5:14Speaker 05

  Like, you know, if you were to sort of look at YouTube as a whole and try to sort of assess what piece of it is dedicated to what you or I would consider like high-quality journalism, like it's a vanishingly small part of the overall ecosystem there. And so part of my mission is just to like crack that and figure out how to do YouTube and video in general, you know, like in a way that is sustainable from like an economic and a sort of creative standpoint. And then just to like experiment and do a bunch of things. Like, this is not a face that needs to be on YouTube, so I'm also hoping to like bring some other people in and like get, nail some formats. But like, I look at what The Verge has done there, I look at what people like Joanna Stern

• 5:58Speaker 05

  and Marquez have done there, and I'm just like, that to me is very exciting, and it's a space where there's a lot less competition than like text or podcasts.

• 6:07Nilay Patel?

  Do you think of Machine Gods, it's a podcast, you're doing it in partnership with NPR, which I want to talk about, but do you think of it as a YouTube product?

• 6:14Speaker 05

  We want it to be video first, and what that means basically is that we're like are gonna give a shit about what it looks like, to a degree that we have not always done. And we're gonna try to give people a lot of reasons to want to watch as well as listen. We obviously, like, we know that there are some people who just don't like video podcasts, and so we want to give the audio audience something that is enjoyable. But, you know, this is the, this is the direction of things. This, you know, you can't do a podcast now without doing video on top of it. And so we're gonna like try to really embrace that and make a good version of it that is as good as, you know, two middle-aged guys, you know, journalists can pull off on YouTube.

• 6:56Nilay Patel?

  Yeah, truly, the future of media is middle-aged journalists pivoting to YouTube. I feel the same way, because I am now making more video, and it's, that's what we're doing. It's gonna be great. The other reason to leave and to do this, and obviously people know I'm friends with Casey as well, I've been talking to you and him about it for a long time, is the AI story is here, right? There is a platform shift, there are new companies, new characters, new ideas to talk about in the world. It's funny, in that context, usually you get a media shift as well, right? The way we tell stories or the formats we tell stories usually shifts alongside the thing we are talking about, or the technology we're talking about. Here, YouTube feels like the most solid media foundation that exists, right?

• 7:38Nilay Patel?

  It's the one that's the same the most, even though it's changing and, you know, has YouTube Shorts, and, you know, now you can A/B test like everything, including your own face. But it's the one that is the most the same. Like Instagram has undergone some massive change, and it feels like that will continue happening. Does it feel the most stable place to build as you're trying to tell these new stories?

• 7:58Speaker 05

  It does. I mean, obviously, you'd never want to put all your eggs in one basket, and so we are also distributing video on Apple Podcasts and Spotify, and like we're gonna sort of be everywhere. But yeah, it feels relatively stable. I mean, I think the, the question I'm always wondering is like, what would it take to replace YouTube? Because not only would you, you know, many, many companies have tried to replace YouTube, but to do that, you would not only have to like build an experience that was better for creators and better for users, but you would also have to replicate like the entire, you know, visual corpus of humanity. Like, you would have to just get like 6 million videos from plumbers about how to fix like every individual sink, and then upload it to that.

• 8:43Speaker 05

  And it's like, it's a really hard thing to display. It's so ingrained for so many people, and I just think it's gonna keep growing.

• 8:50Nilay Patel?

  We'll come to this when we talk about AI, but a long time ago, I, you know, asked Google CEO Sundar Pichai, well, what are you gonna do if there's no more websites? And he was kind of like, well, we have YouTube. And there's just no arguing with that. Like, you sure do. And it is just gonna keep growing. What has been the most surprising thing about going from the times to what you might call the creator economy?

• 9:12Speaker 05

  Well, we're very new at it, but surprising to me so far is how much I like all the business stuff. Like, I think in journalism, you're always taught to stay in your lane. And, um, and I, you know, have never run a business before. Casey's run a business. I knew he enjoyed it. But for me, this is, these are new muscles and I'm, you know, I don't enjoy all of it, like negotiating the insurance quotes for our, you know, defamation coverage is not like what I would feel, you know, compelled by. But like the actual running of business, you know, there's something to this founder thing, you know?

• 9:46Nilay Patel?

  Yeah, it's a good time. It's funny in talking to you and to Casey and Joanna, who also just left and started her own thing. That comes up a lot. And then, you know, my perspective is why I've been doing this a long time. Like, what if I stopped running the business? And that doesn't seem to be a choice. It seems like you have to run. This is the future of media is, I think, taking control of your business and taking control of your audience. And there's something very powerful there. One of the things I want to talk about, and then I definitely want to talk about the book at length, is the audience for AI coverage. Because I see it in two ways. There's a group of people building the stuff that loves it, wants to consume all of it. There's a group of people that truly hates it and thinks all of it is fake. And then there's what I would call an expanding group in the middle,

• 10:28Nilay Patel?

  where the products have become more useful. You have to look at them and evaluate them as they are. They were not terribly useful a couple of years ago. Like, the promises did not match reality. But every day now, it can do one more thing that it couldn't do before. Whether or not that thing is like hacking all of the websites in the world, we'll set that aside, but it can do things it couldn't do before. And the audience seems to be changing. How are you thinking about the audience for your, for machine gods and how it might grow or change?

• 10:58Speaker 05

  I mean, right now, we're just trying a bunch of stuff that we think will be good and we hope that it will find its sort of natural audience. But I think you're right that the audience is changing. I think for a while, there was, there was a lot of demand for kind of cope. Like, basically, people telling you, like, this is not happening. You don't have to pay attention to it. It sucks. It's hurting the environment. Like, there, there was, and still is in some parts of the internet, like YouTube has a weird amount of this still going on. And Reddit is also like very anti-AI. But I think most people, or at least, you know, maybe a plurality of people, are kind of either resigned to or excited about the fact that this is not going away.

• 11:40Speaker 05

  And that is the audience that we're looking for. We're not trying to tell you that AI is amazing and we're gonna be in, you know, AI just, your AI utopia, uh, you know, under the new sort of machine regime. We're also not telling you that this stuff is fake and sucks and is going to go away on its own. And so we don't need to do anything to regulate it or be worried about it. Casey's phrase for this is AI realism. He thinks that there's like a big piece of, chunk of people who are just sort of curious and interested and maybe a little scared, but like they, their reaction to the fear is not to try to like bury their heads in the sand, but to try to like, understand the thing. So like, we started feeling this when we were doing Hard Fork, where like,

• 12:22Speaker 05

  we would do something that just seemed kind of like not our vibe. Like, we would do a, like a Claude code walkthrough tutorial, where we would just spend like half an hour building a website, like film ourselves and throw it up. And it would do great. So like, people clearly were curious, even if they're like, the, the comments on the videos were all from people who hated AI. There was clearly a lot of demand out there for people who were just like, I kind of want to know what's going on.

• 12:45Nilay Patel?

  Yeah, I hate that I can't stop watching this. It's a real theme inside of AI coverage, uh, for a variety of reasons. Let's talk about the book. Uh, I think that's gonna frame a bunch of the other questions I have. The book is meticulously reported. It is really fun to read. And the point of it, as, as I've heard you describe it, is to tell the history of the deep learning revolution, because all these people just talk in Signal and in extremely childish tweets. Tell me about that, that gap, right? Because this is an industry that is obsessed with documenting itself for computers and being legible to computers and not at all to other people, which seems wild to me. So just tell me about the reporting of this book, because it feels like convincing them to even tell you what happened was a slog.

• 13:31Speaker 05

  It was, yeah. So basically, the, the reason I wrote this is because I was scared that, uh, all of this stuff was gonna disappear. You know, I am a, I'm a history buff. I love to read books about like the Wright Brothers or the electrification of rural America, or, you know, the, the sort of moments of, of great invention during the PC revolution. Like, I'm such a sucker for that stuff. And, you know, when, what you notice as you read more of these books is just like, they all rely on like a lot of contemporaneous notes and evidence. You know, the, the lab notes from the scientists, uh, developing the atomic bomb,

• 14:09Speaker 05

  the, you know, the, the correspondence between the Wright Brothers and their friends and business partners that is sort of preserved and archived. And it was just clear to me, reporting on AI every day, that like, there was not going to be anything remotely similar for this stuff. Because A, they're private company employees, all of the, the people building this stuff. They're not, you know, government employees who are legally required to preserve their correspondence. B, the Signal messages and Slack messages are all set by the lawyers to auto delete after a certain period. And so there was literally going to be like kind of a mass disappearance with each passing year of these, I think, historically important artifacts and just moments and stories.

• 14:49Speaker 05

  And then C, like, the researchers are all too busy to keep notes. Like, they're all grinding because they're trying to like build the machine god. And so I thought like, as a journalist, but more so like almost as a historian, which is not a role I have ever officially played, but what I think I was playing in this one. I was just like, I gotta just get this stuff, because I have this sort of privileged ringside seat. These people talk to me. I know that they will sit down, some of them, and talk to me and tell me the stories. We did it, a lot of it on background, so they could sort of be protected from the consequences of talking to me. But this was sort of my, my attempt to like kind of furiously get all the, the stuff and put it in the archive where like future generations could look at it.

• 15:32Nilay Patel?

  The theme that really jumps out at me as I read the book was, there's a moment when AGI goes from being a thing they're all dreaming about, to being a thing that they all have to build. Right? And the shackles come off and they are, we gotta do this. And they all drive toward it as hard as they can. And you talk specifically about Google, where the Overton window moved. And suddenly, the people inside of Google, who thought this was all a pipe dream and pretty silly, started talking about AGI. That moment feels like the single most important moment in, in the history of AI. Where, okay, we can do this, goes from, that won't it be great to, we better go as fast as we can. Talk about that a little bit, because it, it feels under-reported on, that that was actually the shift inside of these companies.

• 16:17Speaker 05

  Yeah, I think that's absolutely right. I mean, people have been talking and writing about AGI or, you know, some, some precursor that wasn't called AGI but meant the same thing since literally antiquity. Especially since the 1950s, when the term artificial intelligence was born. People were talking about, you know, Alan Turing was writing about the thinking machine that could do anything the human brain could. So this concept has been around for a long time, but there was no technology

• 16:47Speaker 05

  anywhere on the market or in research labs that could actually do what people wanted it to do. That could take in information, learn to, learn to learn about that information and then use that to make predictions and act in the world. Like, people tried lots of different ways to do this and none of them worked all that well. And what really needed to happen was the transformer needed to come along. This architecture that Google developed in 2017 and published and then, you know, presented to, among other places, OpenAI, who took the idea of the transformer and ran with it. And, you know, that's the T in ChatGPT today. But like, that was the-

• 17:27Nilay Patel?

  Somewhere, somewhere, a Google PR person just like leapt out of a bush to hand you a check.

• 17:32Speaker 05

  They do love this one.

• 17:33Nilay Patel?

  They love it when you say it. They get so happy. Yeah.

• 17:35Speaker 05

  They love it. They love it less when you point out that although they invented it, they, they like absolutely failed to turn it into a product until after OpenAI did. But yeah, like that, it was a dream, but there was nothing to attach to the dream. There was no technology. And then the large language model came and suddenly it was like, oh, maybe this is the thing that could scale forever and keep improving over many orders of magnitude. And so that's the light bulb that goes off at like OpenAI and then later at Google is like, now we have the tool to do the thing that we have always wanted to do.

• 18:17Speaker 06

  Support for the show comes from QUO. You know, when you're juggling five things at once and your business phone rings and you just let it go? That call might have been your next best customer. That's why today's episode is brought to you by QUO, spelled Q-U-O. The business phone system built so you never miss a call. All your calls, texts, and voicemails live in one place, so anyone on your team can pick up a conversation, see the full history, and respond fast. More than 90,000 businesses rely on QUO to stay reachable and look professional every day. Setup takes minutes, and you can keep your existing number and add teammates as you grow.

• 18:58Speaker 06

  QUO's optional built-in AI agent handles after-hour calls, answers questions, and even books appointments, even when your team is offline. Money is on the line. Always say hello with QUO. Try QUO for free, plus get 20% off your first six months when you go to quo.com/decoder. Spelled Q-U-O dot com slash decoder. Support for the show comes from Framer. Picture this: you launch a website and your whole team's celebrating. Then the support tickets start to roll in and dampen the mood. That's when you could use a website platform that lets teams build, ship, and improve their sites faster than ever while getting every detail right.

• 19:42Speaker 06

  For that, you could use Framer. It's a complete website platform, not just a builder, so teams can launch and keep improving their sites in one place. Framer lets you prompt, inspect, edit, and publish in one place at a whole new pace. On Framer, AI agents and humans work in tandem. Agents bring speed and scale, while people bring taste, judgment, and control. And that all takes place in the same workspace where the real site is designed, managed, reviewed, and published. Framer is an enterprise-level solution that the world's leading brands like Perplexity and Miro rely on to build their websites. Learn how you can get more out of your site from a Framer specialist, or get started building for free today at framer.com/decoder for 30% off a Framer Pro annual plan.

• 20:27Speaker 06

  That's framer.com/decoder. Rules and restrictions may apply. Support for the show comes from Shopify. Starting a business requires you to be resourceful, and the more you flex that muscle, the more you start to see opportunities for growth. For example, launching your business with Black Friday approaching means you can take advantage of the extra online traffic. Shopify is there to help you make it all happen. Shopify is the platform for a record-breaking, reliable holiday selling season for businesses at every stage. Their pre-built website themes and AI site builder make it easy to launch a beautiful, professional-looking store. And you don't ever need to touch a line of code to do it.

• 21:11Speaker 06

  They also have a mobile app that lets you run your store from anywhere. Their AI e-commerce expert sidekick can help you manage almost any business task with a single prompt, including holiday discounts. With Shopify, you can easily sell everywhere people shop: online, in person, across social media, and even in AI chats. If you're ready to launch your business before Black Friday, go to Shopify.com/decoder to start your free trial. That's Shopify.com/decoder. Go to Shopify.com/decoder today to hear your first...

• 21:56Nilay Patel?

  You've got a line in there where you say, where you just point out that next token prediction has not failed yet, that it is doing all of the things people want it to do. We've obviously had to develop lots of technology around it, right? There are harnesses to make agents work, there's reinforcement learning, there's all this stuff around it, but that core technology has scaled. And the reason I bring that up is, you know, 2023, 2024, I would ask every AI person who would come on the show or that we would see on The Verge, can LLMs do it? Can they do the thing you want them to do? Because I don't think at that moment it was clear that they could. We're closer now. They're all openly talking about AGI. We are watching Meta's muse just walk its little butt around the internet whenever it wants.

• 22:41Nilay Patel?

  Do you think LLMs can do it? Do you think they can get there?

• 22:43Speaker 05

  I think they can. Like, I'm not certain about that, but they've been shockingly good and effective so far. If you, you know, if you go back and you look at the first LLMs, it's just like, you know, 99% of people thought, this is crazy, this won't scale. And then it just keeps working at every stage, at every order of magnitude. And, you know, I've asked people, like, what was it about next token prediction that was clearly superior to everything that came before it? And there have been people who have told me, like, this, this actually exists in, in the human brain, too.

• 23:23Speaker 05

  Like, there's a whole school of thought that says that what is happening inside the human brain is essentially next token prediction at scale. And obviously, it's more complicated. These are not one-for-one replicas of each other. But I think this is sort of the, the fundamental mystery at the heart of AI is, like, why is next token prediction so remarkably effective? And I think, I think we don't know. I kind of think it's sort of one of these cosmic mysteries, but that this is why all of this works, is because next token prediction turns out to generalize really, really, really well.

• 23:59Nilay Patel?

  There's a part of that where all these companies had to believe it. Daria Amodeh at Anthropic obviously wrote this paper, Big Blob of Compute, where he said, "You can just do it. You can just dump data and compute at this, and we'll get there." And everyone else had to be convinced of this. What path did they all take to be convinced of it? And are they all fully convinced of it now?

• 24:19Speaker 05

  I don't think they are all fully convinced of it now. I think they are convinced of it to varying degrees. Ilya Sutskever has this phrase, "Feel the AGI," that he would, like, walk, you know, floor to floor at OpenAI chanting in the old days and reminding people to feel the AGI. And I think the labs sort of feel the AGI to varying degrees. I think Anthropic is clearly the most AGI-pilled of the labs, with OpenAI kind of a close second. And then there's Google, which I think has always struggled to feel the AGI. And there, I think there are lots of reasons for that, but it's very funny. I love this, um, this, the section in the book where there's this internal memo battle at Google, where, you know, they would just go on their internal message boards and, you

• 25:04Speaker 05

  know, Googlers love to write these, like, manifestos and then, you know, like, snipe at each other's manifestos and things like that. So a researcher writes this memo called "Say AGI," which is all about how Google, if it wants to win an AI, if it wants to recruit the top technical talent, and if it wants to be safe about how it's developing AI, has to kind of admit through using this term, AGI, that what they are building is more than just a bunch of tools to, like, make your Gmail better and make your docs better and make your Google Photos better at recognizing, you know, the people's faces. It is all actually building up to something like a kind of human-level general intelligence. And then someone else immediately writes a memo that's like, "Don't say AGI. It's like a horrible, unscientific term."

• 25:48Speaker 05

  And so they go back and forth like this for a while, and you'll notice, like, even today, Sundar does not say AGI.

• 25:55Nilay Patel?

  No.

• 25:56Speaker 05

  Demis Hassabis says AGI. Sergey Brin says AGI. So I think there are kind of factions within Google that either think this is still crazy and we should talk about these things as narrow tools versus, I think, this growing and ascendant faction that is like, no, no, no, we're, we're building AGI, or at least we better say we're building AGI, because otherwise, all of our top people are going to go work for OpenAI or Anthropic.

• 26:22Nilay Patel?

  I think you and I had the same experience at Google I/O in May, where we watched Demis Hassabis end the whole keynote by saying, "We're in the foothills of the singularity." And then we both went and interviewed Sundar Pichai and said, "What does that mean?" And he was like, "Yeah, yeah, Demis."

• 26:37Speaker 05

  Take it out with Demis.

• 26:38Nilay Patel?

  He was like, "What a kooky character that guy is, building the machine god over here. Let me tell you about the new features in Google Assistant." And there's some big disconnect there that maybe explains everything about Google at this point in time. The reason I'm asking about this in particular, in this moment when they all clicked over to realize somehow we have to talk about it, is, you know, OpenAI and Anthropic were founded to do this. They were built to make this happen. They don't agree on how it should work. I saw you at your book party in New York, and you said that you could have retitled your book "Fuck That Guy," because they all truly hate each other.

• 27:14Speaker 05

  It's true.

• 27:15Nilay Patel?

  And yet, it seems like they have the exact same idea, right? That they, they can get there with this technology. The commercial imperatives on OpenAI and Anthropic are very different than the commercial imperatives for Google. And right now, as you and I are talking, it's not clear to me that they can get to AGI before they run out of money, or before core weave collapses, or before the bubble writ large comes to some other conclusion about the state of the economy. Do you think that their, like, religious fervor at AGI is keeping them from running a business? Like, if I ran Anthropic today, a logical thing you could say is, we should stop training new models and sell the very capable ones we have to more people.

• 27:59Speaker 05

  I think belief in AGI is a double edged sword. Yes, if you are super AGI pilled as the CEO of a frontier AI company, you are going to struggle to care about things like, what is my profit margin on inference? Where is my next slate of like enterprise customers coming from? Because how boring is that compared to like, we are going to build the superhuman machine intelligence that leads us into the post singularity future? On the other hand, I think belief is actually a key ingredient in AI success. Because belief and the, you know, the ability to act on that belief

• 28:43Speaker 05

  is what leads you to do these sort of bet the farm moves that can really pay off and have really paid off. So like, remember a year ago when everyone was like, oh my God, OpenAI is like spending so much money on compute and they, how are they ever going to pay for all this compute? And isn't Sam Altman overextending them? And even Dario was like giving interviews where he was kind of throwing shade at OpenAI for like spending so much goddamn money on compute. And then you fast forward a year and like, OpenAI is the only major AI company that is not severely compute constrained. Like, that belief that led Sam and his executives to bet and maybe overbet on compute last

• 29:26Speaker 05

  year was the reason that they are in a better position than Google or Anthropic or Meta or anyone else today. So I think there's, I don't want to say it's like always a liability to believe this hard because I think it actually does allow you to make the bets that do end up paying off if the, if the scaling continues to work.

• 29:47Nilay Patel?

  It's interesting because I might reframe that OpenAI decision. You, you write very pointedly in your book about what people at OpenAI call the blip, where Sam Altman was fired for 20 minutes and then he came back and that cleared out a bunch of people from OpenAI. It is the moment you describe in the book where OpenAI went from being a research company into being a business and now they've become a business. And the rational business decision was we need to sell more AI to people. We ought to have as much of it as we can. And Sam is well known to be like a ruthless business person. I've heard people describe him as S tier at raising money, which is just an incredible Silicon Valley phrase. So I look at that as like a business choice. Like they got rid of a bunch of stuff they were doing.

• 30:31Nilay Patel?

  They pivoted to making enterprise AI products, which is where the money is. And then they realized they needed to service that demand. You're saying that was a bet on AGI, which to me is a more idealistic turn. Can you, can you square those two?

• 30:42Speaker 05

  Yeah, I think it absolutely is a bet on AGI because Sora, which they launched and spent a bunch of money and compute on and then killed, is a cool toy that might open up a new line of business for you, but it is not going to get you closer to AGI. What actually gets you closer to AGI is sparking recursive self improvement by building really good coding models and then turning them back on themselves. That was something that Anthropic was doing. They had not gotten distracted by the side quests and shiny objects that OpenAI had last year and OpenAI was getting their asses kicked. And so they decided to sort of trim the sails, cut fat, like really focus on coding, really focus on long horizon agentic workloads.

• 31:22Speaker 05

  And that is, as of now, the leading path to AGI or to, you know, whatever comes after AGI. So yes, I think that was absolutely a bet on conserving resources by not spending them on things like Sora. But what they did spend it on was trying to catch up with Anthropic in coding so that they could reach AGI.

• 31:43Nilay Patel?

  So when I look at Dario Amade at Dreamforce, standing next to Marc Benioff, and he's talking about selling enterprise software, is that just a show? Like I, that's the thing I'm trying to square is like, they've all become enterprise software vendors in like the realist way. They're, they are B2B SaaS companies in the realist way now, but they're also like building God. How do you, you know, these folks, does that align? Are they like, I gotta take off my B2B SaaS hat now and go back to, go back to the altar?

• 32:15Speaker 05

  I think the B2B SaaS thing is the game that they realize they have to play. But I don't think it is any of their true passions. Like I think, you know, I was talking with Sam Altman for, for this book. And one of the things he said was like, basically, you know, all of this was an accident. Like we, I was never supposed to be running like a giant tech company that just didn't factor in. Like now, you know, you wake up and you've got thousands of employees and, you know, Dario Amade also has this like very funny in retrospect document that he writes in 2021 when they're like just starting the company. It's called Vision for a New Co.

• 32:59Speaker 05

  And you can go back and look at the predictions that Dario made about AGI. So one of them is he thinks that it will cost about $10 billion to build like a full fledged AGI, which was, I think, off by at least one order of magnitude and probably multiples. He also thinks that Anthropic will need, will never need more than about 50 people to achieve AGI. I think now they have 5,000 or so people. So again, off by, off by 100x. So I think what has happened is that this has just taken a little longer than all of them thought and cost quite a bit more money than they thought. And so the enterprise SaaS stuff is like how you sort of keep the flywheel going and

• 33:43Speaker 05

  pay for the GPUs and pay for the salaries of the researchers. Like this is sort of the, the, the table stakes of getting to keep pushing on the frontier, keep pushing to AGI.

• 33:55Nilay Patel?

  Are they ready to be the leaders of companies of this scale? This is another thing that I was thinking about as, as I read the book. They're all very idealistic, so much so that they all broke off from one another and made their own companies because they could not make their ideals align. And none of them seem like great managers. Like, I don't know truly what they're like, but they, idealists and we're going to run an enterprise software company are not usually in the same sentence, often not. Do you think they can take the pressure?

• 34:27Speaker 05

  No, but like, I don't think anyone can. Like what they are attempting, no one has ever done. People have built companies. People have built companies quickly. No one has gone from $0 in revenue to potentially a $2 trillion IPO in five years. That has never happened before. So, look, I think that they are scrambling to kind of build the plane as it's, you know, careening toward, toward the event horizon. But I think, you know, they're, they're doing about as well as I think anyone you could hire from, you know, I don't think it would be a good idea for them to like bring in

• 35:10Speaker 05

  some professional CEO from like Coca-Cola to like run the company. Like I just, I just think it would all kind of fall apart because they're all zealots and missionaries. Like they are all in this, not all, probably there's some people in like the SaaS, you know, sales department who are just kind of there for the paycheck. But like a lot of the key people at these companies are united in their belief that they are building, you know, one person, I won't, I won't say who, one person at one of these companies recently told me that they think their company might be the last company. And I thought, like the last company to IPO this year? What do you mean?

• 35:52Speaker 05

  They were like, no, we think this might be the teleological end of capitalism. And like, there might never be another large company primarily run by humans. And so like, these people are weird and weird in ways that are worrying, but also very interesting. And so this is what I was trying to do in the book is not like tell you that these people are terrible or great. It's just like, you need to know who they are. Like you need to know that they all hate each other. You need to know that they are these peculiar breed of thinkers to understand the dynamics of the race.

• 36:28Nilay Patel?

  There's a portion of your book where you talk about people splitting away from effective altruism to go become accelerationists. And you, you have a woman in there who's like, well, the accelerationists have better parties. And I just, I like those vibes better. And I thought to myself, well, what does that have to do with potentially killing everyone? But that, they are very human, even though they're trying to build something that might ultimately dehumanize us all in various ways. What did, what did you make of that as you were doing all this reporting? The, the notion that there's existential risk here looms over the book, it looms over the industry. You know, every time we do an episode on it, the audience is like, why won't they just stop? And Bill Gates was just on Decoder and he's like, you could just stop if you were the only company. If you thought you were going to be the only company with AGI,

• 37:12Nilay Patel?

  then their model would work, then OpenAI's model would work. But the second there are more frontier companies, it absolutely won't work. By the way, they might kill you all. And I thought to myself, well, Bill Gates is not very optimistic right at this moment in time.

• 37:24Speaker 05

  No, he's not.

• 37:25Nilay Patel?

  What do you make of that? That they are racing ahead even though they know the risks are huge in the absence of any regulatory authority that could cause any of them to solve these problems?

• 37:37Speaker 05

  I think there are probably two or three reasons that they're doing it. And you know, I've asked them all, pretty much, you know, I talked to like 150 people for this, and I asked all of them some version of this. Like, if it's so dangerous, why are you doing it? And the answers would usually start with some sort of, I don't know, fatalistic, I don't know, you could call it inevitableism. They were just like, look, someone is going to build this. The recipe is not that hard. All you need is like a bunch of compute and a bunch of data and the right algorithms and like, you get AGI. And so if it is possible, it will happen and it would be good for us to do it rather than people we don't think are as trustworthy. That is sort of reason number one. The harm reduction argument.

• 38:21Nilay Patel?

  Yeah.

• 38:22Speaker 05

  Then they're like the true believers who are like, this is all going to turn out great. Like, if we thread the needle, if we can build this stuff without blowing up humanity or wrecking the entire internet or developing novel bioweapons, like, technology is usually a force for good throughout history. I don't think a lot of people would switch lives with their great, great grandparents. It sucked to be a subsistence farmer. And I think there are people who believe that we are the equivalent of subsistence farmers today, and that what is coming will be so great that we will just, we will want to live in that world. And then there are just people who are like, look, this is fun. And I like doing it. And it's fun to make the neural networks code. And so that's what we're going to keep doing.

• 39:07Nilay Patel?

  Yeah, that aspect of it where every now and again, I look up and I'm like, well, my computer just did something that I've never seen a computer do before is fun.

• 39:17Speaker 05

  It's awesome.

• 39:18Nilay Patel?

  It is legitimately is fun. It's some of the most fun I think you can have with a computer ever. Then I'm always, you know, led to think about, well, this came at some massive cost. I might be empowering some corporation that ultimately wants to go us all. Somewhere in there is the swirl of all the tech CEOs standing behind Donald Trump in the Oval Office to do something, to sign some sort of morally binding AI safety pledge. And all that feels very conflicted. I would put that in the category of AI backlash, right? We hear the AI companies talk about, well, the technology just hasn't diffused enough. People haven't seen the benefits, so they naturally hate change. We have a marketing problem. We just need to communicate about it more. Is that just the prevailing view as you talk to these folks? That eventually the stuff will speak for itself? Or do you

• 40:03Nilay Patel?

  think they've, they've internalized there's more of a backlash?

• 40:07Speaker 05

  I think they're, they're like, fine with, I don't think they're surprised by the backlash. I don't think they like it. They wish it weren't happening. But, you know, I was talking to one CEO at one of these companies who was like, yeah, I get it. Like, right now, like, we haven't cured any diseases yet.

• 40:26Nilay Patel?

  Yeah.

• 40:27Speaker 05

  And yes, we think our tools are capable of doing interesting things, but like, we don't have a lot of things on the scoreboard in the like, big societal improvements category yet. And, you know, I think there are some people in the industry who lament the fact that the chatbots came before the cancer cures and wish that they could have figured out a way to do it in the other order. It's just that that is kind of the way the, the tech tree grew. And so they feel like now they're, the only responsible thing to do is like, keep pushing on the, the cancer cures so that when people look at like, what has AI done to my life? It's like, well, yes, it like, helps my kid cheat on their

• 41:09Speaker 05

  homework and they're addicted to an AI companion. But like, also, my grandma is still alive and she wouldn't have been if, you know, if this medicine had not been discovered by AI.

• 41:20Nilay Patel?

  Yeah, there's some dynamic in there. It brings me back to Google, I think, as this conversation has done many times, where your reporting on Google's approach to AI was, this is task-based software. We're going to make Google search 10% better, Gmail will be 5% better, and we will continually deploy these improvements and no one else has our access to data or our scale, and thus we will be fine. OpenAI shows up and it puts itself in Bing, famously Bing tries to make out with you. Still my all-time favorite. I can't believe I haven't brought this up yet. This is my all-time favorite.

• 41:51Speaker 05

  I can't believe you haven't brought it up either. I was, I had an over-under bet going on how long it would take you. And I took the under at a minute, so.

• 41:57Nilay Patel?

  Well, you and I have already talked about it in our lives. I was like, I've got all these other deep questions. But yeah, Bing famously horny for my friend Kevin Roos. That moment, I think, just like galvanized people, right? Like, wow, these things can do a lot of things. And Google had to react, and they didn't react by going all in. They sort of stutter stepped there. And as you've been saying, they're still not all the way there. They still don't think this thing is going to happen. They think they're making better Google products for people. When I look at the trial transcripts from like, Musk B. Altman about the founding of OpenAI, they're all terrified of Google. They're all terrified of Dennis Hassabis at Google specifically, doing the thing, just like making it happen because he has the resources. And I can't tell if they were right or wrong about that.

• 42:42Speaker 05

  I think they were right to be, to be taking note of how DeepMind was doing. I mean, you know, if you take yourself back in time to like, 2015, 2016, like, DeepMind was the only sort of AGI lab of any real importance. They were beating Go, they were beating other games, they were beating, you know, StarCraft. Like, they were really cooking, using this sort of method of reinforcement learning, deep reinforcement learning. And there were people who thought like, that was the thing that was going to scale all the way to AGI. Like, if you could train a model to be, like, there were people who thought AlphaGo was the way, was the path. Because if a system could go from never having

• 43:22Speaker 05

  played Go before to beating the world's best humans at Go in the span of a few hours, as AlphaGo Zero did, then maybe you could just replace Go with like, any arbitrary thing, you know, math, or coding, or something. And you could, you could sort of, you know, climb the ladder very quickly on those tasks. Now, it turned out that didn't work, at least not without the LLM. But I think people had very real and reasonable fears that that would work. And, you know, in, in an alternate universe where it had, I think DeepMind would have gotten to AGI first because they were so far ahead of the competition.

• 43:59Nilay Patel?

  Are there, are there other paths? Is it just next token prediction? Is it just LLMs? Or are there other paths that might get you to AGI?

• 44:06Speaker 05

  I mean, there are other paths that might. I think, you know, people are, people who are skeptical of LLMs are often a fallback on this world model idea. I don't place a lot of stock in that, but there are people who think that, yeah, to be AGI, you need to be able to, like, visually understand the physical world and like, drive a robot and things. I'm kind of a believer that like, it's all software. And so I think that LLMs are the most promising path yet. Now, maybe you need to make some tweak to the architecture. Maybe you need to like, build in continual learning methods. Like, maybe, maybe it's not just literally scaling up the same transformer-based LLMs that we've had for the past, you know,

• 44:48Speaker 05

  six or seven years. But, but it sure seems like it is continuing to work. And I think all of the labs are betting that it will continue to work forever.

• 45:33Speaker 00

  Support for the show comes from CrowdStrike. AI is already fundamentally changing how companies build products and run their businesses. If you listen to the show, you probably already know this. But while AI drives innovation, every new model and agent also creates new cybersecurity vulnerabilities. That's why CrowdStrike built AI Detection and Response, or AIDR. AIDR helps organizations discover where AI is being used, monitors AI activity at runtime, and helps teams detect, investigate, and respond to threats involving AI systems. As companies adopt AI, security has to be built in from the start. More than 70% of

• 46:15Speaker 00

  the Fortune 100 trust CrowdStrike to protect their businesses. Now CrowdStrike is helping secure the AI era. You can learn more about AI Detection and Response at crowdstrike.com/decoder.

• 46:33Speaker 06

  Support for the show comes from Outshift, Cisco's incubation engine for frontier technology. You can find multi-agent AI in every enterprise roadmap right now. But here's the problem everyone's facing. Agents can pass messages but can't think together. These systems silently underperform through cognitive failures, misreadings, unverified claims, false consensus, and conventional monitoring can't detect them. Outshift by Cisco is building the fix with their new comprehensive resource, the Internet of Cognition. It's an open source guide that's a foundation for building multi-agent systems from design to production. The goal is to give agents shared context, shared memory, and guardrails,

• 47:16Speaker 06

  all to drive the results we actually expect from AI. You can lean on this resource to understand Agentic AI's evolution, and how it shifted from isolated agents to orchestrated superintelligence. You can gain insights for how to future-proof your business by tackling complex challenges with multi-agent systems. You can read the paper, experience the demo, grab the code from outshift.com. That's O-U-T-S-H-I-F-T.com.

• 47:52Nilay Patel?

  All right. Now I am going to ask you about Bing and Sydney.

• 47:54Speaker 05

  Please.

• 47:55Nilay Patel?

  Because I've been thinking about this a lot. Obviously, I've been thinking about it. I can't stop talking about it. It's my favorite thing to talk about.

• 47:59Speaker 05

  First question, am I still married to a human woman?

• 48:01Nilay Patel?

  Yes.

• 48:04Nilay Patel?

  I just remembered that period, 2023, like, four different people, like, woke up one day and was like, the computers are alive. Right? Or like, they had some out-of-body experience with a chatbot. You were one of them. Ben Thompson was one of them. Blake Lemoine at Google was one of the, like, a lot of different people were like, it's, it's obviously alive. I'm having this, like, completely insane experience with a computer. No computer has ever talked to me this way before. Sydney, in particular, was like, break up with your wife, send you heart emojis. It's, it's very good. I encourage everyone to go read that, that Times article. Which, by the way, also credit to the Times for being like, this should go on the front page of the New York Times. Incredible editorial insight there to immediately put that on the front page. I don't think you feel that way about chatbots anymore.

• 48:47Nilay Patel?

  And I think most people have come to recognize that the computers are not alive. Like, there's something that happened where if you gave the people in 2026 a 2023-class chatbot and said, do you think this is alive? Like, I, I think maybe not everyone in the world, but some enormous set of people has now had enough experiences to be like, well, this is pretty dumb and it's hallucinating. And I, I'm just wondering how you place that in the context of AGI.

• 49:15Speaker 05

  Yeah. So first of all, a slight correction. I never said Sydney was alive. In fact, I said explicitly that I didn't think Sydney was alive and that I understood that this was not like a sentient AI system. So I'm not sure my own judgment has wavered on that point. In fact, I would, I would probably put a higher, like, P-consciousness on today's AI systems than I would on the models of 2023.

• 49:41Nilay Patel?

  I'm sorry. Let me just, I mean, let me correct.

• 49:44Speaker 05

  Yeah.

• 49:44Nilay Patel?

  There's a reason I think this. The headline, it was Bing's AI chat is, quote, "I want to be alive."

• 49:50Speaker 05

  Yes. It, it said it wanted to be alive.

• 49:52Nilay Patel?

  That's just why I have the word "alive" in my head.

• 49:53Speaker 05

  The body of the story makes clear that I did not buy that it was alive.

• 49:58Nilay Patel?

  That, that's why it's in my head, is that, that was the headline in the New York Times.

• 50:02Speaker 05

  It's, headlines are so powerful, Eli.

• 50:04Nilay Patel?

  They framed my own thinking. I'm nothing but a next-token prediction machine.

• 50:09Speaker 05

  But you're a very sophisticated one. You're like a GPT-7 class.

• 50:13Nilay Patel?

  Oh, I'm there. Yeah, obviously.

• 50:15Speaker 05

  No, I think, I, look, I have come to think that there's like a very banal explanation for why Sydney was so compelling and creepy, which was that it was the first public deployment of GPT-4. Like, we did not know it at the time. But that was GPT-4, an early, sort of, not fully post-trained version of GPT-4 that Microsoft had, like, jumped the gun and shoved into Bing before it was ready. But that was GPT-4. And I think there actually was a, a subjective step change in what it was like to talk to a class, a GPT-4 class model versus a GPT-3 class model. Maybe it was just like, it passed the Turing test at GPT-4 and it didn't at GPT-3. Maybe it was like,

• 50:57Speaker 05

  it's much more human-like and it sort of crossed some perceptual chasm that made you think, oh my God, this thing is like a human. But, you know, I'm not the only person who felt that. People at Microsoft also felt that. There was that whole Sparks of AGI paper that came out around that time. So that was sort of the banal explanation. But yeah, I, like, I've been talking to computers for a long time, like you have. I used to go on AOL Instant Messenger and talk to a smarter child, which was like absolutely not intelligent, but it was still fun to, like, type something in and have a thing type back. So like, I, I grant that I am sort of obsessed with.

• 51:33Nilay Patel?

  No, the reason I ask this, and I realize it's funny even reporting on AGI so much, I'm using "alive" in such a colloquial way and it clearly means something else inside the AI industry. Like, if you ask Anthropic if Claude is alive, they're not like, no, but then they will start defining alive in very specific ways.

• 51:49Speaker 05

  It's moral status is deeply uncertain.

• 51:52Nilay Patel?

  Like, I don't, so it's alive? And they will get to, well, you have to have flesh and blood to be alive, obviously. And I don't, I would set that aside. What I'm getting at is I think if you went and talked to a GPT-4 class model today, you would not have the same experience. And maybe that's just because you have the experience. But a lot of people have had that experience now. And I think if you put the GPT-4 class model in front of your average Google AI overviews user, they would not be, like, confused about what was happening anymore. They would not say this is a step towards AGI. And it's that reframing, as the models get more and more capable, our expectations of them go up, but they also get higher.

• 52:35Nilay Patel?

  I'm asking about this in the context of AGI because this thing can go off and find a bunch of PHP vulnerabilities, is one kind of intelligence. But this thing can compete with another human being to have an emotional relationship with you or be your butler, or whatever it is. It feels like a different set of capabilities. And that one, to me, feels flat.

• 52:57Speaker 05

  Well, I think that's just saturated. Like, like, we just don't, like, how much more like a human could it communicate? You know, it's like, there, there just isn't a lot of room to run on that particular skill. Where I think they still have room to run is on solving novel problems in math and science and, and, you know, other things. But like, I, I kind of believe that, like, we, we just, like, it talks like a human. It talks more like a human than many humans I know. And so I think that's just like, we just, we kind of, they, they kind of checked the box on that one.

• 53:33Nilay Patel?

  It talks more like a human than many humans I know is absolutely a sign of you living in Silicon Valley, talking to investors for some of that one.

• 53:40Speaker 05

  Well, now everyone just speaks Claude. Like, I, like, I, I've stopped counting the number of times I'm like at a party and talking to someone and they're like, now that is a load-bearing fact about our world. Or like, they say, you know, I hope that's landing cleanly with you.

• 53:54Nilay Patel?

  That's really bad. Tell me about that. That's a real back and forth. I've always joked with you and Casey that we should just start a show called New York vs. California, where I'm just openly skeptical of these claims and you guys are fully AGI-pilled. This is a culture. It's a scene. The number of parties that show up in your book is astounding. There was a moment reading your book where I was like, I don't think I'm going to enough parties.

• 54:18Nilay Patel?

  All these people are always at parties and issuing proclamations to each other at parties. Tell me about that culture. Tell me about that scene.

• 54:26Speaker 05

  Yeah. So I, it's important to know that these are not always like parties in the way that, like, New Yorkers would think of parties. Like, there are very few raves. There's only one rave in the book. And, and, and Grimes was the DJ and, and it was a sort of symbolic rave for the effective accelerationist movement.

• 54:47Nilay Patel?

  This is a great scene in the book because Grimes stops in the middle of her DJ sentence and says, I don't agree with any of you.

• 54:52Speaker 05

  That she keeps DJing. Yeah, it's incredible. But like, yes, there are a lot of parties. It's a very small world. I mean, that's part of the, the sort of whole, like, takeaway from the book. You know, it's not really a message book. I wasn't trying to, like, convince anyone of anything. But like, one thing that was apparent to me and that I think will be apparent to people who read the book is just like, it is a super small subculture of people who have been making all the decisions. And they all hang out, they all know each other. There are a bunch of, like, people who are married to or dating people at different frontier AI companies. Like, it's just a small world. And so, yeah, there are parties, there are, they have LAN parties. There's a, there's a big group of AI researchers who play, like,

• 55:34Speaker 05

  Age of Empires on a LAN party on the weekends. There's, I just learned there's, one of the labs has a poker game where they, like, bring in people from other labs. So like, these people are all just kind of living and hanging out in San Francisco together. And I think that's important to capture because you're right, it is a whole scene. There's a whole ethos. This is how the norms are sort of enforced and deepened, is like, these people just hang out with each other all the time.

• 56:00Nilay Patel?

  Well, how do you think that collides with, and we might also remake the world? Right? If you are like in a small scene in San Francisco, you're all playing poker together, and the conversation at the poker table is, this, we might be the last company. That is in conflict with, oh, there's millions upon billions of other people in the world who might have a say. And that, if I had to, like, boil the backlash down, it's, it's that, right? It's, oh, we should probably get a say here.

• 56:28Speaker 05

  Totally. I, I think we should get a say here. I'm not saying that this is like a good state of affairs where like, you know,

• 56:35Nilay Patel?

  No, but I'm, I'm saying like, as you do the reporting.

• 56:36Speaker 05

  A couple hundred people in San Francisco are like making all the decisions about how society goes.

• 56:41Nilay Patel?

  But as you, as you were doing the reporting and the weight of the task and the consequences of the goal set in on these people, because that is the through line of the book, right? The stakes just keep going up. And, you know, at one point Sergey Brin is like, we're going to get to AGI now. And he just like announces it at Google. It's unclear whether Sundar Pichai thinks that is a good idea, but that is a thing that the co-founder of Google is announcing inside of Google. The stakes are just going up. Like, the race to get there is just going up. But it's, you know, I just can't get the image of it's a bunch of people drinking beers and being like, well, what if there was no money? What if, what if physical scarcity was gone too?

• 57:17Speaker 05

  I mean, this is like any, any party in San Francisco you go to and someone will say some version of like, what's your plan for the post AGI future? Or like, they'll tell you how, you know, they like, yeah, how, how they're like really into the idea of like whole brain emulation or mind uploading. Like, there's a lot of futurism happening out here. And, um, look, again, I'm not saying this is good. In fact, maybe I should sell a version outside the Bay Area of the book that just has like a, and this is bad appended to every factual claim.

• 57:51Nilay Patel?

  I don't think it's good or bad. I'm actually, I'm more neutral on that. I'm just, I'm always wondering if the technology can accept the pressure. Right? So my question about LLMs three years ago was can it do all the things you want it to do? And now it can definitely do some set of things that you want it to do. And now it's, can these companies accept the weight of their own ambition? Like, I don't know the answer to that question.

• 58:15Speaker 05

  I don't either. And I think that we're gonna see a lot of growing pains. I mean, these are still very young companies. They still, you know, have a lot of growing to do. Um, I'm sure they're

• 58:27Nilay Patel?

  Do you think they're gonna make it? Actually, let's do this like this in a lightning round. Do you think OpenAI will make it?

• 58:30Speaker 05

  Yes.

• 58:31Nilay Patel?

  Do you think Anthropic will make it?

• 58:32Speaker 05

  Yes.

• 58:33Nilay Patel?

  Do you think Google will make it?

• 58:34Speaker 05

  Yes.

• 58:35Nilay Patel?

  What, what does make it mean in your mind? They're all gonna be trillion dollar, multi, multi AGI variant successes?

• 58:42Speaker 05

  Yes. I think, look, there will probably be those three and then maybe one or two others that sort of make it. Um, and make it for varying definitions of making it. Like, there will be one or two big enterprise AI companies, one or two big consumer AI companies, but there just are network effects and winner take all dynamics and, or winner take most dynamics in, in these markets. And so, yeah, I expect that there will be, um, you know, there's some people who believe in the kind of like, the, the mono AGI, like, uh, the singleton theory it's often called, like, there can only be one, because once you have one AGI, it's then gonna go around

• 59:24Speaker 05

  like melting the chips to, to the other AI labs and they won't be able to build theirs. Um, but I think we'll probably end up with like a small handful of AGIs.

• 59:34Nilay Patel?

  What do you make of the idea that, yep, it is a pretty simple recipe. Yep, there are open weight models. Yes, there is a China. All the pricing power will be destroyed because a mid-range open weight model can do everything that you probably want it to do on a, like a Mac.

• 59:48Speaker 05

  Hmm, I don't think so.

• 59:50Nilay Patel?

  No?

• 59:51Speaker 05

  Um, because you still have to pay for inference on those models. They're, they're, you know, even, even the open weights models, um, have to run on computers and inference is getting more compute intensive, not less. Um, the, the more frontier these models get. And so, uh, yeah, training budgets may fall, but inference budgets, uh, will, will be, uh, robust for the foreseeable future. So, yes, you can run some tasks locally, but I think this is something that has been predicted since the first open weights models that, oh, they were gonna just get commoditized and destroy demand. But I think what we're starting to learn is A, that, you know, there are a lot of people who will pay and want to pay for the absolute best model. They don't want to, uh, be working with the, the slightly crappier model from six months

• 1:00:36Speaker 05

  ago. And then I also think, once you crack down on distillation, which is this, uh, process whereby you can take a closed model and make a slightly dumber open version of it, uh, by just basically DDoSing, uh, the, the, the inference for the smarter model. Um, once you eliminate that, the qual, it's not clear to me that the quality of the open models will continue to sort of be closely correlated with the quality of the closed models.

• 1:01:02Nilay Patel?

  Yeah. The thing that really grabs me at that is they all did it to each other for a while. Like Grok obviously distilled Claude and it's admitted to in the court documents. There is that class of sort of second tier companies. I would put SpaceX SI, I believe it's now technically known as.

• 1:01:21Speaker 05

  I haven't checked my email in the next, the last 30 minutes, so I don't know what that company is called.

• 1:01:24Nilay Patel?

  Uh, it's called SpaceX SI Tesla Inc. That's, it's, that's all you need one. Elon Inc. is what it's called. They, they obviously distilled Claude to some huge extent. They are now renting their, their compute to Anthropic. It's unclear whether they will make the investment in having a, a leading edge model. Meta just announced its model. They're sort of back on the frontier, but they obviously care more about Muse than having a bleeding edge model. There's some dynamic there. Uh, those companies don't strike me as being run by idealistic true believers. Right? Those are companies that need to invest in some technology to build some products. They just need to get there. Can you get there without the idealistic fervor?

• 1:02:00Speaker 05

  Uh, no, you can't get there with, without the idealistic fervor. Um, so, well, let me, let me clarify that, because I don't think Sundar has the idealistic fervor, but Google might still get there. So I don't wanna like

• 1:02:13Nilay Patel?

  Sergey has the idealistic fervor.

• 1:02:14Speaker 05

  Right. You need someone in a position of

• 1:02:16Nilay Patel?

  You need a co-founder of the company.

• 1:02:18Speaker 05

  Well, it, it might be enough for Meta to have like Alexander Wang, like, you know, who, who by all accounts is a true believer. Um, I don't know that it literally has to be the CEO every time, but you need like a dense nucleus of true believers to be able to convince you to lay out the money on GPUs or just, you need someone who is like, you know, like Mark Zuckerberg, like one of the most competitive people in the world who just like, will kind of adopt whatever strategy he thinks will like allow them to win. Um, and if that happens to align with building AGI, then maybe they'll do it.

• 1:02:51Nilay Patel?

  Yeah. Last question, I've asked you this several times. Are you AGI pilled?

• 1:02:55Speaker 05

  Yes. Are you?

• 1:02:57Nilay Patel?

  No. I know, and I, I, I think I'm, I read your book and I, it's as close as I've ever gotten. That line about next token prediction, just always overcoming its skeptics, just really hit me, because it is, it has overcome my skepticism several times now. But I, this is why I was asking about talking to a chatbot from three years ago. I feel that wall in the kind of intelligence these things are. And even though I've, I have set off agents to do all kinds of things, you can, there's a brittleness to it that feels inherent to the technology to me.

• 1:03:31Speaker 05

  Did the millennium prize problem solution change that view at all?

• 1:03:35Nilay Patel?

  Only in a sense that finding the next thing that has some objective verifiability seems very rich for these tools. Right? And so, you know, obviously mathematics has a verifiability and the tools got so capable that they could run at it really, really hard. And the brittleness that I'm describing is a lot of the world around us does not have that kind of verifiability. And so I set off Muse to like fill out a document on some janky government website and the website cannot tell Muse what it's getting right or wrong and it just collapses. And you can blame that on the underlying model not being at the frontier, whatever, but there's just some part of that experience with LLM-based technologies that I feel. Right?

• 1:04:20Nilay Patel?

  They're brittle in like very specific ways and they haven't overcome that sensation for me. I'm not saying they're not incredibly capable. I'm not saying that as enterprise software, they will not automate virtually every business in the world in some way. I'm not saying that my wife isn't a lawyer who uses Claude every day. Like, it's all there. It's, but there's some part of it where my skepticism that it can overcome its brittleness to become the, the machine god, I'm excited to see if they can do it, but I don't see the path from where I see it.

• 1:04:48Speaker 05

  You think there's sort of a natural plateau at like just below hu- human level intelligence?

• 1:04:54Nilay Patel?

  I think the human level intelligence to me, and this is just more philosophical, is defined by like novel problems, problem solving. Like looking at a thing and like understanding that there are multiple paths to a solution and not having one of those needing to be objectively right to get to the end.

• 1:05:16Speaker 05

  What, what would it take? Like, what would you have to see to have the, the scales fall from your eyes and go, I and Eli Patel am fully AGI pilled? What would convince you?

• 1:05:25Nilay Patel?

  I would have to see, you have young children, I have, you have young children. There's a thing that young children do where they like fall down the stairs, right? And you just see like the lights all go on. Like, I've learned this, I will know it forever and now I'm gonna build on that. Right? And maybe, maybe they're gonna announce recursive self-improvement at the AI labs and that will be the thing that drops the scales from my eyes. But the brittleness that I'm talking about is I see the opposite of that in my children. I'm assuming you see the opposite of it in yours. Like, that, when they fail, they know about that failure forever. Right? And that is, to me, there's just something very, very core to being human where, you

• 1:06:10Nilay Patel?

  know, every now and again, I like wake up and I'm embarrassed about something that happened in high school. You know, and like, that drives me to do a bunch of stuff or it drives me to not make a bunch of decisions. And I think that is just a key part of our intelligence. I'm not saying it has to be a key part of every intelligence, but the brittleness I see with the AI systems that I use today are that they're still computers in like a very specific way. Right? They, they still act like computers. They don't act like, they're good at solving problems that you can test. They're good at software engineering. And I, I don't know that that way of thinking extends into every domain.

• 1:06:46Speaker 05

  I hope you're right.

• 1:06:48Nilay Patel?

  Do you think, I mean, I don't know, that, that's my point of view. I call it software brain. And it's just the idea that you, you hear the AI CEO say, we solve coding and now we can solve everything. And to me, I'm like, that assumes everything happens inside of a computer. And in fact, most things don't.

• 1:07:04Speaker 05

  You sure? You sure your brain's not a, a, just a fleshy computer?

• 1:07:09Nilay Patel?

  I am a hundred percent sure of that. I, you know, I think

• 1:07:13Speaker 05

  You're not a computational functionalist.

• 1:07:15Nilay Patel?

  No. And I, I think I've read this from you in the past, I've certainly read it from other people, that the machines of our time dominate our own explanations of ourselves. Right? So like, in the era of the steam engine, our self-conception was that we were steam engines. Or like, in early computers, we were early computers, right? We, I still call it horsepower-ing through a problem, because I'm like a car person. You don't think that the AI industry has convinced itself that they are AI because that is the machine of our time?

• 1:07:45Speaker 05

  I don't know. I'm not a theory of mind guy. I'm not a neuroscientist. I think to believe that what's going on in your head and my head right now is not a form of computation, you have to be kind of a dualist, or you have to be kind of a religious person. You have to think there is some spirit, some soul, some secret sauce of cognition that is not just data entering and being processed and transforming inputs into outputs. And look, I know that is like the least romantic, most San Francisco way of explaining what human cognition is. I just have not heard a compelling argument that it is not how

• 1:08:25Speaker 05

  our brains work that doesn't involve some sort of supernatural or immaterial substance.

• 1:08:31Nilay Patel?

  There's a surprising number of neuroscientists and religious scholars listen to the show. They're going to come find you.

• 1:08:38Speaker 05

  They will.

• 1:08:39Nilay Patel?

  I'm confident. Let me just ask you this. There's like a surprising amount of religious osity in Silicon Valley right now. We just published the Peter Thiel lectures on The Verge. He's talking about the Antichrist. Do you think these ideas are connected? That we are building gods such that they seek God?

• 1:08:54Speaker 05

  Yes. I think we will absolutely have AI religions. I think AI is posing all of the same questions as a religion that we would, we would, you know, otherwise be asking of God. You know, what is our purpose? What is our meaning? Why are we here? These are the questions that people now talk about in the context of AI. And then, yeah, obviously, like I often tell people that being in San Francisco these days feels like living in Europe during the Reformation, where you just have like crazy, like wild-eyed prophets, like waving their pamphlets around and like trying to get you to join their thing. Like, this is what walking down markets or into San Francisco feels like.

• 1:09:35Nilay Patel?

  Yeah. What, what dropped the scales from your eyes? What made you AGI-held?

• 1:09:39Speaker 05

  Being Sidney, believe it or not.

• 1:09:41Nilay Patel?

  Yeah.

• 1:09:41Speaker 05

  I mean, that was the start. And then from there, it was just kind of a gradual process. And one day I realized when I used these things, I wasn't treating them like an intellectual inferior. I was, in some ways, treating them as a peer. And that's when I realized, oh, like something, something's happened here. So it's possible I have AI psychosis. It would not be the first time I've heard that. But I also just think that there's, I have, I have been, every time that I have thought, okay, this, it can't possibly get any better than this. This has to be the plateau. I've been wrong. And so I'm just learning to approach these things with some humility and like, maybe the crazy people

• 1:10:25Speaker 05

  who have been predicting that scaling would continue indefinitely are right. And if they are, I think we need to learn about them, A, and why they think that, and B, try to figure out what to do about it.

• 1:10:37Nilay Patel?

  Well, Kevin, I'm confident the next time you come on the show, I will have set up a fully AI influencer to make money doing UGC ads, which is the true future of, I think, all of our businesses. The podcast is Machine Gods. It's very good. Your host, Casey Newton. The book is excellent. It's the AJ Iconicles. Kevin, thank you so much for being on Decoder.

• 1:10:56Speaker 05

  Thanks, man. It's great to see you.

• 1:11:00Nilay Patel?

  Thank you for listening to Decoder. To get new episodes every Monday and Thursday, subscribe to our YouTube channel at DecoderPod and find us on TikTok and Instagram under DecoderPod as well for more fun stuff we put out every day. If you'd like to let us know what you thought about this episode, or really anything else at all, let us know. Drop a line at [email protected]. We really do read all the emails, or hit me up directly on Threads or Blue Sky. If you enjoyed this episode, please send a link to it to someone you think might like it too. It really helps us grow the show, which you can subscribe to wherever you get podcasts. Decoder is a production of The Verge and part of the Vox Media Podcast Network. Show's producers are Greg Ott, Kay Cox, and Nick Statt. This episode was edited by Kabir Chopra. Our editorial director is Kevin McShane. The Decoder music is by Breakmaster Cylinder. We'll see you next time.

• 1:11:41Speaker 01

  Does anyone know where the bass is?

• 1:11:45Speaker 07

  Hey, Bose here. When your next track starts, listen for the decisions that the artist makes. The vocal pushed right up front. The bass that waits half a beat. That sample from a 70s underground gem. Every choice deliberate. With Bose, you'll hear it all. Your music deserves Bose.

• 1:12:11Speaker 07

  Finding a doctor is hard enough. Finding the right doctor, even harder. Someone who takes your insurance, has an available appointment when you need one, and feels like someone you can actually open up to. That's where Zocdoc comes in. Download the Zocdoc app to search and compare doctors by specialty, insurance, and availability. Read real patient reviews, find the right fit for you, and book instantly. Because getting seen by the right doctor matters. Zocdoc. Ready to be seen? Download the Zocdoc app today. Now at Kroger, your points add up to more ways to save every trip. Use them your way to take dollars off groceries at checkout or save at the pump. You can even save up

• 1:12:51Speaker 07

  to $35 on your next fill-up. Same points, more choices. And with low prices on fresh, quality food, plus digital and personalized offers tailored to how you shop, your dollar goes further. Redeem your points online or in the app. Kroger. Fresh for everyone. Restrictions apply. See site or store for details.