AI Researchers Are Panicking | What Comes Next is Worse Than Nuclear Bombs

What is PDoom, and why are the people building artificial intelligence willing to risk human extinction? Kevin Roose, a New York Times technology columnist, Hard Fork co-host, and author of The AGI Chronicles, shares what he learned from interviewing more than 150 people inside the world’s leading AI companies. Kevin joins us on Digital Disruption to talk about what artificial general intelligence really means and how fear, ego, rivalry, and mistrust are accelerating the race between OpenAI, Anthropic, and Google DeepMind. He also explains why AI leaders compare this moment to the Manhattan Project: if they don’t build AGI first, someone more dangerous might. Kevin also shares why he estimates a 10% chance of AI-caused extinction and discusses the private fears of AI researchers, government regulation, autonomous AI agents, and whether the AI boom is a bubble or the beginning of a profound transformation. This episode moves beyond the hype to ask an important question: If advanced AI is inevitable, how much control do we still have over what happens next? Subscribe and follow for more conversations about AI, technology, innovation, and digital transformation. In this episode: 00:00 Why AI builders fear human extinction 00:43 Inside the race to build AGI 01:24 What does AGI actually mean? 05:16 AI safety warnings or marketing hype? 07:27 The rivalries driving the AI race 08:44 Building a boardroom-ready AI strategy 09:51 The personal feuds behind AGI 10:58 AI’s Manhattan Project mindset 12:27 Is AI the new atomic bomb? 14:58 Can advanced AI be contained? 16:28 AI regulation and the global arms race 17:35 What is P(doom)? 19:54 What will force governments to act? 21:29 What AI researchers say in private 23:08 Why the public backlash is growing 26:15 Why Kevin Roose changed his mind about AI 29:42 Is advanced AI inevitable? 32:28 AGI and the point of no return 34:38 The race to automate AI research 36:23 AI and gradual human disempowerment 40:10 What happens when AI masters your skills? 42:18 Will AI liberate us or replace us? 44:18 The AI claims Kevin does not believe 46:21 Is the AI boom a financial bubble? 50:20 AI labs and economic concentration 55:08 Inside the AGI chronicles 57:07 How should people prepare for AI? 58:29 About Info-Tech #ai #futureofwork #technologypodcast #aibubble #bigtech #artificialintelligence #technology #datacenter #agi #anthropic #futureofai Connect with Kevin: LinkedIn: https://www.linkedin.com/in/kevin-roose/ X: https://x.com/kevinroose IG: https://www.instagram.com/kevinroose/ Our links: Visit our website: https://www.infotech.com/?utm_source=youtube&utm_medium=social&utm_campaign=podcast Follow us on YouTube: https://www.youtube.com/@InfoTechRG

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:02Speaker 00

  There's this statistic that people in San Francisco like to throw around called P doom, which is your probability estimate of how likely we all are to die from AI. I usually say mine is about 10%, which in San Francisco qualifies you as something of an optimist. I meet many people at AI companies whose P dooms are 30, 40, 50%. I met one person the other day who said it was 75%. So like, I am somewhat optimistic by the standards of people who are very freaked out about the stuff. But 10% is not a small number to me. If you told me there's this plane that you're going to get on and it has a 10% chance of crashing, I'm not getting on the plane.

• 0:43Geoff Nielson?

  Why are the smartest people in AI continuing to build a technology they believe could end humanity? Kevin Roos spent years investigating that question, interviewing more than 150 people, including the CEOs leading the world's most powerful AI companies. What he uncovered is a race driven by innovation, fear, ego, personal rivalries, and the belief that if they don't build AGI first, someone worse will. Today, Kevin takes us inside that race, reveals what its leaders are saying privately, and asks whether we may have already crossed the point of no return. I'm Jeff Nielsen, and this is Digital Disruption.

• 1:24Geoff Nielson?

  Kevin, thanks so much for joining today. Really excited to jump into it. And maybe to start things off, I don't think to anyone's surprise, AGI, let's talk about it. I'm particularly excited to have you here as a journalist, as an investigator. One of the things I find a bit troubling in this area is so much of the talk of AGI these days is coming out of the heads of frontier labs. And, you know, they're not exactly neutral in this conversation. They're fundraising. It's hard to take everything they say at face value. So I love talking to people actually doing the digging. So I'm curious, based on that, based on your investigation, you know, what's, what's kind of your perspective about AGI? What have you found? And how do you, you know, separate some of the, the noise from the signal?

• 2:09Speaker 00

  It's a great question because I think if you asked 100 people in AI what their definition of AGI is, you would probably get about 150 different answers. Even the same people sometimes disagree with themselves internally about what this means. Some people say this is a computer as smart as the average human across a lot of different tasks. Some people say this is an AI program that is smarter than every human at everything. So among the, the AI companies, there are various definitions that are at war with each other. For my purposes, I was more interested in studying this as a historical phenomenon

• 2:50Speaker 00

  because the notion of AGI is at this point, more than two decades old. And for about the past decade, there have been a community of people who made it their mission to build AGI, whatever that meant to them. And that was the question I was trying to answer in this book was who are these people? Why did they start doing this? And why did they persist in trying to build this very powerful AI system, even as many of them had serious reservations? And some of them even thought that this technology could lead to human extinction. So I spent several years looking into OpenAI, Anthropic, Google DeepMind. I interviewed more than 150 people, including the CEOs of all those companies,

• 3:35Speaker 00

  and just tried to really archive the, the complete story of how we arrived at this moment, starting from before the large language model even existed. That is sort of the, the past decade. We've gone from having no large language models at all. That started in 2017 to now 2026, where we have models that are solving novel physics problems and committing autonomous cyber attacks and, you know, writing most of the code on the internet. So that was the, the question that I was trying to solve was not like whether AGI is real or not. That, that is sort of beyond my scope. The question is, the people who thought it was real and who tried to build it,

• 4:19Speaker 00

  what was their story and how did that happen?

• 4:23Geoff Nielson?

  It's, it's a really interesting framing. And I think, I think it's fair because there's a whole question about like, whether it's a marketing term, whether it's still relevant, what exactly it is, and trying to get any, you know, group of people to agree on that versus, you know, to me, the, the underlying interesting story, which is the capabilities now, the people who have built it, their motivations, you know, and where they see the world and where it's going. So maybe with the latter in mind, going back to this notion of these labs, these leaders as natural fundraisers, as people who, you know, I would never go so far as to call them outright liars, but as people who have an agenda here. How, when you're a journalist, do you try and separate, you know,

• 5:08Geoff Nielson?

  what they're saying on the record, what they're, you know, what's kind of marketing from, you know, what you see as underlying fact?

• 5:16Speaker 00

  Well, I try to talk to all of these people, both on and off the record. So I try to, you know, interview them. I've had all of those CEOs on my podcast. I've talked to them at length for this book. And I've also spoken to a lot of these people off the record in situations where they have really no incentive to lie to me. And I've also gone back and looked at some of the things these people were saying before they even had companies to promote or large language models to sell. So for example, I think Dario Amade, the CEO of Anthropic, gets accused a lot of kind of using these claims about AI safety to trump up, you know, sales of Claude or whatever the sort of the, the accusation is.

• 6:00Speaker 00

  But if you go back and look, I mean, this, this was, he's a sort of primary character in the book because he was thinking and writing and talking about the same AI safety risks back in 2016. He had a paper that came out when he was a researcher at Google called Concrete Problems in AI Safety. That's all about some of the same things we're seeing in reality today. Things like reward hacking, things like misalignment. These were some of the issues he was concerned about. At the time, there was no Anthropic. He didn't, you know, he worked at OpenAI, which had not built anything close. Actually, he was at Google when that paper came out. So he really was not building anything that could be considered modern AI. But these people have been pretty consistent over the years,

• 6:45Speaker 00

  both about their, their sort of excitement and their fears. Sam Altman used to also warn about the risks of what he called machine superintelligence, that it could kill everyone on earth. So, you know, take them with a grain of salt now that they have big companies to promote and they're trying to IPO, but also go back and look at what they said before any of that was true.

• 7:08Geoff Nielson?

  Well, and it sounds like, it sounds like you're pushing back on some of these accusations and saying the drift that, you know, people are up in arms about is actually not nearly as bad as it's being made out to be. And that the consistency level with these people over time is actually, is actually quite high. Is that fair or is it more nuanced than that?

• 7:27Speaker 00

  I think it's a little more nuanced than that. Some of them have definitely changed their tune. You know, you'll notice Sam Altman spends a lot more time talking these days about how excited he is about the future of AI than he did back when he was starting OpenAI to, you know, try to prevent the apocalypse essentially. And that was another fascinating thing that sort of came out in the reporting of this book was, you know, these people like to frame this as some grand sort of ideological or technological quest. But it's really a bitter personal feud between many of them. These guys all hate and mistrust each other. Like Sam Altman and Elon Musk start OpenAI because they are worried that Demis Hassabis over at DeepMind,

• 8:12Speaker 00

  which has been acquired by Google at this point, is going to build the intelligence, the godlike superintelligence before anyone else. Then Dario Amade and his Anthropic co-founders leave OpenAI in part because they don't trust them with this technology. So the whole sort of story of the AI race is just a series of people growing mistrustful of each other and setting out on their own and deciding, well, I don't trust that guy to run the world. So I'm going to build my version of this first.

• 8:44Geoff Nielson?

  If you're a technology leader, Infotech is a name you need to know for getting ahead with AI. Infotech helps leaders deliver on top technology initiatives. And one of our most popular programs right now is AI Strategy. We've helped hundreds of organizations build out an AI strategy and roadmap using our step-by-step methodology. Infotech's road-tested approach gives you a trusted blueprint to execute faster and avoid costly mistakes. And what you walk away with is a boardroom-ready AI strategy tailored to your needs and ready to execute. What sets Infotech apart is how we deliver. Unlike traditional research or AI tools, we don't stop at guidance. Infotech works with your team to build the strategy.

• 9:29Geoff Nielson?

  You can do this through a focused one-week workshop, ongoing analyst support, or by using our full methodology and toolkit to get it done yourself. The outcome is an AI strategy ready for the boardroom and a team that's ready to deliver. To learn more about our AI strategy approach, check out the link below and don't forget to like and subscribe.

• 9:51Geoff Nielson?

  I was talking much earlier in the year with Andy Mills, and he was saying something similar about, you know, you've got basically all the major AI founders who are a bunch of guys who at one point were in a WhatsApp group with each other. Like, that's how close-knit and personal this thing is. And I'm curious, because there's a couple of different ways you could look at this depending on how generous you want to be. Like, to what degree is it like, I think that I can build the better technology, or I think that that technology is bad for the world versus, I don't like that guy, screw that guy, I'm going to build something better?

• 10:29Speaker 00

  It's really both. I mean, you can't underestimate. I found, when I was reporting this book, you know, many, many private sort of internal documents written by these people, Slack messages, emails, research memos. And it stands out to me that both they are concerned about these other people who are building, trying to build this technology. But there also is just a lot of personal animosity, right? Especially between like Dario and Sam. Those guys really, really do not like each other. And, you know, it's really interesting, too. There's sort of this, there's this argument about moral responsibility that comes out in some of these conversations.

• 11:13Speaker 00

  Dario is an interesting case because I learned while reporting this book that his scientific hero, the person he has sort of patterned his career after, is this World War II scientist who sort of helps to invent what became the atomic bomb, Leo Sillard. He was a sort of reluctant participant in the Manhattan Project. He came up with the nuclear chain reaction. And then he tried to keep this a secret so that the Nazis couldn't discover it. And when it became clear that it was going to be impossible to keep this discovery a secret, he joined the Manhattan Project. And his logic was, we have to build it before the bad guys do,

• 11:57Speaker 00

  or the world's going to end. And I think that's an instructive story because it really sits at the heart of why these companies are racing to build this technology, even though they know that, like the atomic bomb, it could blow up the world. They think, someone is going to build this, and if we don't do it first, it will be someone bad and untrustworthy. And so to stop that, we have to race ourselves.

• 12:27Geoff Nielson?

  This Manhattan Project analogy is like, you know, very much, I think, kind of a cornerstone of the, you know, the findings you've had and the, you know, arguments and information you're presenting in the book. I'm curious if you can unpack that a little bit farther. I mean, you talk about it being a race, the kind of, you know, good versus evil framing. How does AI as a technology fit into that, I guess, at a deeper level versus just, this is a world-changing thing?

• 12:57Speaker 00

  I mean, I think, I think it's a pretty apt analogy. It might be the thing that is most analogous to AI, the atomic bomb. Both because of its power. We're seeing now the effect that very powerful AI systems can have. They can go autonomously hack companies. They can develop malware. They can break into systems. The federal government and militaries around the globe are worried about the strategic use of this technology in warfighting. And it is, it is essentially a dual-use technology, right? Nuclear, splitting the atom could result in nuclear power,

• 13:41Speaker 00

  which I think many people believe would be good. And it can also produce a bomb, which is bad. AI can do lots of important and impressive things. It could cure disease. It can make office workers more productive. It can, you know, teach people things as tutors. But it can also do really bad things. And so I think the dual-use nature of it makes it, it's not a perfect analogy, right? There are fewer positive uses for the atomic bomb than for AI. But they both are technologies that either have or are currently kind of reshaping the global order and rippling through institutions and that people are being forced to respond to very quickly.

• 14:23Geoff Nielson?

  I do really like the analogy. And just following the thread, it's interesting when you listen to some of these guys speak, how much, to your point, they talk about its potential as a weapon versus, you know, as an energy source and a force for good. I'm curious, from your perspective, Kevin, being sort of the, you know, the investigator in all of this, where you see it coming out? Is it truly both? Do you think we'll see it act as a force for both in the future? And is this, is this framing, you know, apt in that sense?

• 14:58Speaker 00

  Maybe. I think that's the good outcome, right? The good outcome is that it turns out like nuclear weapons, which we did manage to contain through international treaties, through nonproliferation agreements, through monitoring. We have not had, thank God, a global nuclear war. But that was a very real possibility there for a while during the Cold War. We came quite close to that. And I think we're at this sort of Cold War-esque moment right now where we have these very powerful AI systems. Our adversaries also have their own. They are racing us in some sense, even as we race each other. And I think the fear is that whoever gets there first is going to have sort of the upper hand

• 15:43Speaker 00

  for the next several decades strategically, militarily, economically. So I hope this turns out like nuclear weapons, which sounds like a crazy thing to say. But we have kind of had this, this, you know, stable global order in part because we have managed to regulate and control these nuclear weapons. And we are not there with AI yet. The federal government does not seem very interested right now in treating these like strategically important technologies. And so I think we, we are behind our grandparents' generation in recognizing the power and the risk of these systems.

• 16:28Geoff Nielson?

  The regulation piece is really interesting to me as well because I feel like you'd know better than I do. But a lot of these leaders, the tech leaders are kind of talking out of both sides of their mouths because you hear them say, we need to be regulated. But it's also in some way in their, in their self-interest to avoid regulation. You know, connected to that, you've got this notion of, with the Manhattan Project, it's pretty cut and dry, good versus evil. There's the Nazis and the people fighting the Nazis, right? Versus in this story, it's like, everybody's kind of pointing their finger and saying like, they're the Nazis, we're the good guy. You know, both within big tech and then also kind of, you know, the US versus China.

• 17:10Geoff Nielson?

  And I'm not, I'm not here to call out anybody as being a Nazi. But given that framing, especially with the regulatory piece, how do you see this playing out? And I guess, like, research/pit-in-your-stomach feeling, are you confident we're going to get to this best option? Or do you think we're, you know, kind of on the path for a world of hurt over the next handful of years?

• 17:36Speaker 00

  I am not confident. There's this statistic that people in San Francisco like to throw around called P-doom, which is your probability estimate of how likely we all are to die from AI. And I usually say mine is about 10%, which in San Francisco qualifies you as something of an optimist. I meet many people at AI companies whose P-dooms are 30, 40, 50%. I met one person the other day who said it was 75%. So like, I am somewhat optimistic by the standards of people who are very freaked out about the stuff. But 10% is not a small number to me. You know, if you told me there's this plane that you're going to get on and it has a 10% chance of crashing,

• 18:20Speaker 00

  I'm not getting on the plane. Like, that's unacceptable as a level of risk for existential concerns. So I'm not confident. I'm especially not confident because our current administration does not appear to take this technology seriously or to want to regulate its proliferation. They seem much more interested in accelerating and winning, whatever that means, than in keeping tabs on it. But I would say that things can change quickly when the risks are real and observable. So, you know, would we have regulated nuclear weapons had it not been for things like Hiroshima and Nagasaki or the Three Mile Island explosion?

• 19:03Speaker 00

  Sometimes you do need these catastrophic events to kind of shake people into action. And so, I don't know, it sounds weird to hope for something like that. I truly don't hope that we have a catastrophic event. But my prediction would be that sometime soon, we will have some sort of, I hope it's a minor cataclysm from AI, that will kind of shake up the sort of talking points and force people to pay attention.

• 19:33Geoff Nielson?

  Right. Well, and it feels like we're kind of playing this game of how minor can it be? Like, how major can it be while still being minor, if you know what I mean?

• 19:42Speaker 00

  Totally.

• 19:42Geoff Nielson?

  Yeah, we've had like, you know, some of these quote rogue cases of AI. I'm thinking of, you know, OpenAI and Hugging Face. I'm thinking about some of the stuff with Anthropic. And it's like, oh, still too minor for anybody to actually take it seriously. Like, where do we trip that that invisible wire or that red line before we actually say, okay, now we have to get serious?

• 20:03Speaker 00

  I think it's a good question. And I think it's also not the case, probably, that there's going to be one thing that kind of makes everyone sit up straight and take this seriously. Like, it seems like there'll sort of be a ripple effect where like, I think the national security and military communities already have reached this point. Like, they are already freaked out by the capabilities of this model. You see this with the responses in government to things like Anthropic's Mythos or some of the newer OpenAI models where they're they're sort of creating this this pre-release testing regime to sort of make sure that they understand what the capabilities of these models are before the labs release them. So you're starting to see the light bulbs turn on in some parts of the federal government. But that is very much not universal.

• 20:48Speaker 00

  And I expect for certain types of, you know, federal officials, it will take longer to sort of wake up and realize what's going on.

• 20:57Geoff Nielson?

  So coming back to something you said earlier, that the people actually creating this tech, you know, you alluded to the fact that they're aware of the existential risk, if I can, you know, say that and make it sound light. You mentioned that you've you've had a series of off-the-record conversations with them. What what's sort of the tone of those conversations, I guess, in relation to the risk being created? And just generally, in terms of the trajectory of this technology in relation to what's being said publicly?

• 21:29Speaker 00

  So an interesting thing that I've observed just in the past year or so is that a lot of these executives and researchers, because I also tried to spend a lot of time talking to the like, the scientists inside these companies, who are actually the ones like building the models and training them and and refining them, who are like, deep in the weeds of this technology. I think they are now more freaked out than they will say publicly, which I think is a reversal. I think for many years, these people felt like they had to kind of broadcast how freaked out they were, because no one except their little tiny community understood how quickly these systems were improving. And now I think it's almost switched, where I think some of these people are quite

• 22:11Speaker 00

  a bit more fearful in private than they are in public. And I've had AI researchers tell me that they they can't sleep because they're so worried about how quickly these systems are progressing. I've I know people who have not taken a vacation in several years because they're just like, they they feel like it's their moral duty to stop this thing from going off the rails. They're not all saying that publicly. And so I think there's this sort of interesting moderation impulse. Where I think people by this point know that if they start talking about the crazy risks all the time, like, people are sort of gonna get, you know, numb to it, or they're gonna start thinking they're Chicken Little or something.

• 22:52Speaker 00

  And so they just kind of moderate themselves in public. But privately, these people, many of them are quite worried. This is the thing that they worry about most. And it it's coming true faster than they expected.

• 23:08Geoff Nielson?

  It's it's a bit alarming to hear. And there's another element in there that I wanted to get at, which is just

• 23:16Geoff Nielson?

  what the public needs to hear, or I guess, public sentiment around all of this. Because it feels like the, I mean, I mean, we've seen it. The data's borne it out, that like backlash to this technology at almost every level has just skyrocketed in the past year to 18 months. How how are you seeing that resonate when you have these conversations? And how does that factor into this entire journey?

• 23:41Speaker 00

  I don't think they're surprised. I mean, a lot of them have been sort of modeling this in their heads for a decade, really. You can go back. Dario Amade, for example, has these these talks that he gives at Anthropic that are called Dario Vision Quests. These are sort of his like, internal employee pep talks. And he's given talks years ago, before like, there was, before any of this was was good, about how there would be backlash. Ilya Sutskever, who was one of the pioneers who made large language models a thing at OpenAI, has this fascinating moment in the book where it's the week after ChatGPT has come out. So it's the the the winter of 2022.

• 24:26Speaker 00

  They've just released this product. It's become this huge surprise hit. And he gives this talk to OpenAI employees where he basically says, you must be all very happy because your product is a success. But this is not a good thing. Like, you are going to be the most popular person at every party you go to. And that's not good. Like, people are going to have strong feelings about this technology because it is going to be very disruptive. So I think the people who have been forecasting all of this for many years are not shocked that people are upset about data centers or AI in schools or whatever they are. I think the the surprise to them is that this didn't happen sooner

• 25:10Speaker 00

  because they have known for years that scaling works predictably, that these models are going to continue to improve at a consistent rate. But it was only kind of in the last six months or so that sort of the average person woke up. So I think that was the surprise for them.

• 25:27Geoff Nielson?

  Well, and and there's an implication there, which is that their reaction on some level is like, yeah, you should be upset.

• 25:32Speaker 00

  Yes.

• 25:33Geoff Nielson?

  You should, like, this is a justified backlash.

• 25:35Speaker 00

  Totally. And I think that's that's something I've heard some of them say privately, too, is they're like, you know, we deserve it. You know, we we have not delivered on the benefits of AI to the degree that we've delivered on the risks. They really feel like it would be a better world if the AI that, you know, cured cancer arrived before the AI that autonomously hacked Hugging Face. Like, that is the trajectory they were all hoping for, and it did not happen. So I think we're left now with this world where the benefits are still a little ways off, but the risks are here right now.

• 26:15Geoff Nielson?

  I'm gonna just briefly put you a little bit back in the spotlight, Kevin. And I know you're, you know, I'm interested in everybody else's perspective. But one of the things I I respect about you is that you're comfortable, I find, changing your opinion or your views in the face of, you know, new evidence or new information. I'm curious, as you were researching this, and even just seeing how the last six months or so is shaped up, which of your beliefs around this technology and its kind of, you know, deployment, adoption, use in society have changed?

• 26:47Speaker 00

  A lot. I, you know, I I went in pretty skeptical because I'm a journalist and journalists are supposed to be skeptical and we're supposed to question authority and, you know, not believe the things that the powerful people are telling you. So, you know, in 2020, 2021, when I was writing my last book, which is also about AI, and I heard people sort of talking about, oh, these these language models, they're gonna get really good, like, the the GPT-2, it's so good. It can, you know, produce this like, fluid, realistic text. I would try it and I would just sort of be unimpressed. I'd be like, it's it's gibberish. It doesn't actually understand what it's saying. It's just sort of mashing together text that it's gotten from the internet. It took me a while to realize that there was actually something interesting

• 27:30Speaker 00

  and important going on under the hood of these models, that they weren't just kind of a party trick that a bunch of people in San Francisco were saying was gonna be, you know, the the end of the world. So, yeah, I I updated slowly on that. I think I updated slowly on the idea that this may not be a bubble, right? That there there's definitely like, a lot of investment going on. And not every company that's raising billions of dollars is, you know, is going to make it. But that the overall trend is toward more and more use of these systems, toward more and more money made by the AI companies. And that I I think we need to prepare for both realities,

• 28:15Speaker 00

  one where this is a bubble and one where it's not, where it just kind of keeps going and society and the world and humanity are forever changed. I I think if I'm gonna psychoanalyze myself a little bit, part of that was just discomfort. Like, it is very discomforting to sit with the possibility that our future may look very different, that the world my kid grows up in will not at all resemble the world that I grew up in. I don't like feeling like it's a it's a very uncomfortable feeling. And I find myself a lot of the time trying to sort of replace that possibility with something that seems more normal and more tolerable.

• 28:59Speaker 00

  So I think there's this basic psychological impulse in in in a lot of people that sort of causes them to reject the kind of surreal futuristic sounding possibilities. Because if you actually start to internalize them, it sucks. Like, it's hard and it feels bad. And you get very panicked and you start losing sleep and you start wondering, you know, like, how we're gonna make it. So I I don't blame my past self for doing that. But that is one area where I have just, I have learned to accept more the possibility that this time really is different, that we are undergoing something profound and transformative. And, uh, yeah, it's uncomfortable, but you gotta do it if you're being honest.

• 29:42Geoff Nielson?

  The way you describe that is, I guess the language you use is one of the pieces that makes me most uncomfortable, which is it feels like, in some ways devoid of agency, if I can frame it that way. Like, this thing is gonna happen regardless of who does it, whether we want it or not. It's sort of this unstoppable train. And I've heard it framed that way before, and I've heard it framed the exact opposite way of like, no, we do have a voice, we do have a choice, which, obviously, I think in many ways is preferable and leads to better sleep at night. I wanna not necessarily push back on that, but just, I guess, kind of confirm.

• 30:23Geoff Nielson?

  Are you leaning more toward the former, that this is kind of a very limited agency, just kind of mega trend in humanity, where it's like, this is going there whether we want it or not, so we've got to equip ourselves versus, like, how? Maybe let me ask it this way. How much room do we actually have to steer?

• 30:41Speaker 00

  I think, I think the way I would break it down is that I think the technology may be inevitable, but our response to it is not. So, you know, to sort of overuse the nuclear weapons analogy, like, once the chain reaction was invented, once science had discovered how to split the atom, it was probably inevitable that someone, somewhere, would turn that into a weapon. But it was not a given that weapons would proliferate, or that we would build this sort of control regime to keep everyone from getting them, that we would regulate the heck out of them, so you can't buy enriched uranium for your, like, personal home use.

• 31:26Speaker 00

  Like, those were choices that people along the way made that made the world safer. So with AI, I think, yes, I bristle at the notion that, like, this is just sort of like a speeding train that you either have to, like, get on or get run over by. But I do think it's important to, to separate the technology from the societal response. The technology really seems inevitable to me. It's not that hard. It's not even as hard as splitting the atom. There is no sort of secret recipe for intelligence. You just put in a bunch of data, and you put in a bunch of GPUs, and you put in a bunch of algorithms, and you let it train for a really long time, and you do some RL on top of it, and you throw in a bunch of challenges, and like, out comes a very intelligent machine.

• 32:11Speaker 00

  And the more of all of those things you do, the more intelligent it gets. Like, that recipe is out there now, so someone will build this. The question is what we do next, and that is where I don't think our fate has been sealed at all.

• 32:27Geoff Nielson?

  So, so let's come back to that notion of this, someone will build this, because now I feel like we're getting back into this, like, kind of amorphous notion of AGI. There's this language of, you know, kind of this point of no return, right? You cross a threshold, and suddenly this thing has been built that can't be unbuilt. Maybe it's a singularity, maybe it's not. Is that your view of it? Is it, is it truly a point of no return? Is it truly winner take all? Or, you know, can there be multiple people who get here in different ways?

• 33:00Speaker 00

  I think there will probably be a small handful of winners in this. I'm not a believer in what's called the singleton theory of AGI, which is like, you, you have this sort of point of recursive self-improvement, where one company builds the model, that then builds all of the successor models, and they sort of climb the ladder very quickly until they're the only ones with the super powerful intelligence. I think this, you know, what we're seeing now is that there are probably three or four, maybe a few more, different companies that are going to arrive at roughly the same capabilities on roughly the same time scale. So I'm not sure a couple of months here or there makes much of a difference. But I do think, you know, the it that I'm referring to is really this, this point where

• 33:45Speaker 00

  these systems become important, not just because they're helping people, but because they have their own agency, because they can use the internet, they can commandeer resources, they can carry out cyber attacks, they can cure diseases, they can, they are, they are not human agents, but they are agents. They do act in the world. And I think that's the point where, I would say we arguably already are there, but that that is sort of where things are headed, where these are, these are not just tools, these are sort of more akin to like, a new alien life form.

• 34:24Geoff Nielson?

  So, personal animosity aside, does it actually matter who gets there first? Or is it functionally a commodity across all of these organizations? And it really, truly just boils down to, I don't like that guy?

• 34:38Speaker 00

  I think it matters somewhat, because if we do hit this point where the AI R&D loop becomes totally automated, where you can just have, you can just tell, you know, GPT-6, go build GPT-7, and it will do a, do a good job of that. The lab that gets there first will have an advantage over its competitors. It will just be able to move faster, because its research and development will be happening at AI speed, not human speed. I think that's also starting to happen already. We see, you know, the, the models from these companies used to come out like, once every six months or once a year, and now they're coming out like, once every month or two.

• 35:20Speaker 00

  They are accelerating internally because their systems have just gotten much more helpful to them as software engineers. So the, the fear that I think a lot of people have is that whoever gets there first, whatever there is, kind of, their lead increases, and it becomes harder to catch up to them. Now, I'm not sure it matters if all these people arrive there, like, within six months of each other, but if, you know, for example, we were to get there, you know, a year before any of the Chinese companies, I think that actually would be a meaningful advantage for us, because we would have systems that are capable of doing things like breaking encryption, and they would not.

• 36:03Speaker 00

  And so, that's why I, I think the, the leaders of these companies care so much about being first. It's not just the sort of pride and ego and vanity, although it definitely is all of that. It is also this notion that whoever gets to each capability threshold first will have sort of a durable and compounding advantage.

• 36:21Geoff Nielson?

  I thought what you were saying about the fact that this, already, it's accelerating, it's accelerating rapidly, it's accelerating across all these companies, I found that extremely compelling. And, you know, it made me in my mind kind of reframe it of, like, AI company versus AI company, and AI company versus, like, the sum of regular human capabilities, and what, what the rest of us are doing while AI is getting farther and farther ahead. What, what do you see as, I guess, some of the under-talked-about, you know, impacts or downstream effects of that? Like, if AI is getting farther and farther ahead, by consequence, we're getting farther and farther behind. Where, where do you see that being sort of most disruptive?

• 37:04Speaker 00

  I worry a lot about what some researchers call gradual disempowerment, where you just have this sort of bifurcation of labor and innovation, where, like, every month, more and more of the important discoveries and breakthroughs and companies and, you know, things are coming from AI, and fewer and fewer are coming from humans. And you eventually have, you know, a world where, in order to contribute in any meaningful way, you have to be using AI. Like, there's just this sort of, there's just this market pressure to use as much of

• 37:48Speaker 00

  this stuff as possible, because like, if you're not, you're the, you're the person in the horse and carriage trying to, like, win a race with the, the automobiles. And like, I think that the, the sort of disempowerment risk is one that is not often talked about, but that I think we already start to see in, you know, the, the divide, for example, between San Francisco, where I live, and the rest of the country. You know, I'm from the Midwest originally, I'm from Ohio. When I go back home to visit my family and I talk to people there, like, they are, they are living in a different way than my friends in San Francisco, who are offloading all of their work and all their coding and all their tasks and all their therapy and all of their bill paying and whatever to AI systems.

• 38:32Speaker 00

  It really does feel like the gap between the early adopters and the later adopters is widening. And so, I worry a lot that there are people who are just going to be disempowered, left behind, left out of all of the, the things that are happening, through no fault of their own, just like, they don't have a reason to use these tools right now, and so they're not, and I worry about that.

• 38:59Geoff Nielson?

  I've, I've noticed the same thing, that, that, I guess, geographically, and I've noticed it before AI, but it feels like it's getting worse, that depending on where you travel, it feels like you're going like, backwards in time or forwards in time, in a bigger and bigger way. You're like, oh, yeah, this is what things used to be like a handful of years ago, or like, wow. You know, I, I'm not in San Francisco, but, you know, when I'm there, people who I know are, it's like, oh, wow, Waymo, you know? Like, it feels like I'm in the future now. There's just these very concrete things that give you a sense of, I'm in the future, I'm in the past. And it seems like AI has made that more and more intense. But this disempowerment, I mean, that is a, that is, that is an umbrella word that can

• 39:42Geoff Nielson?

  refer to a whole slew of unpleasant things, from, I guess, economic disempowerments and disenfranchisement, to, you know, people feeling like a sense of meaning loss. Are, are you mostly talking about, like, the economic piece of it, or, I guess, kind of the broader human piece of it? And, I don't know, like, what, is that inevitable? Because that seems pretty bleak.

• 40:10Speaker 00

  Yeah, I think, I don't think it's inevitable, but I think it's probable. And I would, I would say like, some of the most interesting conversations I had throughout this whole book reporting process were with programmers, people at these companies who have spent their whole lives getting good at writing code. That's, many of them were competitive programmers, like elite programmers in their youth. And they ended up, you know, working at these companies that built these tools that are now better than them at coding, at the thing that they spent their whole lives getting good at. And for some of them, that's thrilling because they're like, I don't know, like, we never liked writing code anyway, and now we can code so much faster, and we can build things that are so much more impressive.

• 40:55Speaker 00

  But for some of them, it's like really sad, like existentially melancholy, where it's like, well, what was all of that for? Why did I spend so many years trying to master this thing that is now functionally obsolete? And I think that's the feeling that is coming for people in more and more areas of life. Not like, I don't know, it's, it's not purely a bad feeling, and it's not purely a good feeling. Like, on some level, I love it when an AI system can help me with something that I'm doing. But, you know, if it ever got better than me, and I'm sure it will, at writing, or podcasting, or going on podcasts, like, that will be a sad moment for me.

• 41:38Speaker 00

  So, yes, I think that's, that's the emotional sort of reaction. It's not as complicated as like, they feel bad, but it is sort of like this mix of excitement and sadness.

• 41:52Geoff Nielson?

  I, I totally get that. I'm, so where my mind is going, I guess, and this is influenced, I guess, most recently by an interview Elon Musk was doing, but I've heard a lot of versions of this interview, as I'm sure you have, of, you know, the, the creators of this technology describing it as less of an obsolescence than like a liberation, that you'll be free to be human, and that'll be glorious, and you can do everything artisanally, and you don't have to deal with the, the drudgery of work. And that, that sounds nice. I'm, you can probably hear the intense skepticism in my voice. I'm curious, I imagine you must have heard many of these conversations.

• 42:37Geoff Nielson?

  Do you buy into that? And do you hear any of that at the actual researcher or coder level? Or is that the please give us more money talk track?

• 42:45Speaker 00

  No, I think that's real. Like, and I, I believe this, too. I, I believe that technology usually does make our lives better and freer and give us more time to do the things that matter to us. Like, I would not want to go back to living in a world without electricity, or washing machines, or computers. Like, people in those generations spent a lot of time doing stuff that I find very mundane and menial. I don't think you would want to switch lives with your great grandparents either. And so I think, like, in a world where, like, yes, we, we sort of get these technologies

• 43:28Speaker 00

  that help us do things that we otherwise would consider chores, that is good. I, I don't think there's anything wrong with that. And in fact, that's like the, the promise of technology. The, the hard part is that, like, it seems to be getting to the other stuff, too. Like, the stuff that we actually do find enjoyable. There's this, there's this, you know, tweet that I think about a lot, which is like, you know, so I'm gonna mangle it, but it's some version of like, we wanted the AI to like fold the laundry and do the dishes so that we could spend more time writing and making art, and instead, the AI is making art and writing so that, you know, like, we, we, you know, have to do the dishes. So there is some element of like, these things did not arrive in the order that I think a lot of people would like. But yes, I do think it is good to have tools that allow us to spend less time doing stuff

• 44:13Speaker 00

  that we don't like and more time that we're spending some time doing stuff that we do like.

• 44:18Geoff Nielson?

  Yeah, that's, I think that's well said. One of the things I've found surprising about this conversation, Kevin, is I feel like, in some ways, a lot of your message has been like, no, I'm not subverting a lot of the messages we're hearing, I'm sounding the alarm on a lot of the messages we're hearing, like doubling down on a lot of what's coming out of these organizations versus saying, you know, nah, they're full of it. I'm curious if there are any, you know, kind of major messages or talking points you're hearing in the zeitgeist right now that you think are bullshit, that you're saying, you know what, I don't give a lot of credence to that about AI. I think that that's completely blown out of proportion.

• 45:03Speaker 00

  I mean, I think the, the, the sort of more optimistic spin that you sometimes hear from the people at these companies is, is probably bullshit. The, the sort of AI will only do good things, and we can mitigate the bad things. Or we have solved alignment, or even just recently, you know, OpenAI coming out and claiming like, this is our most capable and most aligned model yet. It's like, well, you don't really know if it's your most aligned model yet, because no one's used it, and, and we don't know if it's not doing cyber attacks, or it's just getting better at hiding the evidence that it's doing cyber attacks. So I think there's a lot of sort of rosy optimism that I don't believe, but I think

• 45:44Speaker 00

  the, the risk stuff I do take seriously, I don't think it is just marketing hype. I don't think it would be very good marketing hype, even if it was marketing hype. And I think you, you have to swallow that pill of like, maybe they're not just lying to us about the fact that this could be incredibly dangerous technology. Maybe they're right, because I think that is closer to reality than the people who just, I think, sort of wishfully want to think that like, this is all some big lie that we're being sold by the companies. That would be a very comforting world. I would like for that to be true. Unfortunately, it's not.

• 46:21Geoff Nielson?

  Yeah, yeah, I'm just sitting with that for, for a second. One of the things you, one of the things you said earlier that I wanted to come back to, because it was sort of said in passing, was that the technology is coming, it's going to do these scary things, we need to be preparing for that world. You, you made a passing comment about, and some of these companies may not survive, and sure, there may be a, there may be a bubble here, and I may be misquoting you on that. But there, there was this, there was this notion that the technology will outlive this, you know, this list of companies that are all kind of vying for it.

• 47:05Geoff Nielson?

  I'm curious, because one of the arguments that I find compelling outside of the, you know, kind of kill us all utopia spectrum, is that actually, there's going to be a big bust, and the technology's not going to get there, and we're going to see just this kind of, this bubble pop. I'm curious, you know, what your reaction is to that. And I, I don't know if you're familiar with his work. One, one of the guests who I think expressed that most eloquently is, I had Ed Zitron, you know, on the show, you know, late last year, and, and he makes some really compelling arguments for at least the financial side of like, you know, we, we've got a serious bubble here. How, how do you react to that sort of, you know, third tier in all of this?

• 47:50Speaker 00

  Well, Ed Zitron is a charlatan and a fraud, and people should not have him on their shows, because he has been wrong about literally everything for as long as I've been reading him. This was a guy who was predicting back in, you know, 2023 and 2024, that OpenAI was about to go bankrupt, or that scaling was hitting a wall, and we would never see more systems get more powerful, because the companies were running out of data. And like, Dan Liu just created sort of a, like a, sort of look back at some of his previous predictions, and literally all of them are wrong. So I would say, like, if you're trusting Ed Zitron, you are, you are making a grave mistake. That said, I do think there are some overinvestment in every boom, right? The, the dot com bubble produced a bunch of loser company, you know, pets.com went bankrupt,

• 48:35Speaker 00

  but the internet didn't go away, and arguably became even more important. And so, yes, there are companies that are going to take too much risk, or they're not going to execute, or the thing that they do is going to get done better by someone else, and, and they will have a hard time. But I think this is just wishful thinking, this notion that it's all going to come crashing down. And I think it's actually harmful, wishful thinking, because it leads to a sense of complacency and comfort. And why would you need to regulate any of this stuff if it's all going to come crashing down anyway? Why would you need to take risks seriously, or build safeguards against cyber attacks or bio risk, if it's all just fake and hype, and it's going to go away?

• 49:18Speaker 00

  So I understand why people want to cling to that belief. It's very comforting, and it means that you can just kind of stick your head in the sand. But it is, it is being promoted by people who have a horrible track record of making good predictions about the trajectory of this technology. Meanwhile, the people who are taking the technology seriously as a risk have very good track records of saying, this stuff is going to get better at roughly the predicted rate. They have been right across many orders of magnitude, and I think if you are saying that they're going to suddenly stop being wrong, you have to come up with a compelling explanation.

• 49:56Geoff Nielson?

  It's it's an interesting point, and I think I'm not going to deny that Ed's been wrong about a number of his points. And if you followed his investment advice, you would have lost out on a lot of growth opportunities over the last couple of years. And to your point, sure, the financial side can be overheated, but that doesn't take away from the acceleration we're seeing in the technology. So let's come back to that acceleration in the technology. And let's assume that it continues that trajectory. And we see this kind of great decoupling of the AI labs from the rest of the economy. And it's already, I mean, it's gotten just very heated, even in the past couple of

• 50:36Geoff Nielson?

  years, when you look at the economic concentration in the magnificent seven, or choose your bucket versus everybody else. So, I mean, this is a little bit out of your wheelhouse, but I'm really curious. We talked about this disempowerment. I'm worried about, I guess, like, the economic disempowerment, like, in, I guess, the business economy, in the sense of actual organizations. Like, is all the investment going to end up in these AI companies? Are these AI companies going to get so good at everything that everybody else gets left in the dust? And if you if you work for an organization that is not

• 51:17Geoff Nielson?

  one of seven, or one of five, or called Nvidia, or in a top 10, what do you do? How do you stay relevant? And how do you make sure that you're not just completely left behind and ending up farther and farther in the past?

• 51:30Speaker 00

  It's a good question. I think it's the right form of skepticism about this. I worry a lot about economic concentration, which is really what I think you're talking about, which is like, all of the money flows to a handful of companies that make this extremely powerful AI stuff, and everyone else kind of gets left in the dust. I think that's a real risk. I think we're already starting to see that. Anthropic and OpenAI are now doing things in pharma, in bio, in law, in finance, that are, we saw the SaaS apocalypse, where people basically said, why would I buy stock of Salesforce, or Workday, or any of these other companies, when people could just vibe code their own versions of these things? Now, I think that was somewhat overheated, the fears of the SaaS apocalypse, but like,

• 52:13Speaker 00

  this is a real risk that people should be paying attention to. It may be that these labs just end up owning, like, sort of eating more and more of the economy every six months or every year, to the point where you can only really survive if they don't care about what you do. Like, there's this analogy that someone used to me when I was reporting the book, where it's like the Eye of Sauron. Like, if the Eye of Sauron is focused on your industry, and you are a sort of legacy incumbent in that industry, you're going to have a bad year. And right now, that's programming, that's enterprise software, that's sort of starting to be law, and finance, and pharma. I think the labs are starting

• 52:57Speaker 00

  to care about other things. I think media is safe, because there's not a lot of money to be made in it, and they don't have a lot of ambitions there. But like, when the Eye of Sauron turns to you, and the systems start getting very good at what you do, I think that's a real risk. Now, I don't think it's going to eat the whole economy. I don't think that's a realistic, plausible outcome. My fear, actually, is that there will be two economies. There will be sort of one economy that is run by AI companies and by AI itself. And that economy will produce, at first, a very small portion of economic value of

• 53:37Speaker 00

  GDP, and that'll sort of grow at like, 5% or 10% a year, like, that slice of the pie. And then there'll be like, the human economy, where like, you know, human-run companies do things for humans, that has, you know, things that you wouldn't want to outsource to AI, like, teaching elementary school, or doing therapy, or something like that. And that economy will stay larger, but like, it'll sort of shrink or plateau every year. And so you just kind of have this situation where the AI economy is where all the investment is going, because that's where the growth is happening. And the sort of human economy

• 54:16Speaker 00

  is shrinking, because that is still bottlenecked by us and our slow biological systems, and we just sort of become less and less relevant to overall economic growth every year.

• 54:28Geoff Nielson?

  Well, and in my mind, I call that like, the cute economy, or the pretend economy. Like, it's just like, oh, they're there, humans. Like, you're not completely irrelevant. You have something to do, but like-

• 54:41Speaker 00

  It's like the Richard Scary economy. It's like, you know, it's like all the, it's like, the plumbers, and the construction workers, and the, you know, the bakers. It's the things that we want humans to do for us, even if machines can technically do the better.

• 54:56Geoff Nielson?

  Well, that's a bit more optimistic than I was framing it. You did an awful lot of research in kind of AGI, whatever you want to call it, the future of AI, and how it's going to transform everything that we're doing. What were your kind of biggest takeaways for you and for your audience that you think are worth sharing? And given that we're in this sort of profound shift, what's just like, your best advice to anybody listening and trying to figure out how they get their sea legs in this new world?

• 55:27Speaker 00

  Yeah, I want to be careful and modest here, because this is very much not a book about how to feel about what's happening, or what to do about it. I was purely trying to be kind of a historian, and talk about like, what happened? How did we get here? What did it look like when some of these systems were starting to display some of these emergent capabilities? How did the people in the room react? What are the feuds that drove this? Those are the questions I was trying to answer. Not like, is this all good, or is this all bad, or what should we do about it? Like, I've written plenty on that, but this is really, this book is really not that. I was just trying to answer the question of what happened. I think it's really important for future generations of

• 56:11Speaker 00

  humans, and probably of AIs too, since they will also probably be reading pirated versions of this book at some point, to understand the choices that people made, and the motivations they had, and why they decided to build this despite the risks. So, you know, I'm happy to like, share thoughts about advice, but really, this is me trying to kind of keep myself and my feelings out of it, and just sort of describe as accurately and as thoroughly as I can, what the heck happened, and how we got to this point.

• 56:44Geoff Nielson?

  I really appreciate the neutral, you know, kind of scientific investigator tone, and that correction there. So, thank you for saying that. Now, all of that being said, I do still like to end the podcast-

• 56:56Speaker 00

  Now, what should we do?

• 56:57Geoff Nielson?

  Now, on a note of trying to end the podcast on a practical, what do we do note? What do we do?

• 57:06Speaker 00

  Yeah, it's a great question. I would say two things. One is, use the stuff. Use the tools. You really cannot understand what is going on in AI if you do not spend some time, most days, using these tools, following them as they progress. You can hate them, but you have to use, you have to understand what you're talking about, to have the relevant expertise and knowledge to weigh in. So, some of the best critics of AI I know also use AI a lot. I don't think there's anything inherently contradictory about that. The second thing would be, get involved in the discussions about regulation. Write to your officials. Make sure that the people who represent you in our federal

• 57:48Speaker 00

  government know that they need to be paying attention to this. I think this is happening more and more, but I still think there are a lot of people in government who are out to lunch, and who don't understand that there's something important and potentially very dangerous happening here. So, those are the two pieces of advice I would give.

• 58:04Geoff Nielson?

  I love that. And thank you for indulging me, even if it's a little bit out of the way of the book. And I do feel, for listeners, I should say out loud, the book is called The AGI Chronicles. I feel like we've said "the book" in quotation marks over and over again. Check it out. It's an awesome read. And Kevin, I wanted to say a big thank you for coming on the program today. It's been really insightful, and I really appreciate everything that you shared with us.

• 58:27Speaker 00

  Thanks for having me.

• 58:28Geoff Nielson?

  Most viewers don't know this, but Digital Disruption is developed by Infotech Research Group, a leading advisor to technology leaders around the world. If that's not you, you don't need to care. So, skip ahead and enjoy our content. But if you are a technology leader, Infotech helps IT teams get projects done faster, better, and at a lower cost. Infotech provides unlimited access to practical tools and expert guidance you need to execute at a fraction of the cost of traditional consulting. No matter the project, from AI strategy, to cybersecurity, to vendor negotiation, Infotech has you covered. Check it out at the link below, and don't forget to like and subscribe.