An Insider's Guide to AI Doom

The conversation around AI harms has become incredibly heated in recent weeks. But it's also incredibly confusing. In this short explainer episode, Cal walks through the three major categories of beliefs around AI in the current moment and helps explain who belongs where and how they all relate to each other. If you're trying to keep up with AI news you have need to learn this framework. Chapters: (0:00) Intro (1:01) Why the debate is so heated (2:01) Group 1: The Doomers (rationalists and effective altruists) (9:12) Anthropic, OpenAI, and the doomer civil war (14:03) Group 2: The Accelerationists (17:51) Group 3: The Centrists (24:00) The full landscape and the fight over regulation Links: Buy Cal’s latest book, “Slow Productivity” at ⁠www.calnewport.com/slow ⁠ Get a signed copy of Cal’s “Slow Productivity” at ⁠https://peoplesbooktakoma.com/event/cal-newport/⁠ Thanks to Jesse Miller for production and mastering, Jay Kerstens for the intro music, and Nate Mechler for research and newsletter. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Cal Newport?

  The conversation surrounding AI harms has really heated up in recent weeks, but it's also at the same time become really confusing and hard to follow. There's many different groups involved and it can be difficult to try to figure out who belongs to what group, what group believes what, how do they feel about each other? I don't know how you can follow the current AI discussions about harms if you do not understand this landscape of different players. So that's what I want to help you figure out today. This is a short episode I'm going to record where I'm going to lay out the three major categories of people involved in the AI harm debate.

• 0:43Cal Newport?

  I'll tell you who's in each category, what they believe and how they feel about each other. So if you are interested in the current discussion on this technology, you need to understand how this landscape is actually configured. All right, so let's get into it.

• 1:01Cal Newport?

  All right, so let me give you two caveats before we get started. One, I'm going to make some mistakes here. I'm basically just winging this episode and the boundaries between these different groups I'm going to talk about are a little bit porous and there's some people that fall in between and some people that won't like the characterization. So I apologize in advance, but I believe the landscape I'm about to present you is more or less correct and will definitely help you better understand the current discourse on AI harms. My second caveat is I'm going to try to keep my personal opinion out of it. That's not the point of this episode. I really do want to just give you, to the best of my understanding, the current lay of the land in the sort of AI discourse that's happening at the moment. Okay, so let's get into this. I'm going to bring out my tablet here because this gets complicated enough that we need

• 1:46Cal Newport?

  to actually probably see how these different groups connect. I'm going to start over here by talking about the first of the three major groups that's involved right now in the current AI discourse. And we're going to call these the Doomer Group. I mean, I'll pluralize it. Let's call it the Doomers. All right, this is an important group in the current AI harm debate. Who is in this group? Well, I'm going to start by putting RATS and EA down here. These two philosophical movements give the intellectual underpinning for the group that I'm calling the Doomers. So let me give you the quick history on this.

• 2:29Cal Newport?

  You have to go back to Silicon Valley futurist movements in the 90s into the 2000s. In particular, there is a movement called the Extropians that was futurist and very utopian. It believed that technology was advancing at such a rate that it was going to deliver us in the future, a utopia, probably a transhumanist utopia, where we merge with the machines in such a way that we could have infinite life or whatever else we think is critical to our utopian visions. There was two main technological threads that the Extropians wove together. One was nanotechnology, which was just becoming big in the 80s and 90s when a lot of the Extropian movement had started.

• 3:12Cal Newport?

  They were influenced by a thinker named Eric Drexler, who wrote about this for both academic and public audiences. So they thought nano machines would be able to repair our bodies, reverse aging, reverse disease, et cetera. The other major technological thread that the Extropians held on to was AI. And so maybe we're going to have a super intelligent AI that could act like essentially a benevolent digital deity. So this was going to be the other technological move that was going to deliver us this utopia. All right, so that's the Extropians. They're pretty positive. Then you have a particular Extropian. This was a young man named Eliezer Yudkowsky, who dropped out of school after the eighth

• 3:55Cal Newport?

  grade, was reading a lot of science fiction, found his way to the Extropian mailing list, got really into these ideas, was writing a lot for these email lists, eventually got funding to start his own sort of techno utopian institute. He came to believe as time went on that though super intelligent AI could deliver a utopia, it could also just as easily kill us all. And that it was actually more eschatological than the Extropians realized that super intelligence AI was inevitable, but it's not inevitable that it was going to deliver good things. And so he began to write and think quite a bit about what would be required to prevent

• 4:40Cal Newport?

  super intelligent AI from killing us all and to help make sure that people understood this danger before we created this technology. So Yudkowsky's group became known as the Rationalists, right? So that's why I have RATS written here under Doomerism. It's a shorthand for the Rationalists. All right, but I also wrote EA under there. That stands for Effective Altruist. Now, this is a movement you've also probably heard a lot about recently. This is a little bit complicated, but I'll summarize it really briefly. So the origins of the Effective Altruist is William McCaskill and Toby Ord, who are two philosophers at Oxford in the 2000s, 2010s. They were heavily influenced, among others, by the Princeton ethicist Peter Singer to

• 5:23Cal Newport?

  be, to apply logic and rationality to being altruistic. And so they're trying to be very effective at altruism. So early on, the Effective Altruists, for example, were really big on raiding charities, right? Because they figured you want to give your money to a charity that's going to do the most good per dollar invested. So they worked on projects to help raid charities. They also had another campaign called GiveWell, which said, look, if you have a high paying job, the best thing you can do for the world is keep that job, but give a significant portion of your high income towards really effective philanthropy. So like, these are the type of things the Effective Altruists were thinking about.

• 6:06Cal Newport?

  But pretty soon, they also became, as a general movement, very worried about super intelligent AI. There's lots of different points of influence between the Rationalists and the Effective Altruists. A key one is Nick Bostrom. He was also a philosopher at Oxford, but he was also heavily involved on the Extropian mailing list in the 90s into the early 2000s. And he was very convinced, especially by Yudkowsky's style arguments, that super intelligent AI was inevitable and without care was going to kill us all. And so he was one of several voices in Effective Altruism that moved the movement away from

• 6:47Cal Newport?

  just thinking about effective giving and towards what was known as a long-terminism view that said, wait, the most altruistic thing you could do is prevent the extinction of humanity. And so he started an institute at Oxford that looked at different existential risks. But over time, super intelligent AI killing us all became the dominant risk they cared about. So we're in a world now where Effective Altruism is a very large movement. It's incredibly well-funded, mainly by tech sentimentaries and tech billionaires. It deals with lots of things. It's not only focused on super intelligent AI in the way that rationalism really is, but it's become a dominant theme. William McCaskill gave this well-known speech, I believe it was at Google, where he effectively

• 7:30Cal Newport?

  said, look, if we're going to be super rational, we have to think long-term. And effectively, why worry too much about hundreds of millions of lives right now when we really, if we want the biggest return on our efforts, need to think about preventing human extinction, which would prevent tens of trillions of future lives from ever being born? They pulled on this logical thread long enough and a lot of Effective Altruists said, preventing super intelligent AI from killing us all, that is what Effective Altruism should focus. Now, again, not everyone in EA is focusing on that. Dustin Moskowitz, for example, recently gave a huge amount of money to foreign aid after USAID was effectively denuded by the Doge effort.

• 8:15Cal Newport?

  So it's a bigger movement than the Rationalists, but super intelligent AI is a big part of their portfolio. I've heard from a lot of people that in the Bay Area in particular, the Rationalists and the Effective Altruists are sort of indistinguishable. This is why, for example, if you read the really good reporting by Cade Metz, the lead technology reporter for the New York Times, he often puts those two words together. He will say EA/Rationalist circles are saying X because in the Bay Area, they become more or less the same thing. Okay. So to summarize, the Rationalist and the Effective Altruist think super intelligent AI is inevitable. They've been thinking this long before LLMs. They think that if we're not really careful, it will kill us all.

• 9:00Cal Newport?

  But if we're careful and do this just right, it will deliver us a transhuman utopia that is like the ultimate good and therefore should be a, this should be a major focus of our altruistic instincts. Okay, so I want to return now.

• 9:17Cal Newport?

  To our diagram, and I'm going to add, this is where things get a little bit more complicated. I'm going to add Anthropic and OpenAI under the Doomer category. Okay, so this is really important. There are a lot of AI companies in the world and in America that have hyperscaled LLMs or working on big projects, but those two labs in particular, Anthropic and OpenAI, are distinguished by the reality that they come out of the rationalist, effective altruist intellectual tradition. They were started in large part to try to fulfill the rationalist, effective altruist

• 9:58Cal Newport?

  idea that we have to carefully align superintelligent AI to prevent extinction and to instead get us closer to a transhumanist utopia. That's why OpenAI was founded. You have to remember Elon Musk originally funded OpenAI because DeepMind, which also came out of these circles, was bought by Google, and they said, "We can't trust a big company to shepherd superintelligent AI the right way, so we'll start OpenAI to do that." And then Anthropic was breakaway employees from OpenAI. They broke away two years later after the for-profit subsidiary was declared in OpenAI. They broke away and said, "OpenAI is not sufficiently pursuing the rationalist, effective

• 10:40Cal Newport?

  altruist goals of steering away from bad superintelligent AI and towards good, superintelligent AI." Anthropic in particular is just completely and sensuously connected with these movements. I mean, most of, many of their employees, Dario Amadei himself, he used to live in a group house with other really major effective altruists. His sister, who helped start Anthropic with them, married one of them. A lot of the initial funding for Anthropic came from effective altruist individuals like Sam Bankman-Fried and effective altruist investment funds. There's anyways, there's a lot of connections, which I don't want to go through now, but I would recommend you look at my New York Times op-ed about this from a few weeks ago,

• 11:23Cal Newport?

  or Cade Metz did a really good appearance on the Daily, the New York Times Daily podcast a couple of weeks ago, where he walked through this history. Michelle Goldberg did a good column on this history. The Wall Street Journal last weekend had a good column on this history. Look at Adam Becker's book is very good on this. Listen to my podcast with Adam Becker and Ed Zitron, where we went through this. So there's a lot of sources out there detailing these connections. But what I want us to focus on here is this idea that Anthropic and OpenAI share the same intellectual taproot as the rationalist and the AI-focused effective altruists. Now, what makes this complicated is that the relationship between these two frontier

• 12:04Cal Newport?

  labs and the rationalists and effective altruists who are not part of those labs is strained right now. So you have some people from these circles, right, like Eliezer Yekowsky from the rationalists, who are completely upset in Anthropic and OpenAI. They say, "You guys are moving too fast. You're going to end up with the bad, superintelligent AI." They all share the belief that eventually we want a transhumanist utopia delivered by AI, but like Yekowsky would say, "You guys are going too fast. We're not, we're not ready to do that right. I think you're caring too much about money or winning." And so like Yekowsky is openly hostile towards these two labs. You have a bunch of effective altruists who are sort of in the middle, who maybe are

• 12:48Cal Newport?

  a little bit on board with Dario Amadei and saying like, "Yeah, we need to have regulation that'll allow Amadei to set the pace because he'll do it right, and we need to prevent competitors from going too fast because they're not going to do it right." And then you have a bunch of effective altruists who are out there lobbying hard to say like, "We have to basically shut down these labs. They're doing it wrong." So it's complicated because they all believe the same general things. Superintelligent AI is inevitable. It will either be a good god or a bad god. They differ, however, in whether they believe we can get the good god right now or whether we're not ready and we need to wait. And so the people at the labs think we can deliver the good god, and a lot of the people not at the labs are saying, "We're not ready yet."

• 13:30Cal Newport?

  Now, the reason why that's confusing is that these feel like they're opposed parties, right? So when you see Nate Suarez and other, and then some effective altruist figures surrounding Bernie Sanders as he talks about legislation to try to prevent superintelligent AI, you would assume that these are from a completely different mindset than the frontier labs that this legislation would apply to, but they all come from the same Doomer mentality. They just differ on where we are on this timeline that they both agree about. All right, so that's the AI Doomers, but that's only one of three parties. So let me draw another one up here. Another key party right now, I'm just going to write down the word, I'm going to abbreviate accelerationist.

• 14:14Cal Newport?

  So the accelerationists, and they, I don't know, they'd call themselves other things. The accelerationists are a group that believes that the Doomers are weird and that we need to go as fast as possible because there's a lot of benefits from AI. Why slow that down? And also, there's an economic race occurring with China that we want to win. So who is under the accelerationist rubric? Well, you have a bunch of venture capitalists in here, which makes sense. They want to invest in a lot of AI companies and have them not be impeded in trying to make a return on investment. So, you know, you'll have Mark Andreessen is a well-known accelerationist that would be in here, the sort of the host of the All In podcast would probably put themselves in that category as well.

• 14:57Cal Newport?

  They're all investors or venture capitalists who are not happy with what the Doomers are doing. You also have, interestingly, other AI firms. I'll write that down. So Anthropic and OpenAI are the two, along with DeepMind, but they're kind of a, they're a subunit of Google that's kind of being reorganized. But Anthropic and OpenAI, these are the two firms working on hyperscaled LLMs that are fully in the Doomer philosophy that comes out of rationalism, effective altruism. A lot of the other companies, major AI companies in the US and abroad, are not so directly connected to Doomer philosophies, and they think the Doomers are nuts.

• 15:38Cal Newport?

  And so now there's a kind of a feud between OpenAI and Anthropic and all the other AI companies. This is why Jinson Wong, for example, is coming out and saying, "What's with all this sci-fi stuff? This is crazy. Just don't build unsafe things. This is a fairy tale." Mark Zuckerberg, a couple of weeks ago, had a tweet that was really pushing back. He, I'm paraphrasing here, but he was like, "Why is Anthropic obsessed with recursive self-improvement? Why don't you take that effort and instead put it towards building useful products?" Sort of calling out that sort of Doomer mindset of, "We have to race towards superintelligent AI." The other figure over here that's very important is the White House. So the executive branch is pretty firmly in the accelerationist camp right now as

• 16:23Cal Newport?

  well. So President Trump's rhetoric about AI and what we should be doing is straight up sort of accelerationist-style rhetoric. You shouldn't fully trust the accelerationists. They have an obvious motive here, which is money, right? So if you're a venture capitalist, you don't want impediments on you being able to, you know, your startups to be able to grow and return on investment. If you're Jinson Wong, you don't want chip exports bans, right? Because you want to sell as many chips as possible everywhere you can in the world, and the White House just has a lot of connections to that, the sort of right-aligned corners of Silicon Valley, and they sort of like them better, and they're sort of going along with what they have to say.

• 17:08Cal Newport?

  Right, so the Doomers are very influenced, we want to understand biases. The Doomers are very influenced by these philosophies that go back to the 80s, these sort of futurist, transhumanist philosophies. The accelerationists are motivated by money, money and power. We're kind of comfortable with the accelerationist story because we're used to this. Like, yeah, you know, obviously, like, companies want to make more money, and they don't want stuff to stop them. The Doomer story is a little bit more complicated, and I think it's sort of thrown a lot of people off. This idea that they have an eschatological, futurist, utopian vision that is driving them is way stranger in the American experience than just being driven by, "I want to get a return on investment. I want my company to grow." But those are two major groups. So what's the third?

• 17:53Cal Newport?

  So I'll put the third in the middle here, and I'm going to call this, I'm going to call this the centrist.

• 18:01Cal Newport?

  And I'm only calling this group the centrists because they're literally in the center of my diagram between the doomers and the accelerationist. Okay, so the centrist, this is a group that is concerned about AI, largely is in favor of the idea of we need some sort of controls or safeguards or regulation of AI. So they disagree with the accelerationists who say, let's just let her rip. But they also have no interest in the doomer philosophies, which they feel to be anti-humanist and quite frankly, kind of strange. And so they want to regulate AI like a normal technology. We want to regulate this like we would regulate any other technology that we're worried about being

• 18:44Cal Newport?

  unsafe. And if we think companies are acting weirdly, then we need to investigate and find out why. So they, they share the doomers' interests in regulation, but the style of regulation, the motivation is different. And they don't, they agree with the accelerationist's rejection of a lot of the ideological underpinnings of the doomers, but do not agree with the accelerationist's let her rip philosophy of like, let's just let it grow. All we're doing is holding back good, anything we do to slow down is just holding back possible good outcomes. So who are the centrists? There's sort of three very related groups. I'll write them as three separate ones. We have the AI ethics crowd. We have the, I'm writing NTCS here. This is the

• 19:28Cal Newport?

  normal technology computer scientist. So the computer scientists who understand enough about the technology and enough about the ideologies to be like, look, this is a powerful technology, but normal, we got to treat it normal, very wary of the hype. And then we have cybersecurity experts. Below that, I'll just write the word cyber, who are worried in particular about the computer hacking ability, or the cybersecurity implications of language models that are very good at producing computer hacks. So those are the centrists. All of these groups are related, like for example, some of the AI ethicists are also these normal technology computer scientists, but not all of them. Some of these cybersecurity people also come from the computer science world,

• 20:09Cal Newport?

  but not all of them. Some of them are from industry. So there overlaps, but they're, they're not all the same. AI ethics is a term of art that it's a field that's been around for a while. It's sort of, it definitely predates, obviously, like large language models and OpenAI and Anthropic. The AI ethicists have traditionally focused on proximate harms. Like, we're worried about this thing that AI is doing now, or could immediately enable, right? So they're very worried about proximate harms. They would talk about things like bias in models, like, hey, cultural biases that we don't like as a culture are in the training data sets. And if you train a model on this data set, it might have these cultural biases. And then if you're using that model to make decisions,

• 20:54Cal Newport?

  it might be making biased decisions in ways we don't want it to do. But they're also worried about other applications of AI, like for surveillance, for example, which is happening in China right now. Man, they're using these AI tools to hunt down and persecute in terrible ways minority groups that they don't like. So the AI ethicists have, they care about the proximate cause. They were also the first, I want to give them credit. They were the first to really point out early on, the unsavory aspects of the futurist ideologies that drives the doomers. So if you've heard terms like Tescriel, which is an acronym that captures a lot of related philosophies,

• 21:37Cal Newport?

  the R in there is rationalism, the E in there is EA, but they have transhumanism, long-termism. There's a bunch of sort of related and sub-branches of these various thoughts. But Tescriel, this was Tim Necker and others worked on this, email Torres as well, I believe. They're arguing, hey, you got to look closer at these philosophies, because there's some deep anti-humanist aspects to it. It's almost eugenicist, and this idea of what matters is, doesn't matter how many people die along the way, if we can get these, these like idealized transhumanist individuals to survive and be the apotheosis of the human race. And so they were kind of early at be wary about these philosophies. So it's something that they've been at. I have a foot, you know, I'm in the

• 22:22Cal Newport?

  normal technology computer science circle. You would also have like, Arvind Narian is in here. I would put Gary Marcus in here as well. I'm in here as well. It's like, look, we, we look at the actual technology, we're less worried about superintelligence than we are about good old fashioned, straight up harms that it could or couldn't cause right now. I do a lot of work, for example, on the specific types of long horizon LLM-powered agent experiments that OpenAI and Anthropic are running that are causing problems. Other AI companies are not, they're running these experiments, I believe, because it's part of their race to get to superintelligence first. And this is why I've advocated, we need to get in there, we need to investigate, we might

• 23:03Cal Newport?

  need to regulate, not because I'm trying to stop the bad god from coming, but because I think these are bad actors, that their motivations are leading to dangerous behavior, and we got to get in there. So I think the normal technology computer scientists, you know, I'm one of them, but I also have a foot in the AI ethics world. I'm a part of the Center for Digital Ethics at Georgetown. I also see these things through an ethical frame. The cybersecurity world, like, these are experts at computer hacking and cybersecurity. Here, I'll take this off the screen, which is not my fundamental expertise. But if you talk to the cybersecurity centrists, they'll say two things. One, this is a fairy tale, the superintelligence stuff. Two, these hacking tools are really dangerous, and we got to stop advancing these just so you can do well

• 23:47Cal Newport?

  on some benchmarks, or, or read some, you know, vision you have about where you should be on the route to superintelligence. Like, these are, these tools are very dangerous for very realistic, pragmatic reasons. All right. So again, I'll go back here. This is our landscape then. This is what you have to understand about the debate right now about AI harms, is that it's these three circles, and the accelerationists don't like the doomers, the doomers don't like the accelerationists, the centrists have issues with both, and agree with parts of both. Inside the doomers, you have civil wars, inter-scene battles, where they all believe the same general thing, but this group, the rationalist, the effective altruists, often think that these two doomer companies, Anthropic and OpenAI, are going too fast, or they're going to get it wrong. So it is

• 24:32Cal Newport?

  a confusing picture when you see little aspects of it, little corners of the picture in isolation. But when you zoom out, it makes more sense. So I want to give this lay of the land. I mean, I would say like, probably the most confusing thing about our current moment is the fact that the, there's a call for regulation. And I think the most confusing thing at the moment is that the centrists and the doomers, both are calling for regulation, but they're very different visions and motivations for what that means. And I would say, if you're observing this, and I'm hearing this directly from congressional staffers, Congress that's looking towards regulation, so it's mainly the Democrats right now, but also some Republicans as well, are being

• 25:13Cal Newport?

  very influenced by the doomers, especially the effective altruist AI doomers, are having a huge impact on the types of regulations that Congress is thinking about. They have a lot of money, and they're aggressively lobbying Congress. But also, I've been told by, again, from staffers on the Hill right now, that the effective altruists, through the years and through these sort of fellowship programs, have been placing specifically, this is a plan, they've been placing effective altruists onto congressional staff. So a lot of these congressional staffs have their own staffers in the ear of the reps and the senators, saying, superintelligence is the biggest worry, this is what we have to stop. Why is this a problem from the centrist perspective?

• 25:57Cal Newport?

  Is that the concern is that intervention or regulation inspired from a doomer philosophy is probably not what we need. We need normal technology intervention. What are you doing? Why are you doing it? What's your motivation? And that we could actually have much more effective interventions that are going to apply, to be honest, way more to the two doomer labs than anyone else. The concern about just focusing on superintelligence, or recursive self-improvement, which is an idea that comes out of the futurist circles, not computer science, I think we're going to miss actual, meaningful, probably workable style interventions, which needs to be much more, in my opinion, focused on the specific technologies and techniques with, with

• 26:37Cal Newport?

  specific types of safety guards. And so we're roughly aligned with the doomers who are saying we need to intervene, but we're really not aligned on why we need to intervene and what that should look like. So I think that's the most complicated part of the picture right now, is you have multiple groups arguing for intervention, but they're not all the same people. And we really have to pull that apart. So let me show this picture one more time. Let this sort of sear into your understanding. This is the landscape of the AI harm conversation. So whatever you're reading, whether it is an article, or it is a tweet, or blue sky thread, or it's a podcast, or it's, you know, a buddy of yours at a dinner party, you got to first

• 27:21Cal Newport?

  categorize that person somewhere in here to understand what they believe, why, and what their agenda is. And I'm not going to tell you right now, which of these you should agree with. I obviously, I, you know, I'm a centrist, but that doesn't mean I'm right. Doesn't mean I'm right. I don't even really want to be a part of this debate, to be honest. I find it very stressful. But I do think clarity really matters. So if you're in one of these versus the other, or you're listening to someone in one of these versus the other, you need to know where they are and why, so that you can properly contextualize it, whether that ends up you rejecting their thoughts, or embracing them even harder. All right. That's my best attempt to explain this landscape. Hopefully you found that useful. See you next time.

• 28:05Cal Newport?

  Hey, if you've made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday. You can sign up at calnewport.com/ideas. And when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today, calnewport.com/ideas.