A Stark Warning From a Former AI Doomer

Cal Newport takes a critical look at recent AI News. Cal interviews Emile Torres who provides a stark warning about AI. Video from today’s episode: youtube.com/calnewportmedia (0:00) A stark warning from a former AI doomer (2:01) Interview with Emile Torres (1:13:57) Cal recaps the interview with Emile Links: Buy Cal’s latest book, “Slow Productivity” at www.calnewport.com/slow Thanks to Jesse Miller for production and mastering, Nate Mechler for research and newsletter, and Ben Crain and Rob Montz for video editing. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Cal Newport?

  An argument that I've been making for a long time now is that if we really want to understand what's happening in AI, we have to understand the futurist ideologies that are heavily influencing the top frontier labs, Anthropic and OpenAI. The problem is, is that these ideologies are confusing and there's rumors swirling around them and the excerpts we hear can sometimes be absurd and we think that can't possibly be true. So I thought what would be useful is to bring in an insider to explain to us what is really happening in these labs and what do they truly believe. This is why I'm really happy

• 0:40Cal Newport?

  to have on the show this week Emil Torres. Emil is a researcher and academic who focuses on eschatological and utopian movements. But what makes Emil really useful this discussion is that they used to be involved in these ideologies. Emil comes from the world of futurist ideologies that believed that superintelligent AI would either kill us all or redeem humanity with a post-human digital god. But then Emil's moral compass led them out of that world and now Emil has become, in my opinion, one of the most astute chroniclers, expositors, and critics of these ideologies. So Emil's gonna join me here in a second and they're

• 1:24Cal Newport?

  gonna walk us through what exactly are these different beliefs, how do they intersect, what is their role in the major frontier labs, and why should we be really worried about them? If you have any ambition to understand what's really happening in the world of AI, you need to listen to this conversation with Emil. It is going to, quite frankly, probably shock you, but this is a good shock. This message is one that we all need to hear. So let's get right into it.

• 1:59Cal Newport?

  Emil, thank you for joining me on the show.

• 2:03Speaker 01

  Thanks for inviting me. Great to be here.

• 2:05Cal Newport?

  I've been looking forward to this conversation for a long time. I think both your story and your research is being too overlooked in our current moment. I think it is very relevant to a lot of issues that people care about or are worried about or don't understand about the current AI situation. And so I'm excited to really get into both the meat of your work as well as the origin story, which I think is very important. But I thought we should start briefly with definitions, just so that everyone is on the same page when we're talking about these various movements. So can you explain what the acronym TESCRIL means?

• 2:43Speaker 01

  Yes, I can. So this was coined by myself during a collaboration with the computer scientist Timmy Gebrew, which we initiated in, I think, late 2022 or so. And we were specifically interested in the underlying ideological drivers behind the current race to build ASI or artificial superintelligence. So TESCRIL is a clunky, kind of unsightly term, but it does a lot of important explanatory work. So one way to think about the acronym is it's basically just a name for the dominant orthodox worldview within Silicon Valley. That simultaneously highlights a certain intellectual history that is relevant to

• 3:24Speaker 01

  understanding that worldview. So it explains what the worldview is, where it came from, what shaped it, how it evolved, and so on. And so the acronym itself stands, it represents seven different ideologies. And the letters appear in roughly the same order as the ideologies emerged chronologically over the past almost 40 years. So the T stands for transhumanism, E for extropianism, then singularitarianism, cosmism, rationalism, effective altruism, and long-term, a bunch of like really big, clunky, polysyllabic terms. But these ideologies are really fringe within American society. And I think, you know, if you were to go

• 4:06Speaker 01

  and talk to the average person on the street, they would find them to be so bizarre that they'd consider the grand cosmic vision behind these ideologies as basically a nonstarter. Nonetheless, they are massively influential within Silicon Valley. Like basically everybody, you know, from Elon Musk to Mark Zuckerberg to Sam Altman to Dario Amadei are transhumanists. And many people in Silicon Valley don't even call themselves transhumanists. But that's for the same reason that I've never publicly stated I'm a round-earther, right? It's just like, what else would I be? Obviously, I'm that. So transhumanism really is the water that people in Silicon Valley swim in, it's the air that they breathe, and it's also

• 4:49Speaker 01

  the backbone of the whole TESCRIL worldview. And so, once you understand transhumanism, then all of the other ideologies, some of which are just variants of transhumanism, are sort of easy to grasp.

• 5:02Cal Newport?

  Well, I want to ask you a history question here, because this is, some of this is new to me. The history of this, that I best understand, starts with the E. So the history I've been writing or talking about starts with extropianism becoming a pretty big utopian futurist movement, I guess, like the 80s and 90s. From there, you get rationalism, sort of simultaneously effective altruism, which starts with a bunch of different goals, embraces ideas from rationalism and long-terminism, and becomes more focused on a sort of more of a long-terminism, transhuman, utopian, anti-AI type of orientation. All right. So what pieces am I missing in that really simplified history?

• 5:47Speaker 01

  No, that's a pretty good overview. So when Gebrew and I are talking about transhumanism, we primarily mean modern transhumanism in its libertarian form. That libertarian form, like the first organized libertarian transhumanist movement was the extropians, right? And so this is a movement that, like you said, emerged in the late 1980s, early 1990s. There was, you know, somewhat famously, an extropian mailing list that brought together transhumanists from all around the world. It's where people like Eliezer Yudkowsky, one of the leading AI doomers out there, you know, was able to exchange ideas with people like Nick Bostrom, who's another very important figure in the formation and development of the TESCRIL

• 6:31Speaker 01

  worldview. So you're absolutely right. And then, you know, the other ideologies right after E in the acronym, like singularitarianism, this is basically just transhumanism, libertarian transhumanism with a focus on the singularity. And at this point, I feel like most people listening probably know what the singularity is, right? That wasn't the case like five years ago. So, you know, very quick definition, it's, it actually has several definitions, but the most common one these days is an intelligence explosion, right? Triggered by recursive self-improvement. So you get an AI system that's just clever enough or smart enough to develop the next generation of AI systems, and you get this positive feedback loop, and suddenly, through some kind of exponential or hyperbolic explosion of capabilities,

• 7:15Speaker 01

  you end up with a superintelligence. And then beyond that, you go out and colonize the entire universe, plunder the vast resources of the cosmos, and create this, you know, sprawling multikalactic utopia among the stars. And cosmism, the next letter, is just transhumanism that emphasizes the aspect of this worldview that I just articulated, which is, it's basically the idea of like, well, why stop at just radically reengineering the human organism, which is the central goal of transhumanism, to create a new post-human species? Let's go out and create technologies that enable us to re-engineer the entire universe, you know, whole galaxies, maybe intervening on the, you know, fundamental

• 8:00Speaker 01

  fabric of spacetime to manipulate the world in ways that we find desirable. So it's kind of just like transhumanism with this like broader cosmic vision. And rationalism was founded by Yudkowsky, who was a transhumanist, extropian, singularitarian, who hired the leading modern cosmist named Ben, namely Ben Gersell, who worked at the Machine Intelligence Research Institute, which is originally the Singularity Institute founded by Yudkowsky. So rationalism, like emerged directly out of this movement. And basically, the idea was, okay, if we're going to establish a cosmic post-human utopia at some point, we're going to need to develop these advanced technologies. There's no other way around

• 8:43Speaker 01

  that, especially superintelligence. But to develop them in the right way, such that these technologies do not destroy us, but rather enable us to establish this, you know, sprawling utopia, heaven among the literal heavens, we're going to need to optimize our rationality. So that is essentially the main thrust behind rationalism. And effective altruism, as you mentioned, I mean, it emerged basically from rationalist blocks, like overcoming bias and less wrong. And the focus there is not so much optimizing our rationality, but rather optimizing our morality. And although the story that effective altruists like to tell is that initially, they were focused primarily on global poverty and then animal

• 9:26Speaker 01

  welfare, and then this other cause area called long-termism came into the picture later on, really, the effective altruists from the very origins of this movement were focused on long-termism, right? So long-termism is basically just cosmism, except it has a more explicit ethical foundation to it. And that ethical foundation, I know there are a lot of isms that I've mentioned so far, but that ethical foundation is utilitarianism, specifically total or totalist utilitarianism. And that just says that we have an obligation to increase the total amount of value in the universe across cosmic space and time. How do we do that? Well, one way is if you have a fixed population, it's to increase

• 10:11Speaker 01

  the well-being or welfare of the entities that exist in that population. The other possibility is to just increase the population, right? So if you double the population and everything else is the same, you know, ceteris paribus, then you've doubled the amount of value. So from the long-termist view, you know, cosmism, basically, like one way to think about it, is it's just like, wouldn't it be really cool if we as transhumanists radically reengineered ourselves, became post-human, you know, uploaded our minds to the cloud, went out and colonized the universe? Wouldn't that just be like a really awesome thing to do? Long-termism goes a step further and says, actually, no, we have a kind of ethical obligation to do this. As Will McAskill says in his 2022 book, What

• 10:55Speaker 01

  We Know the Future, the upshot is a moral case for space settlement, right? And so you could maybe see why Elon Musk has called long-termism, quote, "a close match for my philosophy," right? So that's in a little bit more detail, just fleshing out some of the details of your very good first pass history.

• 11:18Cal Newport?

  So it sounds like the tricky part, but both the simple part about understanding this bundle, the use of term that you and Tim that often use, is there's these little differences, which can be confusing to people where the, you know, Yatkowski and the rationalists would be, they were dismissive of a Kurzweil style, singularitarianism, because it was too optimistic, and he was more negative, and the effective altruists are, they're doing more like of these explicit utilitarian expected value computations where the rational, you know, so there's these little differences, and they don't, they're not all the same group, and yet they're really knit together by fundamental ideas about transhumanist utopia, avoiding extinction, and technology being

• 12:03Cal Newport?

  delivered. But what I could never figure out, and maybe you know, what were the cultural and technical forces that really made the Transhumanist Foundation emerge in the 70s and 80s, and give birth to this movement that ended up essentially taking over Silicon Valley?

• 12:18Speaker 01

  May I comment briefly on the remarks you just made, before I answer that question, if you don't mind? So one way that I've explained it in the past is, if you have like, like, let's call it an ideological map, right? And maybe it's like six by four feet. If you are right up close to, and so they're different, these ideologies are plotted geographically on this map. If you are right up next to the map, you will see that rationalism, effective altruism, for example, occupy different locations, right? If you take like five steps back, you find that all of these ideologies are basically coterminous, right? They're like at the exact same location. And just for perspective, if you are looking from a distance at this ideological map, you would find

• 13:02Speaker 01

  that like my view, which is very anti-transhumanism, for reasons maybe we could discuss, or maybe like Timothy Geber's view, is like five feet away, right? So you've got this, this cluster of views, of worldviews that share a whole lot of ideological real estate. And in perspective, you find that they are, they're almost exact, I mean, all of these, these communities have the exact same vision of the future. That vision is a cosmic post-human utopia through the creation of artificial superintelligence, right? And so this leads to my second comment, if I may, which is just that, that one way to understand how all of these different alternative, basically

• 13:45Speaker 01

  alternative interpretations of Tesco real fit together, is that within the Tesco real movement as a whole. So everybody in the Tesco movement is aiming to create this, this cosmic post-human utopia among the stars, right? Then there are differences of opinion about the risks associated with developing this necessary technology, which is superintelligence, right? And so you have two camps, doomers and the accelerations. And of course, it's a spectrum, there are people in between. But the doomers are represented by individuals like Yatkowski, and he is, oftentimes presents himself as pro-human and anti-ASI, but he's neither of those things. He's not pro-human, he's not pro-ASI. Like most people in the Tesco movement,

• 14:30Speaker 01

  he is a pro-extinctionist. Part of the vision, integral to the vision of establishing a cosmic post-human utopia, is that our biological species disappears, right? So in the good case, if ASI is aligned, we disappear. This is the most important thing for people to understand. These AI companies are run by individuals who, who have extensive connections with the Tesco movement, and they are trying to bring about, trying to inaugurate the post-human era. And in the post-human era, our species has no place, right? Utopia is an exclusionary concept. Someone or something is always left out. So what, you always have to ask, like, what is being left out of this proposed utopian vision? In theirs, it's our biological species, right? And that's for two

• 15:13Speaker 01

  reasons, and I know I'm going off on a little bit of a tangent here, but on the one hand, we will become, we will become technologically, scientifically, economically, and politically irrelevant. We will become obsolete, right? In a world that is ruled and run by these digital post-humans that they imagine becoming, we just will serve no function, because they will be, I mean, if these individuals are successful in realizing their vision, these post-humans will be superior, right? And so they will be able to do anything that we do better than us. But the situation is even worse than that, because not only will we be useless, but we'll be in the way, right? So we will continue to suck up valuable resources that they could use for other purposes,

• 15:56Speaker 01

  like building a Dyson swarm around the sun, which Demis Hassabis says we may start doing in the early 2030s, or perhaps covering the entire planet in data centers, which, which Sam Altman has said on record is the most likely outcome of superintelligence. Sam Altman also, by the way, has expressed pro-extinctionist ideas, just like all these other people, because he says in a 2017 blog post, that the only way that we will survive the advent of superintelligence is by merging with machines. Well, what happens when we, we merge with machines and become, you know, uploaded minds and digital, well, we're not going to be human anymore, right? So he's essentially saying that the only way to avoid a certain kind of extinction is to undergo another

• 16:40Speaker 01

  kind of extinction, namely a radical transformation into the post-human form, right? And so I know I danced around your, your question a little bit, but you have these doomers and these accelerationists, and another like really key thing to understand is that we have no place in the future. I mean, there are, there are some people in the community who would argue that biological life as a whole has no place in the future. Will MacAskill, in the aforementioned book published in 2022, says that it's probably the case that most wild animals have lives that are not worth living. And hence, he concludes that it may be good, our systematic destruction of the biosphere,

• 17:23Speaker 01

  right, may actually be good, because the fewer wild animals there are, the less wild animal suffering. So the goal is to ultimately bring about this digital world, in which there may be, you know, non-human creatures out there, but they're all going to be simulated. They're all going to be, you know, within a computer simulation that we, that we live in. So this is a deeply anti-life, anti-human vision. Yeah, that's the most important thing to emphasize.

• 17:52Cal Newport?

  These are the people who watched The Matrix and were on the Matrix's side. Like, oh, that's a good idea. But, but let me, okay, so here, here's a tricky part about this, which, which I think is confusing a lot of people. You could probably help us out here. The tricky part is, you have Anthropic and OpenAI, where, you know, I've documented this, you've documented this, are very strongly connected to Tesco real. I mean, they're basically, they're formed out of Tesco real motivations, right? Like to bring about this, this sort of utopian future. But they're feuding, this is what confuses people, with other people from within the movement. So now you have huge amounts of effective altruists and rationalists who are pushing back against OpenAI

• 18:33Cal Newport?

  and Anthropic. And I think it fools people into thinking, oh, there's, there's a lot of sort of people outside of that movement pushing back on what OpenAI and Anthropic are doing. But actually, a lot of the pushback is coming from people within the movement who agree with the eventual goal of, yeah, we want to get to a transhuman utopia, but just don't think that Dario Amadei or Sam Altman are doing it right, right? And I think this idea that, and I've been arguing, hey, and I've been talking to a lot of, you know, congressional staffers, etc., saying, look, there's a danger in having this team just fighting against other members of its own team. That debate is only going to happen within, so is, is that more or less right? Is that there's a, there's a schism within the movement, but it's a really actually pretty small one. They're all

• 19:17Cal Newport?

  on, they're all on board with the same track. They just think specifically Anthropic and OpenAI maybe aren't doing it quite right?

• 19:24Speaker 01

  That is correct. Within Team Post-Human, there are family disputes. And as many of us know, family disputes can be vicious, right? But there's this whole other team, which is Team Human, right? That's the team that, that I'm on. And the, the debate right now, the public debate about the development of superintelligent machines should be between Team Human and Team Post-Human. But Team Post-Human has, has more or less monopolized the, the conversation. They've hogged the microphone. So you're exactly right. So for example, I mentioned that Yatkowski is not anti-ASI or pro-human, even though if you read his book, if anyone builds it, everyone dies, you kind of get that impression. But he has explicitly said in interviews that once

• 20:09Speaker 01

  post-humanity arrives, then it may be unethical to continue to have biological humans. And he draws this, this, you know, quite offensive, apelist analogy between a post-human choosing to have biological humans or children, or humans continuing to have biological humans in the post-human era, and someone right now intentionally choosing to have a disabled child, right? And in the same interview, he talks about how, you know, it might be good just to like wipe all humans off, off of the, the earth. In other interviews, he said that if the only and reliable way to create superintelligence is flitting about the universe, quote, having fun, is to sacrifice all

• 20:52Speaker 01

  of humanity, then he would make that trade-off, right? So he has this, he's on Team Post-Human, not Team Human.

• 21:01Speaker 01

  The debate has to do with AI safety, which is exactly that dispute, the tension between the accelerationists and the doomers. The question is how to go about developing superintelligence. Everybody agrees, including Yatkowski, that we need superintelligence. As Nick Bostrom says, virtually all paths to a utopian paradise in the future take us through superintelligence. So we can't not build it. The question is when to build it and how, right? And so you have some people who are on the doomer side of the, in the doomer camp within the Tesco movement, who are screaming that we aren't yet ready to build superintelligence. We don't know how to align it with human values. But what they mean by human values is the Tesco worldview, right?

• 21:46Speaker 01

  So for example, if there were a superintelligence that were aligned with my anti-Tesco worldview, and if this superintelligence were to shape the entire future of life in the universe, which is how they imagine, what they imagine superintelligence ultimately doing, they would think that's an existential catastrophe, right? Because what they want is a cosmic post-human utopia, right? So it's not about aligning it to like human values. It's really about aligning it to Tesco values. Anything other than that, anything, any, I mean, literally the definition of existential risk is any event that prevents us from establishing this, you know, post-human paradise among the stars. So if you have an ASI that's aligned with some non-Tesco worldview,

• 22:29Speaker 01

  that is going to be existentially catastrophic, right? So you have people like Yatkowski, who are saying that we need more time to solve the alignment problem, to figure out how to control this superintelligence such that we can direct it to radically re-engineer the human organism, upload our minds to computers, and ultimately go out and plunder the vast resources of the universe.

• 22:48Cal Newport?

  All right, so the people within Team Post-Human that are upset at Anthropic and OpenAI, it's because you don't have, you haven't sufficiently solved the alignment problem to proceed with our shared vision. And the labs are basically saying, no, I think we can, I think we can do it. We'll figure it out on the fly. We can't really slow down now because of, you know, what have you, China? So, but it's, it's a, it's basically, that's the bottom of the feud is like, are we able to solve the alignment problem enough? Though it sounds like if you talk to, you know, these Anthropic employees, what they're really saying is like, hey, we might not, but now we're, now we gotta go, we gotta follow through because other people are gonna start doing it now too. And so we gotta get there first.

• 23:32Cal Newport?

  Because if we don't, then for sure someone else will do it, and it will be the bad version. It will be the version that kills us all.

• 23:38Speaker 01

  One thing to notice here is you will hear like the AI CEOs and others at the company say that there are really three reasons that they can't stop, right? One has to do with market forces. So just another brief, like historical tangent here. The AI race began motivated almost entirely for these ideological Tesco-related reasons. If you look at what the founders and the people around the founders were saying, you know, back in 2010, 2015, et cetera, no one was talking about profit. No one cared about profit because all of them realized that utopia is so much more valuable

• 24:22Speaker 01

  than however many billions of dollars they might gain in personal wealth, right? This is about fulfilling some kind of grandiose eschatological vision. That is why the race started. I know because I was in the community back then, and we had these conversations. And, you know, I knew a lot of these people. And so, but over time, market forces absolutely have crept in and become a major driver of the ASI race. So things have evolved. It started off very ideological. Now, the profit motive is a piece to the explanatory picture, especially after ChatGPT was released in November of 2022. The second reason they'll say is, well, there's some kind of like game theoretic predicament between the AI companies, right?

• 25:06Speaker 01

  And so like, like they all think that they're more responsible than the others. So Anthropic's trying to race ahead of OpenAI. OpenAI doesn't like that. They think that Anthropic is on the wrong path. So they're trying to race ahead of Anthropic. And then the third one, of course, is geopolitical. And that has to do with the, the race between, so zooming out, the race between the US and China. But what, the important thing to notice is that what all of these boil down to is anxieties about whether or not utopia, their version of quote-unquote utopia, will actually be realized, right? So Anthropic is thinking that if OpenAI builds superintelligence in the very near future, that superintelligence isn't going to be aligned. And the probable outcome is going to be an existential catastrophe, maybe human extinction, right? If China wins the AI race, then we also don't get utopia.

• 25:50Speaker 01

  Because, you know, whoever builds the first ASI will then establish a singleton, you know, like a world government. They'll have what Bostrom calls a decisive strategic advantage. And so China will then realize its version of like a quote-unquote communist utopia. Well, that's not the Tesco utopia that they're envisioning. So ultimately, what all of these excuses, all these pretexts for continuing to race ahead boil down to is deep anxieties about utopia. And maybe there's one last thing to mention, if I may.

• 26:22Cal Newport?

  Yeah.

• 26:23Speaker 01

  Which is, which is just that, you know, people have become aware. So the topic has exploded into the public discussion, public consciousness over the past few weeks as a result of Jacob Cox's resignation at Anthropic and an employee there retweeting him and saying, yes, we believe there's a 10% chance of human extinction in the near future. So it's really, really important to properly interpret the terms that these people use. Because oftentimes, the way they use terms that seem to have a straightforward definition is defined in an idiosyncratic manner, right? Safety, what does safety mean? Safety means building in AI, advanced AI,

• 27:08Speaker 01

  that is aligned with the Tesco worldview. What is the likely outcome of successfully aligning ASI with the Tesco worldview? Well, our species will be sidelined, marginalized, disempowered, and ultimately eliminated because we'll be of no use and we'll be in the way. And like Yatkowski says, maybe there are ethical reasons, right? So another term that is, is, that contains hidden ambiguities is the term human extinction, right? So there are, there are, both human and extinction are ambiguous. So if you look at the academic Tesco literature, you'll find that everyone defines humanity according to what I call, for lack of a better term, a broad definition.

• 27:52Speaker 01

  On this broad definition, it denotes both our species and whatever post-human successors might come after us, right? So an immediate implication of this definition is that our species could die out literally next year without human extinction having happened, right? As long as there are post-humans to take our place, then humanity will persist. And if humanity persists, then that's sufficient for human extinction to have been avoided. The terminology is confusing because they count post-humans as humans, right? But the other ambiguity is if you, if you take like a narrow definition of humanity as meaning our species in particular, Homo sapiens, then the term extinction becomes ambiguous, right? So again, for lack of better terms,

• 28:37Speaker 01

  there is a difference between terminal extinction, as I call it, and final extinction. Terminal extinction is just the, a scenario in which our species disappears entirely and forever, full stop. That's the only condition that needs to be satisfied for terminal extinction to have happened. Final extinction kind of builds on this. And it denotes a scenario whereby our species disappears entirely and forever, and we do not leave behind any post-human successors. All the Tesco people care about is avoiding final extinction. That is it. Many of them hold a pro-extinctionist view according to which we should bring about terminal extinction without bringing about final extinction, which is just to say, we should survive for long enough to create our post-human successors.

• 29:21Speaker 01

  And then once our post-human successors are here, we should dip out. We should, you know, follow the dinosaurs and the dodo into the eternal grave of extinction, right? And so, so it's just, it is deeply frustrating to see the public conversation where people are like, you know, all of these like Tesco guys, like Jacob Coxon, who has deep roots in effective altruism and rationalism, which grew out of libertarian transhumanist movement, talking about the importance of avoiding human extinction. Because most people sitting at home are interpreting that. It's like, yeah, we should like, like, do whatever we can to ensure the continued survival of our species. That is not what they're talking about. They're talking about bringing about us surviving long enough to bring about the post-human era, at which point, who cares?

• 30:04Speaker 01

  What happens to the 8.3 billion people on Earth? You know, we're just irrelevant at that point. And from a grand cosmic perspective, we're utterly insignificant, right? So our, our role in this whole eschatological scheme is ultimately to be the transitional species between the biological and digital realms, right? And that is our role. And once our eschatological duties have been discharged, there is no reason for keeping us around. So this is a deeply pro, deeply anti-human position. I hope I didn't ramble for too long, but maybe that was useful, useful clarification.

• 30:38Cal Newport?

  No, this is critically important. And I think for a lot of people listening to it who are not anywhere near the Bay Area and not anywhere near Tesco, this seems almost absurd, or it seems almost, you know, like who would actually hold a view like that? But, but partially, what I think makes you such a good, not only critic, but unders, you know, expositor on this movement is that you became a part of it for a while. And so I was hoping to ask you about that. Because if we could understand what attracted you to it as, you know, incredibly bright academic, you know, I mean, what attracted you to this? And what did you see on the inside? I think this might help people understand how it might be possible that you have an entire essentially sector of the economy

• 31:22Cal Newport?

  and region of the country that all strongly believes this.

• 31:26Speaker 01

  Yeah, absolutely. I mean, the absurdity of this this grandiose vision of the future is part of the reason that Teskril is still kind of outside the Overton window. I think that is shifting. You know, there are more and more people waking up to like, like, learning about effective altruism and rationalism and transhumanism. I'm seeing those terms all over the place. They're in popular media articles like like never before. And so, yes, there are many, many people in Silicon Valley who are true believers. And if you find this like whole story to be absolutely nuts, you are, you are getting it. It is completely nuts. And so for me, like, I grew up in a very religious household. And there was lots of talk about eschatological motifs like the rapture.

• 32:11Speaker 01

  And, you know, I was convinced that Bill Clinton was the Antichrist. In 1992, when he was elected, I was like overcome with terror. I was like, oh, my God, the tribulation is right around the corner, you know. And then in my early twenties, I, as many people do, apostatize my faith. And for, it took another like several years before I came across the work of Ray Kurzweil, who I later ended up being a research assistant to for his sequel, sequel to the book Singularity is Near, which was published in 2005. And at first, I thought that these, this singularitarian view, again, like, singularitarianism is a version of transhumanism that just emphasizes the singularity. I thought it was completely outrageous. There's a point

• 32:56Speaker 01

  in the book where Ray Kurzweil says, he believes there is a greater than 50% chance of us making it through this century without destroying ourselves. And he adds, but then again, I've always been accused of being an optimist. So that means like up to a 50% chance that we destroy ourselves. So you might say like, well, why develop these technologies in the first place? Well, Kurzweil's acquaintance, Bill Joy, he was the co-founder of Sun Microsystems, who's technologist, no, no, you know, technophobe. He published an article, a famous article widely discussed in 2000, on April 1st, 2000, in Wired magazine, of all places and all times, titled Why the Future Doesn't Need Us, in which he argued that these technologies

• 33:38Speaker 01

  are too dangerous to develop. So we need to impose like broad moratoria on entire fields of emerging science and technology because the risk is just too great. The reason that these, the Teskril folks are plowing ahead with developing these technologies is because they are necessary to reach utopia. I alluded to this earlier, right? So if the definition of an existential catastrophe is any event or state, et cetera, that prevents us from reaching cosmic posthuman utopia, then failing to develop these technologies would itself be an existential catastrophe. Right? So we, there's only one way forward. And so initially, when I, when I came across his work and that, that, that quote

• 34:20Speaker 01

  about the risks associated with advanced technologies, I was, I was kind of mortified. But making a long story short, like, I eventually got on board. And it was kind of like, well, if you can't fight them, then, you know, maybe the best thing is to, is to join them. The, the deeper I got into the transhumanist movement through this, through the literature on existential risk, the more compelling I found it because it really perfectly fit the religion-shaped hole that was left behind. There are so many parallels between transhumanism and traditional Christianity from the promise of resurrection, right?

• 35:01Speaker 01

  If you're a transhumanist and you don't live long enough to live forever, as Ray Kurzweil says, you can always opt to have your body or just head and neck cryogenically frozen so that you can be resurrected, not from the grave, but from the vat, right? All the way to the theistic aspects, like Teskril is a, I would consider to be a new religious movement. And I would consider this religious movement to be theistic, right? The, the only difference is that the Teskril worldview inverts the traditional relation between humans and God. Rather than God creating us, we're creating God in the form of ASI, a God, you know, God-like AI, a machine God, a digital deity, et cetera. So I, this is what attracted me. I thought, oh, here's like a

• 35:46Speaker 01

  an ideology that is checking all the same boxes, all of the things that I miss. You know, meaning, purpose, eschatological hope about the longer term future of humanity, the possibility of immortality, you know, of like transcending my biology, becoming this digital, you know, piece of software, living in a computer, something that Sam Altman himself has already taken steps to become by signing up with a company called Decto, to have his brain chemically preserved and then digitized. And, you know, ultimately, just, just like living in paradise with radical abundance and, and no suffering, that just really appealed to me. And so I, you know, it was really around 2009 that I publicly, by which I mean, I published an academic paper

• 36:28Speaker 01

  on, on embracing transhumanism and, you know, at the time, like the early, earlier Teskril movement. And I continued to be a true believer really up until 2019, at which point I defected and realized that, once again, I had been completely wrong about, you know, dearly held beliefs. And so I, since then, have become a critic. And I, if you, if one uses the term a bit loosely, kind of like a whistleblower, you know, sounding the alarm about like just how extreme and radical the, the views are that are held by, you know, people at Oxford and in Silicon Valley.

• 37:08Cal Newport?

  So I want to get soon to, to why you defected, but sort of two follow-up questions about your experience there that I'm very curious about is one asking about the role of, there's a sort of logical and sometimes even mathematical complexity to some of the views, which I've heard makes it very attractive if you're smart, if you're an engineer, because it, you can very much believe like, oh, I can understand this. This is actually kind of complicated. This would explain why maybe the masses don't yet get this. But, you know, Dunning Kruger, I'm smart enough to like, it makes sense to me that I'm, I'm understanding this future. I mean, then two I want to ask you about, I've heard that it's a very self-reinforcing community. When you're in this world, there's a lot of support, there's a lot of social connection, there's a lot of money sloshing around, there's jobs to be had.

• 37:53Cal Newport?

  And so I'm curious about those two aspects about being inside the Teskril world.

• 37:58Speaker 01

  Yeah, the community is very culty, right? That, that obviously ties into the new religious movement aspect. There are many cultish elements. I mean, people in EA are like encouraged to like, you know, get, pursue career paths that are recommended by people in EA. They're, you know, they have their, their, their own like lingo, you know, their own argo, you know, the technical terms, everybody's into expected value theory and Bayesianism. And, you know, they oftentimes date each other. There are a lot of policles. I mean, people are now discovering this with respect to rationalism, because rationalism at EA, I mean, they're like twin sibling communities. Again, the, really, the biggest difference is

• 38:43Speaker 01

  one specifically focuses on rationality in order to do the most good, especially by creating in aligned superintelligence. EA is about optimizing your morality, mostly by focusing on the very far future, which means being concerned with AI safety and AI alignment. So you can see there, you know, a lot of parallels. But, I mean, even when I was in the community, I remember I, in 2019, I spent several months at the Center for the Study of Existential Risk at Cambridge University. And over drinks, I had conversations with, you know, at like a party with 15 people about whether or not we might be in a cult, you know.

• 39:26Speaker 01

  So it's, I mean, it was these conversations that, that did end up playing a role. You know, it was one of many factors, but did end up playing a role in me ultimately defecting from the community. So, yeah, I mean, I'm sure at, if you go on social media and you look, you search for rationalism, I mean, you'll find that like, people are describing as basically like, like a sex cult. And it kind of is, like, there are big orgies that they have and, you know, everyone participates. It's, it's, it's a very, very weird, very weird community.

• 40:00Cal Newport?

  And there's a lot of money, too. I was surprised to see you estimate, it was a bigger number than I had heard, but you know more than I do, that the EA community, for example, has something, you said $26 billion worth of donations have been pledged to that community, which allows them, you know, you go to Oxford and it'll be like, one of the nicest buildings on campus is going to be an EA-funded, you know, building. You look at Anthropic, that's a lot of EA money went in to fund that. There's these Future of Life Institute, these are really large, really wealthy. So is that, we're really talking about billions of dollars, but I assume mainly comes from tech centimillionaires and tech billionaires who like where this narrative puts them, that makes them the heroes of the world. But we're talking about huge amounts of money, too, right?

• 40:46Speaker 01

  Yeah. Just as a quick side note, I mean, long-termism tells billionaires exactly what they want to hear. It says, not only are you morally excused from helping the poor, or, you know, focusing on animal welfare, you know, here and now kind of problems, but you are a morally better person, a superior person for focusing on the far future. And so, you know, again, like Elon Musk is a fan of long-termism. No wonder, when like, a key aspect of long-termism is that it's morally good, it's, you could say it's morally obligatory to go out and, you know, colonize the entire universe, conquer the cosmos. It's, you're absolutely right, that there's a huge amount of money. That figure, the 26 billion, that was before FTX collapsed, right? So Sam Bankman-Fried was one of the most prominent

• 41:31Speaker 01

  EA long-termists, right? He never had like a global poverty phase, like some EA's do. I mean, EA has been described by Molly Lieberman as a bait and switch, where it sort of draws people in by advertising that they care about, you know, alleviating global poverty. And then once people are in, they say, well, actually, there's a much better way to do the most good. If you can affect, like positively affect a tiny, tiny percent of the trillions and trillions and trillions of future digital people spread throughout the universe, the amount of good you'll do, like in terms of expected value, is just many orders of magnitude greater than if you were to lift the 1.2 billion people in multi-dimensional poverty today

• 42:12Speaker 01

  out of poverty. So it's really, it's kind of always been focused, but so, you know, Sam Bankman-Fried was a leading long-termist. So they lost some of the pledged money. But since then, I mean, they've, they continue to attract just truly mind-boggling amounts of money. And I mean, if you count, if you count Anthropic as an EA long-termist organization, and you should, right? Dario Amade was one of the very first effective altruists, right? He was one of the first to sign the Giving What We Can pledge. I think he was 34 on

• 42:50Speaker 01

  the list. You know, Amanda Asko is a prominent philosopher at Anthropic. She, her ex-husband is William McCaskill, right? So if you count like, like companies like Anthropic, then there's even more money, right? If you start to include all of that money. And of course, Anthropic is about to have its IPO, which the figure I saw most recently, I think was 2 trillion, is what they might aim for.

• 43:14Cal Newport?

  It's coming, it's coming down now because people are, institutional investors want a little risk mitigation because they think they're sounding like crazy people. Because they, if you're a pension fund, you're not on board with the Tesco L trains. You're like, whoa, whoa, whoa, whoa, wait, what do you guys believe? But, but a lot, over a trillion for sure.

• 43:32Speaker 01

  Yeah, over trillion. So there's going to be this massive influx of money. And also, just as a side note, there's, there's another, as it were, using the term loosely, whistleblower, who was heavily involved in the Tesco, EA, rationalists, et cetera, communities like I was, named Holly Elmore. And she has known personally some of the people at Anthropic, and she's also out there sounding the alarm. One of the quotes, this is what I wanted to mention, was that the people at Anthropic decided to pursue superintelligence to ultimately create utopia for post-humans, right? I've already mentioned that. And she goes on to say that many in the company are concerned that the superintelligence that

• 44:13Speaker 01

  they create will just completely usurp or supplant humanity, and that this is why it's really important to figure out how to, to transfer our values. Like basically, our ethical and axiological values to the superintelligence, so that it can go out and do what we, what we ultimately think is the right thing for it to do, ethically speaking, namely, like colonize the universe, you know, create all these computer simulations and maximize value. So anyways, Effective Altruism is about to get an enormous amount of money. And this is a community of people who oftentimes present themselves as living humble, modest lifestyles. And some of them do that, right? But there's also like a kind of televangelist

• 44:58Speaker 01

  aspect to this. You know, so like McCaskill, when he published his book, he went out to a vegan restaurant in, in New York City and paid, I think it was, it was more than $400 a plate, right? Also, the Center for Effective Altruism, well, technically Effective Ventures, which is the umbrella organization over it, secretly bought a, I think it was 15 million pound castle, so a palatial estate in Oxfordshire, you know, to, to host conferences and get togethers, et cetera. And when a New Yorker journalist asked them about this,

• 45:39Speaker 01

  they said, this was before FTX imploded, they said, well, if you do the math, right? If you crunch the numbers and you, you, you know, figure out the expected value of this, if there's even a small probability that, that this castle brings in the next Sam Bankman-Fried, the investment will have totally been worth it, right? And that's not only, like, there was a chateau, and I think the Czech Republic or something, that the EA's also bought. So like, once you, you, you know, EA, like from a distance, like doing the most good you could possibly do, that, that, that sounds great. And I'm all for that, right? And I'm all for like donating one's income. You know, I'm committed to doing that as well, like all of my disposable income to charities. But when you actually like get close and

• 46:23Speaker 01

  look at the details, it's, it's just a deeply flawed view. I mean, the whole reason Sam Bankman-Fried went into crypto, I mean, he initially worked at Wall Street, was to, to follow this career path that EA's called Earn to Give, whereby you try to make as much money as possible, even if you work for, quote, immoral organizations, as Will McCaskill says, so that you can donate more money to, to charity, right? So this is what, what got Sam Bankman-Fried on Wall Street, then he worked for Center for Effective Altruism, and then he got into crypto. And he was, you know, a hardcore utilitarian. And if you're a hardcore utilitarian, you believe that there is no act that is intrinsically wrong. Like that is, this is a key aspect of the ethical theory. All that matters are

• 47:08Speaker 01

  the consequences. So there's nothing intrinsically wrong with fraud, with lying, with cheating, there's nothing intrinsically wrong with murder or genocide. If the, the outcome is overall good, you know, all things considered, then you have a moral obligation to engage in fraud, and, and lie and cheat and steal and so on. So you can, you can see now how Bankman-Fried ended up, you know, justifying in his own mind, his, his actions. Because, you know, he was a long-termist, and he was donating, I mean, FTX invested in Anthropic, right? So he was really concerned about ensuring that we build and aligned with the Tesco worldview, superintelligence, so that we could ultimately maximize value from now until the heat death of the universe.

• 47:52Cal Newport?

  So how did you turn around on this? So you're, you know, in the 2010s, 2017, you write a big book that's sort of explaining some of these ideas in a positive light. What was it that led to your second, you know, apostasy in your life? Why did you, what turned you?

• 48:13Speaker 01

  Yeah. And apostasy is the right term, because I do think that, that I was involved in, in a new religious movement. So there were, there were really a bunch of, I guess there were like maybe three considerations that resulted in this drastic pivot away from the, what I consider to be my intellectual home for a decade. One was the realization of the poverty of utilitarianism, which really underlies like basically the entire Tesco movement. Even some of the far-out accelerationists who argue that the ultimate goal is to climb the Cartagev scale, which is a measure of how much, basically how much

• 48:55Speaker 01

  energy a civilization uses, to, to bring about the heat death of the universe faster. Like ultimately, it's a kind of utilitarian view, because it's saying that, that there is something of value that we need to maximize, and that thing is entropy, right? So it's not like, it's not like happiness or pleasure, as traditional utilitarians would say, but it's, it's, so it's still like deeply utilitarian in, in flavor. And so utilitarianism, like, like, consider its, its view of, its conception of persons, of you and I. Like, we have no intrinsic value. There is only one thing that has intrinsic value in the universe, and that is well-being or welfare, right? You and I are nothing more than the containers

• 49:39Speaker 01

  or vessels or vehicle, vehicles of value. And so I matter as a means to an end, not as an end in of myself. Insofar as I am bringing value, like pleasure, into the universe, then I am, I am fulfilling my ethical obligations. And, and I matter only insofar as I'm doing that, right? And so this gets back to the idea of like maximizing the future population. Because, so if we are just these fungible value containers, then if you create a whole bunch of new value containers, right? Especially if they're digital, because you create more digital people per, you know, volumetric unit of spacetime or whatever than you can biological beings. So if you create all of these, these value containers, and they all have like net positive amounts of value on average, then you're going to, you're going

• 50:21Speaker 01

  to also increase the total amount of value in the universe. So like this, this view of people as mere means to an end, it, it didn't sit right with me. I guess it did for a while, but the more I reflected on it, the more convinced I was, this is just a deeply impoverished way of, of thinking about things. And maybe I'll mention the one other reason, which is that Which is that I began to realize that the the vision of the future that these people had articulated is essentially techno-capitalism on steroids, right? It's about going out and conquering the universe, you know, subjugating nature, maximizing economic productivity. I mean, literally, Bostrom at one point defines

• 51:05Speaker 01

  existential risks as any event that prevents us from reaching technological maturity. And technological maturity, he defines as a situation in which we have maximized economic productivity and fully basically subdued nature. It's very Baconian. And one thing I noticed is was kind of completely obvious in retrospect. It's that this vision was the product of almost entirely the product of white dudes who are very privileged, you know, quite wealthy in Oxford and in Silicon Valley. And if you look at the Tesco literature, there's virtually zero reference to what the future ought to look like from alternative perspectives. Feminism, indigeneity, disability, queerness, and so on and so on. There's a

• 51:50Speaker 01

  huge list. So I mentioned before that utopia is inherently exclusionary. So I started to think like, gosh, like, is the future that we are trying to bring about through in aligned superintelligence really going to include like all of humanity? Like, who's going to get left out? Which, because, you know, we're not like, like, trying to determine what the future should look like in a democratic manner through some kind of democratic process by actually going out and asking people, what do you want the future to look like? There's a colleague of mine named Monica Bilschita, and she has this great line, which she's specifically referring to the future. She said, "You cannot design for, you must always design with."

• 52:30Speaker 01

  Right? So from the very start, you have to include people or representatives from every tradition, culture, religion, gender, you know, socioeconomic status, and so on and so on, from the very start, right? And that is, that is the only way you can go about creating a genuinely good and inclusive future that benefits all of humanity. But the, the next step of this line of thinking was recognizing that, oh, actually, yeah, it's that no humans have a place in the future. Right? Obviously, like, we're going to create these, these post humans. And like Recurswell says in The Singularity is Near, "If you don't want to upgrade your genes or, you know, become posthuman in some other

• 53:15Speaker 01

  manner, that's fine with me, but you won't be around for very long to influence a debate." Right? So it's essentially saying, become posthuman, or just die out. Right? And so this, this seemed even more troubling than the already troubling conclusion I came to that like, yeah, our, our species just doesn't have a place in the future. And I'm not sure that's right. And furthermore, I mean, the only plausible way that we could transition fully from a human to a posthuman era is through some involuntary means. Because the vast majority of humans out there are not going to consent or give their permission to allow posthumans to take over the world. Right? And that's the whole point, again,

• 53:58Speaker 01

  for posthumans to take over the world, to run, you know, the politics, the economy, science, technology, and so on. So there's going to have to be some kind of involuntary, forceful, coercive push by those of us in the Tesco movement, I was thinking at the time, to actually bring about this transition. And well, that obviously introduces all sorts of profound ethical problems, right? Violation of rights, you know, there are 8.3 billion people on the earth. When you start to like, list the cities, you know, like, okay, so like, everyone in, we're talking about like, everyone in Toronto, in Singapore, and, you know, you go through, it's like, like, there are a lot of people around the world. And

• 54:41Speaker 01

  how hubristic is it for us to say, well, you know, we know best, we're smarter than everyone. And so, you know, these, like, the dumb masses, I mean, this is the way a lot of the people think, the dumb masses, they just don't understand the stakes, they don't understand the ethical importance of bringing about this cosmic posthuman utopia. And so kind of like, you know, forget them, like, kind of screw them, because they're just, you know, as, as Yatkowski would say, they're NPCs, you know, non-player characters. So they just don't really matter. So that was, that was the other thing that, there's a consideration that really, you know, put a nail in the coffin and led me away from the community and encouraged me to become a critic.

• 55:22Cal Newport?

  Interesting. So, I mean, is this going too far to say, because this is what's occurred to me, is that when I think about this type of long-termist, expected value style utilitarianism, where, you know, like, an individual human's life, so you move away from that sort of fundamental natural law, religious principle at the core of, you know, all normative moral thinking of humans are infinitely valuable, a human life is infinitely valuable. When you move away from that liberal conception, immediately what comes to mind is the 20th century utopian movements. So if you look at far right fascism, if you look at the far left movements, the mid-century, we're completely willing for many millions of people

• 56:05Cal Newport?

  right now to die because, yeah, but the utopia, because the utopia they were going towards was worth it. So if you're in Nazi Germany, you're like, well, yeah, maybe it's not the best, we have to kill, you know, six to eight million people, but it's what matters is it's going to give us this sort of utopian state, and that's worth more. Or if you're, you know, Soviet Russia at a certain point, like, sure, like, half of Ukraine is dying from starvation. But the point is, you have to break a couple eggs, make an omelet, and the omelet is going to be this sort of utopian, you know, socialist communist world. And, and so there is, as soon as you move away from the precept of life is, you know, a human is infinitely valuable and has their own natural rights, well, as soon as you move away from that, we've always, we've tried this experiment, it ends in a place

• 56:50Cal Newport?

  that we just universally look back and say, that was really bad. Am I going too far to put this in the same ontological category within the world of ethics?

• 57:00Speaker 01

  In my opinion, absolutely not. The utopia, the thing to understand is that utopia justifies anything and everything, right? Because utopia is marked by infinite or near infinite, or as Tescoalists would say, astronomical amounts of value, there is no act, however extreme, however violent, et cetera, that can't be justified, right? If you, you know, if you do the expected value calculations, like, committing genocide might actually be the, like, like, absolutely the best thing for you to do. There's a statistician who, named Ole Haxstrom, who is generally sympathetic with long-termism, but he has this moment

• 57:42Speaker 01

  of, what I would describe as moment of clarity, in his, I think it was 2015 book, Here Be Dragons, where he talks about, somebody, you know, Nick Bostrom has said that, that reducing, because there could be so many digital people in the far future, and hence so much value, again, his estimate is like 10 to the 58, digital people at, at least, that, that decreasing the probability of an existential catastrophe by a tiny, tiny amount, which is just to say, it's, it's just another way of saying, increasing the probability of this cosmic posthuman utopia by a tiny, tiny amount, is worth the lives of billions and billions of actual living human beings. Right? So Haxstrom says, if

• 58:22Speaker 01

  there's a politician out there who, who takes seriously this line of reasoning, and is told by, you know, intelligence agencies, that there's a one in a billion chance that some lunatic in, somewhere in Germany, they don't know, they don't know where, is about to bring about some kind of existential catastrophe, destroy this, as Toby Ord says, "vast and glorious future among the stars," then the, the precedent might be justified in just nuking all of Germany. Right? So, so there's absolutely an enormous, and I mean, that was sort of hypothetical back in 2015, but if you fast forward to 2023, you find Yatkowski publishing an article in Time Magazine, in which he says that

• 59:07Speaker 01

  the, the almost certain outcome of misaligned superintelligence is that everyone on earth will die. And again, his concern here isn't that 8.3 billion people will die. I mean, he would say, like, that's a part of it. But much, much, much more important is the loss of what he calls our glorious transhumanist future, right? All that, that future value, which, which is, again, another way to say, he cares about avoiding final extinction, not terminal extinction. Right?

• 59:34Cal Newport?

  And so, not to interrupt, not to interrupt, but he, there's a tweet of his about this that's going around that makes his stance much more clear, where he said, "If we had a fear that misaligned superintelligence was emerging, thermonuclear, global thermonuclear war was not a bad option. Sure, it would kill most people, but probably there would be some humans who would survive a thermonuclear war, and then they would have a shot thousands of years in the future of going for better aligned superintelligence." So he said, "I'm completely fine with killing 99.9% of the population right now, if I thought that was going to stop a misaligned superintelligence. And the, the survivors that are thrown back to the Paleolithic, like, yeah, give them a couple thousand years, they'll rebuild civilization, and that, we have another shot at getting superintelligence." Right? So

• 1:00:17Cal Newport?

  he explicitly said, "Yeah, I would discount, the lives don't matter, it's exactly what you're talking about, is the glorious future."

• 1:00:24Speaker 01

  Right, so it's exactly the kind of profoundly pernicious reasoning that Hekstrum highlights in the passage, right? So Yatkowski is saying like, okay, there's a thermonuclear war, like probably, it's probably not the case that everyone will die. And the best calculations that I've seen, you know, from the past like six, seven years, support that, like, like maybe 5 billion people would die, but that would leave a reassuring 3 billion to rebuild civilization. So he's saying that like, we should risk thermonuclear war if that is what is necessary to prevent misaligned superintelligence, then we should do that. He was asked online, quote, "How many people are allowed to die to prevent AGI," end quote,

• 1:01:07Speaker 01

  "from being built in the near future?" Because it would be misaligned, he says. And his response is like, basically, everyone, as long as there are enough survivors to, in close proximity, to carry on the human species, to rebuild civilization, and then I'm quoting, "then there's still a chance of reaching the stars someday." Well, if you, like, just consider, like, what that that means. The minimum viable human population is estimated to be to be between 40,000 and maybe as low as 150 people, right? So pretty, like, nobody knows. But even if it's just 40,000, if you subtract 40,000 from 8.3 billion, you get, just rounding off, 8.3 billion, right? So he's saying 8.3 billion people should be, quote, "allowed

• 1:01:51Speaker 01

  to die to, you know, ensure that that superintelligence is aligned." Again, he is pro-superintelligence. If, if we figured out a way to align superintelligence to the Tesco worldview tomorrow, he would come out tomorrow and say, pedal to the metal, build ASI as quickly as possible, we'll create post-humanity, you know, it'll be unethical to have biological humans, we'll wipe them off Earth, we'll replace them, and so on. And so it's, it's just important to understand how, like, absolutely extreme that that view is. But you're absolutely, you're completely right, that utopia is just a, I think, a very dangerous concept. And the other thing I would add to this is, I have been speaking so far as if the real, successful realization

• 1:02:33Speaker 01

  of the Tesco vision is possible, right? And I've been saying that, you know, if they successfully realize their vision, then the outcome of safe, safe superintelligence is that will become completely irrelevant, or, according to many of them, like Peter Thiel, like, we should, we should disappear, right? I actually am very skeptical that they could successfully realize this vision. But you brought up utopian movements of the 20th century, and examples go back many centuries before that. These utopian movements never reached their goal, their goal. But that doesn't mean that they weren't profoundly dangerous. They left a trail, in so many cases, they left a trail of destruction behind them. I mean, Hitler

• 1:03:17Speaker 01

  promised a thousand-year Reich, you know, he basically was borrowing from, you know, Christian eschatological, the Christian eschatological narrative, which promises a millennial kingdom at some point in the future, that one of the bloodiest battles in human history is the Taping Rebellion of the 19th century in China, and resulted, I don't know, like, something like 30 million deaths, I can't remember the exact figure, but that was, this was driven by the Taping heavenly kingdom, which was essentially a utopian apocalyptic movement, just like, just like Tesco. You know, terrorist, terrorist groups, doomsday cults, you know, the Islamic State, you know, they were an apocalyptic movement. Well, from the religious point of view, the other side of the apocalypse is utopia. So it's

• 1:03:59Speaker 01

  just another way of saying that they were motivated by utopian visions of the future. And that's why they were trying to, literally, they were trying to start the Islamic version of the Battle of Armageddon in this small Syrian town in northern Syria, near Aleppo, called the Debeek, which is said to be the location of their version of the Grand Battle. So it's just, my, my main concern is that, you know, so far, the AI companies have left a trail of destruction behind them. I mean, some people would argue that they are, like, tearing apart society, they're flooding the internet with, you know, disinformation, deep fakes, they're working with the US military to engage in illegal activities like kidnapping Maduro and bombing Iran. They're partnering with Palantir to sort of, like, build the

• 1:04:43Speaker 01

  infrastructure for totalitarian state. And so, so all of these are, I mean, not, not to mention the massive, massive intellectual property theft that they're all guilty of. So these are all, these are like harms that we can point to right now. And I think the harms are going to get worse, but from their perspective, like, like, they're marching to utopia, and sometimes the march of progress, like, it's going to trample some people. And also, like, from the grand cosmic point of view, all of these harms that they're causing right now, they are mere peccadilloes, right? They're, they're just small, like, little mishaps that utopia will forgive, because utopia forgives everything.

• 1:05:23Cal Newport?

  So this brings me to my, the last thing I want to ask you about before I let you go, is trying to understand, okay, what's happening right now. And I, and I'll say, I'm sure you've experienced this as well, but I find if I, if I write or talk about some of these movements, a very common response I get is like, you're just being ad hominem. Sure, these are weird people, but they were right. The things they were warning us about are starting to happen. So who cares if they're weird or not, they were right. Now, I have this alternate view I want to run by you, that actually, it's a self-fulfilling prophecy, right? So that they're, they're talking about AI in the ways that matches the Tesco sort

• 1:06:05Cal Newport?

  of predictions, because they are from Tesco, and they're trying to bring those predictions about. And I also am increasingly believing, if you look at Anthropic or OpenAI, is that they're specifically running experiments and developing this technology in a way that's creating dangers, like you were just talking about, the dangerous experiments, hazard experiments, unpredictable experiments, exactly because they're trying to rush forward a movement towards their predicted sort of superintelligent future. The strong version of this argument would say, if we go to a counterfactual world, where you don't have these two Tesco-aligned labs at the frontier, and you just have other technology companies unrelated to these philosophies pushing for generative AI, we would be having a completely

• 1:06:49Cal Newport?

  different conversation. There would be no conversation about existential death, the sort of really unpredictable, scary experiments, like what happened at OpenAI in the summer, just would not have happened, no one responsible would have run those. And that, that, so that we have almost, we're already seeing this damage on the route to utopia. It's not that they predicted this right, it's that they're, they are trying to make their predictions happen. But I want to get your final take on this, is how do we understand what is happening right now, the things that are concerning people in AI world, how that connects to the Tesco philosophy?

• 1:07:22Speaker 01

  Yeah, I think you were exactly right. So one way I put it in the past is if you want to understand, this kind of brings us full circle, if you want to understand the behavior of companies, virtually all companies in our capitalist milieu, you have to look no further than the profit motive, right? Why are the fossil fuel companies destroying the world for profit? Why are pharmaceutical companies jacking up prices while they, they just, you know, want to increase their revenue, and so on, and so on. If you look at the AI companies, they are different, they are anomalous. And I can't think of any other examples, right? I've, I've said before that profit has become increasingly significant factor that explains kind of their, their behavior, the dynamics, the race dynamics they're in.

• 1:08:04Speaker 01

  But if you look at the origins, and if you look at the core of these companies, what is really ultimately driving them, it is ideological, it's Tesco. Tesco is integral to an explanation of why there's an ASI race, why these companies arose in the first place. All of this is to say that because these companies directly emerged out of the Tesco movement, claiming that Tesco advocates back in the day were prescient for talking about AI risks and so on, is, is, is a complete misreading of the situation. It's sort of like, if, if someone were to be impressed that, you know, I woke up this morning, and

• 1:08:46Speaker 01

  I prophesized, I will soon have a cup of coffee. And then, and then amazingly, I got up, I made a cup of coffee and I drank it. So it was like, wow, that's, that's incredible, that you accurately predicted what was going to happen. Well, yeah, because, like, you know, I mean, DeepMind got its funding because Demis Hassabis and Shane Legg, two of the three co-founders, gave talks at the Singularity Summit. Singularity Summit was founded by Yatkowski, Ray Kurzweil, and Peter Thiel. Yatkowski was funded by Peter Thiel. The connections here are absolutely convoluted and completely incestuous. I've written an article about this, but it's, it's, it's stunning when, I mean, it looks like a conspiracy,

• 1:09:29Speaker 01

  when you actually look at the details. And, you know, not all conspiracies are conspiracy theories. But it's, it's just like an incestuous network of these people, like exchanging money and supporting each other. So they founded the Singularity Summit. Demis Hassabis and Shane Legg were invited. They gave a talk about superintelligence, because Shane Legg had just published his dissertation on superintelligence, for which he won a prize from Yatkowski's Singularity Institute for $10,000. And then after the Singularity Summit event, they followed Peter Thiel back to his mansion. And Peter Thiel is as Tesco as, as anyone else in this community. They asked him for funding, and he, he gave them funding, right?

• 1:10:11Speaker 01

  So I mean, DeepMind, which was the, the company that, that in a sense, you could say, kind of initiated the, the superintelligence race, because OpenAI was founded because they were, there were concerns about Demis Hassabis establishing what they refer to as an AGI dictatorship, right? Basically, just taking over the world. And, and so, the whole reason that DeepMind came into existence is because Shane Legg was deeply involved in this community, Demis Hassabis was into the idea of the singularity, and they gave talks at this conference, co-founded by Yatkowski. So to say that, like, Yatkowski was, was, you know, clairvoyant in some sense for having anticipated what's, what's going on now, is, is just to completely

• 1:10:55Speaker 01

  misinterpret the situation. You're totally right that this is a self-fulfilling prophecy.

• 1:11:01Speaker 01

  And so we have them to blame for the current predicament. You know, nobody has done more than the Tesco people, including the doomers. Nobody has done more than the doomers to launch, sustain and accelerate the race to build superintelligence. So rather than saying they're prescient, you should, you should point your finger and, and blame them for putting us in this situation because my personal view is that if there, if there had not been a Tesco movement, there would not be a race to build these like general systems that exceed human capabilities in every cognitive domain of interest. I don't think that would, would exist right now. So they are, they are responsible for the unfortunate plight that we all now find each ourselves in.

• 1:11:42Cal Newport?

  Well, I think, Emile, this is, uh, it's vitally important information. Uh, it's confusing, so I'm, I'm glad we got to spend some extra time to try to pull this apart. And I've been saying this to my audience a lot. You have to understand this, this is not ad hominem. This is not, well, they might be right, but they're weird. It's the actions we are seeing by certain companies, the rhetoric we're seeing by certain companies, as you said, is in a almost unprecedented situation. I also tried to come up with an example from, you know, American history at least, where you had a really large corporation that was not being driven by the profit motive, but instead by a sort of philosophical, sort of religious motive. You have to find, I guess, a sort of explicitly religiously aligned organization, I guess.

• 1:12:25Cal Newport?

  So it's really important to understand what's going on and you've seen it from both sides. So I appreciate you coming on. I want to point people towards all the excellent work you do on this. Your website is, it's X riskology. Am I saying, am I getting that right?

• 1:12:39Speaker 01

  Yes, but people should really go to my newsletter.

• 1:12:42Cal Newport?

  Newsletter, okay, go for it. Yeah.

• 1:12:44Speaker 01

  Yeah, which is, which is titled the real time tech apocalypse newsletter. And by, my podcast is dystopia now. So it's sort of like the darker the topic, the greater the chances that I'll have something to do with it.

• 1:12:58Cal Newport?

  I'm also going to push people, if you want a really good sort of article linked introduction to Emile's thoughts, check out the article against long-termorism. There was an Aon magazine and I think it's like a 6,000 word manifesto that covers a lot of these points, but very, very systematically. It's, it's a, it's a great work of sort of modern applied ethics. I would also point people towards that as a good introduction as well.

• 1:13:23Speaker 01

  Yeah, thanks. And I think the only thing that's missing, cause I hadn't fully developed these thoughts, is the pro extinctionist element that is inherent in the Tesco worldview. So by more recent, like there's one half a pro-human manifesto is, I think another like good, I mean, insofar as anything I write is, is worth reading. I feel like that is, is very much worth checking out.

• 1:13:44Cal Newport?

  All right, well, you heard it here first. Uh, check it out. You need to understand this world. Emile can be your guide to it. All right, thank you again for joining us.

• 1:13:52Speaker 01

  Thanks a lot.

• 1:13:55Cal Newport?

  All right. So that was my interview with Emile Torres. That was really useful. I've been writing about these ideologies for a little while now. I had a big New York Times article. I've done a bunch of podcast interviews where I've given my best rendition, even last week's Thursday episode of this show got into some of those details. But I think what I learned today that wasn't as clear to me before was just how extinctionist all these beliefs are. I think Emile is very good on this point, that the goal, the ultimate goal of all of these Tesco role bundle of ideologies is a post-human utopia where biological humans are gone.

• 1:14:36Cal Newport?

  And what we're left with are these exalted digital angels, these sort of digital post-human versions of humanity. But what, as Emile points out, that means at some point, these are tech overlords who are trying to bring this, this future to fruition are going to have to kill all the other, the meat, biological people. Cause most biological people in a world of eight billion people don't want to say, I'll just disappear and we can have a select few go off into a transhumanist future. They're like, we like our world as we have it. So if you really pull the thread, this was new to me and quite frankly, it's, it's disconcerting. But I think Emile is right about this.

• 1:15:18Cal Newport?

  If you pull the thread on the Tesco role bundle of ideologies, you end up at some point killing billions of people on purpose because they're getting in the way of your dream of a digital utopia. All right. The other key point that came out of this that we both emphasized, neither Emile or myself think that this vision is actually possible, that you actually are going to, if you have a good enough agent on your LLM or something, you're going to end up with a super intelligent, God, there's a lot of good computer science reasons to say that's kind of fanciful. But Emile and me both agree on this point, pursuing utopia can be devastating. That's what we've seen. The Nazis did not get the thousand year Reich, but tens of millions of people died.

• 1:16:02Cal Newport?

  The, the Soviet communists did not get their, their socialist utopia, but tens of millions of people died on the way. So we've learned this lesson well. The pursuit of utopias where you're saying the value of humans alive right now pale in comparison to the value of the utopia that I am pursuing. We have seen that exact configuration throughout history. No one has ever successfully got to their utopia. Every time it's been pursued with any sort of well-funded vigor, it's been a blight on humanities history. So we really need to be woke to these ideologies and what's really happening here. I am a fan of AI. I think generative AI can produce any number of useful tools and the whole thing is being

• 1:16:46Cal Newport?

  sullied and hijacked and made into this unstable dynamite bundle of terribleness by this weird utopian anti-humanist, pretty dark philosophy that's incredibly well-funded and the two top frontier labs are completely driven by it. So I think this is a very important message. It needs to be out there. This has to be a part of the conversation we're having when we're trying to understand the AI moment, but also when we're trying to understand what to do next. So, uh, Emile, thank you for coming on the show. Hopefully you enjoyed this. Uh, see you again next time. Hey, if you've made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday.

• 1:17:30Cal Newport?

  You can sign up at calnewport.com/ideas. And when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today, calnewport.com/ideas.