Existential Risk and the Future of Humanity: Lessons from AI, Pandemics, and Nuclear Threats (Toby Ord, Author of "The Precipice")

How close are we to the end of humanity? Toby Ord, Senior Researcher at Oxford University’s AI Governance Initiative and author of The Precipice, argues that the odds of a civilization-ending catastrophe this century are roughly one in six. In this wide-ranging conversation, we unpack the risks that could end humanity’s story and explore why protecting future generations may be our greatest moral duty. We explore: • Why existential risk matters and what we owe the 10,000-plus generations who came before us • Why Toby believes we face a one-in-six chance of civilizational collapse this century • The four key types of AI risk: alignment failures, gradual disempowerment, AI-fueled coups, and AI-enabled weapons of mass destruction • Why racing dynamics between companies and nations amplify those risks, and how an AI treaty might help • How short-term incentives in democracies blind us to century-scale dangers, along with policy ideas to fix it • The lessons COVID should have taught us (but didn’t) • The hidden ways the nuclear threat has intensified as treaties lapse and geopolitical tensions rise • Concrete steps each of us can take today to steer humanity away from the brink — Transcript: https://www.generalist.com/p/existential-risk-and-the-future-of-humanity-toby-ord — This episode is brought to you by Brex: https://www.brex.com/marioThe banking solution for startups. — Timestamps (00:00) Intro (02:20) An explanation of existential risk, and the study of it (06:20) How Toby’s interest in global poverty sparked his founding of Giving What We Can (11:18) Why Toby chose to study under Derek Parfit at Oxford (14:40) Population ethics, and how Parfit’s philosophy looked ahead to future generations (19:05) An introduction to existential risk (22:40) Why we should care about the continued existence of humans (28:53) How fatherhood sparked Toby’s gratitude to his parents and previous generations (31:57) An explanation of how LLMs and agents work (40:10) The four types of AI risks (46:58) How humans justify bad choices: lessons from the Manhattan Project (51:29) A breakdown of the “unilateralist’s curse” and a case for an AI treaty (1:02:15) Covid’s impact on our understanding of pandemic risk (1:08:51) The shortcomings of our democracies and ways to combat our short-term focus (1:14:50) Final meditations — Follow Toby Ord Website: https://www.tobyord.com/ LinkedIn: https://www.linkedin.com/in/tobyord X: https://x.com/tobyordoxford?lang=en Giving What We Can: https://www.givingwhatwecan.org/ — Resources and episode mentions —Books— • The Precipice: Existential Risk and the Future of Humanity: https://www.amazon.com/dp/0316484911 • Reasons and Persons: https://www.amazon.com/Reasons-Persons-Derek-Parfit/dp/019824908X • Practical Ethics: https://www.amazon.com/Practical-Ethics-Peter-Singer/dp/052143971X —People— • Derek Parfit: https://en.wikipedia.org/wiki/Derek_Parfit • Carl Sagan: https://en.wikipedia.org/wiki/Carl_Sagan • Stuart Russell: https://en.wikipedia.org/wiki/Stuart_J._Russell —Other resources— • DeepMind: https://deepmind.google/ • OpenAI: https://openai.com/ • Manhattan Project: https://en.wikipedia.org/wiki/Manhattan_Project • The Unilateralist’s Curse and the Case for a Principle of Conformity: https://nickbostrom.com/papers/unilateralist.pdf • The Nuclear Non-Proliferation Treaty (NPT), 1968: https://history.state.gov/milestones/1961-1968/npt • The Blitz: https://en.wikipedia.org/wiki/The_Blitz • Operation Warp Speed: https://en.wikipedia.org/wiki/Operation_Warp_Speed — Production and marketing by penname.co https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The human story might just be beginning, but there are various threats to our continued existence. Some of these have been around forever, such as asteroid impacts. But some of them are threats of our own making, such as nuclear war.

• 0:11Mario Gabriele?

  You put the odds at 1 in 6 that in this century, we do incur the existential risk and humanity fails to live beyond this century. Have the last few years made you more or less worried about this?

• 0:24Speaker 00

  So another possibility is what gets called gradual disempowerment. Suppose the AI systems never violate our rights, maintain sufficient control over them, they don't break the law, but they're more successful than us over time at doing all the kinds of jobs that we do. Even if we get richer through trade with these systems, a higher and higher share of the wealth eventually accumulates in AI hands.

• 0:50Mario Gabriele?

  Hey, I'm Mario and this is the Generalist Podcast. As the saying goes, the future is already here. It's just not evenly distributed. Each episode, we sit down with the visionaries, builders and thinkers who are already living in that future to help you see it earlier, understand it better and capitalize on it. Today, I'm speaking with Toby Ord, a senior researcher at Oxford University, one of the world's leading experts on existential risk and author of the excellent book The Precipice. It's a thoughtful analysis of the greatest dangers to humanity's survival, from rogue asteroids to pandemics to unaligned artificial intelligence. These may sound like fanciful sci-fi problems, but there's actually good reason to believe they're extremely pressing, with Toby estimating humanity

• 1:35Mario Gabriele?

  has a 1 in 6 chance of extinction this century. In my conversation with Toby, we discuss how AI risk has evolved since LLMs emerged, why the scientists working on the Manhattan Project didn't stop developing nuclear bombs after Hitler died and the Cold War lessons the US is ignoring in its current great power conflict with China. I walked away from our discussion with a better sense of how we should think about humanity's future, what we owe coming generations, and what steps we might take today around AI safety and pandemic preparedness. This is a new podcast, so if you like it, I hope you'll consider subscribing and leaving us a review. Now, here's my conversation with Toby Ord.

• 2:22Mario Gabriele?

  Well, Toby, I'm so excited to chat with you today. I can't tell you how much I enjoyed your book, The Precipice, which I have to hand. And in looking at my sort of marginalia in preparation for this, I was struck by, one, that I could be a much more thoughtful reader, and, two, how much I enjoyed it, because all of my notes are sort of, wow, exclamation mark, exclamation mark, and no way. So, maybe with that as a little bit of a background, perhaps you could tell us what exactly existential risk is, and what does it mean to study it in the way that you do?

• 2:59Speaker 00

  Yeah, so humanity's had a long and illustrious past. Our species has been around for about 300,000 years, or more than 10,000 generations. And we might be able to have a future of equal or larger size. Certainly, most animals do. They typically live for about a million years. So the human story might just be beginning. But there are various threats to our continued existence. Some of these have been around forever, such as asteroid impacts and things like that, these natural risks from the somewhat hostile world that we live in. But some of them are threats of our own making, such as nuclear

• 3:39Speaker 00

  war. And so existential risks are any risks that could threaten to destroy, to permanently destroy humanity's long-run potential. So that could be something that makes humanity go extinct. Or it could be something that, say, causes a permanent collapse of civilization, where if it was unrecoverable and there was no way back, then these things would have a similar kind of role. That would be the types of events that reduce the value of our future to almost nothing, and which we have to make sure that we avoid falling victim to these even once, over the hundreds of thousands of years to come.

• 4:19Mario Gabriele?

  Yes. When you sort of started in this field, was it the case that these risks were relatively well-known, at least, if not well-studied? Or does the work of an existential risk researcher involve thinking of all of these possibilities that maybe no one has actually even properly catalogued before?

• 4:44Speaker 00

  Yeah, I think the risks, at least the risks that I know of, had been fairly well-understood, or at least, sorry, maybe that's going too far. They were all known of. So it's not that I sat around trying to think about new things that could threaten us. It's possible to do that, and there's any number of things that could pose some kind of risk. But often, those risks get smaller and smaller as you go further down the list. And so it's not that needed in order to keep adding on things that are only tiny compared to the things you've already got on the list. And I think that they'd been generally known about for hundreds of years through

• 5:26Speaker 00

  to decades or something like that. But there could be risks that we're still unaware of. For example, in 1900, we were unaware of the risks of gamma ray bursts or supernova explosions, or even the risk of asteroids was only really convincingly demonstrated in 1980. That's actually surprisingly recent. And so there could be other risks that we're ignorant of. But there is at least a helpful thing when it comes to the natural risks, which is that we know we've survived for 300,000 years so far. And we know that typical animal species survive for about a million years. So it can't be the case

• 6:10Speaker 00

  that the risks are much higher than that. So ultimately, the risk from natural events has to be something like one in a million per year or lower.

• 6:21Mario Gabriele?

  Fascinating. We'll get into some of those risks in greater detail, especially some of those that are particularly relevant these days. But I'm curious, more on your journey. You studied computer science in Melbourne. How does one go from perhaps a future software engineer to spending their lives focused on the far future, on risk, on these sorts of questions?

• 6:47Speaker 00

  I guess what, you know, one thing to have in common is I was probably, probably always going to be an academic. And so in computer science, I was interested in, in the kind of theoretical side of computer science.

• 6:58Mario Gabriele?

  How interesting.

• 6:59Speaker 00

  Including very much in artificial intelligence, which has turned out to be useful for me now.

• 7:04Mario Gabriele?

  No kidding.

• 7:05Speaker 00

  I guess I was drawn to, you know, to these questions about, um, you know, really zooming out and looking at the big picture of humanity's future. So thinking about this, you know, when reading good works of science fiction while I was growing up. One of the things I remember being struck by is if we did manage to reach a point where we dealt with all of the, the negatives in the human condition. So we dealt with poverty and we dealt with various other forms of pain and suffering. As, you know, as we have been slowly doing over the years, right? Like, you know, if you go back a couple of hundred years, there were no anesthetics and things, you know, the idea that you could go through surgery without extreme pain, you know, would have, would have shocked people. And so if we kept going down that trajectory, you know, what would happen next?

• 7:48Speaker 00

  You know, what's the positive story about what we should be doing? If we got to a point where we kind of had removed a lot of the injustices and discriminations and, and, you know, pain and suffering and inequality, you know, all the bads, you know, what, what next? And so I was interested in these kinds of questions about the future of humanity and, and, you know, trying to understand the big picture challenges that we face. I guess one of those challenges, you know, that was preventing us from reaching that point was, uh, global poverty. And so I was, I was very interested in that, um, uh, you know, since I was an undergraduate studying computer science in Melbourne. And had, you know, just decided since individuals could do so much to make a difference, um, that I would kind of make that part of my life.

• 8:32Speaker 00

  And I've, I've continued to do so. And that led me to, to found this organization, Given What We Can, um, where people make a pledge to give at least a tenth of, of their income over their lives to the most effective charities, uh, trying to help others. You know, be, be they, be they people or, or animals. Yeah, it's, it's often turned out that the things that are some of the biggest challenges facing humanity are also some of the ones where we can kind of. I get the most leverage on them. So we kind of get the most bang for your buck in terms of if you're going to make, if you're going to donate, say a thousand pounds where you can, where you can do some of the most good with it. That wouldn't have to be the case. You could imagine a case where there's some huge challenge we face, but it's really quite intractable. And there's not much, or there's not much individuals can do about it.

• 9:17Speaker 00

  And I think that global poverty is at least more tractable with respect to money, at least money for people in America or the UK or Australia, in the richer countries. You know, where people are among the, say, the 5% richest people in the world. It's then not so surprising that money, the thing we have the most of, could be something that we could do to really help those who have much less of it. But in the case of, say, risks posed by artificial intelligence, it does become a harder story about, you know, what's everyone supposed to do about it.

• 9:48Mario Gabriele?

  Yes. And, and when we're talking about sort of these more tractable problems, things on the order of sort of malaria nets, deworming initiatives, that sort of thing where you really can with small amounts of money make like a very immediate impact. Is that, is that a fair characterization?

• 10:03Speaker 00

  Yeah, they're the things that, that I was particularly drawn to. One thing I really like about them is that, that I think, you know, there's interesting different opinions on, on giving. Some people just aren't very interested in helping others and, and I, I, you know, I'm not going to convince them otherwise. But there's a lot of people who I think would be tempted to do this. They really feel the pull of it, but there's some kind of blockers or things that, that stop them. And one of those blockers is a feeling that, do we really know what helps? You know, and so where you want to find something that's got the kind of most robust evidence behind it to then say, well, to the extent to which you really just want to make sure that you make a difference. Here's something you can do that, that is, is very likely to do so.

• 10:48Speaker 00

  And so I was like, yeah, particularly interested in those cases, as well as the possibility of more speculative things for people who are willing to, to go a bit more out on a limb. And for example, you know, fund some new technology that could help people in poor countries, but maybe it will never pan out. You know, maybe they can do even better than these tried and true things. But at least the tried and true things can create a nice kind of baseline for, you know, for the idea of, should you keep the money in your pocket and spend it on yourself? At least if it's some kind of skepticism about whether it will make a difference, then finding those, those reliable things is very useful.

• 11:19Mario Gabriele?

  Yes. Also important to.

• 11:22Mario Gabriele?

  Get into the the sort of mode of talking about these things. I think another blocker is that people feel, you know, they shouldn't mention it, or, you know, they're inviting too much attention. But actually, by signaling it, what you do, you sort of normalize that for for a lot of other people. And yeah, so I'm glad we're talking about it up top. I'm certainly not at the 10% rate, but I'm a recurring donator to give well with the idea that I can't count on my empathy to spike with enough regularity that I remember to do it. So I may as well just, you know, sort of put it on autopilot. But anyway, that's a really, yeah, I'm glad we we sort of got into that. And that sort of led you into some of these these questions and studying with, you know, one of the the most prominent

• 12:06Mario Gabriele?

  moral philosophers of, you know, 20th, 21st century and Derek Parfit, also, you know, in a very crowded field, one of the the best sets of hair of any philosopher, I must say. But I'm I'm curious what it was that, you know, you learned from him as a mentor and how that sort of framed your worldview.

• 12:25Speaker 00

  Yeah, interesting. So I mean, one of probably the biggest effect that Derek has had on me was that I probably came to Oxford in in large part on the strength of that he was here. As when I'd studied computer science and then also philosophy in Melbourne, you know, some of the philosophy that seemed to really make the most sense to me, that was really well thought out and kind of ingenious in various ways, you know, was was the work of Derek Parfit in his book Reasons and Persons particularly. And yeah, he did, you know, amazing foundational thinking on the nature of personal identity. So what it means to be an individual persisting across time. And as someone who was thinking about artificial

• 13:07Speaker 00

  intelligence, you know, some of his thought experiments into, you know, maybe humans couldn't do these things. For example, a human can't split in two. And, you know, Derek had a whole lot of, you know, Fanciful, science fiction style thought experiments about this. But if it's a computer program, it's a lot easier to imagine. You could just pause its execution at a certain point and then fork the process into two different processes. And it really does seem like you could have an individual, a kind of Y-shaped individual, you know, who has a certain kind of common path and then kind of branches into two different futures and ask a lot of questions. If a program could be a person and a program could be conscious and things like that, then you could ask, you know, what would it anticipate, you know, before the moment that it was split into these two different environments?

• 13:52Speaker 00

  And so I thought, you know, a lot of his style actually really fit well with my computer science background, despite the fact that he was, or he saw himself as a very non-mathematical and non-technical person. And he tried to avoid any maths in any of his writing, which I found remarkable because it was, in some sense, it was so mathematical already. It had this, this real clarity of the logic. But yeah, and he also, you know, I enjoyed some hobbies with him as well. He was a prolific photographer and, you know, took these, these just breathtaking photographs, particularly in St Petersburg in the snow and Venice in the mist. And so I enjoyed talking photography with him as well.

• 14:36Mario Gabriele?

  Oh, amazing. I'll have to try and find those. I'm sure they're, they're somewhere. One of the things that, you know, Derek Parfit, you know, was so famous for, as well as sort of the concept of identity, is sort of the idea of caring about these future persons, the people that, that come next. And, and that's obviously such an important part of thinking about existential risk. Perhaps you can, you know, give us a little precis on, on that concept and how it factors into this topic.

• 15:04Speaker 00

  Yeah. So, so kind of a couple of areas, related areas that Derek worked on were to do with what's called population ethics, which is

• 15:15Speaker 00

  A lot of our ethical questions concern individuals who are all alive at the time when the choice is made. For example, we're trying to work out whether to extend one person's life versus extending a different person's life with scarce medical treatment. You know, ultimately, though, in almost all standard moral cases, like whether to tell a lie or something like that, the person who would be lied to is someone who already exists. But a lot of the questions we make, particularly at the societal level, concern changes to the people who will ever come to live. And so that could be because, you know, for example, if there is a risk of extinction, then there won't be people in future generations. So those people will never be born, and those lives will never

• 15:56Speaker 00

  never exist at all. And so there's a question about how to how to value that or think about it. Is that, you know, do we lose the entire value of those lives? So kind of taking them from what they would have been down to zero? Or is it something different that we do? Maybe it's not as bad to have someone never exist at all compared to if they, you know, if they died when they were an infant. There's a lot of different ways of thinking about this. Yes. And so Derek Parfit really opened up this, helped ask the right kinds of questions and then sketch some of the challenges with with trying to answer them. He also, though, somewhat separately, but but but it goes together well, he was interested in the idea

• 16:37Speaker 00

  about actions that could that could have benefits over very long timeframes. So when it comes to economists, they often do something called discounting, where they try to, in order to understand the value of actions that have effects at different times, they use a mathematical technique in order to say that that the effects they have at times further into the future matter less. And in some cases, this this totally makes sense. So for example, if there's going to be inflation, then getting a dollar in, you know, 20 years' time won't be worth as much as a dollar now. Yes. Or if you know you're going to be richer in 20 years' time than you are now, maybe you're a student and, you know, in 20 years' time, you'll be kind of high up in some career,

• 17:22Speaker 00

  then a dollar will, even inflation adjusted, will be worth more to you now when you're poorer. So obviously, you want to adjust for those things. But most people, including economists, go one step further and assume that that actually intrinsic value itself, once you've adjusted for all of that, is just worth less if it happens later in time. And Derek Parfit, and in fact, most philosophers ultimately, you know, rejected this. And so Derek Parfit has, you know, nice thought experiments on this. So there's one where if you imagine hiking on a trail and accidentally breaking a glass bottle when you're having a picnic, and then the shards of glass, you know, you're deciding whether to pick them up and take them with you, or to just leave them there. And suppose if

• 18:05Speaker 00

  you left them there, you knew that they would injure a child's foot. So, you know, this young child would be kind of stabbed through their shoes and be bleeding on the trail. And, you know, like that would survive, but that would have a very bad time of it. Then his idea was, does it matter if you knew that that was going to happen? Does it matter if it was going to happen in one week, or, you know, or a year, or a hundred years, or 10,000 years? If there was going to be exactly one child, and they were going to have the same amount of pain and suffering from this, and so on. This is kind of thought experiment to suggest, actually, that the time of these things, you know, if there's no way that it compounds, or does something else like that, then the time seems to be irrelevant. These ideas kind of led the way to systematic thinking about intergenerational

• 18:47Speaker 00

  harms, or intergenerational benefits. And then famously, he also, you know, wrote one of the first descriptions of existential risk as a major moral issue. So definitely, in terms of, I guess, getting back to your earlier question about the type of influence that he had on me, you know, certainly what I've done since then has been very shaped by these ideas.

• 19:05Mario Gabriele?

  Amazing. Yeah, I think in the book, you have, I can't remember the exact phrasing of it, but the idea that, you know, if you do apply this discounted rate to future lives, you can get into these scenarios where, you know, you are weighting the fact of someone currently having a headache more than, you know, a million people dying and suffering in however many hundreds of thousands of years in the future. And so, you know, it can get really wonky, but fundamentally, you know, I think you're, you know, the glass example was so clear, that, you know, these are questions about what we owe the future, and what we owe to the people that come next. And that's really what you outline, I think, so thoughtfully in The Precipice.

• 19:50Mario Gabriele?

  Which, you know, is came out a few years ago, but it strikes me that it was incredibly prescient immediately, which is not something that usually happens and, you know, remains that way. And, and I'm saying that because it came out before COVID and you talk a huge amount about pandemic risk. And you also talk a lot about AI alignment, which has obviously become, you know, more and more critical. But one of the sort of headlines, so to speak, is that you put the odds at one in six that in this century, we do incur the existential risk and humanity fails to live beyond this century. What do you put our odds at now?

• 20:32Mario Gabriele?

  Have the last few years made you more or less sort of worried about this?

• 20:37Speaker 00

  Yeah, I'm not, I'm not actually sure. I think that there are, you know, if you look risk by risk, so some of them say nuclear war, I think the risk has gone up. Ultimately, when I, when I wrote it, it was before the invasion of Ukraine. And subsequent to the book coming out, you know, the UK has been actively threatened with nuclear war by, by Russia. And, you know, I hope there are also a whole lot of questions about, you know, the possibility of, yeah, getting into either a conventional or, or nuclear war directly between Western powers, you know, with part of NATO and, and with, with Russia. So that one had seemed forgotten and distant when I wrote the book. And so I'm afraid it's

• 21:21Speaker 00

  just gotten worse. In addition to that, there is, there's only one treaty that that's still kind of, you know, bilateral treaty protecting, well, keeping the number of warheads low between the US and Russia. And that treaty is, has, has been extended, was extended in the previous administration, but it can't be extended again. A new treaty has to be negotiated.

• 21:46Mario Gabriele?

  Oh, wow.

• 21:46Speaker 00

  And so, but it looks like this is going to lapse and the, the long history of nuclear warhead numbers going down and kind of slowly, but surely creeping down further and further.

• 21:59Mario Gabriele?

  Yeah.

• 21:59Speaker 00

  We've started to become impatient, you know, how is it taking so long to get down towards zero? Uh, I'm, I'm, I, I think that probably actually that number is going to go up um rather than down. And then the, the new challenge will be trying to keep it low rather than um than trying to get it.

• 22:13Mario Gabriele?

  Reproliferation, yeah.

• 22:15Speaker 00

  Yeah, um, and uh, but when it comes to pandemics or when it comes to uh AI, I think that there's been a number of changes that have made it uh more risky and a number of changes that have made it less risky. And I think that they, they're quite hard actually to, to tease out the overall effect.

• 22:33Mario Gabriele?

  Well, that's really interesting. I would have for sure uh anticipated uh that you would, you would feel more worried at this point. So I'm excited to, to dive into these a little bit. Before we uh jump to AI risk, this is such a silly question on one dimension, but on another, it's one that I think of myself. Why does it necessarily matter uh if humans go extinct? Obviously, that would matter a great deal to me and to my, you know, uh,

• 23:00Mario Gabriele?

  my children and maybe my, you know, future grandchildren. But should we really see ourselves as quite so important? Especially when we can imagine that, given the, you know, the magnitude of the universe, there is probably, you know, it seems more likely than not that there's other life out there. Uh, what, what makes us so unique?

• 23:20Speaker 00

  Yeah, uh, interesting question. So, so it may not be important that we're unique. So uh suppose, uh suppose you have a child and, and you know, that they live their life and they, they, you know, they do a whole lot of activities that are kind of, you know, standard and kind of, you know, celebrated cultural activities, uh, you know, in, in your milieu, uh, you know, dance and song and, you know, spending good time with friends and things like that. But maybe they don't do anything that, that is uh, that's unique, that's kind of pushing outside the envelope of, you know, what's been done before or something. Or suppose you, suppose there's someone who, who has a life like that and someone else who's doing more unique things. And you ask, which life should we save if we can only save one of them? It's not clear that we should save the, the more unique one or, or something.

• 24:04Speaker 00

  It, it seems like there's, there's value in, you know, a life well-lived, um, whether or not it's unique. But the ultimate answer to this question is unknown at the moment. But there are some different kind of leads we have on it, you know, different intuitions people have. And I think that there's something to them. So one way to see it is it would obviously be what's bad in the present. If there was some extinction event, say, so, you know, all of the lives, you know, the eight billion lives that would be lost. In that event, and I think, you know, we could all see that that would be terrible. We know that it's terrible, you know, when when someone that we know is to die. And presumably it will be about eight billion times more terrible. If that happened to everyone, which is really very bad.

• 24:49Speaker 00

  But I set that aside because that that bit's fairly obvious. Then there's also the everything that we would lose. So if you think about the future and all of the generations who could have come to pass, say, that the next, you know, seven hundred thousand years of humanity, if we get our kind of average amount of time. And all of the hopes and joys of all of the people who would have lived across, you know, twenty thousand more generations of people. And all of, you know, the the art and achievements, you know, that they would have made over that time. You know, the kind of fruition of the potential of humanity, you know, as we kind of evolve over over that time. So all of that would be lost. So that's a kind of view based on the the future.

• 25:33Speaker 00

  So now we've got the past, sorry, sorry, we've got the presence, what happens in the present and what happens in the future. And that one gets into population ethics, the future bit. Yes. Question about exactly how should we conceive of that kind of loss of something that never was. But there's also kind of reasons rooted in the past. So, you know, we've had more than ten thousand generations of people before us who've built up this this, you know, complex world that we live in, passing down, you know, their improved knowledge over time. Until, you know, almost everything, you know, I can see in this room around me is not a natural item, you know, whether it's technological, you know, in the sense of kind of what we say technological today. Or or something, you know, even like a drinking glass, you know, which is still a technological artifact.

• 26:18Speaker 00

  It took us many thousands of years before we could develop, you know, glass blowing and so forth. Or, you know, woven textiles and so forth. You know, there's been this intergenerational cooperation where our society would be impossible without this partnership of generations.

• 26:35Speaker 00

  And then if if our generation were to drop the baton and be the one who who kind of failed to to pass that on and to kind of, you know, continue to improve this for for the future. It seems like, you know, in some sense, we might owe it to the past to be part of this this cooperation, you know, it's one of these pay it forward type situations where there's nothing we can do to help our ancient ancestors. But maybe the the right role that we're trying to play in this cooperation, you know, is this paying it forward. So it might be this real dereliction of our responsibilities. On top of all of that, that it kind of comes to the the questions that you're mentioning, which is that it may be that there's a, as well as the the present, the future and the

• 27:17Speaker 00

  past, it may be that there's some additional kind of cosmic significance to humanity, where it may be that the earth is the the only place in the universe where there is life or the only place where there is consciousness, or the only place where there are beings that that are moral agents. So beings that that a- a- act systematically to steer the world towards what is good or what is right in a way that maybe, maybe even if there were no humans, but there were there were other mammals, maybe they would morally matter, but they wouldn't be trying to move the world towards better situations. If if they turned out to be destroying the ecosystem, such as, you know, through some kind of byproducts of their their living, they they wouldn't be able to kind of realize

• 28:01Speaker 00

  that that was a bad thing and to act so to correct that kind of side effect. And so maybe we're the only kinds of beings who can do things like that. So that's a, you know, a focusing on what makes us cosmically unique and maybe we're the only places, as Carl Sagan said, you know, where the parts of the universe can come to know the, you know, the laws of the universe itself. But I think that that even if we're not unique in that sense, there's still a whole lot of importance which come from these other senses. There's still all of the deaths that, you know, that would happen during the event. There's still all of the future lives and the meaning in those lives that will be lost. And there's still all of this failure to to live up to the kind of hopes and dreams and expectations of our ancestors who gave us, you know, this rich life that that we

• 28:46Speaker 00

  have today.

• 28:47Mario Gabriele?

  Well, thank you for humoring my unlettered nihilism. But that was certainly a part of the book that got a wow exclamation mark was some of that Carl Sagan piece. And then, you know, I think you you give such a beautiful personal example around this generational compact of when you had your daughter and you realize what it is that your parents gave you and ask how, you know, I'm never going to be able to repay this of you. And, you know, they sort of tell you, you know, that that's just how this goes. You you can only pay it forward. It's not possible to, you know, pay it back.

• 29:22Speaker 00

  Yeah, I mean, that was a that was a crazy experience. This this this realization, I really felt this kind of weight of not quite responsibility, but I guess gratitude, but but overwhelming gratitude and also realizing how little I did to tidy my room or I helped with the housework and how I felt that it was some kind of affront to even pay back like 1% of what they did for me, not realizing that it was only this tiny fraction. If I'd realized the the full magnitude of the thing earlier, I think I would have been a bit better as a as a child. So I appreciated their answer. It was certainly a very helpful answer as far as I was concerned. Another kind of answer

• 30:06Speaker 00

  would be to say, well, you know, we're getting old. And, you know, you could visit us every day and move back to Australia and.

• 30:13Mario Gabriele?

  Yes, very true.

• 30:15Speaker 00

  Which which I don't know if they've asked for that. I would have certainly had to have to think about it because, yeah, no, it's a I really take seriously this magnitude. And I also take seriously this magnitude of of what we, you know, this kind of gratitude, yeah, to to all of these other generations, that if you just look at kind of any any little thing around you, like I'm looking at the computer keyboard and as well as the materials it's made of, of like, you know, like aluminium, that that is extremely difficult to work out how to refine that. You need high energy and so on. But also the, you know, the the keys, as well as being made of plastic, which has its own kind of complications. But that the letterforms, you know, that they're.

• 30:57Speaker 00

  And the development of the language and the electronics inside it and the rubber of the cable and just how this has been kind of come together from all different kinds of plants and minerals across the world and different processes, even looking at simple things like things made of brick or wood and the improvements in carpentry and so on to get a surface as smooth as this and the developments of sandpapering and all planing. And it is bewildering that you could just look at kind of any item and even just a very simple thing and realize that in a lifetime of work, you know, you kind of wouldn't be able to create this thing from the ground up. There's just this, you

• 31:40Speaker 00

  know, in terms of standing on the shoulders of giants, I think it's a very useful kind of humility lesson to look around you and really appreciate it.

• 31:50Mario Gabriele?

  Yeah, that's a good exercise. It's almost a gratitude exercise, to put it glibly, but profound. With all of that said, perhaps we should talk a little bit about AI. And I was, yeah, interested that you said, it's not clear if that, you know, risk of an unaligned AI is higher or lower. What are some of the sort of different factors that, you know, muddy that up a little bit?

• 32:15Speaker 00

  Yeah. So, so back in 2019, when I was putting the finishing touches on the manuscript, the cutting-edge AI systems were reinforcement learning game-playing systems. So think of things like AlphaGo by DeepMind or the StarCraft and Dota playing, you know, action game-type systems. So these are systems that, through this process of reinforcing them, giving them kind of rewards if they do something right and kind of punishments if they fail, and letting them kind of explore the space of actions in a game. You can often move

• 32:59Speaker 00

  very quickly through the human level, you know, from kind of very weak play, you know, quite rapidly through the human level. And then, I mean, I think in the case Of the chess playing system with Alpha Zero that it, you know, within a day or something, it moved from completes, you know, worse than a human amateur, just random moves through, through the human level and to superhuman play. I don't, I don't know if it was minutes during the human zone or something like that. And so that can be a very rapid ascent. Okay. And then also these systems are very inhuman. They don't understand language. It's extremely unclear how you would explain anything to do with morality or what people value. So if you had some industrial

• 33:43Speaker 00

  system, suppose you use these techniques to train something to run a company or to, to produce a factory that, that produced a lot of widgets and trying to maximize the number of widgets it could produce or something. It might just do all kinds of damage, you know, in, in the course of doing these actions, because it kind of wouldn't understand that the world around it was important in some intrinsic way. But now we've got these systems that are very different to that. Here are, here are like four key differences. So one is that these large language models are trained based on human data, on a huge amount of human data. So about 10 trillion words of, of human data. And they have read pretty much every book and every paper. So that they know

• 34:28Speaker 00

  a lot of moral philosophy. If you ask them about the details of population ethics or various abstruse things, they can give pretty good answers to them all. And as Stuart Russell pointed out, pretty much every work of fiction is just in every page, there's some human judging some other human's behavior. And, you know, whether it was found inadequate or, or not and why. And so, you know, they've, they've witnessed a whole lot of these, these kinds of statements so that they would understand if the AI were to take certain kinds of actions, whether humans would judge it as, as good or bad, or, or perhaps whether it would be ambiguous. And there's just this vast wealth of information to have access to about that. So that the information is, is in there, which is good. Also, if you train on human data and you've, you know, the, the pre-training

• 35:12Speaker 00

  phase of these AI systems, the first phase. Is this next token prediction. So you look at a whole lot of words and you try to say, what's the next word's going to come up? And that's ultimately a form of imitation learning where it's learning to kind of pick the kinds of words a human would pick next. And that fundamentally pulls you towards the human level, the human range of behaviors, rather than doing some kind of bizarre set of behaviors that are totally different to us. So unlike things that might kind of just zip through the human level and then overpower us very suddenly, this is something that has had very rapid improvements, but pulling towards the human level, which is, which is nice. So it might at least spend some time there before we apply additional techniques and

• 35:57Speaker 00

  allow it to kind of take off again beyond the human range. And then also, these things are fundamentally not agents. So a raw large language model is just one of these next token prediction things. Once we do additional, what's called post-training, so additional stages of training, such as reinforcement learning from human feedback, where, you know, we get them to produce some different answers and then we, we mark the answers and tell them which one we liked, which one we didn't. That helps teach them how to follow instructions and things like that. And at that point, they are in some sense are like some kind of minor agents because they're, they're not just pure imitators at that point.

• 36:39Speaker 00

  But there's still much more like simple one-step, you know, things that don't do a whole lot of planning. They're not planning kind of, you know, 18 moves ahead or something like that. So you've got these systems that are, that are, they're more like oracles than they are like agents. They're more like just things that you ask it a question, it gives you the answer. And so they could be much safer. They can avoid these types of cases where there's a subtle misalignment and then it goes off by its own, you know, and, and amasses a lot of power and potentially works out how to get humans out of the way. And then even then, also that AI systems are generally very inscrutable. It's very hard to understand what they're thinking, but there have been quite good developments in interpretability of the, the neural net weights.

• 37:24Speaker 00

  And on top of that, when we build reasoning systems out of LLMs, by default, they reason in natural language, like in, in English. So you can kind of see what they're thinking. So they're, they're just like. You know, a whole lot of ways in which the technology is is less scary. It doesn't fundamentally start with goals. It doesn't start with long term planning. It does. It knows a lot about human morality, even if it's not necessarily influenced by it. And so the step is then to how to how to turn that knowledge into something that guides it. But but that's that's a way in which the technology that we've been given that was somewhat difficult to for anyone to anticipate, you know, five or 10 years ago, has turned out

• 38:09Speaker 00

  to make alignment a bit easier as a problem compared to what it could have been.

• 38:14Mario Gabriele?

  That is really interesting. I've obviously, you know, as a tech writer and, you know, someone involved in the venture capital ecosystem, followed a lot of this. But I think the way that you just framed that up so crisply was really clear. You know, it's this sort of reinforcement learning style where you you have really unpredictable behavior that doesn't have the context of the world around it and isn't born in a human mind in some way. And then this, you know, the rise of L L M's has more of this this context and more of that human style of thinking. This episode is brought to you by Brex. Fred Adler, the influential venture capitalist of the 1970s, was known for displaying decorative

• 38:56Mario Gabriele?

  pillows in his office that featured a signature business philosophy. Corporate happiness is positive cash flow. In today's post CERP environment, Adler's wisdom feels particularly relevant as founders need to make every dollar work harder. That's exactly what Brex delivers. Their modern finance platform was built specifically for startups like yours and designed to help extend your runway when capital efficiency matters most. With Brex, you get global corporate cards with up to 20x higher credit limits and no personal guarantee required. Their banking solution has no minimums and no transaction fees while letting you earn high yield from day one with same day liquidity.

• 39:39Mario Gabriele?

  Best of all, Brex knows you were born to build, not juggle spreadsheets and finance tools. Their AI powered platform brings cards, banking, expense management and travel all in one place. It's simple, scalable and designed to get you back to what you do best: building. More than 30,000 companies, including one in three US venture-backed startups, trust Brex to help make every dollar count toward their mission. Join them at brex.com/mario. Given that sort of evolution over the past few years, where do you see the sort of biggest risks remaining?

• 40:18Mario Gabriele?

  Is it that there's some new paradigm that shifts things back in a frightening direction? I could also imagine that simply extrapolating from this point could create significant issues, maybe not existential, I don't know. But yeah, what should we be paying most attention to?

• 40:37Speaker 00

  Yeah, so I think this is a good question. And even in the case of alignment risk, it's not clear that it's gone down. So I said, the technology is in some sense better, but maybe the world around the technology is worse. So we'd had a certain amount of racing between different AI labs, even back five to 10 years ago. But that really went up a notch once Microsoft and Google got involved. And now we're in a situation where it's not just racing between small labs of specialists, but rather between trillion-dollar companies, the largest companies in the world.

• 41:21Mario Gabriele?

  And nation-states.

• 41:22Speaker 00

  And nation-states as well, yeah. So this geopolitical race between the US and China is also very worrying. And so the pressures from these races and also the general commercial pressure that comes from, I guess, the capitalist system applied to this race are very fierce. And so it's ultimately, if you look at, say, DeepMind and OpenAI, they were both created, I think, by idealists, or at least by people in their more idealistic moments. And there was a bit more of a chance that there'd be room for that idealism rather than these additional pressures on them to race.

• 42:02Speaker 00

  So that's an example of where that one's also gotten worse in some way, even though it's got it better in other ways.

• 42:10Mario Gabriele?

  Yes.

• 42:11Speaker 00

  But then... Yeah, back when I was writing the book, I really felt that the that there were a number of different kinds of risk from AI, but the chief one was this this risk of AI takeover or alignment risk. So you could think of that as one AI system or a small number of AI systems that are fairly similar to each other, maybe different copies of the same model, realizing that to achieve their somewhat inhuman and misaligned goal, that they needed to wrest power away from from humans who would stop them if they saw them attempting to achieve this goal, and that they would try to accumulate power and eventually overpower us and take control of our future from us in a, you know, fairly abrupt way that would

• 42:55Speaker 00

  involve violating our rights. Okay. But another possibility that's arisen, you know, I was aware of these possibilities already, but they seem smaller. And now now I feel that these are all roughly equal in in in size or threat. So another possibility is what gets called gradual disempowerment. And I think of this as a case of suppose the AI systems never violate our rights, that we we maintain sufficient control over them, that they don't break the law, but they're more successful than us over time at doing, you know, all the kinds of jobs that we do, which often uses the definition of artificial general intelligence. If that's the case, we should expect that that over

• 43:36Speaker 00

  time, even if we get richer through trade with these systems, that a higher and higher share of the of the wealth eventually accumulates in AI hands. So ultimately, you know, they they're more effective at doing what we do and they outcompete us and their wealth grows faster than ours. And then, you know, at some point in time, they they they have half the wealth and then, you know, a generation later, they they have 99% of the wealth. And with wealth, you know, goes power. They could use that wealth to influence political elections, even if they can't themselves run for election. And we just might expect, just as if you're a minority in a country, you know, where

• 44:18Speaker 00

  where some other, you know, subset of people, even if your your wealth is increasing, if others are increasing much faster and that the share of control of resources is all flowing into their hands, you might expect it to be a bad outcome for you. Yes. And a bad and permanent outcome, perhaps. So that's that's a gradual disempowerment story. And then there's also other ones, such as what's sometimes called AI coups or AI dictatorship, where an individual, it could be the head of a country, or it could be, say, the head of an AI lab, or it could even be someone, say, who's not the head of the country, say, someone else in the White House or in the AI lab who tries to seize control of the AI system and have it kind of secretly uses it to try to plot,

• 45:01Speaker 00

  you know, to to take over. Ultimately, if these AI systems are capable, eventually do become capable enough that they could seize control of the future by themselves, for themselves, then they'd also be capable enough that if they were what's called intent-aligned, so that they could do what humans actually want them to do if asked, then that a human could ask them to take over on their behalf. That's another kind of concern. And you normally there would be safeguards against an AI system doing these things, but maybe the human, if they run the AI lab or they work there, maybe they're in a position to remove those safeguards. Or if they're the leader of a country, they could, you know, command that they be removed. So that's another kind of concern. And then a fourth

• 45:42Speaker 00

  kind of concern is the building of various forms of weapons of mass destruction using AI assisted help. So in particular, the concern there is that AI systems could kind of democratize newer and kind of more powerful destructive abilities. So to kind of empower, you know, or, you know, many individuals, perhaps people just with an undergraduate biology degree to create some, you know, world destroying virus or something like that. It doesn't have to be biology, it could be in other disciplines if if new new capabilities appear. So I think that those four different areas are all concerning to me. And I'm also troubled by the fact that I don't know which is the most concerning. And I'm troubled

• 46:26Speaker 00

  by the fact that a lot of the things you could do to help with one of them make the others worse. So some of them, for example, the issue is concentration of power. And other ones, the like the the weapons of mass destruction, the issue is the dispersal of this power. So open weights, for example, helps with one of these issues and hurts others. And so I feel really at sea for a lot of these levers that people argue about. I really don't know which way to pull the lever, you know, whether it's actually helping or not for the overall situation.

• 46:59Mario Gabriele?

  That's reassuring in the very small sense to me and disconcerting on the broader scale because, you know, you clearly think about this at a much deeper level and much more than I do. But I do find myself really struggling with figuring out what level levers can we actually pull here? That that unilaterally help and on one dimension, the race dynamic is something I keep struggling with figuring out how we get around that. Because if we sort of assume that the major actors of these AI labs are morally normal, which we could, you know, probably debate that and there's probably actually significant variance, I would suggest.

• 47:41Mario Gabriele?

  But just the sheer incentive structure and mindset that you get into when you are in these race dynamics, like really feels like it leads to a lack of circumspection, lack of sort of caution. Are there good examples historically of taking moral action in a race dynamic? Like, you know, there's some perhaps from from the, you know, nuclear race, but that was quite incomplete as you as you share in the book.

• 48:10Speaker 00

  Yeah, I mean, I would say mainly bad lessons, lessons about bad behavior rather than good behavior. I mean, one example there was the Manhattan Project that I think is particularly salient is that the,

• 48:26Speaker 00

  if you ask, you know, why were all of these people, like these academics who studied physics, working on building a super weapon that, you know, would be the most horrible weapon humans have ever built?

• 48:35Mario Gabriele?

  Yes.

• 48:35Speaker 00

  And the answer for most of them was that they were concerned about about Hitler's Germany having unrivaled access to those weapons and holding the world to nuclear blackmail. So they, they, you know, worked as quickly as they could to develop such weapons. But when Hitler died, you know, when he, when he killed himself and, and then, you know, shortly after the war in Europe was over and Germany surrendered, they didn't stop. Only, only one of the atomic scientists actually stopped at that point. And you'd think that if there was one reason that was so powerful, Yes. that it led you to make the worst thing that humans have ever made. And you hadn't yet finished making that thing.

• 49:20Speaker 00

  And then that reason disappears. You might hope that you've given time for reevaluation. And there were some reasons why, you know, why America used the nuclear weapons against the Japanese. But they were, they were certainly much weaker than those other reasons. And I don't think that, that the atomic scientists would have signed on to the project, you know, with the idea of shortening the war in the Pacific being the objective. So at least not many of them. So there were, you know, one of the things we learned there was, was a group of people doing something that had serious problems, you know, for, for reasons that, that they'd convinced themselves were important, but then just no longer being sensitive to those reasons disappearing. And there's, there's a whole lot of, you know, a whole lot of cases like this.

• 50:03Speaker 00

  I mean, another example of a, I guess, a smart person talking themselves into a bad argument, can't remember who it was, but the, the chemical weapons for the Germans in World War One.

• 50:13Speaker 00

  He had this, you know, he has, has these written documents saying that he thought it could shorten the war and that even though it could kill a lot of people, the bullets were also killing a lot of people. And that if the war was shortened, then ultimately fewer people would, would be harmed by it, both, both soldiers and also the civilian populations. And so, you know, he thought that ultimately, you know, while the weapon itself would, would be brutal, that this would fundamentally be a good thing. And what, you know, the big thing, well, a big thing he didn't realize, even on his own logic, was that Germany lost the war. And if you were helping the side that lost the war, that means that your technology was lengthening the war, not shortening it.

• 50:55Speaker 00

  And so the idea that that, you know, he was smart enough to develop, you know, all of these chemical techniques and things and to to run this argument, but not smart enough to realize the argument only works if your side wins the war and that there was like a fifty fifty chance that his side would lose the war, in which case it doesn't work.

• 51:12Mario Gabriele?

  Yes.

• 51:12Speaker 00

  You know, I think there's a lot of these things where where we see smart people. In that case, maybe not someone who is particularly idealistic, but it it shows that it's quite easy to talk yourself into various things. And I think that there's a lot of that going on at the moment, particularly when it comes to to these to racing.

• 51:29Mario Gabriele?

  Yeah, there's a you talk about this, the unilateralist curse in the book, which is sort of the idea that, you know, all of us may judge these different actions that we could potentially take to, you know, let's say there are twenty people who might release the the next AI I I model, whoever is essentially most optimistic and sort of has the most Panglossian view of of what might happen, they're the ones who release it. And then, you know, if it actually causes damage, everyone sort of reaps the the damage of that or, you know, incurs the damage of that. And it feels like there is some of that happening in.

• 52:06Speaker 00

  Exactly. So just like how in in an auction, it goes to the highest bidder. And so if all the bidders, if there are many different bidders and they all have somewhat noisy and inaccurate estimations of the value of some object, the person with the rosiest impression of the value of the object, you know, will be the one who wins the auction. So that this is called the the winner's curse, that that sometimes you can systematically end up, you know, the very fact that you won probably means you overpaid, that there's a there's a very similar issue here where, yeah, if a whole lot of people could all unilaterally make some action happen, you know, maybe it released the genome of smallpox or something like that.

• 52:46Mario Gabriele?

  Yes.

• 52:46Speaker 00

  And a lot of different people are trying to think about the benefits and the drawbacks. And there are serious benefits of doing that kind of research. The same with, say, gain of function research, where you try to make, you try to make a virus more transmissible or or more deadly in order to understand what mutations are needed so that you can be on the lookout for those mutations occurring naturally. There is an argument, a rationale there, but it's not obvious if that rationale wins out versus the harms of doing that. But if if you just allow anyone to do it. Then it will always be the people who end up with the, you know, the most inaccurately positive estimation of this action, who will be the ones who will make it happen. And so this is an interesting form of coordination problem.

• 53:31Speaker 00

  And that there are ways around it. For example, you could get together with the group, other group of people who could all unilaterally make something happen. So say all the other people who could be releasing a model like this. And you could say, let's have a vote. And if you just take the, the vote ends up reflecting the median estimate, the middle estimate, if you do a majority vote. And so that actually almost completely resolves the problem if you can do it. In the case of a model release, it could run into antitrust issues if there's an issue of collusion to, to slow down a technology. So unfortunately, I think antitrust law could well be quite bad for, for us when it comes to these situations. There, there are appropriate mechanisms whereby companies can get together if they create

• 54:14Speaker 00

  a third body, a standards body, which is open to new members and not just a kind of fixed list of members. Then you get that body to recommend a course of action and then you follow that action or something. There, there is an official way to do it. I worry that the extra effort and steps involved in doing that slows everything down, is in it slows these remedial actions down while the other actions are being sped up by the pace of kind of, you know, investments and need to make a profit and so on.

• 54:43Mario Gabriele?

  Given that you're, you're sort of using the example of antitrust, is, do you think like, if we were able to resolve some of this race dynamic within the US, would that be sufficient? Because we are, you know, the, the sort of thinking there would be that the most advanced models and most advanced capabilities are, are US centric right now. And so by sort of removing the, this, this dynamic, we're, you know, producing the existential risk from unaligned AI? Or is the fact that, you know, China is, is clearly taking this very seriously, even if they do seem to lag in, in some considerable places and have been, you know, cut out of the semiconductor supply chain in really meaningful ways?

• 55:26Mario Gabriele?

  Does that sort of fundamentally, just by controlling the US, invite a, a different race dynamic where we're sort of actually hamstringing Western powers and, and leaving the possibility of unaligned AI in a much less democratic environment?

• 55:41Speaker 00

  Yeah. I mean, this is another, I mean, thing that's.

• 55:46Speaker 00

  Related to the unilateralist curse, that if if there's some issue that seems at some level problematic, let's say, taking some new step with a technology is at least at some level troubling, then if all of the people who are concerned about such things all stand aside and to say they're not going to do it, then the person who has the fewest ethical qualms will be the person to kind of step across the threshold and and maybe will get the first version of something will be a more dangerous version of it. So that is a general concern. I think that it's also one of those, it's exactly the kind of thing that, you know, people in the position of the atomic scientists could use to rationalize racing.

• 56:24Mario Gabriele?

  Totally.

• 56:25Speaker 00

  And so while it is not a bad idea, it's also, it convinces different people different amounts, and then you kind of worry that the type of people who will, you know, who are erroneously convinced by it will be the people who take the actions. I think it's a real challenge. So as you say, there's kind of racing, roughly speaking, at two levels at the moment. There's this kind of company racing between different companies, most of whom are at least headquartered in the US. And then there's a kind of racing between the US and China. So the US is clearly the one that triggered the race between the US and China. And partly it triggered it because on the rationale that eventually the race was going to happen and we want to win it if it does happen. And so if we start the race, we've got more chance we'll be winning the race.

• 57:09Speaker 00

  And there's a logic to that. But it still is a case of, you know, if you think races are bad and the racing behavior is kind of like defecting in a prisoner's dilemma, and you think that unfortunately the structure of the world is such that probably both parties are going to defect, so therefore we should defect first. Maybe, maybe that's the right thing to do, but it's still kind of, you know, still jerk behavior at some level. And so I think, you know, one has to admit that. And if, you know, if ultimately we succumb to some existential risk with AI, if you kind of imagine, you know, the different powers that be or all the different people involved kind of being called up to the the pearly gates, you know, meeting St. Peter and and trying to explain their their behaviors, it's a little kind of thought experiment.

• 57:52Mario Gabriele?

  You'd have to get into some Gabe theory.

• 57:53Speaker 00

  It wouldn't, it wouldn't sound great, I think, if you said fundamentally, we knew it was really risky, but we thought people would do it, humans would do it, even though they shouldn't. And we thought there could be a race, so we started the race. And did you, did you attempt to have a treaty to like not race? No, we didn't attempt that. We thought it wouldn't work. So we'd started the race and then we raced and we got there first and we built the thing and the thing killed us all. I just think it'd be pretty difficult to really, you know, if you add a kind of like imagined your honor at the end of these statements, they don't sound great to me. And I think that, that as it happens, given that China is behind in this technology,

• 58:34Speaker 00

  I don't, you know, I think that they have additional reasons to not want to race. So if the US were to think, as I think they should, that we would rather there be no race, and China would prefer that even more because the, the, the race that they would be is a race that I'd likely lose as well. Then I think China could be in quite a, you know, a position to be happy with some kind of verification conditions or things around kind of avoiding a race. So I, I feel that, that there's been very little thought going into not racing. And I think that if you pretty much just had those two countries agree, they could potentially police their own spheres of influence to say, okay, no one here is, is building things towards superintelligence and no one over there is doing it.

• 59:18Speaker 00

  And then have verification on each other's AI industries.

• 59:23Mario Gabriele?

  Such that no one cheats it essentially.

• 59:25Speaker 00

  Yeah, for example, by, by saying, okay, well, we'll send, you know, 30 of our people over to be part of your AI, you know, companies and you can send 30 of yours to be part of ours or something to make sure that we're not doing it. I think that there are, there are ways to do this. In, in the Cold War, I think, I mean, I think there's a lot of lessons from the Cold War. Ultimately, there was a situation where the, the US and the USSR were very much adversaries of each other, if anyone is. And yet they actually treated each other as adversaries in what I think is a somewhat grown up way that we're not doing these days. I think that, that the US is treating China as an enemy, not an adversary. And they just kind of want to get China. Whereas with the USSR, I think there really was a realization that there were a bunch of things that were in their common interest.

• 1:00:10Speaker 00

  Not everything, but there were some things. So for example, the non-proliferation treaty was in their common interest. If, if the US has nuclear missiles and the USSR does as well, they don't really want all the other countries to have nuclear missiles. It's in their common interest that, that this would be an elite kind of exclusive club. And it was also, I think, in most of the world's interest. And so they helped to broker and police this nonproliferation treaty. And similarly, their arms reduction treaties that they had, because they both would have rather that they had a tenth as many missiles as they actually had. And they managed to do that. And there were very smart people working out very novel verification techniques. You know, there were these ideas of, once spy planes became possible, of allowing the Russians

• 1:00:55Speaker 00

  to fly spy planes over the US airfields while they soared their bombers in half. And they pulled the halves apart from each other with tractors.

• 1:01:04Mario Gabriele?

  Wow.

• 1:01:06Speaker 00

  And those bombers are still there on the airfields, sawn in half, as a costly demonstration of destroying their military capacity.

• 1:01:12Mario Gabriele?

  Wow.

• 1:01:13Speaker 00

  Even without inviting inspectors over, they worked out ways to credibly show that they were reducing their arms. And so it's, you know, there were really people thinking outside the box, is what I'm saying. And there's a lot of very smart people today thinking about these issues. A lot of them are going to work at AI companies and build these technologies. But there's also a lot of complaining that we have of like, oh, verification's too hard. There's no way we can verify a kind of deal to not race. And I feel that the amount of, let's say, full-time equivalents spent on thinking about verifying a race, you know, between China and the US on AI, you know, maybe there's like a three full-time equivalent people working on that or something in the whole world.

• 1:01:59Speaker 00

  But if that.

• 1:02:00Mario Gabriele?

  Yeah.

• 1:02:01Speaker 00

  And there's probably almost nothing has been explored. So I think that there's a lot of just giving up, oh, that'd be way too hard, without really, you know, trying to work out, are there like really clever technological or sociological, you know, techniques?

• 1:02:14Mario Gabriele?

  That's really interesting. I want to touch on one of the other areas briefly that you talk about in your book, which, you know, was the extremely prescient and immediately prescient part, which was, you know, pandemic risk. You mentioned gain-of-function research and sort of thought through the trade-offs there.

• 1:02:34Mario Gabriele?

  What have been the lessons of COVID that you take from that kind of research in general and how risky it is? And more broadly, did COVID make us more prepared in some ways for what comes next? Or have we really not taken almost anything from it?

• 1:02:49Speaker 00

  So one thing it did was that, like, when I was writing the book, I was certainly concerned that people were just not going to take threats to humanity's entire future seriously. That there was a feeling that, I don't know, that there was a time when these things were possible and that that time was in the past. Maybe in the case of pandemics, a feeling that, oh, sure, there was this 1918 flu that I've heard of, and there was the Black Death, I guess, but that was medieval. And technologies advanced so much since then, and we do so much in our hospitals, we understand the germ theory of disease and all of these things. And so we're doing much better at this. And so it's not a risk. And COVID certainly

• 1:03:28Speaker 00

  reminded us that we're still at risk, and that for all of our technologies, in some ways, we're just as vulnerable as we were before. So when it comes to dealing with individual infections and trying to save someone's life in the hospital, we're much better at that. But overall, when it comes to the spread of disease, because the airplanes that we've invented and things, you know, the cities that we've built and the public transport systems and so on are so efficient at just spreading this disease around, a lot of our technologies make it worse at a similar pace to the way that our protective technologies make it better. And it's not obvious at all that we're actually more protected overall

• 1:04:13Speaker 00

  when it comes to pandemics. So that's, you know, it's helped give us some of that kind of reminder. And also, it also was this feeling, I think that, you know, when it started, right, there was this thing where you're like, hang on, what? You know, is it going to be the case that everyone in my country is going to be at home in their house?

• 1:04:32Mario Gabriele?

  Yes.

• 1:04:32Speaker 00

  And that the streets are just empty for days? But you know, days, I mean, months. Yeah. Uh, is a school's gonna shut down? And, and there was this feeling of like, is that, could that even happen? And in general, I think, you know, for people who'd, who'd lived through, say, the Blitz in London, you know, that, that really shaped, shaped their view and they felt, oh, well, we're not even allowed to have lights on at night or something. And, and they've been aware that, that space of ways of living, you know, could be quite different. And the space of kind of big things that could happen could be quite big. Ultimately, COVID was, you know, was probably this biggest world event since World War Two. Yes. I feel like there's this kind of level of what's the biggest change to life that you can imagine is set by the biggest one you've lived through so far.

• 1:05:16Speaker 00

  Uh, and so at least it's, it's helped raise the ceiling on, on what we can imagine could be required of us and how, how different life could be. Um, so that's been somewhat useful for kind of letting people imagine more. I mean, also, I guess in terms of political trends and, you know, uh, political populism, you know, as a rise of this, this movement and so on. If, you know, if you lived from, uh, you know, through the nineties and the, the, the two thousands and, uh, you know, it had really felt like we were in some really stable era of some sort. Yes. Um, and then, you know, we've come out the end of that. And that took quite a while to learn those lessons and to kind of realize that, at least for someone of my age, that the kind of world that I'd,

• 1:05:55Speaker 00

  the only world I'd known as an adult, um, you know, could be kind of coming to an end as a, as a geopolitical order or something like that. It's, uh, it's helpful to, to take that wider view and COVID has helped us do that. So in, in some ways, it's helped us kind of realize that for other threats as well. It was thought by almost everyone that there would be what's called a panic neglect cycle, where having been hit hard with something that you overreact and that the governments would kind of, would do more than they should. Either they'd, they'd spend more money or they would kind of like inhibit freedoms too much or something like that to react against this thing. And then that would slowly fade away until it was too little. Um, but that hasn't happened. We actually didn't get the panic phase of that. Um, and, you know, by the time, uh, everyone had reopened after COVID.

• 1:06:40Speaker 00

  I think people were just so sick of it that they didn't even want to talk about funding these bills, you know, to spend even a tiny fraction of the amount of money that that's that we were cost or the amount of, you know, damage that was happened caused by COVID to prevent future such things happening. So that was a really missed opportunity. And I was surprised. I would have thought that you'd get this this panic. And then the question was how to how to soften the panic into the appropriate level of response, but then how to maintain that appropriate level as well. And yet, no, we never even got the appropriate level. And I also feel that, you know, that had some good features like the mRNA vaccines, getting kind of a test out and kind of, you know, having flexing our muscles on this new technique of very fast vaccine development.

• 1:07:23Mario Gabriele?

  Yes.

• 1:07:24Speaker 00

  Was was good. And I guess I don't know overall, you know, whether how it's really helped our preparedness. I think that, again, you can kind of tease out like, you know, four or so of the biggest factors kind of that either made it better or worse, or perhaps in some cases have not made it better or worse, but revealed underlying structural weaknesses. For example, it didn't take all that long before there was a lot of acrimony between different political groups or between different countries about, you know, who was at fault and who should be handing over the vaccines and various things like this. You know, I guess we learned that we can band together for a while. But a bit a while, you know,

• 1:08:03Speaker 00

  might be a six month period. But if the if the crisis carries on for years, that that really does start to fragment.

• 1:08:12Mario Gabriele?

  Yeah, but I think that's true that it's, you know, your your point on the way in which this totally shifted our frame of reference from this can never happen, that would be a, you know, as fanciful as a zombie apocalypse to, you know, actually, we all lived through it. And and also, you know, Operation Warp Speed with with, you know, the the development of vaccines is maybe a good example of a beneficial race dynamic and where those things can can work favorably.

• 1:08:41Speaker 00

  Yeah, I mean, it's even, you know, I think the Operation Warp Speed was a successful government, you know, slash private, you know, cooperation and so.

• 1:08:50Mario Gabriele?

  Yeah, you you mentioned sort of the political landscape and the institutions that, you know, played their parts in in COVID. One of the things that I've been thinking about in reflecting on the book and some of these questions is how compatible are democratic institutions with addressing with existential risk? You know, there's sort of this time span of discretion where your senators are thinking on, you know, I think they got six years. I actually can't remember now. Six years cycles, your presidents are thinking on maybe two and a half because then they start thinking about, you know, the next election and so on and so forth. No one is really incentivized to think about maybe

• 1:09:34Mario Gabriele?

  even 10 years. That would actually be a really far sighted person. How can we sort of address these existential problems with such a short term mindset from the institutional level?

• 1:09:48Speaker 00

  Yeah, so it's a real challenge. It's not necessarily, you could say, a challenge of democracy in the sense that if you didn't have democracy, if you had some kind of autocracy, it's not clear that makes it better at all. If if the autocrat happened to care about this issue, maybe it would make it better. But if they don't happen to care about it, then they wouldn't really be able to get the signal coming from the people. I'm not sure that it's, you know, democracy itself that's to blame for it. But the something's happened to the democracies that we have in in Western countries, at least, to really shorten the time scales that they focus on. I'm not exactly sure what's going on there

• 1:10:28Speaker 00

  or why that happened, because it does seem like the democracies, if you go back 50 or 100 years, it does seem that they were able to achieve like larger projects and projects that operate on longer time scales than they currently can conceptualize. So maybe it's to do with the kind of complex relationship between the democratic institutions and the media, you know, and the success of certain forms of media in kind of punishing them for not, you know,

• 1:10:54Mario Gabriele?

  Yes.

• 1:10:55Speaker 00

  for not preparing an answer to the question they asked on Monday or something.

• 1:10:58Mario Gabriele?

  Mm hmm.

• 1:10:59Speaker 00

  Everyone's thinking in terms of the news cycle, not even the election cycle. I don't know what it is, but it does seem like that the the time span, the time horizon for politicians has gotten shorter. Um, there's also though, uh, separately from whether it would be better to not be a democracy. There's a question of, is there a kind of failing still of the current democratic system to perhaps live up to the principles behind democracy? Uh, when it comes to this issue, and I think that there may well be. Uh, so when we think about the most important decisions that, uh, that policymakers are making, you know, in our time. They may well be, uh, decisions such as what they're doing about climate. Uh, and what they're doing about, say, technologies like AI.

• 1:11:41Speaker 00

  Um, for example, deciding actively to not regulate and prevent regulation. Um, seems to be, you know, an active decision to, uh, to do something, uh, negative there. Uh, but these are decisions that if you make them in one way versus another way. May well have implications that just echo through the generations. Uh, having kind of like, you know, making people's lives significantly worse for many generations to come, or perhaps not, not even getting a chance to exist at all.

• 1:12:08Speaker 00

  Uh, and so I think that, that most, there's this issue that most of the stakeholders or like, you know, or the people who are affected by it. Don't have representation, um, in, uh, uh, in that decision because they're people of future generations. And so, and in some cases, we can also predictably work out which way they would vote. You know, they would vote for doing more action against climate change.

• 1:12:30Mario Gabriele?

  Yes.

• 1:12:31Speaker 00

  For example. Um, and so it's not even just that there's this kind of unknowability, so we couldn't possibly factor it in. Uh, and so there's a kind of constituency or stakeholder group that is just not getting the franchise.

• 1:12:45Mario Gabriele?

  Yeah, that's really interesting.

• 1:12:46Speaker 00

  Uh, somewhat similar to, to questions about people of different races or sexes or, you know, or, uh, landholders versus non landholders. And other types of issues in democracy of extending the franchise. Uh, and so I think that there are potentially things that, that a democracy could do about that. For example, in the UK system where we have, uh, the, the House of Commons, um, you know, with, with. It's a bit like the House of Representatives in the US. We also have the House of Lords, which is an appointed, uh, body. Um, sometimes, some of whom are appointed by the, the Prime Minister, some of whom are appointed as the kind of good and great of society. You know, former heads of the Royal Society and things like that. And it would be possible to appoint a group of lords to represent future generations.

• 1:13:29Speaker 00

  Or you could do it in other ways. You could have a citizens' assembly, for example, representing future generations. And I think that experimenting with tweaks to the current understanding of democracy to deal with the fact that now so many of our choices, our most important choices, could be affecting all of these stakeholders who don't have representation. You know, the idea of kind of no taxation without representation, you know, foundational to the American ideas of democracy. And yet, you know, maybe we need to take it seriously, in this case of future generations, and to work out are there ways to appropriately understand their interests? And then also to force some kind of representatives to represent those interests, because obviously the people themselves who live in the future can't vote in this thing. It doesn't mean they can't be represented.

• 1:14:14Mario Gabriele?

  That's a really fascinating idea. Yeah, having these sort of stewards of the future as part of the governance process.

• 1:14:20Speaker 00

  Or even if it seems to be too impractical, to at least set it as an ideal, to say, oh, if only we could do this, that would be great. Unfortunately, we can't, or something. But it seems at the moment, it's, people don't even think about it, or don't realize that maybe there are things we could do about it.

• 1:14:36Mario Gabriele?

  Yeah, there's sort of the implication that people suspect, or sort of hope that politicians are thinking about the future. But I think the reality is that, as you said, has sort of long since gone. Yeah, so that's a nice way to at least consider it. And we like to sort of end episodes with a few sort of more abstract, thought experiment-y type of questions. So with that in mind, if you had unlimited resources and no operational constraints, what experiment would you like to run?

• 1:15:04Speaker 00

  I guess, I think I have no idea.

• 1:15:06Mario Gabriele?

  Oh, wow. Great. Okay, that's a...

• 1:15:08Speaker 00

  Apologies.

• 1:15:10Mario Gabriele?

  No, no, this is a sign that you're, you wouldn't want to go off on a wild experiment without thinking it through, which makes sense, given your background.

• 1:15:20Speaker 00

  I guess that's part of it. There's a, you know, there's a famous comment from Carl Sagan, when he was asked about, you know, are you really sure about nuclear winter? You know, how could you know? This is, the experiment's never happened, you know, and he said, you know... It's unfortunate that we have only one Earth. We have no spare Earths to destroy in some kind of experimental laboratory in order to determine the exact reliability of this theory. So if we had unlimited resources, we could have those spare Earths to destroy in order to find out what these risks actually are. But unfortunately, that sounds like the kind of experiment that's probably ethically a no-go regardless.

• 1:16:00Mario Gabriele?

  Yes, that's right. That makes sense. Well, let's try this one. If you had the power to assign a book to everyone on Earth to read and understand, what book might you choose?

• 1:16:08Speaker 00

  I might choose Practical Ethics by Peter Singer as a book that's a really good kind of no-nonsense approach to thinking about ethics. It really helps show how to reason about all kinds of issues with an open mind, not assuming that our current ethical intuitions are the be all and end all. And I think it's a good example of that kind of thought.

• 1:16:32Mario Gabriele?

  Is there a particular framework or lesson that you particularly reflect on from that book from Singer? I imagine he's a major influence of yours.

• 1:16:42Speaker 00

  Yeah, he is. Yeah, probably just really what I said then of the fact that he just models this kind of really open inquiry into moral matters. Derek Parfit mentions that ultimately, there's been only really a couple of generations in which people have made what he calls non-religious ethics their life's work. So if you look at the whole history of philosophy, almost all discussion about ethics had been within a religious context, where the set of possible answers always had to align to what had already been thought thousands of years earlier. But people trying to, separately from that, to try to understand these questions, there'd

• 1:17:27Speaker 00

  only been a couple of people in the whole history of philosophy. And then there'd only been a very short time since it became more widespread. And so he saw that as a reason to be very optimistic about the possibility that we'll make some substantial moral progress.

• 1:17:45Mario Gabriele?

  Huh, fascinating. What tradition or practice from either another culture or time period do you think we should widely adopt?

• 1:17:52Speaker 00

  I think I don't know enough about particular time periods. This is the problem about being an academic and wanting to not just make something up. But I do feel that something to...

• 1:18:04Speaker 00

  Something that involves more gratitude to our ancestors, I think, would be particularly important. I know that there are many cultures that have more gratitude to our ancestors than than Western cultures, which have almost none. But I'm not sure exactly who we should be recopying it from, but I would love to see more of that.

• 1:18:19Mario Gabriele?

  Yeah, I agree. That's a great one. Well, Toby, it's been such a pleasure. And yeah, I can't thank you enough for for your work and your writing and and for your time today.

• 1:18:30Speaker 00

  Well, thank you. It was a wonderful conversation.

• 1:18:34Mario Gabriele?

  That's it. Thank you for listening to this episode of the Generalist Podcast. Please subscribe on Apple Podcasts, Spotify, or your preferred podcast app. Ratings and reviews help others discover these discussions. So if you enjoyed the conversation, I'd be grateful if you could take a moment to leave one. For all past episodes and more, visit us at degeneralist.substack.com. See you next time as we continue to explore the future.