How To Avoid Destroying Humanity - Rob Reid | Modern Wisdom Podcast 346

Rob Reid is an entrepreneur, podcaster and an author. The last 15 months have been a terrifying taster of just what a global crisis is like, except it wasn't lethal enough to be a threat to our long term survival - but just because this one wasn't, doesn't mean that more deadly existential risks aren't out there. Expect to learn how synthetic biology might be the biggest risk to our survival, what we should have learned from 2020, whether Artificial General Intelligence is an immediate threat, Rob's opinion on my solution for saving civilisation, whether we should totally stop all technological development, if synbio is preventable, how we can avoid civilisation’s destruction through nuclear bombs and much more... Sponsors: Get 20% discount on all pillows at https://thehybridpillow.com (use code: MW20) Get perfect teeth 70% cheaper than other invisible aligners from DW Aligners at http://dwaligners.co.uk/modernwisdom Extra Stuff: Check out Rob's Podcast - https://after-on.com Follow Rob on Twitter - https://twitter.com/rob_reid?lang=en Get my free Ultimate Life Hacks List to 10x your daily productivity → https://chriswillx.com/lifehacks/ To support me on Patreon (thank you): https://www.patreon.com/modernwisdom #existentialrisk #syntheticbiology #pandemic - 00:00 Intro 01:19 The Thrill of Existential Risk 07:00 Why is Climate Change our Focus? 10:19 Humanity’s Close Calls 20:31 Democratising the Apocalypse 30:46 The Threat of Covid-19 and Pandemics 54:00 Is the Research Worth the Risk? 1:02:00 Would Moon Labs Reduce Risk? 1:08:18 Helpful Lessons from Covid-19 1:16:45 What if China Leaked Covid-19? 1:22:15 How to Prevent Destroying Humanity 1:37:47 Creating Silo Communities 1:46:54 Lesser-known Existential Risks 1:51:17 Making Existential Risk Sexier 2:01:50 How Can Individuals Help? 2:08:07 What’s Next for Rob? - Listen to all episodes online. Search "Modern Wisdom" on any Podcast App or click here: Apple Podcasts: https://apple.co/2MNqIgw Spotify: https://spoti.fi/2LSimPn Stitcher: https://www.stitcher.com/podcast/modern-wisdom - Get in touch in the comments below or head to... Instagram: https://www.instagram.com/chriswillx Twitter: https://www.twitter.com/chriswillx Email: https://chriswillx.com/contact/

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We ignored the warning shots of SARS and MERS and Zika and a whole bunch of other things. Covid is a very, very difficult warning shot to miss. The whole world has been traumatized by this. There will be much greater seriousness applied to pandemic resistance in the future. The question is, will it be adequate attention and will it be sustained intention and will it be intelligent attention?

• 0:28Chris Williamson?

  So I read something on Reddit the other day that I want to dictate to you here.

• 0:33Speaker 00

  Uh huh.

• 0:34Chris Williamson?

  The decision to use CFCs, chlorofluorocarbons, instead of BFCs, bromofluorocarbons, was pretty much arbitrary. Had we decided to use BFCs, the ozone layer probably would have been totally destroyed before we even knew what was happening, killing all life. BFCs destroy the ozone at over 100 times the rate of CFCs.

• 0:57Speaker 00

  That's amazing. I've never heard that before.

• 1:00Chris Williamson?

  How sick's that?

• 1:01Speaker 00

  I mean, and CFCs were scary, but obviously they moved slowly enough that we were more or less able to fix the problem before we were all dead.

• 1:11Chris Williamson?

  Someone replied and said, maybe that was the great filter that all the other civilizations just chose the wrong coolant medium?

• 1:18Speaker 00

  That's funny.

• 1:20Chris Williamson?

  So today, we're going to be talking about existential risk, my favorite terrifying topic and also your, one of your areas of expertise.

• 1:28Speaker 00

  Uh huh. Definitely. And it's amazing how seductive the topic is to a lot of us. It's, it's like, we can't take our eyes away from it. We get fascinated, like what you just read to me. This is a, probably says something bad about me psychologically, but my main reaction was like, how cool. I mean, obviously we dodged the bullet, so that's pretty nice, but like, wow, another existential risk that I didn't even know about.

• 1:58Chris Williamson?

  What do you think it is about that? Because I have the same fascination.

• 2:02Speaker 00

  I, you know, maybe it's something that was, you know, drilled into us when our, you know, distant ancestors were growing up on the savanna. Maybe there's something about being fascinated by things that can annihilate one, oneself, that conferred some kind of survival advantage. And I'm just riffing here and I'm just, I'm just gonna make this up, but, you know, particularly the head of the clan, the hunter-gatherer clan, whoever the boss was, chieftain, whatever you want to call that person, really needed to think about what could kill us all. And the head of the clan probably was a man and probably fathered far more children than

• 2:46Speaker 00

  people who were not head of the clan. And so, we all have a lot of head of clan DNA in us. I'm making this up as I'm going along, but I like that, I like that theory. So we probably do, as a statement of fact, all have a lot of head of clan DNA, because there were thousands of generations and the heads of the clans were the people who probably had the most progeny. And the heads of the clans really did have to think about, not just what could kill me, a saber-toothed tiger on a hunt, or whatever, but what could wipe us all out? Really need to think about that. And the successful ones continue to have progeny. So that's my answer.

• 3:24Chris Williamson?

  It's like a, we've got the anxiety bias, right? That we're more scared of things than we are hopeful about things. But this is like a macro level version of that.

• 3:32Speaker 00

  Macro level anxiety bias, exactly.

• 3:34Chris Williamson?

  We've got it, we've worked it out. Okay, so.

• 3:36Speaker 00

  All right, good.

• 3:37Chris Williamson?

  Given the fact that me and you were obsessed with it, and a ton of people that are listening will be as well, why do you think we're so blind to how close we can come to total civilizational destruction? Generally, it's not at the forefront of what we're talking about every day as much as me and you might wish that it was?

• 3:54Speaker 00

  Well, I think it's because it's so new. There is really no plausible step that I can think of that humanity could have taken before, let's say, the mid 1950s to wipe everybody out. And at that point, it was one thing. So after Hiroshima and Nagasaki, there's one nuclear power, the United States. It had precisely two bombs. It used them both. So there's no way to destroy the Earth, right? Then along comes the hydrogen bomb. And there are very few of them, and only the US has them. And then the Soviet Union gets them. And then all of a sudden, there's this insane push to put them on long-range bombers, missiles, and so forth. It's probably late 50s by the time H-bombs were proliferate enough.

• 4:37Speaker 00

  And, you know, two sides had enough capability that truly wiping out society became a problem. So as a, you know, quarter-million-year-old species, we've been facing this for 60 years. So it's probably, it's, even though what I said about the plan notwithstanding, to put it on a global level, that's a pretty new development. And I would also say that the attention that's given to, the careful attention, academic attention, serious thought in industry attention, governmental attention is far less than what it should be. But nonetheless, the amount of attention that is given to existential risk today,

• 5:18Speaker 00

  to me, feels like it's 10 to 30, or even more, times what we, what we gave to it, let's say, 15 years ago. I mean, 15 years ago, I don't think people like you and I even knew the term existential risk. So I think we're, we're developing that muscle pretty rapidly at this point. And that's a good thing, and hopefully, it's not too late.

• 5:40Chris Williamson?

  That's your hopeful optimism, your unbeatable optimism coming through there?

• 5:45Speaker 00

  Yeah, I'm, I'm pathologically optimistic sometimes.

• 5:49Chris Williamson?

  How much of it do you think could be a hubris as well? You know, by definition, we haven't destroyed ourselves yet. Therefore, we're probably fine at surviving any future destruction potentials also.

• 6:01Speaker 00

  Yeah, I mean, the response to that is like, at a boy, at a girl, it's been 60 years. So you've dodged numerous bullets in 60 years. One or two, maybe sort of by design and quite a few more by accident. And so do you want humanity to last another 60 years? Or do you want it to last another quarter billion years? And if the answer is you've been dodged, you dodged one bullet for 40 years. And you've dodged more than one bullet for maybe 20 years. Is that the kind of track record that gives you confidence? That the civilization or the species is going to survive another quarter billion years? That's absurd.

• 6:44Speaker 00

  That's like saying, I wish we could do the proportions and back of the envelope math could probably reveal it. But that's kind of like saying, one to two seconds into what Americans call a soccer game. And the rest of the world calls a football game. We haven't given up any goals, so we're fine.

• 6:59Chris Williamson?

  Yeah, yeah, yeah, yeah.

• 7:00Speaker 00

  Yeah.

• 7:01Chris Williamson?

  Given the fact that existential risks generally don't have a ton of global sort of attention paid to them. Why do you think climate change is given so much attention? When there's more imminent threats that aren't even really in the conversation?

• 7:16Speaker 00

  Well, I think the attention to climate change, first of all, developed in a compounding way over a longer period of time. And so the whole earth catalog with the picture of the earth on it, the first earth day, which I think was 1970, et cetera.

• 7:31Speaker 00

  You know, that's a great deal more time. And I think these things, you know, like successful investments. When a school of thought really, really plants its roots and takes off. It's like compounding returns. You know, it's like, so the number of people who were environmentally aware in 1971 was probably pretty small. But it was, it was a meme that the world was ready to hear. And it had a lot of committed people all from the very beginning. And that meme spread and it resonated and more work was done and that spread. It was like an investment that compounds at 20% per annum. Like wow, 20% per annum.

• 8:13Speaker 00

  10 years in, you know, you think you're in Fat City, but holy cow, 50 years in, it's ginormous. And so I think it's, a lot of it is the fact that that compounding awareness has had, you know, more years to grow exponentially. And then the other thing is you get to a certain point in any of these fields and you start developing very significant industries and economic interests around them. And so now there is a very, very large number of people who are making their living off of protecting us from climate change. Whether they're making electric cars, whether they're academics who specialize in climate models,

• 8:55Speaker 00

  whether they're politicians who fired up, you know, their base and got elected in part on that message. There is a very, very large interest group that is believes, I'm not saying it's cynical, but believes in this, but is full-time committed to making this stuff work. And the number of people who are currently full-time committed to preventing existential risk is probably a minuscule handful of academics. But that is a hell of a lot more brain power, persuasion power, intellectual output, et cetera, than we had 15 years ago. So it's just starting.

• 9:29Chris Williamson?

  You think that first mover advantage for climate change in that case? In 20 years' time, then are we going to have the Church of Nick Bostrom? And everyone's going to be praying to the control problem and looking at nanotechnology and Grigu?

• 9:42Speaker 00

  One hopes. I don't know if Nick or I want there to be a church dedicated to him. We could talk about that later in the conversation if you wish. But yeah, I think, I think spreading and compounding awareness of this is the only thing that will, will protect us. Because the policymakers will be the last ones to the party. But if this becomes something that, you know, a million people are really interested in, aware of and informed about, 10 million, 100 million, et cetera. That's, you know, in compounding out and eventually seeping into government, that's ultimately the only way we can dodge these bullets.

• 10:20Chris Williamson?

  You mentioned that we'd had a couple of close calls over the last 50 years. Can you take us through a few of those stories?

• 10:25Speaker 00

  Yeah, I mean, I think, you know, starting with, well, first of all, the BFC thing, that's a new one. And if that redditor was correct in what they said, and I'd love it if you could send me that, because I want to dig into that, it is intriguing and frightening. There's one right there. I mean, the ones that are most chilling to me, because I grew up during the Cold War, are the nuclear ones. And, you know, there are a couple of particularly famous incidents. One during the Cuban Missile Crisis, in which there were nuclear armed subs, Russian subs, patrolling the area outside of Cuba as the American blockade settled in.

• 11:06Speaker 00

  And one of the American boats was sending, it's funny, they knew the subs were down there, and they didn't want to escalate. So instead of sending depth charges, they were sending something like practice depth charges, or something weird like that, but they were dropping these depth charges and really menacing these submarines. And there were a lot of American boats, like Path the Fleet was in a pretty compact area at this point. And I think there were four Russian subs in the submarine fleet that were down there. And these depth charges start coming down. And on one of the subs, there was a decision to nuke the American fleet, because they had these nuclear torpedoes, and if they sent them up, it basically would have wiped

• 11:51Speaker 00

  out the American fleet. And in most submarines, three of the four, I believe, it required two people to say, yes, let's do it, in order for it to be done. Now, you're underwater, you got depth charges, you're not on the phone to cruise ship. Like, these are people who are fully empowered to start a nuclear war. And the stopgap measure to prevent that was, not one person, but two people have to say, we're gonna do it. On this one particular sub, where for whatever reason, and I wish I knew the details better, the decision was being taken, there was a third man. And I'm saying man, because I'm sure it was all men, in the Russian submarine crew in the 1960s. There was a third man who got a vote.

• 12:34Speaker 00

  Because he was like the party cadre, or something, like, he was like the head of the, you know, the Communist Party, as opposed to merely the military, you know, delegation of the submarine fleet. And he said no. And had that third person not been on the submarine? Or had that third person said yes, the next step would have been for the Russian submarines to fire a tactical nuke and basically eradicate that part of the American fleet. Now a nuclear weapon has been used during the Cuban Missile Crisis. And that, you know, it's very, it's almost difficult to imagine how that would not have escalated to doomsday. So, holy cow. And then the other famous incident was the guy, I've seen a movie about him, I've heard

• 13:18Speaker 00

  interviews with him, so I should remember his name. Do you remember his name? The guy on the Russian side who saw, yeah. So, another story in the States too, I want to say the eighties, probably early eighties, there was a, basically a Russian equivalent of NORAD. So basically, the operation where they detected incoming American bombers and missiles and so forth, and they saw American missiles coming over. And they, it's like, oh my god, it's Armageddon has started. There's only one missile, or two, or something weird like that. And so it basically went up to the person who was in charge of that facility, whose

• 14:02Speaker 00

  name is escaping me, and his instinct told him, this is not the start of doomsday. They wouldn't just send one missile. And then more missiles started showing up, but it was not an all-out attack. And this guy was in contact with his superiors in Moscow, and they said, launch everything. And he said, no. And his instinct was screaming at him that this is some kind of, something's gone wrong with our systems. They're not starting, they're not going to send two or three or a handful of missiles over here, trigger an all-out response. If they were, if they were taking a first strike, they'd be sending everything they had. So he stood his ground and did not launch. And had there been somebody different, almost probably anybody in that role, because the

• 14:45Speaker 00

  job, your job was to say yes to Moscow. Your job wasn't to think. You know, so that was an unbelievably close call. There was another less close call that I think a little bit more is made of than should be. Where at NORAD on the American side, there was a first strike, a pretty full, full-blooded first strike was detected and it turned out to be a test sequence. And there was actually journalists in NORAD when that happened. So there are, there are a lot of these things that we got through by the skin of our teeth. And these things could still happen. I mean, there's still an inordinate amount of ordinance, sorry for the pun, on the Russian and American side and increasingly on the Chinese side.

• 15:27Speaker 00

  And one hopes that our safeguards and our software and our, our protocols have improved since the eighties, but I don't know that they have. And, you know, that risk still sits out there. Now, I'm focusing on nuclear because those are, I think those really are the bullets that we've dodged so far. Um, we're just getting into the rain, the area in which synbio could take us out. And particularly with the rogue actor is a scenario that, as you know, I worry about particularly, um, although bio, you know, bioterror, bioerror, both of them are very, very, very dangerous. But we're just getting to the point where synthetic biology could take us out. And super AI is not there yet. Um, and I, I feel like that's at least a handful of decades out.

• 16:09Speaker 00

  Um, and nano is definitely not there yet. But we are going to have an increasing number of these, of these risks facing us. And the real danger is the proliferation of the ability to hit, we'll just call it euphemistically, the flashing red button, a hypothetical, probably non-existent flashing red button that we imagine, you know, Mr. Biden and Mr. Putin and Mr. Xi have at all times available to them to destroy the world. Um, when you look at the Cold War. Let's think of this. We spent trillions of dollars preventing two people, I'm obviously oversimplifying, but preventing two people from hitting that flashing red button.

• 16:53Speaker 00

  What did we spend it on? What we spent on all those detection systems, but we also spent it on enormous conventional armies to deter, you know, you know, small acts that could snowball into large, large conflagrations. Um, we spent money on regional wars to prod each other and test each other and to show resolve and, you know, to hold each other at bay, the diplomatic apparatus, all these things were in place to, to stop two people from hitting that button. And those were two people who were highly inclined not to hit that button. And obviously it was more than two because we just talked about scenarios in which people down in the chain of command had that power, but we spent a lot of money making sure that a very small handful of people didn't hit that button.

• 17:38Speaker 00

  And so far we've succeeded, it became terrifyingly close, but so far we've succeeded. The danger with things like synbio and super AI is that that decision not to do something unbelievably dangerous or even something deliberately destructive is suddenly going to be in the hands of thousands of people perhaps. In the case of synbio, I believe it will be thousands of people. Um, and probably pretty soon, in the case of super AI, it's probably going to be a smaller group of people who don't take the right, you know, precautions about not letting the genie out of the bottle. But with synbio, I'll focus on that for a moment. The tools and the methodologies are improving so rapidly

• 18:21Speaker 00

  that the things that only the most brilliant academic synthetic biologists at the pinnacle of, you know, laboratory budget, equipment, know-how, et cetera, things that would elude that person, be impossible for that person, will be today. We'll be child's play in a high school bio lab in quite a bit less than 20 years because this is an exponential technology. And that's the frightening thing. And all the wisdom and complexity and the Nobel-worthy work that is done by prior generations will start getting embodied in simpler and simpler and more and more common and cheaper and cheaper tools.

• 19:04Speaker 00

  And so, all of a sudden, all that wisdom and genius that eludes most of us is embodied, like, you know, we're talking on laptops. How much Nobel-worthy work was done to create, you know, the computers that we're using to speak to one another over many, many generations? You and I are smugly sitting here with computing power that the most brilliant, you know, computer engineer, electrical engineer could only dream of, you know, 25 years ago. And it's all embodied in this simple tool. And that goes into CenBio tools in wet labs at lower and lower levels of academia and in, you know, a higher and higher number of lower and lower budget companies.

• 19:46Speaker 00

  We are relying on an impossible number of people not to screw up or not to do something deliberately evil. And if one says, "Well, why would somebody ever do something deliberately evil with CenBio after we've just been through?" The answer is, "I don't know. I don't know what motivated the Columbine kids. I don't know what motivates the, you know, more than one master degree per year, per day that strikes at the United States. I don't know what motivates those people, but they're motivated and they're killing everybody they can. They just don't have tools to wipe us all out." So, anyway, that was probably a very long-winded answer. But we are going to have to worry a great deal about the ability to do something catastrophic

• 20:27Speaker 00

  being in a lot of hands rather than just two that we can watch very closely.

• 20:32Chris Williamson?

  Is that what you call democratizing the apocalypse?

• 20:35Speaker 00

  Yes, that is exactly what I call democratizing the apocalypse, or privatizing. I call it both, privatizing the apocalypse and democratizing it. And privatizing, you know, just drives home the fact that saving the world from destruction or destroying the world is no longer a public good.

• 20:57Speaker 00

  You know, it's in private hands. And so it's a slightly playful and slightly perverse way of putting it. But that game of chicken that the superpowers played with each other during the Cold War was quote unquote a public good. For all of the terror that anybody who grew up under a nuclear threat, and by the way, that's everybody alive right now because that threat is still very present. We just don't feel it the way we did during the Cold War. The terror that anybody felt growing up with a nuclear threat, and it was billions of people who were some degree, I'm sure traumatized by that, is nothing compared to the horror that would have been inflicted by more and more conventional wars. So if

• 21:41Speaker 00

  conventional wars between the superpowers, you know, more Vietnams, more Korea wars, et cetera, eventually, you know, followed by an enormous World War III smackdown in Europe. So imagine nuclear weapons were impossible. It's probably likely, it's almost, I mean, it's highly likely in my mind, looking at the rhythm of geopolitics stretching from, let's say, 1840 to 1940, if nukes had never been developed, it's hard to imagine we wouldn't have continued to butcher ourselves in our tens of millions.

• 22:14Chris Williamson?

  That's a really good point. Imagine just how much massacre there would have been if we didn't have this capacity to do wide scale preventative destruction.

• 22:24Speaker 00

  Yeah. So I am sure that there would have been an all out war between the Soviet Bloc and the Western Bloc in Europe, probably in the 50s, probably in the 60s, without that threat. And there probably would have been far more Vietnams and Koreas and stuff we can't even imagine. And so that game of chicken was quote unquote a public good. And it was owned and operated by governments. And that most, you know,

• 22:56Speaker 00

  Terrifying of decisions was concentrated in a tiny number of hands. And humanity did shoot those rapids. And we probably would have had a far, far, far, far, far more gory and traumatizing second half to the 20th century. And to this day, I bet we'd still be clobbering each other with conventional weapons. And now they're getting to the point that they're terrifying with automated, you know, like, so, you know, public good. All of a sudden, and when that suddenly is in private hands, things change in frightening ways. So let's pivot over to super AI risk. The, the parties that will be in the position at some point in the future, if super AI

• 23:38Speaker 00

  is indeed a possibility, which I personally absolutely think it is, the parties who will be in the position to create the genie in the bottle, step one, and B, screw up and let the genie out of the bottle. Or let's just go with the creation step, not, maybe they don't know that they're inches away from creating the genie. Those people are almost certain to be in some form of private company in my mind. At least in the United States. And China, we know less of what they're doing there. And there probably are government labs that are recruiting extraordinary talent and proceeding headlong down paths that we can't necessarily proceed on.

• 24:19Speaker 00

  I mean, but today, the greatest talent in computing is not working for, you know, the United States government or any government. It's working for, you know, DeepMind, it's working for Google, it's working for startups that we're not aware of. And that means that the person or people who are in a position to say, ooh, that's really risky, huh? But it's kind of tempting. They have huge economic incentives to take what they might perceive to be a tiny risk and, you know, probably get away with a tiny risk. And, you know, as a result of that, the gazillionaires.

• 25:00Speaker 00

  You know, that economic incentive did not exist for anybody who felt like they were chancing it with nuclear Armageddon. We don't have to worry about Putin saying, "Ooh, it's kind of risky if I take this insane step and like invade the rest of the Ukraine from the eastern Ukraine, could lead to nuclear war, but I get an IPO and I'm rich if it doesn't..." That incentive isn't there if it's a public good. And so what I worry about is lots and lots of private actors taking what might be like, "Oh, it's a sliver of 1% risk that the world ends, but that's not going to happen, or it probably won't happen, and if it doesn't happen, and let's face it, it probably won't,

• 25:45Speaker 00

  holy cow, here comes glory." And suddenly, there's much, much, much more incentive for lots and lots of people to take tiny, tiny risks that could kill us all. And so that's why the privatization really, really worries me.

• 25:57Chris Williamson?

  It's because you've got privatized gains but socialized losses.

• 26:01Speaker 00

  Socialized losses, exactly. Privatized, that's what, that was our economic crisis, right? The financial crisis, for years, people in various positions in Wall Street, on the buy side, on the sell side, on the fund side, all kinds of things, were taking odious risks with the world economy and getting great returns because, you know, higher risk leads to higher returns in finance. And so they were inhaling money for themselves and putting them into super yachts at Picasso's. And then when everything fell apart, the bill came due to us, all of us. The financial crisis, that bailout was, the cost of that was borne by taxpayers throughout the world. And so we see what happens when you have privatized gains and socialized losses.

• 26:44Speaker 00

  And people will take tiny risks on their own account all the time. I mean, we want to be, you know, hair splitting about it, we all take a tiny risk on our own account when we hop in a car. You know, it's like, I really am hungry and I want to go and buy some popcorn. And they've got that great microwave popcorn down at the Safeway. I'm famished. I love popcorn. I'm gonna go get it. You're not thinking, I am putting my life on the line for frickin popcorn, but you are. It's a tiny risk and you take it. Um, if you dial that up, there are people who get involved at extreme sports, uh, who take very, very significant risks in order to prove to themselves that they're great, in order to get public accolades.

• 27:26Speaker 00

  You know, in some cases, some extreme sports probably have nice tidy purses that can be made. You know, nothing like, you know, in professional basketball or football, but, you know, people on their own account will take tiny risks. And particularly, I think, when you have somebody who doesn't have deep family ties, doesn't have children. Um, you know, who is, you know, earlier in life or is more solitary in life or whatever it is. They, when they're facing that risk of like, I could annihilate the world by mistake or make gazillions of dollars and the risks of annihilating the world are miniscule. Their psychology, because again, we weren't trained on the savanna to think if I screw up, all humans die. Their psychology is probably thinking very much in terms of like, I'm taking a tiny risk here.

• 28:11Speaker 00

  They're probably thinking about their own risk of annihilation. That probably is at least half of the calculus in their mind because we're all individuals and we don't want to die. They say, God, that's miniscule. They might take the risk of a daredevil. You know, the daredevil might take it, extreme sports person take like, okay, I'm kind of putting it on the line here, but I think I can shoot these rapids. And when lots and lots of people are in that position, you start arithmetically adding up all those risks. And at some point, it becomes untenable. So that's why the privatization is really, really dangerous.

• 28:41Chris Williamson?

  Because it democratizes the technology to the stage where you include so many potential agents that one of them or multiple of them are going to be outside of whatever overton window of safety that we have. And they're going to lie there and decide, and this is just talking about

• 28:57Speaker 00

  Mistakes.

• 28:58Chris Williamson?

  Mistakes. Yeah, this isn't malignance.

• 29:00Speaker 00

  Yeah.

• 29:00Chris Williamson?

  This is negligence, not malignance.

• 29:02Speaker 00

  Negligence is not malignance and I do worry more, worry more about malignance, particularly in Cenbio. Because if we think about COVID, let's think about COVID. Um, we've all been thinking about COVID for a while. We're very practiced at thinking about COVID. If we think about COVID. It is remarkable on a number of levels how benign this horrific thing is. It is not very lethal compared to a lot of things out there. It's not lethal at all compared to SARS. You know, SARS is, depending on what numbers we run, 10 to 20 times more deadly. It's also coronavirus. MERS, Middle East respiratory system, kills at a rate of about

• 29:43Speaker 00

  30% case fatality rate. H5N1 flu, which as you know, I'm quite grimly fascinated by, kills about 60%, 60% of people. And with COVID, case fatality rate, according to the World Health Organization, somewhere between half a percent and 1%. So COVID could have been far, far worse merely on a lethality basis and also on a transmissibility, contagiousness basis. If somebody were malignant, if somebody were malicious and really deliberately developing something to be maximally destructive, it would be worse than COVID. In an imaginable

• 30:24Speaker 00

  near future where somebody is sophisticated enough or the tools that they're using are sophisticated enough to allow them to basically dial that up, they're not going to unleash something that kills a half a percent. They're going to release something that is so much more deadly and so much more dangerous that it could have civilization toppling potential.

• 30:46Chris Williamson?

  Here's something that I've just thought of. Have you considered the potential that the lab leak hypothesis or some variant of it for COVID-19 could be true?

• 30:55Speaker 00

  Yeah, and it's entirely plausible and it's undeniable that it's plausible at this point.

• 31:00Chris Williamson?

  But that the reason that it was released was some big-picture-thinking, fair-weather-saint human who said, "Bill Gates has told you at the end of his TED Talk, and we've been warning you for years about the dangers of engineered pandemics and natural pandemics as well. So what I'm going to do is I'm going to give you a very moderately transmissible but very not lethal pandemic, which is going to act like a global vaccine. It's going to cause you to have a very benign dose, a coordination problem dose of how to deal with this sort of a pandemic, and maybe this will make people wake up." Have you thought about that?

• 31:40Speaker 00

  Well, have you read somehow, have you hacked into my computer and read the outline of a novel that I'm working on?

• 31:47Chris Williamson?

  Oh, damn.

• 31:48Speaker 00

  Because I've actually, that's a story that I've fleshed out a great deal. So.

• 31:53Chris Williamson?

  Familiar with it.

• 31:54Speaker 00

  Yeah, I'm familiar with it with that scenario and it's a very, very interesting one. And I don't think that that was COVID, and here's why. I think if somebody wanted to unleash an engineered pandemic to freak everybody out and realize how dangerous it was, they would make sure that the world knew it was engineered. Because, you know, right now, the prevailing wisdom in science and policymaking circles is that this wasn't engineered. And therefore, the response has been more about zoonotic, like how do we prevent more zoonotic transmission? So if somebody wanted to create a mild engineered pandemic and freak out the world, they would

• 32:36Speaker 00

  absolutely make sure that the world knew this was engineered. So I don't think that's what happened. That doesn't rule out the lab leak hypothesis.

• 32:45Chris Williamson?

  They would have put like, sort your shit out world into the RNA or something like that.

• 32:51Speaker 00

  Or they would have released some message online or whatever it was.

• 32:54Chris Williamson?

  Yeah.

• 32:55Speaker 00

  And, you know, done, tried to do a pinprick assault and of course, pinprick attack of some kind, and of course, the danger with that is that thing mutates and gets out of control and annihilates us anyway. Not that that's necessarily going to be a plot twist in the book that I may or may not write.

• 33:09Chris Williamson?

  Well, we've ruined it now. All right.

• 33:12Speaker 00

  Yeah, yeah.

• 33:13Chris Williamson?

  So if COVID was one of the more benign, what was the most dangerous or lethal virus in history that you've come across?

• 33:20Speaker 00

  Oh, I mean, you know, probably one of the influenza pandemics and probably 1918. I'm not saying COVID is necessarily, but I mean, like 1918 flu killed so many more people, so many more people in a much smaller world population that proportionately it's so much worse than COVID. And that, we don't really know if that's because we have better tools and better detection and better public health. You know, practices today than they had in 1918, that's possible. Or if 1918 was simply much more virulent. My guess is it's a little bit of both. Because it's not like we really implemented amazing best practices in public health in most of the world.

• 34:04Speaker 00

  You know, Australia and New Zealand did far better than most of us. But we kind of botched a lot of things. So I'd say 1918 is probably worse, but I'm thinking more in terms of, you know, again, SARS. If you had SARS level lethality and COVID level transmissibility. And there's no reason that that nature, when it spins the roulette wheel, won't come up with that.

• 34:26Chris Williamson?

  With a nice big incubation period as well, probably, where you're still able to transmit. Yeah, so maybe like seven days, 14 days without showing symptoms.

• 34:34Speaker 00

  Longer asymptomatic period could be catastrophic. You know, measles, the thing about transmissibility, measles is unbelievable. And you've probably heard me say this before in another conversation, but basically, the comparison, as I know, first of all, I think we all know at this point that if we got into an elevator that somebody with COVID occupied 10 minutes before. And that elevator's went up and down, the doors have opened and shut, people come in and out. We go into the elevator, like even five minutes after somebody with COVID was in there. And we're not wearing a mask. The odds of us catching COVID in that elevator from that person who was there five minutes ago are essentially zero.

• 35:17Speaker 00

  That's the understanding of the science today, and there's no reason to doubt that. I mean, we learn more and more about COVID every day, but that's, that's pretty solid. If you were unvaccinated for measles, or you didn't have immunity, and somebody had been in an elevator hours before, you could catch it. That's my understanding. And I've read that in a couple of places. And I'll just assume that Snopes wouldn't allow that to be on the internet if it weren't true. But measles is radically transmissible. So the, you know, the three dials to me are basically, how long is the person asymptomatic? How transmissible is it? How deadly is it? And so there's a lot of dials that nature has actually hit without engineering. The high transmissibility measles, the towering lethality of H5N1 flu,

• 36:00Speaker 00

  the unbelievable, you know, I don't know, I don't know what disease has a super long incubation period. Oh, tuberculosis, tuberculosis, people, it usually is. Now, this isn't necessarily incubation, but the time from onset to detection is usually about two years. And this is a scourge that we have been fighting forever. Right? It's about two years between when somebody starts getting pulmonary respiratory tuberculosis and when they get definitively diagnosed and start going on drugs. So, yeah, long incubation periods are out there, too.

• 36:37Chris Williamson?

  That's dangerous, man. And then when you think that someone can play with that, I mean, there's even an iPhone game, right? Where you kind of design your own pandemic. It's been around for ages and you see how it. So maybe it's not just me and you that are, that are obsessed with this. Talking about some of the leaks from labs that we've had, this can, this is going back to negligence now. We've had some fairly close calls with that. We had anthrax just after nine 11 ended up in the Senate Majority Leader's office or something like that in an envelope?

• 37:07Speaker 00

  Yeah, an office that I had the great fortune to pass through on the very day that the envelope arrived. Just by sheer coincidence. Yeah, I was in DC, I had business at Tom Daschle's office. So, yeah, I was at Tom Daschle, Senate Majority Leader. This is a single digit number of days after nine 11. And so let's talk about lab leaks and let's talk about what happens when you've got a malicious insider who is leaking things and how hard that is to prevent. So for those who don't remember, this is now getting to be a long time ago. Immediately after nine 11, some envelopes containing anthrax were delivered to a small number of places, including the office of Tom Daschle, who was at the time Senate Majority Leader.

• 37:51Speaker 00

  Patrick Leahy, I think, another senior senator, also Democrat.

• 37:56Speaker 00

  Certain media outlets, including The National Enquirer, it was just a ghastly publication. So it was weird. Immediately, the pattern was strange. Why would Al Qaeda, and it was, it was the attempt, and we'll talk about the attempt in a second, was made to make this look like Al Qaeda sending it. Why would Al Qaeda have a beef with the National Enquirer? You know, people were, a lot of people were annoyed with the National Enquirer, but they were generally annoyed with it because it was kind of gutter celebrity journalism with lots of paparazzi and seemed to violate the privacy of lots of people and seemed to dumb down the readers. And there was a lot of sort of superior sniffing on the part of people who read the New York Times about those who read the National Enquirer, blah, blah, blah. I doubt that that was much of a preoccupation of Osama bin Laden.

• 38:41Speaker 00

  But there was weird scrawling on the letters that said, you know, something like, the thing that immediately told me that this is, this is, somebody's faking it, was like, it wrote, Allah is great, you know, this is, we're doing this for Allah. And Arabic speakers even who are learning English, even ones with very, very, very remedial English, very early in their path of learning English, will quickly learn that the word for Allah, I studied Arabic for years and years and lived in the Middle East for a long time. Allah, you say God, so. God is great. Arabic speakers who are learning English and know enough English

• 39:25Speaker 00

  to write ransom notes or, you know, threatening things. Find anthrax. Will have learned probably in their first day of instruction that the word for Allah in English is God. They don't write, hey, I believe, it's, it's just not done. It was just, it was like a, you know, just a remedial mistake of, of a non-Arabic speaker trying to pretend that they were an Arabic speaker writing English. It was so flamingly obvious. So anyway, these, these envelopes go out and it eventually becomes clear that the anthrax spores, which were very deadly, they were milled in a way that they, that the spores got into suspension in the air. So respiratory transmission was far more possible.

• 40:08Speaker 00

  Natural anthrax is not necess, it's hard for a dust of anthrax to get, go into suspension in the air in a way that could kill a lot of people. That's very careful military-grade processing or, you know, you know, very sophisticated academics that have to do that. So it turned out these anthrax spores came from a US Army lab. Probably at Fort Dietrich, Maryland, almost certainly, but it might have been another Army lab. But so let's think about that for a moment. You have the United States immediately after nine eleven, probably one of the most security minded and security capable nation states in the history of geopolitics. You have the United States military that has

• 40:53Speaker 00

  a lot of jobs, but among its jobs is to make sure the anthrax spores in its own laboratories don't kill anybody, right? And so, you know, we can say all we want about how inefficient the United States military is at certain times and, you know, make jokes about the bureaucracy and so forth, but it's a pretty security capable organization compared to most places. And you have this nation and military on absolute high alert, yet it can't keep the anthrax spores from its own pocket from getting into the office of the Senate Majority Leader, who is one of the, you know, five-ish most powerful people and most important people in the United States government. That was one malicious party who had access

• 41:37Speaker 00

  to that lab. And it's, it's generally considered that we know who it was, the person committed suicide before they could go be indicted and go on trial and so forth. So it was never proven in a court of law, but it seems pretty clear who it was. It was a disgruntled senior person who had no problem getting that stuff out of that lab and getting it to the Senate Majority Leader's office using the United States Postal Service. So if that can leak when you have a malicious actor from the United States military after nine eleven, high alert everywhere, to somebody who I believe is in the line of to succeed the president, you know, the Speaker of the House is, I'm not sure about Majority Leader, that tells

• 42:17Speaker 00

  us that any lab can leak, right? You know, if we can't keep that out of a US military lab after nine eleven, what chance do we have of keeping it out of a? You know, mid-grade state university bio lab if something equally lethal is there. So that's a malicious leak. And then, and what's dangerous about a malicious leak is that almost all the biosafety practices that are out there that govern bio labs are designed to present, prevent accidents. So there's four levels, biosafety level one, two, three, four. And as you go up the chain, the precautions get greater and greater to the point that biosafety level four labs, the highest biosafety level that we have.

• 43:02Speaker 00

  You know, a lot of money and training goes into those precautions and, you know, people are where there's negative pressure suits and all kinds of things that look good on television that go into biosafety level four. I've heard biosafety level two is something on the level of a dentist's office. Not sure if that's a precise analogy or, you know, or if that's accurate or not, but you go biosafety level one, two, three, four. And what we need to realize is that there have been leaks out of biosafety level four labs. So the, and accidental leaks. And so the probably, the, the one that most attention is drawn to is foot and mouth, sometimes called hoof and mouth disease. There had been a devastating outbreak in the United Kingdom, wiped out a very, very

• 43:47Speaker 00

  high percentage, a very high percentage of the cattle herd throughout the UK had to be culled, billions in damages. So much like the, the United States against any threat after nine 11, one hopes that the life sciences infrastructure in the United Kingdom is on very high alert when it comes to foot and mouth disease in the immediate wake of this outbreak. And not long after this outbreak, foot and mouth spores escaped from a BSL four lab in the UK. Like the, they got out. Now, they didn't start another, they, they were contained, they didn't start another outbreak. But that tells us BSL four level labs can leak by accident. So certainly BSL one, two, three can leak. And with a malicious actor, I don't think there's really any protection against that.

• 44:31Speaker 00

  And so again, like when we think about. Do we want to do experiments, for instance, that could result in a civilization toppling pathogen? You know, let's say with the transmissibility of COVID and the deadliness of H5N1 flu. That would probably topple civilization. We can get into why in a second. But do we want those spores or those viruses to exist anywhere? Even if it's people with the brightest of white hats on, doing it for the best possible reasons, even government people, so it's not being privatized? Do we want those spores or those viruses to exist anywhere at all on planet Earth? The answer has to be no. Because if a malicious person ends up in that lab,

• 45:13Speaker 00

  decides to go Columbine with that bioweapon or de facto bioweapon, or if the lab has a boo-boo, as labs have all the time, we're done. So anyway, a little bit of a riff. But yeah, all labs leak. All labs at every biosafety level can absolutely leak. And particularly if we get malicious actors in there, then any lab can leak.

• 45:37Chris Williamson?

  So do we just glass ceiling how lethal or how deadly every person across the entire globe, every synbio researcher is able to get to?

• 45:47Speaker 00

  Wouldn't it be nice if that were in any way possible? And unfortunately, it's not. So what you can do, you can do some pretty simple and basic things that significantly reduce risk. So I'll start with gain-of-function research. Gain-of-function research is a branch of research. It can mean a diversity of things. And nitpickers, you know, will say, "Well, you know, it's gain-of-function when you create any GMO crop. It's gain-of-function when you hone an antibiotic to be more effective." But generally speaking, when people in the field use the term gain-of-function, they are referring to a narrow branch of research, which is one in which generally viruses,

• 46:30Speaker 00

  but let's say any deadly microorganism, is tweaked in a way to make it much more dangerous and deadly. And I'll use an example that is a very powerful example and in many ways, the most significant one that we have, unless we find out something about coronavirus being gain-of-function research, which we'll get to in a second, I'm sure. But in 2011, two independent teams, one in Holland, one in Wisconsin. Did gain-of-function research on H5N1 flu, which is the one that I mentioned before, that kills roughly 60%, 6-0 of the people it infects. And it's rational to hate and fear H5N1 flu. But if you look at that little critter, there's one thing that we can all agree is kind of adorable about it,

• 47:15Speaker 00

  which is that it's almost zero transmissibility. You need to really be in a very, very specific set of circumstances that are unbelievably rare for any human being in order to catch it. And there was a World Health Organization survey that was done, it was a few years back, that documented each and every case of H5N1 flu that led to death. And over the course of the decade, it was roughly 500 cases worldwide. And it's transmitted rarely, but very dangerously when it is, generally from poultry to people who work in poultry farming. And it doesn't transmit from one person to another. But if it did, oh my god, 60% fatality rate.

• 47:58Speaker 00

  So what these researchers did is to oversimplify things, but to cut to the point, they made it transmissible through the air. Now, they were working with ferrets, not human subjects. But ferrets are a good model for virus transmissibility in humans, which is why they're used in virus research. And so they both, both of these teams created an H5N1 that could be transmissible through the air. And why? Because science, you know? Information wants to be discovered. And there was a great deal, when these people came under criticism, and boy, did they come under criticism, there was a great deal of smug fist-waving of like,

• 48:39Speaker 00

  "How dare you prevent science from doing science?" And the answer is, because if this shit got out... We could all be dead. And the reason to do gain-of-function research, if you talk to the virologists who do it, and there are quite a few of them, not just these two teams, is like, "Well, we're getting ahead of the game here." You know, we're figuring out what worst thing can happen so we can do ingenious things to prevent it." To which the response is, there have probably, there have been tens of thousands of generations of Homo sapiens, and influenza's probably been with us the entire time, and not once has H5N1 become airborne transmissible. So you're not exactly creating something that is inevitably going to be dealt from nature's deck.

• 49:24Speaker 00

  In fact, you're creating something that probably never will be dealt from nature's deck. And you're putting in it, by the way, both of these projects were done in biosafety level three labs, so not even BSL4. So response number one is you're creating stuff that probably would never exist otherwise, and you're putting it into a leaky vessel, which is a biosafety level anything lab, because science! Okay? You don't get to do that. You don't get to take that chance. Okay, you want to take that chance with yourself? You can get drunk and go skiing. And that's probably okay-ish, but if you get drunk and go driving, we're going to lock your ass up. And you can't say, "But driving!

• 50:05Speaker 00

  How can you interfere with my ability to drive the open road, the American dream?" Like, no, that's not cool. So that's a really good reason not to do it. And the other thing is, if H5N1 ever does become transmissible, God forbid, naturally, there are countless metabolic paths that it could take to get there. There are countless peculiar sets of mutations that could happen. It's not like there's only one possible genetic code that would create transmissible H5N1. There are probably, you know, countless different paths that can be taken. So it's not like this critter that these two research labs created was something that

• 50:46Speaker 00

  we could then immediately go and create a vaccine for and say, "We're done with H5N1." No. The peculiar version of H5N1 that might arise naturally, we can't anticipate that. And so this research is unbelievably dangerous.

• 51:05Speaker 00

  To the extent that it's useful, I'm not going to say it's completely useless, but to the extent that it's useful, it's unbelievably limited. And the danger of a leak is profound. So after this work was done, both of these labs had papers teed up, one to go into science and one to go into nature, which are the absolute pinnacle of research science to get a paper in either. Those are the two publications out of thousands that one wants to be in. So, oh my God, they're going to get the superstar treatment. And then the American government in particular flexed its muscle, and I believe others did as well. I think maybe they did something in the UK and Holland as well. But basically, Science and Nature were told, you're not publishing that. And a great deal of, you know, concerned thought ensued.

• 51:50Speaker 00

  And there were very, very stringent warnings issued by, you know, different sort of consultative bodies that are, that think on behalf of the US government, but they're people outside of the US government about bio risk. We really sounded the alarm. But eventually, and funding for gain-of-function research was eventually paused. Now, that word paused, I didn't say stopped. So for a period of several years, in the wake of all this, and in the wake of a couple of bio errors that the United States government made, I won't get into them, but there were like a few blunders that happened that were like, kind of, oh my God, how did that happen? That came to light, and it was after this research, and then after these, these sort of widely publicized blunders.

• 52:35Speaker 00

  The pause lasted a handful of years. That's it. And the pause was in the United States government funding of gain-of-function research. There was never a ban on it. A private actor could do it if they wanted to. Governments, other than the US government, could fund it if they wanted to. There was just a pause. And both of these projects, the one in Holland and the one in Wisconsin, was getting NIH funding, National Institutes of Health funding. They were both getting funded by the US government. So the government said, okay, we're not going to fund this for a few years. But eventually, those papers were released in Science and Nature. And eventually, that pause was lifted. So the US government is now funding gain-of-function research. And those two projects got their funding switched back on. Those two very projects got their funding switched back on,

• 53:20Speaker 00

  I think something like 15 months ago, maybe a little bit longer. So the world that said, oh my God, this is scary, let's hit the pause button, has since said, it's cool, they hit the play button. And so gain-of-function research is happening. And it's not irrational at all to think that gain-of-function research is happening with coronaviruses in places where coronaviruses are researched. No entity, no national government, no, you know, major body of scientists is saying, "Thou shalt not do gain-of-function research." It's considered to be fine right now. It's considered to be completely fine.

• 53:58Chris Williamson?

  Given the potential dangers, the expected value of this benefit versus cost, I don't see

• 54:04Speaker 00

  It's insane.

• 54:04Chris Williamson?

  I don't see how any scientist that's able to do the complex level of synbio that you need to, to probably be able to sequence these genomes and mess around with the capability of microorganisms. If an idiot like me can understand existential risk, why can't geniuses like them understand the risk in what they're doing?

• 54:25Speaker 00

  Again, I think you've got a semi-privatization issue, even though these people are getting public funding. So let's try to eject ourselves into the brain of, you know, the scientists in Wisconsin who decided to go ahead with this research. In his mind, I'm not saying he's a bad guy. I am sure his motivations are pure. I'm sure that he thinks what he's doing matters and is helpful, et cetera, et cetera. But he's living in the bubble of his own life and his own career. And he has the overconfidence that any expert has in their own expertise. I am an expert driver, right? I've driven tens of thousands of miles. I have unbelievable confidence in my driving ability, and it's probably misplaced to some degree.

• 55:08Speaker 00

  He is an expert wet lab dude. You know, he's been running a laboratory that has his name on it, that gets all kinds of funding from different bodies, and gets grants from competitive sources, and does excellent work and has never had a leak of any kind from his lab. So in his mind, the risk of a leak is negligible.

• 55:30Speaker 00

  He probably would not say nonexistent. He's gonna say it's so low, it's silly. And also in his mind, he has got, you know, utility function. He has got, you know, things that he's trying to maximize in his life, as all human beings do. And he wants his career to move forward. And he wants to publish in Science and Nature. And he, you know, he wants to do all these wonderful things. And so he is, his own personal risk-reward curve is out of whack with the rest of humanity. If he does this gain-of-function research and gets published in Science and Nature, and does more gain-of-function research, and gets more celebrity and so forth, his career is going to be much, much more fun. And maybe not much more remunerative, because he's an academic scientist.

• 56:15Speaker 00

  He's probably capped out. But it's the things that motivate him, accolades, papers, that kind of thing, are gonna come in greater, greater, and greater cadence. And so his utility curve says, yes, let's go down this path. And he faces the same risk that you and I face if the world ends. He dies. Now, he's got, he'll have a lot of guilt, maybe. So maybe slightly worse for him than you or I. But basically, he says, tiny risk, I die. Very high chance, my career gets more awesome. And I'm convinced I'm doing the good thing. And I'm convinced I'd never let anything leak. Just like Rob Reid has been driving for decades. He's never had an accident. I've been running a lab for decades. I've never had a leak, so I'm never, ever, ever gonna have a leak, right?

• 56:59Speaker 00

  So he's got misplaced confidence of any expert. He has got strong incentives to do things that incur a tiny little risk. But that tiny little risk doesn't merely apply to him. It applies to all of us. And so the expected value curve that we all run in our own brains, whenever we do anything, is generally our own interests. Yeah, if I do this, you know, 10% chance I lose this much money, 1% chance I win this much money, 89%, you know, whatever. We're generally thinking for ourselves. He's thinking for himself. But he's got all of us in the risk curve. And he's not calculating that expected value of what happens if this goes wrong. So there's a bit of a privatization thing there.

• 57:37Chris Williamson?

  Even to go back to the first atomic bomb test, they did run the numbers.

• 57:41Speaker 00

  They did.

• 57:42Chris Williamson?

  And even with the numbers in front of them, that there was a non-zero risk that the entire atmosphere could be set alight. Permanently curtailing not only all human life, but everything. Literally obliterating the atmosphere and setting it, turning the earth into the sun briefly. And I think the number was 14 million? One in 14 million?

• 58:06Speaker 00

  Um, I believe that um, one of the scientists put it actually a little bit low. I think more like one in three million chance of that. Um, but they didn't know, and that's, that's the important thing.

• 58:17Chris Williamson?

  Non-zero.

• 58:17Speaker 00

  This is was non-zero. Now here's again, we get to the public, private decision. And this is really significant. So at that point, um, everybody, you know, Enrico Fermi, Teller, all of these people at the very beginning of the process. So 1941, as the Manhattan Project is just getting going, the first atmospheric test is still years away. Um, I think it was, it was either Fermi or Teller or Oppenheimer, one of them suddenly realized, oh my God, we could, we don't know if we'd set the atmosphere on fire or not. When we do the first explosion, which turned out to be four years later. And so it was immediately determined that the odds of that were miniscule. And I think that a lot of the scientists really said there's zero, right?

• 58:59Speaker 00

  But they were running numbers up until the day before the very first test, the Trinity test. They were running numbers right up until then. And they did the test and lo and behold, the atmosphere didn't ignite. Now, did they do something irresponsible? Well, let's think about it. Um, they did take what they thought was an incredibly low, but real chance of igniting the atmosphere. But particularly back in 1941, when they first confronted that danger and decided to proceed down the path anyway. At that point, the most, you know, the most facile nuclear brain power was concentrated in Germany at that point. And the only heavy water plant in the world, I think at that point was in Norway and Germany had conquered Norway.

• 59:43Speaker 00

  It's 1941, you're looking at a one in three million risk that you might ignite the earth. And you're looking at a much higher risk that Hitler's going to develop a nuclear weapon before you are. And we can all imagine what the world would have looked like if Hitler had developed a nuclear weapon first. So that's big, real possibility. This is also bigger real possibility, but miniscule. And ultimately, the decision was made at the highest level of, you know, a flawed but functioning democracy. You know, people who had been empowered by, you know, 100 million voters or whatever the number of voters was back in the 1940s, probably less than that.

• 1:00:23Speaker 00

  But nonetheless, a very, very careful public good, you know, it wasn't like Oppenheimer was gonna be like, oh, my God, you know, 1 in 3 million chance, the atmosphere ignites, you know, the remainder chance, IPO.

• 1:00:37Chris Williamson?

  It could get published in Science and Nature, yeah.

• 1:00:40Speaker 00

  Yeah, not gonna be published in Science, not gonna go public and make a billion dollars, you know. It was like, he's facing the same risk as all of us, and he is thinking on behalf of the planet. They were thinking very, very, very carefully about that risk. And they ended up saying Hitler with a nuke versus this miniscule chance. And, you know, looking back on it, people could argue both sides, but that's the point. People could argue both sides. I'm really glad Hitler didn't get a nuclear weapon before, you know, his enemies did. And, you know, was that risk worth taking? You could certainly argue that it was. And I think that ultimately, the Manhattan Project people said, teeny tiny risk, we need to win this war, you know, let's go. And again, I don't think that that's, that could be debated eternally.

• 1:01:24Speaker 00

  But the fact that it could be debated tells us it was not a crazy thing to do. Whereas, gain-of-function research, I might end up on Science, and I'm taking that whole, like, you getting on the cover of Science magazine is not as awesome as beating Hitler. It just isn't. Beating Hitler is really, really good. You getting on the cover of Science doesn't matter to anybody in the frickin' planet but you. And taking a similar 1 in 3 million risk, let's say it's 1 in 3 million risk, is obscene when you're not taking that risk to do something as important for humanity as defeating Hitler. And we are going to be giving a lot of people 1 in 3 million roulette wheels. We already have.

• 1:02:06Chris Williamson?

  To me, bringing a virus...

• 1:02:10Chris Williamson?

  Into existence that doesn't currently, in an effort to inoculate us from the chance that it might come into existence.

• 1:02:18Speaker 00

  It's stark raving mad.

• 1:02:19Chris Williamson?

  Thank you. Why aren't we talking about putting BSL3 plus labs on the moon?

• 1:02:26Speaker 00

  Yeah, no, okay, so, well, this is, this actually started with, when you say, can we do a glass ceiling? And I was starting out by saying, well, there's a really simple thing we could do, but we're not doing it, which is no gain-of-function research, period, at all. The world should agree on that, just like the world agreed on, you know, nuclear non-proliferation and, you know, other, there have been treaties that, you know, hundreds of nations have signed, over a hundred nations have signed. That's how we got rid of chlorofluorocarbons. International agreement that we're not going to use this stuff anymore, which more or less stuck, although there are signs that they're being used in China now, on the sly, but whatever. So we've done this before. It shouldn't be that hard for all the nations of the world to agree, no frickin' gain-of-function

• 1:03:11Speaker 00

  research. And that probably stops it, because there's not much of, there's no motivation for somebody for a private company to do it. It really is academic, it's generally government-funded, it's generally done in relatively transparent areas, it's generally done because people want to get published, it's generally done or because the government has an agenda. It would be an easy thing to say, let's stop all of that. And now you have taken one source of risk out of the whole life sciences equation of bioerror with gain-of-function. So that should be frickin' easy thing for us to agree on. But have we, so let's start there, we're certainly not there yet, but if we do that,

• 1:03:52Speaker 00

  then there's a whole nother set of risks that are out there that like, okay, we're going to get better and better and better at designing bugs in silico, or designing bugs in lab, because we weren't smart enough to not do gain-of-function, or we're going to continue to just publish the genomes of things like the 1918 flu and smallpox and have that information out there, like, we've got recipes for unbelievably lethal things. And instead of it being, you know, something coming to be in a, in a wet lab and escaping, we're not that far from a time when people, and this is oversimplifying what they would do, but we're not that far from a time when a lot of people in academic and private company settings will be able to de facto hit a print.

• 1:04:35Speaker 00

  Button and get the genome of whatever arbitrary critter they want out and then have the tools to boot that up and the mech- and the mechanism of a virus and have it start replicating. And so gain-of-function is one lid that we need to place, but we also need to really, really harden the entire SinBio infrastructure to make it very, very difficult for people to, you know, to print or obtain dangerous DNA. And there are quite respectable early efforts, generally originating with private industry, to limit the ability of any random person to get dangerous DNA, but they are not wide

• 1:05:16Speaker 00

  enough spread, they do not have the force of law, there are self-regulatory steps that the biotech industry, life sciences industry has taken, and they're not really necessarily envisioning the day in which printable, highly distributed DNA and RNA synthesis capabilities become widespread. And so that's another lid that we have to put on that's a much trickier lid than merely not doing something stupid and dangerous. You know, it's kind of like, you know, imagine that you're raising a kid who loves to get drunk and drive and also loves to go to school and breathe, okay?

• 1:05:57Speaker 00

  So your kid, stopping the gain-of-function research is like stopping the kid from drunk driving. Okay, we got that off the table, but he also goes to school and breathes. So there's a, there's another risk that's much more complex that he's going to catch a deadly virus at school. And, you know, okay, we've gotten rid of the drunk driving, nice, that's a good thing, stopped doing the self-destructive stuff, but now there's this much more diffuse, harder to define risk that we need to work on.

• 1:06:28Chris Williamson?

  It's going to be difficult to survive the next century, isn't it?

• 1:06:31Speaker 00

  It is, it really is.

• 1:06:35Chris Williamson?

  Why aren't BSL3 plus labs on the moon?

• 1:06:38Speaker 00

  On the moon? Yeah. Well, because most of the work that they do is not with apocalyptic, I mean, these really truly apocalyptic microorganisms are rare and most of the work that they do isn't with things that deadly. We don't have much stuff going on on the moon right now. So shipping that stuff up there and maintaining it up there and, you know, all that payload and, you know, the tons and tons of matter that would need to be transported from here to the moon is currently beyond our capability. And, you know, when you look at all the things, the good things that are being done with therapeutics and, you know, academic research that has an unambiguously good agenda without disastrous consequences, you can rationally

• 1:07:21Speaker 00

  say that like the work that happens in BSL3 and BSL4 labs is valuable to humanity. It's largely contained and the danger of most of what's in there getting out is highly, highly local. And compared to what we're talking about, extremely minor. So that's probably why we don't. Now, once we get to the moon, we get to Mars and we're ferrying things back and forth quite easily and naturally, perhaps another conversation should happen at that point. But for now, I think the easiest answer is let's not have any apocalyptic microbes anywhere. And when we find, you know, semi-apocalyptic microbes like the 1918 flu,

• 1:08:01Speaker 00

  let's not publish their genomes to the internet. That's getting rid of the drunk driving. And yeah, that's, but yeah, it's risky. It's risky as the stuff proliferates if we don't build really, really great safeguards into the tools before they proliferate.

• 1:08:19Chris Williamson?

  So given the fact that we've had COVID and that this has been a coordination inoculation, if we want to call it that, that it's taught us, we aren't able to shut down travel sufficiently quickly, that we weren't able to produce PPE sufficiently quickly, that culturally, given the technology of now, we didn't have any archetypes for how people should behave, that people didn't understand what social distancing was, people didn't understand why you should wear masks, about what staying at home and isolation was and quarantine and in the way that we get vaccines out and stuff like that. Do you feel like we're in oddly a better position post-COVID? And if so, how much?

• 1:08:58Speaker 00

  Yeah, in some ways, we are hypothetically in a better position. If we take a set of actions in response to COVID to harden society against the next pandemic, if and only if we take those steps. And so, you know, we ignored the warning shots of SARS and MERS and Zika and a whole bunch of other things. You know, we kept ignoring the warning shots. COVID is a very, very difficult warning shot to miss. The whole world has been traumatized by this. Trillions and trillions of dollars in economic damage, millions and millions of lives lost. There will be much greater seriousness applied to pandemic resistance in the future.

• 1:09:43Speaker 00

  The question is, will it be adequate attention? And will it be sustained intention? And will it be intelligent attention? And so, there's a, as you know, I'll briefly plug another appearance that I did. Sam Harris and I did this four-hour piece that was a very unusual podcast format in that about 100 minutes of that was a monologue that I researched and wrote and recorded. And I did the research, I interviewed over 20 scientists, I read thousands and thousands and thousands of pages. And in that episode, I propose a set of steps that collectively are trivially inexpensive

• 1:10:24Speaker 00

  compared to the cost of even the annual cost of the flu, let alone a true pandemic. And I believe if we take those steps and surely other steps that I wasn't smart enough to identify, we will really, really, really harden ourselves up. I'll use one example of something that we should be, there should be a global headlong effort in right now, and I've heard absolutely no sign of that. People who are deep in virology are quite convinced that there's a very high likelihood that with the right amount of research and the right amount of dollars, we could create panfamilial vaccines. What do I mean by that? Well, coronavirus is a virus family, influenza is another.

• 1:11:07Speaker 00

  There are untold thousands of virus families, but only a few dozen present lethal risks to humans, coronavirus and influenza being two of them. So let's say there's 20 of them, it's roughly 20. We don't currently have what we could call a universal flu vaccine. What a universal flu vaccine would be, or will be hopefully, if we develop one, is a vaccine that attacks the core infrastructure of the entire influenza family. And so what we have with the vaccines that get issued every year is there's lots and lots and lots of mutations in influenza. I mean, so many mutations. It mutates, you know, frenetically throughout the year.

• 1:11:49Speaker 00

  And when we develop the vaccine for the northern hemisphere, we're looking at what's brewing in the southern hemisphere. There's a lot of, you know, influenza surveillance that's going on throughout the world. And a panel of extraordinarily talented scientists make their best predictions of what elements of influenza are going to be predominant. Let's say in the United States, it's probably the whole northern hemisphere that gets the same vaccine, but let's just say in the United States to simplify it, so I can be parochial as well, here I am. What strains are likely to be predominant in the United States in the coming flu season? Let's protect against that as best we can in this year's vaccine. And some, you know, maybe 50% of Americans get the flu vaccine, probably less than that.

• 1:12:33Speaker 00

  Some percentage of people be immunized. And in a good year, that vaccine will be about 60% effective. Now, a pan-influenza, you know, a universal flu vaccine would say, screw the strains, we're going for the jugular of this, it's called a species, you know, of influenza as a family. I talked to one person who's very, very deep in the world of lobbying for and doing initial work for a universal flu vaccine, a guy named Harvey Feinberg, used to run the Harvard School of Public Health, like all kinds of titles and accolades, I can't remember right now. But he estimated to me that if we really went all in on this, it would probably cost about $200 million and take 10 years.

• 1:13:18Speaker 00

  To get there or not, and he felt that there was a 75% chance that we would get there, not 100% chance. I said, "Well, Harvey, let's go crazy, worst-case scenario. Could it be 10x that?" He's like, "Yeah." He's like, "Maybe it's $2 billion over 10 years, and there's a 50% chance of getting there." Okay, the flu cost the United States $365 billion a year in lost productivity and medical bills. If you have got a chance, let's say Harvey's worst number, to invest $2 billion with a 50% chance of relieving yourself of a annual $365 billion burden, there should be no thought necessary at all. You take that chance, and hopefully Harvey's right, and it's actually more like $200 million and 75% chance of success.

• 1:14:04Speaker 00

  What we should be doing right now is let's take worst-case numbers. Let's say it's $2 billion per virus, and there's 20 of them. Let's invest those $40 billion over the next 10 years and get, you know, pan-vaccines for every virus that infects and kills humans. And let's throw in another 20, the zoonotic viruses that are out there that are most threatening. Let's get pan-familial vaccines, or do our very best and at least have a 50/50 shot with each of them. You know, $40 billion over 10 years, $4 billion a year, that's chump change in the context of the American budget. That's chump change in the context of $365 billion lost to flu and, you know, one credible

• 1:14:48Speaker 00

  economist estimated $14 trillion of damage to the United States economy, US alone, from COVID. But you do that. I don't see that happening anywhere. There are a couple of academic labs that are doing, working on a pan-coronavirus vaccine, but they don't have a $2 billion budget. This is not happening. And so, you asked me, "Are we better off for having had COVID?" Theoretically, yes. Theoretically, we've gotten a wake-up call that's unmissable, and now we're going to take really smart preventative steps. But this shriekingly obvious step...

• 1:15:23Speaker 00

  You know, there may be some governments that are calling for it. I can't read every news story every day. I haven't detected any concerted effort to say, let's just take every virus family off the table that we can. And if we're not doing that, and that's really cheap and really obvious, I could certainly see us not taking a lot more, more expensive and slightly less obvious, but equally important steps.

• 1:15:46Chris Williamson?

  After the last fifteen, eighteen months as well.

• 1:15:50Speaker 00

  Yeah.

• 1:15:51Chris Williamson?

  The most obvious flare fired up into the air directly over our position, highlighting what the problem is, highlighting all of our insufficiencies and our poor coordination. And yet, nothing.

• 1:16:05Speaker 00

  Yeah. Pan-corona vaccine. Hello, people. Why is that not happening now? I mean, that research should have started in April of 2020. We didn't have to wait for, you know,

• 1:16:21Speaker 00

  the mRNA vaccines and so, you know, that research should have started as soon as we said, holy cow, SARS and now this coronavirus, big deal. But no, that, you know, there is research, like I said, I've seen a couple of papers coming out of academic labs, Herodgo academic labs, but it doesn't have the public support, like the public funding support that it needs. And that's, yeah, it's bonkers.

• 1:16:46Chris Williamson?

  What would be the implications if the lab leak theory was proven to be true for China and for sort of the fallout politically and the safety in future as well?

• 1:17:00Speaker 00

  Okay, so let's, um, let's play this out. Let's assume for the sake of this thought experiment, this was a lab leak. And it gets proven definitively that it was. It would almost certainly be, if this was the case, it would almost certainly turn out that COVID was a result of gain-of-function research. Because it is a novel virus and, you know, it came out of nowhere. And if it was a lab leak, it almost by definition would not have been something that was circulating naturally and just didn't happen to cause a pandemic. It would be something that was novel that was created in that lab. So I think if that got out and was proven beyond a reasonable doubt, there would. One hopes that there would be a global ban on gain of function to start with,

• 1:17:43Speaker 00

  like this bizarre thing that we haven't done yet would be done. And I think banning gain of function maybe eliminates 2% to 5% of the total risk that we face from synbio run amok in evil hands or good hands. But that's a great step in the right direction. So I think that would happen. I think that, you know, we talked about the compounding spread of climate risk awareness. I think there would be an unbelievable jump in global awareness and concern about synbio run amok. And so I think that there would be a much better regulatory apparatus, there would be much more self-knowledge within academic and private circles. Like, a lot of really, really good stuff would happen.

• 1:18:26Chris Williamson?

  Is there a part of you that hopes that it does get proven?

• 1:18:28Speaker 00

  Well, I mean, if that's in fact what happened, all of me hopes that it gets proven. You know, if that's in fact what happened, yes, absolutely. China would have a great deal to answer for to every single country in the world. And, you know, China should be held accountable, and the Wuhan Institute of Virology should be held accountable, and the very practice of gain of function should be held accountable, and the notion that BSL4 labs are safe should be held accountable. And so, yeah, if that's in fact what happened, absolutely, I would want that to come out. So that we, you know, the world says, China, don't do that anymore. The world says, no more gain of function.

• 1:19:13Speaker 00

  The world says, BSL4 is only a best effort. And those three things right there would significantly reduce the world's risk. So, yeah, if that's what happened, I'd want it to come out.

• 1:19:25Chris Williamson?

  Thinking-

• 1:19:25Speaker 00

  But we don't know, we don't know that that's what happened, that we probably never will.

• 1:19:29Chris Williamson?

  Well, there's a lot of opaqueness, right? A lot of opacity around.

• 1:19:32Speaker 00

  There's incredible opacity, yeah. Incredible opacity, which is, which makes one suspicious. But then again, authoritarian governments are opaque by nature. It's their instinct.

• 1:19:41Chris Williamson?

  It's almost like, if you could have picked, probably except for North Korea, if you could have picked a country that you didn't want it to start in, it would have been China.

• 1:19:53Speaker 00

  Yeah.

• 1:19:54Chris Williamson?

  In fact, maybe even more so that it's China, because they have more sophisticated resources, probably fewer bad actors that are prepared to turn mole and actually be whistleblower about stuff, better coordination, better surveillance.

• 1:20:10Speaker 00

  Yeah, I mean, it's the opacity surrounding the investigation of where this came from is near, almost total. And, you know, it makes one wonder, why, what are we hiding, you know? And so for those who, and again, I don't, I do not pretend to know whether it was a lab leak or not. I want to be very clear about that. A lot of very, very, very smart people think it was. A lot of very, very smart people think it wasn't. So, and I don't have the level of biosophistication to enter that debate. So I will plainly state, I don't have a theory on that. But a very strong argument in favor of it is like, what in the world are you hiding? Because you're hiding something, why in the world will you not allow, you know, a very,

• 1:20:53Speaker 00

  very serious outside investigation into the very early cases? You know, it seems that a couple of WIV, Wuhan Institute of Virology people were hospitalized of something like an awful lot like COVID in December. And, you know, why is that information, why is that not being explored? You know, why did the World Health Organization delegation that went there have zero access to anything that, well, to a lot that would have shed light on the first five weeks? Why all the opacity? You know, and it's easy to imagine something's being hidden. It's, but we also have to acknowledge that, you know, authoritarian governments are opaque by nature. That's their instinct. And they could be opaque

• 1:21:34Speaker 00

  for reasons that are rational to them that don't have to do with us.

• 1:21:39Chris Williamson?

  Yeah, yeah.

• 1:21:40Speaker 00

  Like, it could be that they, that it's completely zoonotic, completely natural. And when they did their own investigation, they're like, oh my God, our safety protocols at Wuhan Institute of Virology kind of suck. It didn't come out of there, thank God. I'd feel really guilty if it did. But oh my God, like, it could have, it didn't, but it could have. We don't want anybody seeing that. It could be something like that.

• 1:22:02Chris Williamson?

  The fallibility all the way down, just humans, humans and our biases the whole way. So zooming back out now from just Sin Bio into some of the broader strategies that we have for X risk, I think, looking at all of the different ways that we could potentially manifest our own extinction, on top of the natural risks as well, that are just background and ambient and constantly going on, whether it be a supervolcano, or a gamma ray burst, or an asteroid that's going to come to hit us, it seems like making the situation already worse for ourselves is probably a bad idea.

• 1:22:37Speaker 00

  Yeah.

• 1:22:39Chris Williamson?

  Would there be an equivalent of reducing, putting a glass ceiling on gain-of-function research, or putting a glass ceiling on the sort of research that we do entirely? Should we be looking at perhaps considering that across the board with regards to technology? Should we curtail our technological progress for a few thousand years until our wisdom can catch up with it?

• 1:23:02Speaker 00

  Um, I would say no for a diversity of reasons. One is, I think it's thoroughly impossible. It's strictly in the realm of thought experiment, because if, let's talk entirely equally impossible scenario, but let's say, you know, the Western democracies and Eastern democracies, you know, the democracies of the world all agree to do that, China ain't gonna stop, you know, and it just isn't. And if China stops, Russia's not gonna stop. And if China and Russia stop, North Korea is not gonna stop. And, you know, if North Korea stops, then somebody that we never thought of, like, you

• 1:23:42Speaker 00

  know, maybe suddenly, you know, Egypt, like, takes the lead.

• 1:23:48Chris Williamson?

  As Bette Stann decide that they're going to become a world power, yeah.

• 1:23:51Speaker 00

  Who knows? Like, if we all stopped it, you know, and Egypt didn't, a lot of smart Egyptians, I used to live there, at some point, they're gonna be number one in it, and then being, you know, like, you just, you know, there's a coordination problem. And then I also think that there's a human flourishing problem as well.

• 1:24:08Speaker 00

  Sin bio has so much extraordinary promise for us as humans. It has so much extraordinary promise when it comes to, you know, therapeutics. I think sin bio, when cancer is eventually beaten, and if we don't destroy ourselves by some other mechanism, one day we will completely defeat cancer. And it's going to be insights and wisdom that come out of synthetic biology and adjacent fields that do that. It's going to be insights and wisdom, you know, that come out of synthetic biology and adjacent fields that will allow us to eventually create clean meat that is so much less damaging to the planet and so much less immoral in terms of the horrible suffering that's inflicted on the conscious systems that we call cows and chickens. You know, there's so much human and non-human flourishing that's going to result from it,

• 1:24:51Speaker 00

  that it's a path that we should go down with lots of safeguards. And, you know, and if we wanted to cease technology, we couldn't solve the coordination problem. And if we did, there's no guarantee that we would have the wisdom that we seek in a few thousand years. We've been trying to seek wisdom for thousands of years already, and we haven't gotten there yet. So the Greeks and Romans probably would have thought, if you asked them, like, oh, man, 2,000 years from now, we're so great at ethics already. Imagine where we'll be in 2,000 years.

• 1:25:22Chris Williamson?

  We must have nearly finished. We must be at the back of the book soon.

• 1:25:24Speaker 00

  We must be done. Yeah, we've got to be very close to it. Except, of course, almost all of those philosophers owned slaves, so we can question their ethical purity, but that's a whole nother conversation.

• 1:25:35Chris Williamson?

  So, thinking about that, if the global coordination problem is so tough that essentially this is untenable, it's not a strategy that we can look at doing. You've got this increasing democratizing or the privatizing of the apocalypse. You have the Overton window of potential bad actors being, more and more people being brought in so that they're perhaps outside of that. We have technology that is further emancipating the ability to bring about our own destruction, right? On an individual, on a local, a national, international, global level, all of this.

• 1:26:08Speaker 00

  Yep.

• 1:26:11Chris Williamson?

  The future doesn't look bright. Yeah, what do we do? When you take in this broad spectrum view as opposed to just looking at synbio, just looking at AGI or nanotechnology or natural risk or whatever, take a more broad scale perspective. And I often think to superintelligence and machine extrapolated volition with this, that trying to pick apart individual strategies and tactics to try and plug the holes in the bottom of the boat, one cork at a time, is not tenable. Because for every cork that you put in, there are 100 that will appear far quicker than you can even notice that they're there. Is this how you feel? Is this how I feel?

• 1:26:51Speaker 00

  No. So what I think you need to do, and so the approach that I take when I think about synbio, is you need to create a really, really agile, adaptive and multi-layered defense strategy. And I think great inspiration can be taken from our own immune systems. Our own immune systems are confronted daily with all kinds of pathogens, many of which they have never, our immune system hasn't encountered before. And most people who are, you know, not extremely old and are well, don't get sick most of the time. And that's because our immune system is multi-layered and adaptive, very, very adaptive.

• 1:27:34Speaker 00

  And so to take synbio as an example, first of all, you stop doing gain-of-function research. That's like, I want to live to be 90. Okay, stop drunk driving every night. Okay, so great, we've taken one thing, self-inflicted, really obvious off the table. And that, like I said, I think that takes a few percentage points of risk out of the equation. What I think would be very, very big would be something that the industry has already tried to do on a self-regulatory basis, which is to create a set of standards of what is dangerous DNA. And let's not let anybody get it unless they have a very good reason to have it. So there's something called the I G S C, which is an industry body that a number of DNA makers are voluntary members of.

• 1:28:18Speaker 00

  Uh, DNA makers, what do I mean by that? Most of the complex DNA and RNA nucleic acids that are ordered or used in both private and public settings, um, generally are not made if it's, if it's a long and complex strand, that will generally not be made in the lab that wants to do the experiment with it. It will be made by an expert body, a company and the, a good, one example is Twist Bioscience, publicly traded company. They make a lot, that's all they do is make, you know, long strands of error corrected nucleic acid, or that's a lot of what they do. So they're very good at that. They're better at that than any particular academic or private lab is likely to be. And so there's several companies, like they're probably number of the low dozens who do

• 1:29:02Speaker 00

  this for a living. And so those are the places where dangerous nucleic acid is going to originate for now. Um, a bunch of them are in this organization called the I G S C. And the I G S C has maintains its main product, if you can call it that, is a database of dangerous sequences. And whenever any I G S member, they've all agreed to do this, gets an order for something that is on that list, a series of internal safeguards go into effect. And basically every order is rated red, yellow, or green. Vast majority are green, nothing wrong with it, no problem. Yellow order comes in and those companies have teams of bioinformaticians, generally

• 1:29:46Speaker 00

  PhDs, who look at that yellow case and usually in most cases it's like, okay, this is fine. You know, they're taking part of a gene that's dangerous, but they're just taking a regulatory part of the gene or whatever it is, it gets, it gets passed through. And when you get a red flag, then a lot of hours are spent and probably the customer's contacted and work is done to make sure that this is again benign usage. And in the worst case scenario, these companies will call the FBI. That's really good. And the companies are investing significant money in this. It's about, my understanding, talking to a few people, it's about $15 per order. Of work that goes into this whole, if you, if you take the cost of all those bioinformaticians

• 1:30:30Speaker 00

  and the red, yellow, green system inside a company like Twist Bioscience, you total up that full budget and you look at the number of budgets, about 15 bucks an order. And, you know, that's real money. These are staffs of people that, this is great, self-regulation. But there's problems with it. One problem is that as the ability to synthesize DNA and RNA gets better and better and cheaper and cheaper, which is what's happening, has happened with computing, you're getting more and more orders and the order size is shrinking. And so 15 bucks is becoming a bigger and bigger and bigger part of the order cost. The cost of that apparatus is significant enough that there are a lot of companies that aren't members of the I G S C for cost reasons.

• 1:31:12Speaker 00

  And so the I G S C says that they account for 80% of the DNA in the world, or at least in their industry, a provision. But that number was kind of pulled out of the air a decade ago when the I G S C was founded. Nobody really knows. And because there's only one Chinese, only one Chinese member of the I G S C and there's a lot of DNA synthesis going on in China, there's no way it's 80%, right? So it's, it's a hell of a start. It's a very good start. It's an admirable start. But we need to dial that game up dramatically. It needs to be 100%, which means that it, it, it needs to be mandatory. So voluntary and 80% ain't gonna work. Gotta be mandatory and 100% because if you're a malign actor,

• 1:31:55Speaker 00

  it's really easy to go to the 20% who are doing this. Like, there's, there is essentially no protection unless there's total protection. So if we could get, this is a harder agreement to get passed than a global agreement against gain-of-function. But if we can get past the notion that like, we as a, as a species are going to take this I G S C model, dial it up and make it mandatory. That's a big step forward. Now, the next step forward that is very, very important. Is that we are going to go to a distributed model of DNA creation. What I mean by that is, you know, the people at Twist Bioscience hate when I say this, but I don't see there being much of a role for central service bureaus 10 to 15 years out.

• 1:32:36Speaker 00

  Just as there isn't much of a role, you know, we all send a lot of text messages today. We send more text messages than at any time in human history before. But despite our obsession with text messages, there aren't a lot of telegraph companies anymore. Right? Because to send a text message in the 1920s, you had to use this very expensive centralized infrastructure to send your text message. And we went to that centralized infrastructure, we sent our text message, and that's the way it worked. Eventually, that centralized infrastructure disappears because that capability goes out to the edge. That happened with photography, that's happened with a lot of recording techniques, it's happened countless times. It's just what happens. And so benchtop DNA synthesizers are in their infancy right now.

• 1:33:21Speaker 00

  But there's a pretty good one out there now called the BioXP, which allows scientists to synthesize the DNA that they want in their own damn lab without being wizards. I mean, because the scientists who are using the DNA, they don't want to be great at synthesizing DNA. They're synthesizing it for a purpose. They want to get that synthesizing out of the way as quickly as possible to do their actual work. Synthesizing the DNA is like brushing and flossing their teeth. It's, you gotta do it, but it's not, you don't wake up saying, "Today, I'm gonna brush and floss my teeth. It's gonna be the pinnacle of my day." They want as little hassle as possible. That's why they're using Twist Bioscience rather than developing those capabilities themselves. And if there's a little machine that they can have in their, in their lab

• 1:34:05Speaker 00

  and just print out what they need, that's gonna be even better than Twist Bioscience, much as the iPhone is better than a telegraph office. So the next thing that we need to do as a species, as a global society, is say, "Yes, those distributed printers are coming." They need to be mini IGSCs. They need to have those protocols built into them. So that dangerous DNA also goes through this process. And the makers of the BioXP, which is the best printer on the market right now, they are an IGSC member. And when you're printing something on your BioXP, that print desire goes back to the company that makes the printer, and it goes through the red, green, yellow process. And so that's a doable step. Now, let's imagine a world 20 years from now,

• 1:34:49Speaker 00

  where these printers are everywhere, even in high schools, and they've got these very, very good protections and safeguards in place. Does that mean it's impossible to create a doomsday bug? Certainly not. It's still possible. But what we've done is we've made it so much harder that the number of people who can pull it off has gone from, you know, untold, you know, hundreds of thousands, even millions, to a tiny handful of experts. That's an unbelievably powerful protective step. Because, you know, particularly that tiny, narrow group of experts, and we prefer it to be zero. But those people have generally come through life following a path of continuous self-improvement, career development.

• 1:35:32Speaker 00

  They're not the type of people, in general, they're not the type of people who go Columbine, anthrax, leak notwithstanding, they're generally not. So you've diminished a great deal of risk. And also, with the passage of more time, the ability to create a complex virus genome from whole cloth will even vanish from the high pentacles of academic science, because it's just not necessary anymore. There was an incident a few years ago where a Canadian researcher created the horsepox virus, that genome from whole cloth. Horsepox is not a big deal to humans. But what's interesting and important about it is it's a very close cousin to smallpox. And so if that particular academic scientist in Canada

• 1:36:17Speaker 00

  was able to synthesize the horsepox virus using mail-order DNA and other techniques and his own extraordinary, you know, skills, that means by definition, he could have synthesized smallpox if he wanted to. He just elected to synthesize the equally complex.

• 1:36:36Speaker 00

  Horse pox and very closely related horse pox virus. And part of the reason he did that was to put the world on notice, hey, this is possible. And some idiot put the smallpox genome on the internet back before this was possible. And hello, world, wake up, right? Now, the skills that he has to create that are going to, they're not going to become more and more widespread as printers and things like twist bioscience get better and better. They're actually going to diminish because those sort of wet lab skills to create things from a whole cloth and so forth, people aren't going to develop those skills because they're just going to be able to hit print. So if the world, if the academic world and the life science and symbiote world become dependent on these distributed printers, which I believe they will, and the printers

• 1:37:21Speaker 00

  are really, really good at preventing bad stuff from happening, now that's a pretty big protective step.

• 1:37:27Chris Williamson?

  So you're actually utilizing humans inherent laziness or their paths of least resistance to distract them with that. I mean, the sequence that you went through with Sam, that four hour special is one of the, it's an absolute masterpiece.

• 1:37:42Speaker 00

  Thank you.

• 1:37:43Chris Williamson?

  And the work that you guys did on that is phenomenal. It'll be linked in the show notes below for anyone that wants to do a deeper dive on this. I want to, I want to change tack for a second. Have you read Seven Eves by Neal Stephenson?

• 1:37:55Speaker 00

  Seven Eves is the one where the moon.

• 1:37:58Chris Williamson?

  Explodes. On the first line, the first line is the moon exploded.

• 1:38:02Speaker 00

  The moon explodes. Okay, I'm really embarrassed and I hope Neil's not listening because we've met a few times and I think very, very highly of him, but I love Neal and his writing. That was one I couldn't get through. And the reason was, um, sometimes I think, you know, Neal's work turns into mechanical engineering porn. And, um, I am not very adept when it comes to mechanical engineering and it's not a fascination of mine, unlike bioengineering and other things. I'm obviously not a life science expert either, but that'll get through. But like, there is a lot of like, this device is this and there's this gear that has this leverage ratio and that's really cool because this. And so I was like, I just can't. But it was a hell of a start. I'll tell you, I really enjoyed the first 50 pages.

• 1:38:46Chris Williamson?

  Yeah, so.

• 1:38:47Speaker 00

  And I have, I have, I have, that's the only Neil Stephenson novel I have ever started and not gotten through.

• 1:38:53Chris Williamson?

  That's crazy. So yeah.

• 1:38:54Speaker 00

  So anyway, yeah.

• 1:38:55Chris Williamson?

  It's, it's, I wouldn't say it's a slog, but there's certain thing. I mean, you learn the finer points of orbital dynamics, right? In this, you're learning about the Nazir and all of the different ways that the front and back of a three, a three dimensional boat, essentially, which is what a spaceship is, moves and how when you swing back around, you're in different positions. So yeah, hard sci fi can be hard at some times. But everyone that's listening, it's in the new reading list, which is about to get released, which everyone can pick up for free quite soon. But in it, this thing occurs and they need to find somewhere to get people off earth, right? They need to put them up into space. It's the equivalent of an existential risk. There is an existential risk.

• 1:39:34Chris Williamson?

  The fact that it's the moon exploding is irrelevant. It's something. The difference is that they have almost exactly two years from when it happens to when the, what they call the white sky or the hard rain begins.

• 1:39:45Speaker 00

  Yep.

• 1:39:46Chris Williamson?

  And what they actually do is they say, at the very, very beginning, the president says, we need to have a two-pronged strategy. We need to send people underground and we need to send people into the sky. Given the fact that we are constantly up against an increasing anthropogenic risk from ourselves, an ambient background, natural risk. We don't have a second community on Mars. We don't have another planet. We do not have a backup. Would it be advisable to create a siloed community somewhere, which is totally self-sufficient, defended and air gapped from the rest of the world?

• 1:40:24Speaker 00

  I have never thought of that and shame on me because that's such a cool idea. And anybody who thinks about existential risk should be thinking about this. That is a really fricking interesting notion.

• 1:40:37Speaker 00

  Wow. Yeah, we should. We should have that. You know, the Long Now Foundation has this really cool library they've created, which has the, you know, 1,500 books that they believe would be necessary to reboot civilization. And they've got, if you go to the Interval Cafe in the Fort Mason area of San Francisco, and anybody going to San Francisco, I urge you to go to the Interval Cafe. It's awesome. It's run by the Long Now. And downstairs is the cafe and bar, and upstairs are all those books. They've got this library, but you can't go up and look at them. There's a, you know, unless you know somebody from Long

• 1:41:18Speaker 00

  Now is going to take you up there. But that's almost a symbolic gesture. And, you know, yeah, I mean, creating a community like that, and maybe it's like national service, like you rotate through for two years. Like, nobody would want to live their entire life there. But, you know, maybe it's, you know, a whole bunch of young people, because you want a lot of young people who can, you know, procreate and all that other stuff. But maybe it's like, you know, people go on a two-year rotation, which would probably be kind of fun and cool, like a neat thing to have done for two years to be in that community.

• 1:41:45Chris Williamson?

  It seems dumb to me that we're waiting for some sort of potential risk to occur before we can save ourselves from it. Like, I know that, I know that the problem is fully air-gapping yourself whilst you're attached to the same piece of rock is going to be difficult. If you've got a malignant AI that's got a control problem that's running around, it'll probably know where it is. It might be able to manipulate it. But, again, the same thing as you're trying to achieve with the Sin Bio Solutions. It's not about perfect. It's about reducing the risk as much as you can, because that's all that you can do. Just because you can't get perfect doesn't mean that you shouldn't try. So for me, defended, you know, really heavily defended, walled off somewhere in a mountain bunker shit, you know, all of the seeds, all of the books, all of the, everything that you

• 1:42:29Chris Williamson?

  need. And just, if you've got it there, yeah, ethically, is there a question? Yeah, of course there is. It's the same as the generation ship. You know, if you're going to fly to Alpha Centauri and decide to commit the next thousand generations of your progeny to living on this.

• 1:42:44Speaker 00

  Tedium.

• 1:42:45Chris Williamson?

  Yeah, exactly. To be born, live, and die in this steel cage floating through space. Is that ethical? Well, if you decide to take the grandest view of humanity as a whole, if the choice is between some people suffering and humanity dying, I think that that's, so this is something I reckon more people should be considering, like, because there's bunkers. Isn't there a? Have you looked into these rumors that billionaires are buying up, is it New Zealand or Australia that they're buying up ranches or bunkers or something? Have you heard about this?

• 1:43:17Speaker 00

  Yeah, well, the whole prepper movement is very real.

• 1:43:21Chris Williamson?

  This is preppers, this is preppers on steroids.

• 1:43:23Speaker 00

  And this is some, and some preppers are going to be billionaires and they're going to take crazy, yeah. So I'm sure that there's some crazy prepping stuff going on. What's, what's more relevant to this conversation, there was a big effort in the very early 90s called, I think it was Biosphere 2. Have you heard about this?

• 1:43:40Chris Williamson?

  No.

• 1:43:41Speaker 00

  So some, some, you know, kind of eccentric billionaire or near billionaire funded another eccentric's dream to create a completely independent biosphere. And it was in, somewhere in the southwestern United States, like Arizona, New Mexico, something like that. And the bones of it still exist. And it got a great deal of press at the time. And so they created this huge bubbled community. And it was, the bubble was huge, the, the physical footprint was huge, the budget was huge, and it was privately funded. The number of people was not. It was something like nine or 11 people went in there. And the objective was to completely

• 1:44:22Speaker 00

  decouple from the Earth's biosphere. Now, it wasn't bunkered, it wasn't guarded, it was, you know, but just decouple from the Earth's biosphere.

• 1:44:29Chris Williamson?

  Like an ecological rather than a defensive experiment, yeah.

• 1:44:32Speaker 00

  Yeah. And, uh, you know, have agriculture going on in there and, you know, have all these, you know, to try to create a biosphere imbalance that these people could live in and be completely self-sufficient in for two years. And it failed. And there's a really good documentary that I saw toward the beginning of COVID that I'm happy to try to dig up and you could put in the show notes. It was well done. It failed for a bunch of reasons, but basically the agriculture started failing. Their, their ability to generate enough calories in there, you know, was meant to, you know, they had huge, huge greenhouses and so forth. That started failing. And then also the carbon in the atmosphere, the internal atmosphere

• 1:45:14Speaker 00

  started getting out of, out of whack. And so at some point, this small group of people were slowly starving and asphyxiating.

• 1:45:23Speaker 00

  And they eventually gave up. And there was also this controversy where one of the people needed to have surgery. And so obviously left the biosphere to get the surgery she needed and came back in laden with like a Santa Claus like bag full of stuff that that was missing. So people like, oh, you guys are cheating. But, you know, okay, this did not have the budget or scientific power of even a lightweight government effort. I mean, when you when you watch the documentary, you realize kind of how absurd it was like a Don Quixote type of thing. It was charming, but a little absurd. And they just didn't have the firepower to pull this thing off. But that's an interesting, relevant example.

• 1:46:06Speaker 00

  But I think what you're describing would, you know, it could be a national effort. The US has the the resources and budget to do it and China does and a couple of other places. But, you know, maybe ideally it'd be international. And you wouldn't be sentencing people to life or thousands of generations to life. I imagine people would rotate in and out. And I imagine rotating in would be a pretty cool experience to have in life if you were there for a year or two. There'd probably be lots of other young, single people in there. It'd probably be kind of fun. You know, there's people who spend the winter in Antarctica. You know, if you're in Antarctica at one of the research stations, once winter settles in, there's nine months where you cannot get out. And that's a small group of people and it's really intense and a lot of people have horrifying experiences, but some people have awesome experiences with that. You know, so it'd be kind of like that.

• 1:46:51Speaker 00

  That's a really interesting idea. I like it.

• 1:46:54Chris Williamson?

  Cool. What do you think is a big X risk, which is rarely talked about within the community? So you might not be able to predict the unknown unknowns, but what about the barely known unknowns or the recently known unknowns?

• 1:47:05Speaker 00

  Well, you know, there is there was an experiment run at CERN on Brookhaven back in the 90s that the best documentation I've seen of it is in Martin Rees's book, Our Final Hour. And that's what's called in the United States and it's got the much more intelligent title in the UK of Our Final Century. So I guess that sort of reflects the perceived detention spans of the American and British populations, according to the publishers who released that book. But it's an amazing book. It was one of the first really Um, popularly accessible contemplations on, on exerisks. Um, and it really was more about like things that could wipe out.

• 1:47:50Speaker 00

  He was also very interested in things that could wipe out tens of millions of people. But there was this experiment run that had a tiny chance. Uh, again, let's talk about Trinity experiment in New Mexico level chance of creating something called a strangelet, a hypothetical form of matter, which is very much like ice nine in Cat's Cradle by Kurt Vonnegut. So Cat, Cat's Cradle is a brilliant novel that I'd recommend to anybody who enjoys speculative witty crazy fiction. Um, and in I, in Cat's Cradle, there was this thing called ice nine that somebody developed that if it touched, it was a form of ice that if it touched water,

• 1:48:29Speaker 00

  the adjacent molecules would also turn to ice nine, basically at a melting temperature of, I don't know, 90 degrees Fahrenheit. You know, 35, 40 degrees centigrade. And the, the, the danger with ice nine was if it touched normal water, the adjacent molecules would immediately turn into ice nine. And the adjacent molecules to those would turn into ice nine. Like Midas touch. Yeah, it was like Midas touch. Almost instantly, any water it touched would turn into ice nine and would only be meltable if you got it really hot. And so this is a ticking time bomb that runs throughout the, the novel. Because obviously if somebody drops a shard of ice nine into the ocean

• 1:49:11Speaker 00

  or even into a tributary, a minuscule tributary of a, of a minuscule river, that is going to flow through the, all the water of the earth with unbelievable rapidity and game over. So strangelet is ice nine. And if it exists, if it exists and if it has, if it's possible to exist and if it has certain properties and if, if, if, and they if, if, if this down at, at the Stern and Brickhaven Institute, um, experiment to say, oh, it's just really unlikely. And they proceeded with the experiment. And if the strangelet thing had happened, it would have turned all of earth into strangelet matter. And it's unlikely, but possible that the ice nine effect would have propagated further out and might have destroyed the entire universe.

• 1:50:05Speaker 00

  That's not the reaction that humans need to have to existential risks in order to prevent it. If we think existential risk is just excellent forms of comedy, we're done, Chris.

• 1:50:14Chris Williamson?

  I know, but it's so interesting.

• 1:50:16Speaker 00

  It's so interesting.

• 1:50:17Chris Williamson?

  I know, I'm so fascinated by it.

• 1:50:19Speaker 00

  Once again, we probably have a set of risk and payoff in the minds of the scientists who are making this decision, like, this is really unlikely. And if it doesn't happen, and it almost certainly won't, it might end up on the cover of nature, might end up on the cover of science. And there was no, so they went ahead with it. And the danger with that is only this cloistered elite is in a position to say no, because only this cloistered elite is in a position, even though that the experiment's happening, and to know its ramifications. And if some outsider came in and said, no, don't do it, they'd be like, okay, nincompoop, what did you get your PhD in?

• 1:50:55Chris Williamson?

  Yeah, exactly.

• 1:50:56Speaker 00

  Oh, you studied Arabic in college, and now you're a government regulator. Well, you're too stupid to even understand the risk that we're taking, blah, blah, blah. And yeah, so it's things like that. You know, it's, very few people know about that.

• 1:51:09Chris Williamson?

  That being said, and this is where I want to pivot into something that I'm quite passionate about with regards to ex-risk. And this is what I see as the Future of Humanity Institute, which is phenomenal, and the work that Nick and Toby and the guys do there is outstanding, it's world-changing. But what I see as one of the big holes in their worldview with regards to trying to enact the most successful existential risk protection strategies is a lack of an understanding of the importance of public idols. So if you were to look to someone like Greta Thunberg, not a climate scientist, not a specialist in her field,

• 1:51:49Chris Williamson?

  and yet, because she was right place, right time, right delivery, she was able to garner support behind a particular movement. What I think is missing and what you've just highlighted there, the fact that there aren't sufficient people, if you had a million people that had PhDs in Arabic and were wagging their finger at somebody, you'd go, actually, there's quite a lot of people outside of the office, maybe we have to listen to them. But when you have small number of people and the type of research that's being done is very specialized and gated within these particular intellectual communities, the intellectual proletariat are not going to listen to the plebs in the street, especially if there's only three plebs, me, me and you and Sam Harris, like waving our hands outside, like that's not going to work. So I think that there needs

• 1:52:33Chris Williamson?

  to be a, I think existential risk research and the movement at large needs to be made more sexy and more popular. That's not to say that the guys at the, you know, Nick and Toby are handsome gentlemen, but someone who's going to be out there, you know, Anders Sandberg is a perfect example of this, although he's busy doing real research and actually getting the work done, but more of these sort of public facing talking points, the stuff that, the things that you did with Sam, Josh Clark, the end of the world with Josh Clark was

• 1:53:08Speaker 00

  Podcast series?

• 1:53:09Chris Williamson?

  Yes, a nine part podcast series, fully soundscape, oh, beyond outstanding, but it needs to be like endless amounts of that. Get people interested because the compulsion that I have and you have, we're not outliers, like this is something that a lot of people would just naturally be interested in. And then on top of that, when you say, this is potentially the most important thing that any of us can work on at all in our lives, the most important thing, the successful continuation of the human species, or the ensuring that we do not neuter, permanently neuter our own ability to reach our full civilizational potential, is what else is there? Like, other than enjoying your life and, you know, loving the people that are around you and stuff like that. As far as callings go, I can't see much that is bigger than that.

• 1:53:54Chris Williamson?

  So there seems to be a delta between the fact that it's so compelling and so important and naturally interesting to some of us. And also, there is this huge vacuum when it comes to the public conversation about it and the pressure that's being placed on artificial intelligence research and nanotechnology and synbio. And have we got the first time that you, a person who is steeped in this world and has written novels about it, has heard, and there should be no way that there should be any idea that hasn't been heard already, when the consequence is the outcome of our species.

• 1:54:29Speaker 00

  Yeah. And the other thing that we could use more of is storytellers. And so as a science fiction writer, that's something that I can do a bit of. And, you know, I believe that a very significant reason why we survived the Cold War as a species was movies like War Games and movies like Doctor Strangelove, that really, really resonated with the popular consciousness and really got people freaked out at all levels of society. Those stories can travel in a really, really powerful way. I think one reason why the world dodged the bullet of totalitarianism in, you know,

• 1:55:07Speaker 00

  that the resistance to totalitarianism, like really methodical resistance to totalitarianism, you could date it, you know, to World War, can't really date it to World War II because we're fighting alongside Stalin against Hitler, right? It really kind of dates to the 1950s. You know, if you look at the intellectual landscape in, I don't know about all countries, but certainly in the United States, in the 30s and 40s and early 50s, it was unbelievably fashionable to lionize communism. It was unbelievably fashionable to lionize Stalinism. And lots and lots of thoughtful, caring people, really, really smart people, became communists.

• 1:55:51Speaker 00

  And not merely communists, but a lot of them would have done anything to hasten a communist revolution in, you know, in Western democracies.

• 1:56:00Chris Williamson?

  Full-on card-carrying communists, yeah.

• 1:56:01Speaker 00

  And bring on a Stalinistic government. What changed all of that? Well, a lot of things did, but I think a very big thing that changed all of that was the novel 1984. 1984 was written in 1948, and it was a monster global sensation. And the intelligentsia, like all novels, it was read more by the intelligentsia than by the extreme mainstream, but that really shifted the dialogue of the intelligentsia. It painted such a plausible and horrifying picture of what Stalinism with slightly better technology would look like. And it was a science fiction novel. People forget that. 1984 was a frickin' science fiction novel.

• 1:56:44Speaker 00

  First of all, it was set in the future, people. And secondly, the telescreen technology that the thought police used to keep an eye on everybody, that was impossible in 1948. So it was near-future sci-fi, and it shifted the debate. Anybody who read that book could no longer say, "I want to live under Stalin." You just couldn't. It really inoculated the Western democracies, or, you know, free countries throughout the world. I keep saying Western mission. It really inoculated most of the intelligentsia, and it really inoculated most of the free world against the lure of totalitarianism. What was the allure of totalitarianism?

• 1:57:26Speaker 00

  Go back a couple decades, in the 1930s, the world's going through the Great Depression, and, you know, all you see are... Stalin was industrializing a non-industrial nation of the Soviet Union at an unbelievable pace. And there was a lot of low-hanging fruit that kind of worked in the 30s and 40s. And you have a lot of people who are worried about starvation in, you know, outside of the communist bloc, and says, "That gets seductive." And it gets seductive if you're, you know, a brilliant student at Cambridge, or Harvard, or whatever, and you encounter these ideas, and they're all about, you know, the good of the many and so forth. That inoculation was really powerful from 1984. And so I think we need stories,

• 1:58:08Speaker 00

  more stories like the Terminator. People chuckle and sneer and snigger about how unsophisticated the Terminator is. But guess what? It brought the notion of super AI risk to, basically, everybody understands the Terminator story in Skynet. You know, I mean, some people are not, are so disconnected from popular culture, they don't. But you have much less, if you want to talk about super AI risks to somebody who's not inculcated in the world, you have a much shorter path to that, to getting them up to speed and understanding the risk, because we've all seen Terminator. That was a pretty damn good story. We haven't really had the Sin Bio Terminator franchise yet. That would be really useful.

• 1:58:48Chris Williamson?

  Have you seen?

• 1:58:49Speaker 00

  Be really, really useful.

• 1:58:50Chris Williamson?

  Have you seen Next? I think it's an NBC series.

• 1:58:54Speaker 00

  No.

• 1:58:55Chris Williamson?

  Okay.

• 1:58:55Speaker 00

  Tell me about Next. I'm already excited.

• 1:58:58Chris Williamson?

  So Next is a nine or 10 part series about a misaligned, super intelligent, artificial general intelligence that goes rogue and starts taking over the world. But it's done in a very, very smart way. So the guys, they're using the language of Nick, they're talking about recursive self-improvement. It's trying to find, it's trying to find the correct architecture with which it can deploy itself onto multiple servers because it knows it's got the algos, but it doesn't have the computing power to be able to do it. It's trying to manipulate people psychologically. It's trying to manipulate people with regards to their logistics. It uses different actors to play off against each other. It knows, it is phenomenal. And it's, it's a bit, for want of a better term, it's a bit Americanized,

• 1:59:41Chris Williamson?

  right? It's very dramatized. It's sort of glitzy and everyone's got really nice teeth and so on and so forth. And there's a kind of an Elon Musk type character in there who's a little bit crazy and he's trying to fix things. And, but Next, anyone that's been interested in this, if you want to watch it, like a 10 part series, I think it's NBC.

• 1:59:58Speaker 00

  It's Fox. I just checked. I'm looking at the Wikipedia page right now. It sounds fabulous.

• 2:00:02Chris Williamson?

  It's legit. And they know, they know their stuff. They, they understand about the alignment problem. They understand about recursive self-improvement and, and what it would try and do. Again, there's some dramatization, but that, again, you're right. I think pushing the culture in that sort of a way. So I came across this, this quote, and it was,

• 2:00:21Chris Williamson?

  Someone criticizing the intellectual commentary out of the intelligentsia and saying, "Just because you've beaten bad philosophy doesn't mean you've beaten bad psychology." And what they were highlighting was the fact that people who can proslatize about ideas academically are still at the mercy of cognitive biases because they're still human, deep at heart. And I think that what you've just hit upon there uses the same, it leverages the same type of mechanism, right? So if you can move the culture in a way where it becomes desirable to have a particular mode of thinking, because all of the norms have been pushed in a particular direction

• 2:01:05Chris Williamson?

  through things that people don't even consider, like how much did those movies to do with the Cold War in the late 1900s? How much did that really push public consciousness? Or how much did The Terminator really open people's eyes to the fact that, "Oh, my God, like, this is, this is what supercomputers perhaps could do." And that was well, well, well before its time.

• 2:01:26Speaker 00

  So long ago.

• 2:01:27Chris Williamson?

  Yes.

• 2:01:27Speaker 00

  So prophetic.

• 2:01:28Chris Williamson?

  Yes. Unbelievably prophetic. The same thing that happened with 1984. You can shift culture, not just through academia, but through entertainment as well, not just through education.

• 2:01:38Speaker 00

  Oh, yeah, absolutely. No, I think entertainment is an enormous lever. Yeah.

• 2:01:44Chris Williamson?

  It shapes culture in that way. So, yeah, I think, I think that would be a very useful way to do it. So let's say that someone's taken the existential risk black pill. Perhaps we've begun, some people at the beginning of their dominoes sort of toppling journey today, or there might be some people that have joined us who are already partway up the mountain or down the mountain, depending on how you look at it. How would you advise them if they're passionate about this topic? Generally, general existential risk? How do you think that we should conduct our lives to contribute best to a cause that we care about?

• 2:02:17Speaker 00

  Wow, that is an interesting question. I mean, I think, you know, people who, like, there are relatively few avenues right now. I mean, they could contribute money to the few centers that are working on this. I mean, that's something that anybody could do with five bucks. And if millions of people did that with five bucks, then the budgets that support people who think about this full-time would rapidly, rapidly expand. I think that, you know, public pressure on governments everywhere to take this seriously would be a pretty highly-leveraged outcome. I mean, let's imagine that, you know, the United States and Australia continue to hit the snooze button on this, whatever. But, for whatever

• 2:03:01Speaker 00

  reason, Denmark gets with the program, okay? The Danish government is very small compared to the US government, but it's very, very, very large compared to most organizations in the world. You know, a mid-sized country with lots of resources, lots and lots of brilliant people, you know, very progressive social policies. If the Danish government said, "You know what? We're gonna try to lead the world in taking this seriously," and then they started directing government-sized budgets, as opposed to academic-sized budgets at the problem, you know, any government of mid-size or greater that decided to make this a priority and fund a lot of basic science and public

• 2:03:44Speaker 00

  education and so forth, that would really be a major domino to fall. So I think people, even particularly people in countries like Denmark or Estonia, you know, that are smaller countries that, you know, have maybe, you know, a higher average IQ than my country, I don't know, like, you know, but small, well-organized countries that have a history of, you know, being a little bit ahead of the curve when it comes to social policies and other things. Any government that takes this seriously and uses the might of government and the megaphone of government could have, would have a significant impact on public awareness. It just, that would be a hell of a start.

• 2:04:27Chris Williamson?

  So you're talking lobbying, local officials, bringing it up more. I suppose the problem is, again, the chasm between most people. When you talk to them about existential risk, it's just not, the same way as 100 years ago, climate change wouldn't have been a word. So I suppose that's something, that's definitely something I'll take away from this conversation. I think I'd looked at existential risk and the furthering of existential risk as a purely academic or intellectual conversation, which then needed to essentially just be communicated to people through shows like this or YouTube videos that are kind of like edutainment or whatever it's called.

• 2:05:07Chris Williamson?

  But the cultural side of that, you know, moving people in pure entertainment is something that I totally didn't consider, but is probably even more powerful and, and more accessible.

• 2:05:18Speaker 00

  Yeah, it's part, it's part of the, the packet of things you want to do. What was the name of the guy who created the podcast we were just talking about, Into the World podcast, which was?

• 2:05:27Chris Williamson?

  Josh Clark.

• 2:05:28Speaker 00

  Okay, so let's talk about really local government now. So let's say somebody is influential with their local school board. And they say, we're going to do something crazy here. We are going to, in our school system, have a required 10 week course for all seniors, all, you know, all people who are about to graduate. It can be fun class, but it's a 10 week class they all got to take and they're going to listen, listen to Josh's podcast and there's going to be a smart, you know, social studies teacher who leads it. They're going to have great discussions and debates and, you know, we're going to just do that in our school system because that's what we can influence. And then maybe some state or provincial government says, hey, this is cool. It starts spreading like, again, it's like compound spreading.

• 2:06:13Speaker 00

  And so nobody was aware of existential risk 15 years ago, really. Now, I think low digit millions of people are aware and interested in this subject. And at some point, you know, maybe one of those people is the chairman of a school board. And implements this. And so you suddenly have a local school system studying this. And maybe adjacent towns say that's neat. And you know, this is a compounding of it. Or people who did that high school class, nine years later, one of them's in a position of influence. That's how the environmental movement over the 50 years, people became aware through this or that. And they made more people aware. So I think that, you know, anybody who can implement some kind of policy, whether it's local government, school board, national government,

• 2:06:57Speaker 00

  to take this more seriously, that's, that's a big step. Oh, here comes Ringo.

• 2:07:01Chris Williamson?

  Oh, we've got Doggo.

• 2:07:02Speaker 00

  Popped up. Yeah, at least popped up on the, hasn't gotten into the camera yet, but he's in imminent danger. I'll just lift him up.

• 2:07:08Chris Williamson?

  There we are. What a way to feed. He looks very placid.

• 2:07:13Speaker 00

  He is right now, but I think he's telling me that he wants to go out and I'm going to obey that signal because, you know, that means something with a dog. He doesn't want to make a mess. Ringo is new to potty training. I've had him for a month. He's a little rescue dog. He's 12 months old and he came to me unpotty trained. And he's learned very quickly, but oh no, I think he's just saying he wants to cuddle. Okay, well, we may not have to go out. Oh, he's barking at somebody in an adjacent deck. Anyway, well, we got Ringo now. Now you're not going out if you're going to bark, dude. Anyway, I'll deal with Ringo. I think we're good for now.

• 2:07:50Chris Williamson?

  Cool. Uh, Rob Reid, ladies and gentlemen. Rob, thank you so much for coming on today. This has been everything that I wanted to try and get out of it, but there's still yet more to talk about. So round two may have to happen in the future.

• 2:08:03Speaker 00

  I'd be very open to that. It's been great to be here. It's been so much fun talking. Yeah.

• 2:08:08Chris Williamson?

  Talk to me about what we can expect, what people can expect from you coming up. You're working on books and have you got any other super secret projects that you can tell, hint at?

• 2:08:18Speaker 00

  So I'm working, there's a pretty serious effort that's pretty far along, but about to hit one of the really, you know,

• 2:08:29Speaker 00

  significant, you know, go no go decisions to turn one of my novels into a movie that I've been very engaged in. And if that happens, there's some prospect that I would be involved in, you know, the further creation, because it'd be a feature-length animation, which is something that happens over a period of years rather than 10 weeks of production. And so that's a cool thing that I've been working on. My podcast is called The After On Podcast. It's not unlike yours. Yours is more similar to my podcast than anything I can think of, really. And I interview, you know, world-class thinkers and scientists and occasionally entrepreneurs about their life's work and try to, you know, create an interview that makes all that really accessible.

• 2:09:13Speaker 00

  I had a long hiatus from that because I was doing a bunch of other projects, including this thing with Sam. But that's now up and running. So I've got new episodes coming out with that. I've been very busy recently writing and recording a lot of music with a band called The Godchildren. Hello, Ringo. Ringo is not our drummer. Ringo's a drummer in another band, not this Ringo. With a band called The Godchildren, we are invisible online as of yet, but we've got a bunch of music that we're about to start releasing, which I'm pretty excited about. And the thing that's been taking up a great deal of time recently is I started an investing partnership

• 2:09:48Speaker 00

  with a guy named Chris Anderson, who runs and, for all intents and purposes, kind of owns. And he bought the TED Conference. He's a TED Conference guy. And he bought the TED Conference, which was a profit-making concern, and put it into a nonprofit entity. So nobody owns it now, but he's the one who bought it and this guy runs it. So he and I have created an investing partnership we call Resilience Reserve. And we're investing in startups that, in some meaningful way, have the potential to make the world more resilient. And, you know, so I'm investing with that fund. So there's a lot going on. Lots of balls in the air.

• 2:10:24Chris Williamson?

  Dude, that's so cool. I knew about the music thing, but I didn't know about you and Chris and creating a fund and yeah, there's so much, it's so impressive to see somebody who's obviously just worked and worked and worked at different things and managed to get themselves to a place where they have expertise and contacts and some leverage and some capital. And then when you pool all of that together, you can do some, you can do some pretty cool shit.

• 2:10:50Speaker 00

  Yeah, and you're doing some pretty cool shit. I'm amazed at the, just the quality of, you know, the quality of your guests, the quality of the interviews and just the frequency with which you're coming out with episodes is unbelievable. So yeah, I mean, I'm, I'm also in quite a bit of awe as a podcaster who could never release episodes with a frequency that you do, even if I were working a thousand hours a week. I'm in awe of what you're doing. So you keep up the good work too.

• 2:11:19Chris Williamson?

  I appreciate it. Rob Reid, ladies and gentlemen. Rob, catch you next time.

• 2:11:23Speaker 00

  All right. See ya.