Pod Insights Modern Wisdom Shocking Ways AI Could End The World - Geoffrey Miller About this episode Geoffrey Miller is a professor of evolutionary psychology at the University of New Mexico, a researcher and an author.
Artificial Intelligence possesses the capability to process information thousands of times faster than humans. It's opened up massive possibilities. But it's also opened up huge debate about the safety of creating a machine which is more intelligent and powerful than we are. Just how legitimate are the concerns about the future of AI?
Expect to learn the key risks that AI poses to humanity, the 3 biggest existential risks that will determine the future of civilisation, whether Large Language Models can actually become conscious simply by being more powerful, whether making an Artificial General Intelligence will be like creating a god or a demon, the influence that AI will have on the future of our lives and much more...
Sponsors:
Get 10% discount on all Gymshark’s products at https://bit.ly/sharkwisdom (use code: MW10)
Get over 37% discount on all products site-wide from MyProtein at https://bit.ly/proteinwisdom (use code: MODERNWISDOM)
Get 15% discount on Craftd London’s jewellery at https://craftd.com/modernwisdom (use code MW15)
Extra Stuff:
Get my free Reading List of 100 books to read before you die → https://chriswillx.com/books/
To support me on Patreon (thank you): https://www.patreon.com/modernwisdom
#ai #existentialrisk #chatgpt
-
00:00 Intro
02:43 Is AI Viewed as an Existential Risk?
06:14 How the Perceived Risk of AI Has Evolved
15:47 Rapid Advancements in Neural Networks
20:07 The Next Level for ChatGPT
28:25 Helping the Public Understand the AI Arms Race
37:16 Who Will Be the First to Create AGI?
40:53 Is the Alignment Problem Still a Problem?
50:42 The Opposing View to AI Safety-ism
54:55 Most Concerning Current AI Advancements
59:36 How AI Will Influence Content & Social Media
1:04:57 How Accurate Can AI Expert Predictions Be?
1:08:40 Something Much Worse than Existential Risk?
1:11:43 Why AI Won’t Save the World
1:19:29 Where to Find Geoffrey
-
Get access to every episode 10 hours before YouTube by subscribing for free on Spotify - https://spoti.fi/2LSimPn or Apple Podcasts - https://apple.co/2MNqIgw
Get my free Reading List of 100 life-changing books here - https://chriswillx.com/books/
-
Get in touch in the comments below or head to...
Instagram: https://www.instagram.com/chriswillx
Twitter: https://www.twitter.com/chriswillx
Email: https://chriswillx.com/contact/
Transcript 12,045 words · 2 speakers
Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:00 Speaker 00 So what we're facing is potentially AI systems that are smarter than any human at doing a wide range of things, but also that are potentially 100, 1,000, maybe 100,000 times faster than humans. I think when we confront very fast, powerful, general-purpose AI systems, that's the kind of situation we're going to be in. We're going to be outclassed, not just in terms of intelligence, but also reaction speed.
0:25 Chris Williamson? Why are you, as an evolutionary psychologist researcher, talking about AI?
0:32 Speaker 00 I just love getting into trouble and making making trouble on Twitter. No, I actually, look, long story short, when I started my PhD program at Stanford in way back in 1987, right, I was studying cognitive psychology and cognitive science. And pretty quickly, we got one of the leading neural networks researchers at Stanford Psychology, a guy named David Rumelhart, who worked very extensively with Geoffrey Hinton and lots of other people. And so I started in grad school doing a lot of work on neural networks and machine learning and genetic algorithms and so forth. And then I spent most of my postdoctoral years
1:16 Speaker 00 at University of Sussex, also in a cognitive science department, doing autonomous robots and sort of applying genetic algorithms to evolve neural networks. So a long time ago, I was sort of an early adopter of machine learning. And then I got sidetracked into this evolutionary psychology thing, you know, for about 30 years. But recently, since about 2016, I've I've become concerned and fascinated by rapid progress in AI, particularly deep learning and the large language models. And I sort of fell in with this gaggle of effective altruists, right? This movement who are quite concerned about existential risks, risks to all of humanity.
2:00 Speaker 00 And a lot of them were very concerned about how AI could play out. So the last few years, I've been reading a lot about this. Since last summer, I've been publishing a bunch of essays on the Effective Altruism Forum and gotten pretty active on Twitter the last few months about AI ex-risk. So to the casual observer, right, they might go, who is this psychology dude suddenly interested in AI? Well, honestly, I've been fascinated by AI ever since I was a high schooler reading science fiction, you know, and ever since grad school, learning about cognitive science. So that's my long story short, that wasn't actually very short.
2:41 Chris Williamson? Is it right to say that most of the researchers in the existential risk world see AI as one of the, if not the premier, primary risk that we're facing?
2:55 Speaker 00 Yeah, absolutely. There's a great book by Toby Ord, O R D, who's an Oxford moral philosopher, but he has worked a lot on existential risks. So he did a book called The Precipice, right? And he actually tries to quantify the different extinction risks that we face. Some of them are really, really low probability, but really hard to fight. Like if there's a local gamma ray burster, right? Or a supernova, very, very low likelihood, very hard to defend against. Other stuff like asteroids, right? Which get a lot of attention, very, very low likelihood. Like, there's probably less than a one in a million chance we're going to get a dangerous asteroid in the next century. If it comes, that could
3:40 Speaker 00 be bad, but you know, there's stuff we can do about it. Whereas Toby Ord estimates estimates that the risk of extinction, human extinction through AI in this century is about one in six. And I think that's sort of in line with a lot of the estimates that many experts give. The other, the other big risks are basically nuclear war, which is still an issue, right? After, you know, 70 years of thermonuclear weapons being around. So nuclear war, possibly genetically engineered bioweapons could be really bad. It'd be like COVID on steroids that could wipe out a lot of people. But the other, the other one is AI.
4:22 Speaker 00 So those seem to be the big three, AI, nukes, and super germs.
4:28 Chris Williamson? Yeah, it's an interesting one, man. I remember that sheet, that chart that he has, it's burned into my memory. It's one of the five books that I think everybody should read in my reading list that I've sent to a million people. It's one of the five books that I think everybody should read. If you want a primer on exorisk, the precipice is the place to start. And yeah, one in six chance that humanity goes extinct within the next 100 years due to AI. And I think that the word the precipice is about a squeezed bottleneck, a treacherous path beyond which there could be this sort of glorious future. But right now, there is a very particular, very important forced function for, we don't know how long it could continue,
5:13 Chris Williamson? but it's definitely, we're definitely not far off it at the moment.
5:17 Speaker 00 Yeah, the way I visualize it, I like the precipice metaphor that it's this narrow path, sort of up a mountain. But if you've ever seen the movie Free Solo about, what's his name, Alex Holnold, climbing up Half Dome, right? I almost think of it as like, we've been doing a walk through the woods as humans for like, the last 200,000 years, and the level of risk we face is relatively low. And suddenly, we're climbing up Half Dome without ropes, or peatons, or any safety gear. And if we can just kind of make it to the top, then I think we'll be, you know, at a relatively lower risk state, hopefully in several decades. But a lot of people in the effective altruism movement think this is what they call a key
5:59 Speaker 00 century, a time of particularly high elevated risk when humanity has to be extra, extra careful and smart and risk averse and very, very self-aware about what we're doing.
6:13 Chris Williamson? Okay, so lay out the landscape of AI risk. Why is it something that we should be concerned about? How have we got ourselves here? What's changed over time?
6:24 Speaker 00 I think the basic intuition that lots of people are developing now is that there are gradations of intelligence in the natural world, right? Squirrels are smarter than squid. Monkeys are smarter than squirrels. We're smarter than monkeys. But we are not the ultimate level of intelligence. We can easily surpass human reasoning and planning abilities in lots of ways. And in fact, right? Everybody on their smartphone already has apps that are better at doing certain narrow things than we ever could. Like Google Maps is better at figuring out where to go than I would be with a paper map.
7:05 Speaker 00 Face recognition, right? Is way better than than I am at face recognition. I'm I've got a little bit of prospecnosia, like recognizing people. You know, like at the the Human Behavior and Evolution Society Conference we were both at two weeks ago. It's a little challenging for me. Computer vision systems have gotten very good at it. So one issue is we are approaching the point where AI systems are getting more and more general purpose and smarter and smarter across many, many different domains. And that represents a kind of major evolutionary transition in intelligence, where we could be outclassed. A second thing that I worry a lot more about than some people seem to
7:50 Speaker 00 is just the raw speed issue, right? If anybody's played around with ChatGPT, right? And you've asked it to write a little essay or an outline or any kind of language. You know, oh, my God, it is so fast. It's way faster at writing material than humans could be. And it's not even particularly optimized to be fast in that way. So what we're facing is potentially AI systems that are smarter than any human at doing a wide range of things, but also that are potentially 100, 1,000, maybe 100,000 times faster than humans.
8:29 Speaker 00 And the way I like to think about that is, you know, you might be familiar with the the speedster superheroes like the Flash, right, or Quicksilver, where once they do their kind of super speed and they're running around, it's as if everybody else is just frozen in place. I think when we confront very fast, powerful, general-purpose AI systems, that's the kind of situation we're going to be in. We're going to be outclassed not just in terms of intelligence, but also reaction speed. Right? You could potentially have an AI trading bot, right, that's trading equities or crypto
9:07 Speaker 00 or whatever, just way, way faster than any human can follow. You could have military AI applications, right, that can kind of simulate scenarios in terms of tactical applications or firefights, where they can kind of run through like every possible way that they could engage with an enemy and kind of spin out these simulations and then just completely outclass an opponent in terms of tactics and strategy. So I think these two things, right, AI is being smarter, potentially, and also extremely fast. It takes a while for the full implications of that to sink in, but I think they're
9:52 Speaker 00 kind of worrying.
9:53 Chris Williamson? Well, let's think about a JCB digger, right? It's larger and stronger than a human, but it is a tool that is under our command. So why should we be concerned about an AI which is faster and smarter than us? It just makes us do things quicker and better than we would have known how to do it, surely.
10:12 Speaker 00 Yeah, if we're just outsourcing, you know, suggestions to the AI, like with Google Maps, we're saying, how do I get from A to B, right? And Google Maps is not actually taking control of our Tesla. It's not driving for us. We're not outsourcing our agency, right? That's relatively safe, right? The AI could still manipulate us in lots of ways. It could nudge our decisions in certain directions for its own ends or for the interests of somebody else, like the people who develop the AI. But once you give it agency, once you outsource decision-making powers, that's where I think the real danger starts to come in.
10:58 Speaker 00 So for example, you know, there's a big difference between like a military AI that has the ability to suggest certain courses of action like steer this carrier strike force towards this ocean for this purpose. Right. Versus an AI that actually has control and decision-making power over like, okay, it's time to launch a bunch of F-35s to go bomb this country. The trouble is because of the huge speed advantage of AI, there will be very, very strong incentives to give it agency. Right. To make it able to make decisions in a sort of
11:39 Speaker 00 perception, decision, action loop that's like way faster than a human could do.
11:46 Speaker 00 There'll be commercial incentives to do that if you're in any kind of competitive environment like finance or military applications or even you know, market research. So that you can do
12:02 Speaker 00 optimization of some, you know, anything you want to optimize better by, by sort of outsourcing some agency to the AI.
12:12 Chris Williamson? As of yet.
12:12 Speaker 00 That was a little vague, but if, if that made sense, let me.
12:14 Chris Williamson? It does. But as of yet, you haven't talked about artificial general intelligence or some Terminator Arnold Schwarzenegger apocalypse or it becoming self-aware recursive self-improvement, machine extrapolated volition. I haven't heard any of that. So presumably in between now and a concern that I think was very prevalent, maybe a decade ago or so, when Toby or Nick Bostrom's superintelligence book came out, which was that once you reach singularity, there's going to be all of these problems. It seems like there's some pretty dangerous gradations between now and there. So what, what's happened in the world of AI risk concern over the last decade in terms of how that
12:58 Chris Williamson? was a problem, it's changed. And then what, what are the gradations between now and where we could get to in future?
13:05 Speaker 00 Yeah, so when Nick Bostrom, an Oxford philosopher, published Superintelligence, which is a great book, another must read, it's a little technical, it's challenging.
13:15 Chris Williamson? Audible it, audible it, don't try and read it.
13:17 Speaker 00 Okay, yeah, that was 2013. Bostrom emphasized very heavily the notion of like a self-improving takeoff, where like an AI could get pretty smart and then it starts optimizing its own software and potentially hardware, and then you get a kind of explosion of capabilities that could potentially happen very, very quickly, like the AI could sort of go from like human level intelligence to superintelligence that's smarter than all humans who have ever lived in a matter of perhaps days or weeks or months. Now, that's a super scary scenario. I think it's legit to worry about it. I take a slightly different view, and I think a lot
14:01 Speaker 00 of people are also taking a different view, that even fairly narrow AI, that's like not even as smart as humans in many ways, could still be extremely dangerous. For example, if you have a narrow AI that is very, very good at designing bioweapons, right, at doing kind of gain-of-function research through computer simulating, here's how a virus could spread through the world, you could have like a bioweapon, narrow AI, that actually invents an extremely dangerous pandemic, right? And you might think, nobody sane would want to do that. Well, there are lots of insane people in the world. There's lots of nihilists and terrorists and, and you know, degrowthers and people who don't like human civilization,
14:46 Speaker 00 who might be motivated to use that kind of narrow AI. You can also have narrow AI applications that could be very politically destabilizing. You know, you could have extremely good video deep fake technology that makes it look as if Vladimir Putin or Xi Jinping or, or, or Joe Biden has declared war on some other major nation state, and that provokes an extremely panicked military response, right, that could lead to like global thermonuclear war. And we're not really very ready for that kind of thing. And, and that stuff could happen.
15:29 Speaker 00 You know, within a couple of years, like people debate, when will we get artificial general intelligence? It could be five years, 10 years, 50 years, who knows? But these narrow AI applications are coming down the pike very, very quickly and they could be pretty destabilizing.
15:46 Chris Williamson? What has happened with the development of neural networks and large learning models that maybe caught a lot of AI X risk, safety, alignment researchers unawares and kind of how's that changed the viewpoint of concerns around AI?
16:08 Speaker 00 What we're basically seeing is very rapid advances in hardware, you know, memory size and speed. So when I was doing neural network research back in the late 80s, right? We were playing around with networks that had like maybe a few dozen, maybe a few hundred units and if maybe a few hundred or a few thousand parameters, meaning the weights in the network that kind of connect the little simulated neurons together. So a few thousand parameters back then. Now it's like trillions of parameters in the large language models because the hardware is just better, right? We can make GPU chips that are much, much
16:48 Speaker 00 more powerful. And once you get to a certain size of network with enough hidden layers learning, using the deep learning methods, you get kind of powerful emergent properties popping out that nobody was really quite prepared for. And that's where we've been blindsided by ChatGPT.
17:09 Chris Williamson? What like?
17:11 Speaker 00 Just like, people kind of thought, well, if you make these large language models kind of predict like the next language token and you feed them a lot of information about the whole content of the internet, then maybe they might be able to answer some basic questions, right? Kind of like a Google search. But lo and behold, they're way more powerful than people expected, right? Like you can
17:40 Speaker 00 Ask ChatGPT to do all kinds of things that really wasn't designed to do that people didn't kind of grow it to do, like write an outline for a screenplay, figure out how to write a summary of AI existential risks. GPT is pretty good at that in a kind of self recursive way. It can do math, it can do computer programming, it can do an awful lot of capabilities that people really, they thought like, maybe that's 10 or 20 years away, and boom, here it is in 2023, available to everybody, available to the 100 million
18:19 Speaker 00 users of ChatGPT. So that's where the shock has come, that once you get to, you know, a multi-trillion parameter, large language model, man, it's looking pretty close to artificial general intelligence. It's not quite there yet, and it's still fallible, and there's still a lot it can't do. But, you know, if you, if you had a time machine, and you took ChatGPT back in a laptop to 10 years ago, in 2013, and you ask people, does this look like artificial general intelligence? They'd go, holy moly, yeah,
19:00 Speaker 00 that's pretty effing close. That's way more advanced than we expected it would be by 2023. So that's the concern. Like, the fact that you can be so surprised by the pace of development means maybe the next step, whatever that is, is going to blindside us as much or even more.
19:24 Chris Williamson? And this is the same group of people who said it's one in six as the chance that humanity goes extinct due to AI within the next century. So you
19:36 Chris Williamson? I don't think by anybody's standards, that would be considered a conservative estimate. It sounds crazy, you know, you're rolling a dice and on one side is a button that destroys humanity. So, you know, again, just that that's quite a stark claim, I guess.
19:53 Speaker 00 Yeah, one in six is literally Russian roulette. So if you're playing Russian roulette and and the head is not like a single individual's head, but your whole species, that's that's what you're doing.
20:04 Chris Williamson? So what is the definition of AGI and why doesn't ChatGPT yet breach it?
20:16 Speaker 00 AGI means artificial general intelligence. And the way that the research groups themselves, like DeepMind and OpenAI and Anthropic, the way they define it is basically an AI system that can do just about everything a human can do in terms of cognition and perception to about the standard that a professional would do it in a job, right? So with an AGI, you should be able to train it to be as good at medical diagnosis as a doctor, or as good at teaching a class as as I am as a professor, as good at playing chess as a chess grandmaster,
20:58 Speaker 00 as good at trading equities as a good Wall Street trader, and everything else, right? And the real power comes from the fact that once you have an AGI that can do all of that, that's fairly general purpose, you can copy it, right? You can copy it and you can make one copy of it do one human job, another copy do another human job. And then they could potentially even trade information about how you how you do this. So, you know, the explicit goal of OpenAI and its its CEO, Sam Altman, is create AGI as fast as possible
21:41 Speaker 00 and make it so it can automate most human jobs as as fast as possible.
21:50 Speaker 00 So that raises lots of issues about unemployment. But it also raises lots of issues about existential risk. Because among the many jobs, right, that an AGI could learn to do would be like be a really good terrorist at making bombs, be a really good military strategist in terms of overcoming Russia or China or America, be a really good spy, be a really good propagandist who can shape the outcome of elections, et cetera. So the point of AGI is it should be able to do anything people can do about as well as people can do it. And that includes not just all the good stuff, right?
22:34 Speaker 00 Like being a good veterinarian who fixes dogs, or being a good nurse who takes care of people with psychiatric disorders. It also means all the bad stuff that bad people can do.
22:46 Chris Williamson? What do you think of Sam Altman?
22:49 Speaker 00 I think he's a brilliant guy, and I think he has what to him is a compelling vision of the future. And I think he genuinely believes that developing AGI will lead to a kind of awesome human utopia. He talks as if he understands the extinction risks. But I don't, I think there's some deep, deep cockney dissonance, right? Because on the one hand, if he really took the extinction risks seriously, I think he would shut down OpenAI, they would no longer do research, they would say, this is radioactive,
23:33 Speaker 00 this is toxic, this is crazy, this is Russian roulette, let's not do this. And he's not doing that. He's not shutting it down. He's sort of pushing more or less full steam ahead, and he's making some little noises about we need to be careful, and we need to be, you know, we need to get good regulation, blah, blah, blah. But I don't think it's really a heartfelt appreciation.
23:58 Speaker 00 Of the kind of Toby Ord point that dude, you're talking about one in six at least risk of human extinction this century. To most people in the world, that sounds absolutely insane and reckless. And it's something we did not consent to.
24:14 Chris Williamson? What do you say to the people that would push back and say, well, look, Sam Altman, at least he's fighting for the good guys. He's somebody that's from the West. The choice is between us getting there first, or Russia getting there first, or China getting there first. This is a winner takes not only all, but takes the world and takes the world for the rest of time. And then you've got potentials of bad social lock in, or whatever it's called that William McCaskill came up with. He's on the side of the good guys. If the good guys don't win, the bad guys win. Therefore, we need to make sure the good guys win.
24:51 Speaker 00 I think it's worth asking, okay, like, I'm American. Lots of Americans think we are automatically the good guys by definition. Anything America does is good. Anything that a rival does is bad, and we must win at all costs. One thing I would say is, you know, if it's an arms race into a concrete wall, it's not really a race you want to engage in, right? It's kind of like a game of chicken, but where nobody can swerve. I think if the likely outcome of the arms race is extinction, then everybody who's contemplating the arms
25:34 Speaker 00 race should try their best to avoid to avoid getting caught up in it.
25:38 Chris Williamson? How are you going to coordinate this? Are you going to coordinate China and Russia to get on board with your very laudable altruistic desire to get Sam Altman to just go to the Canary Islands for the rest of his life without a laptop or an internet connection?
25:55 Speaker 00 I think there's the traditional approach to this, right, is what's called the AI governance model, where you get a bunch of policy wonks and Washington DC insiders and people in the UK government and Whitehall thinking really hard about how to regulate AI so it's benign and how to reduce the arms race dynamics. And I think that's fine and that's a good worthy thing to do. Unfortunately, I think it's way too slow and it's way too easy for the AI industry to to sort of capture what happens in their own interests. And honestly, the politicians who are involved in this simply don't understand AI well enough,
26:40 Speaker 00 I think, to have a very sensible way of approaching this issue. A second strategy, which is something I've been advocating recently, is a little more informal, a little more grassroots and kind of bottom up, which is I think there are certain industries where it's okay to just look at them and go, that's bad, that's evil, that's reckless, we should stigmatize that industry, we should stigmatize everybody who works in that industry, who supplies anything necessary to that industry, who finances that industry, and it's just a bad industry and we want to try to slow it down.
27:22 Speaker 00 We have done that with many, many, you know, industries in the past. Crypto has been handicapped very, very successfully by adverse PR and moral stigmatization campaigns by politicians and central bankers and so forth. The arms industry, right, has been heavily stigmatized. There's lots of ethical investment criteria for investors that say things like, hey, let's not invest in alcohol, tobacco, gambling, arms trading, human trafficking, et cetera. Right? And I think at the moment, there's quite a bit of popular opposition to the AI industry and concern about these risks.
28:06 Speaker 00 And I think we should kind of normalize.
28:10 Speaker 00 People being able to say, I didn't vote for this. I don't support this. I don't want these extinction risks imposed on me and my kids. And the people who are doing it should be stigmatized.
28:23 Chris Williamson? I suppose one of the problems is that with weapons, or crypto, or gain-of-function research coming out of a BSL Level 2 lab, or something that's not sufficiently secure, people are able to observe, experience, and imagine the problems quite easily. At the moment, all that they've found is a really cool way to get them to write short essays, or tell them jokes for their best man's speech. So, as of yet, the experience of AI, especially neural nets and large learning models, has not been what you are concerned about moving
29:06 Chris Williamson? forward, which means, where is the incentive for people to get on board with this?
29:10 Speaker 00 Yeah, I think there's an important role here for imagination, and fiction, and scenario building. So, remember when people were very worried about global thermonuclear war? Like, when I was in college in the 80s, this is all we talked about, and all we worried about. It's like, how long is it going to be till the US and the Soviet Union have a massive exchange of ICBMs, and we all die? We could visualize that stuff very clearly, because Hollywood movies and TV series did a pretty good job of showing what that would look like. None of us had personal experience of Hiroshima or Nagasaki, right? We kind
29:53 Speaker 00 of read the accounts, and it was horrible. But when you're trying to imagine potential harms from new technology...
30:02 Speaker 00 All you really have to go on is sort of what experts say the risks are. And then the way that that that screenwriters and directors visualize those risks in a way that the the public can understand.
30:15 Chris Williamson? I don't disagree. The problem you have is that you're not only trying like, no one has ever said, look at that atomic bomb. It gives me so much pleasure and it's so cool. No one has ever said the same thing about an engineered pandemic. But when it comes to chat G P T and its iterations downstream, people are finding benefits in it in the now. And they're trying to be told you need to let go of this thing which you see as a positive in order for something which you can't foresee as a negative.
30:45 Speaker 00 Yeah, and this is where I think it's important for people to kind of tap into their their kind of, I think, increasing sophistication that there can be very, very seductive technologies that can have very toxic kind of social side effects, even even that, you know, the discussion about social media itself, I think, has moved from, oh, wow, this is cool. We can connect with our grandparents and we can find people to date to, oh, my God, is this creating like mass mental illness in Gen Z in a way that we we really need to rethink how TikTok and Instagram and so forth operate. So I think people have the ability to understand some
31:29 Speaker 00 new technology can be very seductive. It can look great. It's new and shiny. But, you know, there might be kind of a viper hiding inside it that could be pretty poisonous.
31:48 Speaker 00 The other thing I would I would add is, you know, when I was I was teaching online courses for Chinese University of Hong Kong, Shenzhen, a couple of years ago, and we talked quite a bit about A I extinction risks. And this is a bunch of Chinese undergrads in Shenzhen, very smart. They understand existential risk. They're tuned in to nuclear war. They understood bioweapons because the COVID pandemic was raging. And they think quite a bit about A I, right? Because China had said, we want to be the leading A I superpower by 2030. That was their plan a few years ago.
32:26 Speaker 00 And these Chinese students were like perfectly willing to understand the risks and to stigmatize AI. Right? They had a kind of moral imperative and concern for the future of humanity that was just as strong as what my my American undergrads have. So I think we have this stereotype in the West that other countries like China are sort of full of
32:54 Speaker 00 unthinking automatons who have been programmed by their government and their authoritarian regime so that they will just do whatever, you know, Xi Jinping says. But I think actually there's quite a bit more room for a kind of global grassroots opposition to AI, not just in America and Britain, but also in other countries that are kind of key, you know, potential arms race players.
33:25 Chris Williamson? Well, isn't China a country in which they've refused to make public a lot of the neural nets and the developments in AI, unlike in America where citizens can just go in and play around with it? China said, well, we can't be sure that this thing isn't going to start telling everyone about Tiananmen Square. So because we don't have control over it, we're not going to let it loose with the public, which is it has to be curtailing the development because it's not able to do any of the learning that it would have done had it have been iterated over 1.25 billion users, all of whom are asking it how to make a cake this evening.
34:06 Speaker 00 Yeah, I think it's
34:10 Speaker 00 it's very interesting because insofar as the Chinese government really wants AI mostly for purposes of of of social control and social stability and and censorship and reducing crime and reducing terrorism. Right? To have a much narrower range of AI applications than your typical American AI company wants. Right?
34:32 Speaker 00 Chinese government, I think, is not really interested in the kind of techno-utopian vision of like the singularity and transhumanism and let's all like upload our minds into the matrix, the way that you see in a lot of the Bay Area AI enthusiasts, right? They're not really into that. They just want like China to be stable and prosperous.
34:57 Speaker 00 So here again, what's happening, I think, is the American AI companies are at the cutting edge, right? They're far and away more advanced, as far as we know, than anything happening in China, much less Russia or Brazil or the UK.
35:15 Speaker 00 And they're just kind of trying to play catch up. What's happening is like we're far out in front in terms of the arms race. If I was a Chinese policy expert, I would be freaking out. I'd be wanting to play catch up. I'd be very concerned about AI having a kind of a, America having a kind of AI hegemony. And it would, it would look like a threat to me that I have to respond to. So we are setting the pace in terms of the arms race. If we slow down, right? China might go, huh, thank God. Like, we don't have the talent or the resources to play this arms race. We can, okay, if America's relaxing, we can relax too. We can focus
35:58 Speaker 00 on other issues like, no, how do you get Chinese people to have kids? Stuff like that, right?
36:06 Chris Williamson? Use AI.
36:08 Speaker 00 Yeah.
36:11 Speaker 00 AI matchmaking, right? That might, that might be a key, key application for them.
36:17 Speaker 00 So I think here again, Americans have to look in the mirror and go, to what extent are we really setting the pace with the arms race? Are we forcing other countries to try to catch up?
36:25 Chris Williamson? That's a really good point.
36:26 Speaker 00 Maybe if we slow down, we can all, you know, take a step back, take a breath.
36:34 Speaker 00 Pause, think, what exactly are we doing? Should we be playing this Russian roulette?
36:40 Chris Williamson? Yeah, that's a very good point that I hadn't I hadn't considered. I suppose as well that the wake of what is on the internet and what is available, what is public knowledge about what's happening at all of these different neural net companies will be slip streamed in some regard by foreign actors. You know, the the source was, I think, a long time ago, open sourced, and then it's no longer open sourced, which means that you can't fully see what's going on side in terms of workings. But I'm sure that you can determine a good amount of stuff, or at least a non insignificant amount of stuff from that.
37:15 Chris Williamson? One of the words that we haven't used yet that would have been used an awful lot 10 years ago was alignment problem. So how relevant is discussing the alignment problem now? And actually, before you can get to that, does it look like if you were to put your money on the table, the front runner contender for creating AGI, would it be neural networks? Can a large learning model become sentient and be the AGI thing?
37:51 Speaker 00 I don't know on this point, right? There's a big variety of opinions. And on the one hand, people like my friend Gary Marcus at New York University argue that large language models just based on deep learning and just based on neural networks cannot do many of the key cognitive tasks that humans do, so can't reach AGI. And he's got various arguments for that and some pretty good recent books on AI. On the other hand, you have people saying, well, people keep underestimating deep learning and keep underestimating what it can do. And as far as we can see, there aren't really very many hard constraints, you know, against it. And moreover, the human brain.
38:37 Speaker 00 Looks an awful lot like a large neural network. We don't exactly learn using deep learning methods as, you know, GPT does. But we're basically just a bunch of, you know, neurons with connections.
38:53 Chris Williamson? So what do you, what do you think? Is it just, can we just layer transistor upon transistor upon reinforcement? And will we arrive at something that approaches sentience and or AGI?
39:06 Speaker 00 Well, most of the work I did in neural networks back in the day was, was based on the assumption that, no, you can't just train a big, random, kind of incoherent, formless, blank slate neural network to do anything. It needs a bunch of structure. It needs an architecture. It might even need an evolved architecture where you have to try a bunch of different ways of wiring up large networks before you can get something that's really smart. Now I'm more humble. I don't know. You know, I was excited about that because I studied a lot of evolutionary biology and animal behavior and it looked like, uh, simple nervous systems often had quite a bit of architecture to them.
39:51 Speaker 00 You know, you look at a bumblebee or an ant nervous system and it's, it's highly architected. It's not just a glob of neurons. There's like ganglia doing different things. There's perceptual, um, clumps of nerve cells doing particular things. But at this point, I don't know, like, it wouldn't surprise me that much. If, you know, GPT eight or nine proved to be able to do pretty much everything that, that humans can do without having a whole bunch of intrinsic structure. On the other hand, Gary Marcus might be right, um, that you actually need a radically different approach.
40:31 Speaker 00 I'm not sure it matters all that much, though, in terms of the AI safety issues. Cause one way or another, the AI companies will figure out how to do AGI. If, if we give them the money and the talent and the resources and the social support, they will figure it out sooner or later.
40:51 Chris Williamson? Why haven't we talked about the alignment problem? Why is that a term that I'm seeing less of as well? Is it just that there's so much AI progression that it's not a problem? Or is, is there a new issue with regards to this that neural nets have created?
41:07 Speaker 00 People are still talking about AI alignment. It's just, um, there's so much more buzz right now about, you know, the amazing new capabilities, not just of the large language models, but also of like Mid Journey and DALL-E and the other generative AI systems for creating images and videos and the ones that can create music and amazing audio. So people are excited about that. People are extra, extra alarmed about the AI risks. And everybody interested in so-called AI alignment is still trundling along, working on it, right? I'm still writing essays about AI alignment.
41:48 Speaker 00 It's just gotten slightly overwhelmed by all the other, you know, shiny new issues.
41:53 Chris Williamson? What is AI alignment for the people who aren't inducted?
41:59 Speaker 00 The basic notion is how do you get an AI system to be quote aligned with human values and preferences and goals? And it's kind of like the, the so-called principal agent problem in, in companies. Like if you have a bunch of investors, right? And they are supporting a company and you create like a board of directors and the board of directors gives power to a CEO. How do you make sure the interests and decision-making of the CEO are aligned with those of the board of directors and in turn with shareholders? Right? That's a kind of corporate alignment problem. A political alignment problem would be how do you elect a president that actually
42:44 Speaker 00 serves the interests of the voters instead of their own interests? In terms of AI, it would be how do you create an AI system that actually respects what the, uh, the human users want it to do? Or kind of what they really want it to do, even if they can't quite articulate everything about what they want it to do.
43:10 Speaker 00 Right, and there's, there's lots and lots of myths and legends going back thousands of years about the pitfalls of getting some power, like some genie that pops out of a lamp, where it says, you know, what wishes would you like?
43:25 Chris Williamson? King Midas turns everything into gold.
43:27 Speaker 00 Right, and you, you give it wishes and then it, it interprets what you want sort of over literally, right, in a way that's absolutely disastrous. So AI alignment in principle is about making sure the AI is kind of doing what you want and, and even in ways that you can't necessarily articulate, that fit with a whole background of like human common sense and, and moral norms that we might not even be able to articulate to the AI system, but that we would still want it to act as if it understands.
44:02 Chris Williamson? Yes, however, different individual humans have got different values and conflict of interests happen a lot. So which human values should AIs be aligned with?
44:16 Speaker 00 Yeah, so this is exactly the issue that I've, I've been writing about in my Effective Altruism Forum essays for the last year or so. I've pointed out, like, you can say, we want the AI to just respect the views of its, its end user, the consumer who actually buys it, but what if the end user is a terrorist? What if the end user is a political propagandist? What if the end user is in a hostile nation-state? What if they're a bad actor, right? Do you really want the AI to do what they want it to do? And second, if you try to aggregate like the collective will of humanity, right, where
44:55 Speaker 00 you go, well, the AGI should sort of do what people in general would want it to do if it could summarize all of our preferences. Well, then you get into interesting issues like, well...
45:10 Speaker 00 Men and women have like different interests. So should the AI go with the 50% of people who are male or the 50% who are female or whatever? People have different political views, right? And in one essay, I pointed out 80% of humans are still involved in organized religion. Right? And yet, the AI industry is dominated by secular atheists who largely have contempt for religion. So if you have an AI that's trying to be aligned with people in general, and people in general have religious values that are being completely ignored and dismissed and mocked by the AI industry itself. Right?
45:54 Speaker 00 How do you actually get alignment with with people? They don't really mean alignment with what people believe. They kind of sort of mean like, we want the AI to be aligned with what a good, liberal, secular, humanist, democratic, Bay Area tech bro values. Right? That's really the bottom line. That's that's what AI alignment actually boils down to in practice.
46:21 Chris Williamson? Yes, that does it split the difference? I remember to not spoil the end of superintelligence, but the like the good guy that that comes in at the end is machine extrapolated volition, which is in a world in which we can't be sure that we're going to give a program an instruction, and that program is going to take the instruction and turn us all into paperclips or or kill us so it can make a cup of coffee.
46:47 Chris Williamson? Telling the machine, do what you think we would have asked you to do had we have had sufficient wisdom to ask you to do it. That is roughly machine extrapolated volition. But even that modeling human preferences needs to be done based on a group of humans. And which preferences do you mean? And when preferences come into conflict, which ones win? And if it's 50 50 between men and women, and you split the difference between the two, like, is that optimal? Is that actually optimal? Or is it more optimal to swing it to more toward one way or another? One group might be incredibly vehement about something, but it might be immoral. Okay, so given the fact that we've been trying to work out ethics and virtue and morality ourselves philosophically for thousands of years
47:30 Chris Williamson? and have made some progress, but not got anything that's too definitive. Please try and put that into code for me. And which morals do you mean? You mean the morals of modern, secular, 21st century, Western, industrialized, educated people? But why don't we use the ones from the Roman era? Or why don't we say, well, let's wait for another 3,000 years and see which ones come up there?
47:55 Speaker 00 Yeah, and I think there's two additional sort of alignment problems I've been worried about. One is what I've called embodied values. So when we think of values and preferences, we typically think of things we can articulate verbally. Like if someone says, what do you want for dinner? Like, we have a verbal answer we can give. But I made the point that like, our brain is only 2% of our body mass, right? Our body is full of all these other tissues and organs that have evolved their own kind of values and agendas in a way, right? And I call these embodied values. Like what your immune system really wants to do is fight off pathogens that might infect you.
48:36 Speaker 00 So ideally, you'd want an AI to be aligned, not just with the values that our brain can articulate through our words, but that is also kind of like biomedically aligned with the interests of our bodies, right? So that we are kept healthy and well and live a long time. Now, if you ask people, how do those embodied values actually work? Like, we have no idea, right? We can't even articulate those. So methods for training AI system based on human verbal feedback cannot even in principle align with all these embodied values of our bodies.
49:17 Speaker 00 A second issue is, you know, from an evolutionary viewpoint, the development of AGI would be a major evolutionary transition. It's a big thing that's comparable to the evolution of like DNA, or the evolution of multicellular life, or the evolution of nervous systems. Like, it's a big deal that doesn't just affect humans. It also affects like the other 70,000 species of vertebrates and the other 10 million species of invertebrates and all the life on the planet.
49:47 Speaker 00 Now, if you ask, okay, how do you align AI not just with humans, right, but all the other kind of organic stakeholders, all the other life forms on Earth, who might be affected by AI? Okay, how does the AI learn the true interests of an elephant, or a dolphin, or a termite hive, or, you know, all the other life forms that that matter? I've never seen anybody in the AI industry even sort of seriously talk about this. And yet, they portray AI as this kind of world historical thing that will re-engineer the entire, you know, planetary ecosystem.
50:30 Chris Williamson? Yeah, it could quite quickly become the most super intelligent, powerful Greta Thunberg ever created, just screaming from the top of some transistor hill. I I read a thing from Bern Hubert and Tobias Huber talking about AI safetyism being more of a risk to society than anything that AI doomers are predicting. Why, it seems like you've put forward a case that suggests this is something we should be concerned about, et cetera, et cetera. What is the other side of the fence to this? What is the side of the fence that says, stop shitting yourself about safetyism? Let's just crack on.
51:13 Speaker 00 I think if I was going to like, put forward the best possible case against myself, right, to steel man the the anti-AI doomers, I would say something like,
51:27 Speaker 00 if you pause or stop the development of AGI, there are huge potential opportunity costs. There are lots of human problems maybe that AI could help solve, right? And the usual ones that people talk about are, oh, AGI could help us like solve climate change, it could help create peace and prosperity, it could reduce global conflict, blah, blah, blah.
51:52 Speaker 00 I do not actually find those very compelling. I think there are often ways to to solve lots of those problems without developing AI. The one potential application that does give me serious pause that I've talked and thought about a lot is longevity issues. Okay, if you had an AI that could seriously help.
52:15 Speaker 00 Develop longevity treatments and anti-aging treatments and help regenerative medicine research and help biotech so that we don't all have to die, right? I'm 58, I would like not to die. I would like to live another 100, 200 years. However long I want to live, I would love to live that long. Maybe if we pause AI, I don't get that benefit, right? Now personally, I'm willing to die for my kids. I'm willing to forego longevity treatments if we reduce existential risk. Some people might have a different view. Some people might be like, well, screw you, Miller. I don't want to have to die just because you're scared of AI. I understand that. I respect that viewpoint.
53:00 Speaker 00 Let's have that debate. Let's talk about it. My personal hunch is that if we invested as much directly into longevity research as we're investing in AI, we could actually probably fix aging within like 20 or 30 years. But people, what's happening instead is people are like, we know it's really hard to get people to directly support longevity treatment. People are in a kind of pro-death trance. They think it's weird and creepy to head towards immortality. So we know we can't sell that as a research program, right?
53:41 Speaker 00 So instead, we're going to sell them on AI and then AI will like magically deliver the anti-aging cures, right? I've often seen this argument, right? People won't directly support longevity research, but AI can solve longevity. So that's why we need AI. Does that make sense?
54:02 Chris Williamson? Yeah, it does. It's, um, I don't know, man, I'm really trying to stay open-minded to this, but coming from a Bostromite background, you know, I've always for nearly a decade now been so wary and touchy-feely about anything that even looks remotely smart that comes out of, you know, Siri. Siri was something that I remember there was a, oh my God, like, you know, is Siri going to be conscious? What sort of problems are going to be caused, et cetera, et cetera?
54:32 Chris Williamson? But there feels a part of me that's like, well, look, if you're going to work incredibly hard to fix longevity problems so that you keep people alive, you keep some people alive that are alive right now, and the outcome of that is that everybody is dead within two centuries, that seems like a bad deal. It seems like a relatively pointless deal. So we're talking about terrible outcomes that could occur, the tip of the spear or the top of the mountain, so to speak. Looking at some of the stuff that's going to occur in the interim between now and then, you know, because this is, how many transistors is it going to take? Will the large learning models get us there? Is it GPT-8? Is it GPT-never?
55:16 Chris Williamson? But some of the things that we probably can be certain about are what the tools and the technologies are enabling at the moment, stuff like deception, deep fakes, information sanitization and misinformation, elections, politics, persuasion, friend bots, loneliness. What are you most concerned about that is very high likelihood? Maybe it's already here. What are the things that people should be looking out for with the rise of AI over the coming years?
55:45 Speaker 00 I think the 2024 election cycle in America is going to be absolutely wild and shocking and is going to involve a lot of narrow AI applications in political propaganda and ads and deep fakes and speech writing and sort of the mass customization of political propaganda. Because one thing AIs can do is track, you know, individual preferences and values and priorities through social media interaction and get a pretty good model of what every potential voter really cares about. And then potentially customize political
56:27 Speaker 00 messaging directly to each voter in a way that like pushes their own hot buttons very effectively, right? So like if you're a Democrat and you care a lot about racism, but you don't really care about abortion, then you'll you'll start to see customized ads that are like very anti-racist, but they don't talk about abortion. That will be new. We haven't seen that before. Before, political ads were kind of like lowest common denominator TV spots or magazine ads that sort of tried to hit like the typical undecided voter. But now, I think in 2024, my humble prediction is, we're going to see a lot
57:09 Speaker 00 more narrow AI used for purposes of political manipulation. And I think people will be shocked at how effective and persuasive it is. And I have no idea what the outcome of the election will be, but I think the outcome for a lot of voters will be to go, oh my God, we like don't have a sort of political immune system that can fight this very well.
57:38 Chris Williamson? Does that mean that AI at the moment has got theory of mind?
57:46 Speaker 00 So theory of mind is like your ability to understand the beliefs and preferences of others. I think large language models do have, pragmatically speaking, theory of mind in that they have absorbed, you know, all the lessons that are available from the internet about how to understand people's beliefs and desires. So I think if you ask, like a GPT, can you please write really good ad copy to advertise a particular good or service? It's pretty good at that. Like, I know people in advertising who are like, oh my God, GPT is better at ads than than than we are.
58:28 Speaker 00 And so we have to use it because our rivals are using it. And I think that also applies to political ads and speech writing.
58:35 Chris Williamson? Yes, functionally, it is able to achieve the same thing as somebody that has theory of mind. And this comes back to, you know, a conversation that everyone is having, which is when do you know if something is conscious or not? You know, the Turing test turned out to kind of be a bit of a Insufficient barometer for working out whether or not something is sufficiently intelligent, because there's no way that you can speak to a well-trained ChatGPT model and not think that there is something inside of that. But you don't know if there is any there, there, right? Is this just a P-zombie? Is it just making all of the movements and the sounds and the shakes? And you roll this forward into a sufficiently fleshy, sufficiently
59:17 Chris Williamson? pink, sufficiently right proportioned robot that can move around your house. And you go, that's a human. That's a human. I know it's a human. It looks like a human. It walks and talks like a human. So at the moment, yeah, that the function and the outcome that these things are able to achieve is so close to somebody that is able to do that. So 2024 election, lots of persuasion, lots of speech writing. What about social media and the news landscape and the information landscape and people both producing and consuming content? You know, we're coming, we're still in this world of content creators at the moment. How do you think that that's going to be influenced?
59:59 Speaker 00 You know, I think what we've got at the moment is, well, you know, Noam Chomsky wrote this book 30 years ago about, you know, manufacturing consent, about the way that in so-called democratic societies, the way that power actually works is through subtle semi-voluntary propaganda, right? People are willing to go to school to absorb the indoctrination that we get in public schools. And we expose ourselves voluntarily to certain kinds of newspapers and magazines and TV news shows. And then they sort of guide us into what we should think and value. And the engineering of all of that has been
1:00:43 Speaker 00 done before by humans, by people who do, who are Hollywood screenwriters and political speechwriters and ad people, marketing people, public relations people, right? There are millions of humans involved in that venture of trying to manipulate public opinion in various directions. But now they're going to have these AI systems. That hugely increase their reach, their ability to customize messages to particular individuals, their ability to capitalize on big data gathered through social media
1:01:23 Speaker 00 that can do extremely fast iterative testing of messaging, right? Not just split tests or just focus groups like in traditional marketing, but that do the kind of things that Facebook does in terms of testing all the time, which of millions of ads are most effective. So we're going to have kind of this AI-powered hegemony warfare, like worldview warfare, where people are advocating their political, ideological, religious belief systems and they're fighting against enemies, right?
1:02:03 Speaker 00 And there's going to be a massive, ongoing culture war, which in a way is the only war that matters anymore. But it's going to be heavily, heavily shaped increasingly by AI tools.
1:02:16 Chris Williamson? What about friend bots? People who just abscond the real world for some virtual best mate or a girlfriend or whatever in their apartment? Do you think that this is something that's likely?
1:02:29 Speaker 00 Yeah, absolutely. This is something I wrote about in my most recent essay on a likely anti-AI backlash. One thing that will provoke, I think, a backlash is people using AI as fake boyfriends, girlfriends and friends and getting a kind of pseudo-intimacy and sort of validation from AI systems.
1:03:03 Speaker 00 And you might think, oh, surely people can't be that gullible that they would prefer interacting with just a smart chatbot. But if you think, like, we're talking about a chatbot that could potentially remember every single personal detail about you, that remembers all previous conversations with you, that can try out different ways of interacting with you to see what you like and what you don't like, that is much more attentive than any real-life boyfriend or girlfriend, that has infinite patience for listening to all your shit and all your dumb stories and all your neurotic woes, right, in a way that no living lover would ever put up with. And it would be like the perfect, you know,
1:03:48 Speaker 00 combination psychotherapist and friend and girlfriend and mentor and confidant and all of that. And we're pretty close to being able to do that. So I do worry that real-life social interaction will look like a pretty poor substitute for those kinds of AI pals.
1:04:11 Chris Williamson? Why would that cause a backlash?
1:04:14 Speaker 00 I think it'll be a backlash because people who don't get caught up in having an AI boyfriend or girlfriend will look at people who are caught up in it, like Gen Z staying alone in their apartments and never going out on dates and not getting married and not reproducing. And they might go, oh, my God, this is the most socially toxic technology ever invented. Like, the birth rate is dropping. Nobody's dating. Nobody's having real relationships. This is not sustainable. And therefore, we are going to do a moral backlash, or religious backlash, or political backlash that says, this is not the way we want civilization to go.
1:04:56 Chris Williamson? I saw a headline which you'll have seen, 50% of AI researchers believe that there is a 10% or greater risk that humans go extinct due to our inability to control AI. And the Centre for AI Safety had a ton of researchers agree with the statement, mitigating the risks of extinction from AI should be a global priority alongside other societal scale risks such as pandemics, nuclear war. The open letter has been signed by more than 350 executives, researchers and engineers working in AI, plus Elon Musk as well was a part of this movement. How effective are AI experts at predicting this AI development and the risks moving forward?
1:05:39 Chris Williamson? And if 50% of the people say that there is a 10% chance that the technology that they're working on is going to end us, what happens next? Are there going to be picket lines of disgruntled terrified AI researchers outside of OpenAI's lab? What's happening?
1:05:57 Speaker 00 Yeah, I mean, there is a movement actually to do literal protests and demonstrations and picket lines. I'm involved in a Slack group that, you know, some members of that are actually organizing like pickets in front of OpenAI or in front of DeepMind in London.
1:06:17 Speaker 00 Now, of course, the point of these letters, and I think I've signed all the letters that are going around myself, we know the letters themselves will not stop the AI industry from doing what it's doing. However, the whole point of the letters is to draw public attention to the issue, right? To get press coverage, to get the general public thinking about these things, you know, hopefully tuning into like podcasts like this and reading the books they need to read and becoming...
1:06:47 Speaker 00 Politically motivated to take this seriously as as an issue. And I think in that regard, the letters have been surprisingly effective, like the amount of press coverage given to AI risk in the last couple of months is hugely greater than anything in like the previous 10 years. And the government response, like, you know, Biden inviting AI industry executives to the White House and Prime Minister Britain taking this very seriously and tweeting about it and having like AI safety summits in London. There's a lot of public support for slowing down and the AI industry being held accountable
1:07:30 Speaker 00 and people asking hard questions about like, what exactly is the end game here? Like mass unemployment and then extinction? Is that really the direction we want to go in? And I think, you know, the AI experts themselves, I hope have a new humility about their ability to predict things because they they'll know they've been blindsided by GPT. They they didn't expect these kind of capabilities this quickly. So when certain AI researchers like, let's say, Yann LeCun, who makes fun of AI doomers
1:08:12 Speaker 00 a lot on Twitter, hopefully people like him will stay open minded enough that they might actually kind of reexamine their biases. And maybe he'll have a kind of Jeff Hinton moment, right? Like leading AI researcher Jeff Hinton going at age 75, oh, no. Oh, no, I think my life's work might have been really actually kind of evil and imposing risks on people and I'm going to resign from Google and blow the whistle and make a big fuss about this.
1:08:41 Chris Williamson? Yeah, there was a tweet as well from Tim Urban that you responded to. Tim said, whether you're optimistic or pessimistic about the future, the fact is that we're facing an existential risk at worst and a vastly different future world at best. The world that we're used to very well may not be around for much longer. So let's really enjoy the world that we have it. Visit the places that you've always wanted to visit. Dive into that hobby you've always wanted to try. Spend quality time with your loved ones. Savor each sunny Saturday, each great meal, each moment of fun. If we end up looking back on these days with great nostalgia, we want to at least know we made the most of the time that we had. And you responded and said existential risk isn't anywhere near the worst thing that we could have.
1:09:23 Chris Williamson? There's something called S risk as well, above and beyond X risk. What's S risk?
1:09:30 Speaker 00 S risk is suffering risk. So with extinction, right? Everybody's dead and then they don't experience anything anymore. And that's like pretty bad compared to experiencing things and being happy. But as anybody who's ever suffered chronic pain or torture will will attest, like there are things worse than death. There are levels of suffering that could potentially be imposed by new technologies that would make us wish we'd gone extinct. I personally haven't taken S risk that seriously. I don't know that much about it. There's other people who are more expert on it than me.
1:10:11 Speaker 00 I've only read like a few science fiction novels that depict like really, really bad S risks that could be imposed. There's a, there's a novel called Surface Detail by E and M Banks, where, long story short, in the far future, some religious fundamentalists decide heaven and hell don't really exist, but we should make them exist. So we're going to upload everybody's brain before they die into a simulated reality. And if we think they've been bad and naughty, we're going to make them live in a simulated virtual hell for like subjective millennia and like thousands of years of suffering and torture and mayhem and death. And like, that would be worse than being dead forever.
1:10:55 Speaker 00 So that's S risk. Some people worry a lot about it. I haven't really focused on it very much, but I think when people like Tim Urban say, yeah, it'll either be great, or we'll all be dead. So just smell the daffodils and enjoy your steak and visit beautiful Austin, Texas, or whatever, whatever they think we should be doing. I think, no, that like,
1:11:25 Speaker 00 I want my daughters to grow up being very confident that A, they won't suffer an extinction risk, and, and B, they won't suffer, you know, an S risk.
1:11:38 Chris Williamson? Yeah, the, take it a little bit more seriously. One final person who waded in was Marc Andreessen. I'm going to guess that you read his big essay, Why AI Will Save the World recently. What were your thoughts on that?
1:11:56 Speaker 00 You know, I used to have so much respect for Andreessen. And I think he's just so witlessly stupid about the extinction risks regarding, it truly baffles me like how you can have a brain that big and be so, honestly, I think what, what happens with a lot of these sort of well-respected, rich elders like Lacun or Andreessen is, like, they've had a great career, they've done well in business, they've, you know, made a bunch of money, they are technically savvy, I have, I have no doubts, they have high IQs, but I think they radically overestimate
1:12:40 Speaker 00 their ability to understand issues that they just haven't read very much about. And I think if you haven't read, you know, Nick Bostrom's Superintelligence, you haven't read Toby Ord's The Precipice, you haven't read some of the other key ideas, if you haven't read Eliezer Yudkowsky's work, then, like, you're just going to be reinventing a bunch of very amateurish objections to that kind of work that like were already addressed 10 or 20 years ago
1:13:09 Speaker 00 by the actual experts who have been thinking about this. So this is one of those topics where...
1:13:18 Speaker 00 It's very important not to just defer to people because they're rich or famous or smart. Like, you really want to dive deep on have they thought through these issues? Have they engaged in meaningful conversations with other experts in the area? And if they haven't, they probably do not know what they're talking about.
1:13:38 Chris Williamson? What would you do if you could step in? What would your prescription or policy be if you had an omnipotent, omniscient God's eye view?
1:13:50 Speaker 00 I think I would love for the general public just to tune into the issue, to apply their natural survival instincts and their parental instincts to go, "Wow, this looks like a threat to me, my family, my kids, my grandkids. This is a legit threat. I should take it just as seriously as I would take a local crime wave, or just as seriously as people took nuclear war back in the Cold War." And this is a matter of potentially life and death, you know, for me and my family. I want them to personalize it. I don't want them to just think this is some abstract science fiction scenario
1:14:33 Speaker 00 that is in like the distant future. This is the stuff that like could affect, you know, my 26-year-old daughter, or my 15-month-old daughter, or like my actual kids and cousins and nieces and nephews and whatever. And once you take it seriously, then you're motivated to learn more and to start moralizing the issue and to go, "Are the people who are charging headlong into this, right, through some combination of like greed and hubris and prestige and whatever, are they on our side? Are they pro-human? Are they fighting for my family? Or are they just caught up in some kind of..."
1:15:17 Speaker 00 Sort of delusional project that is at heart reckless and evil.
1:15:22 Chris Williamson? And then what?
1:15:24 Speaker 00 And then if they decide this is reckless and evil, let's let's effing morally stigmatize it. Let's say if you work in the AI industry and you are not spending most of your time worried about AI safety, then you're a bad person and I don't want to associate with you. I don't want to date you. I don't want to be your friend. I don't want to invest with you. I don't want to supply you with hardware or software or anything else.
1:15:49 Speaker 00 I don't want you to be a blooperate in our city, state or country. And I want to, you know, shut it down. Now, I'm not advocating for like a violent so-called like Butlerian Jihad where people rise up and they're like smash all the machines and kill all the AI like, no, I don't want that. Mostly because it would be counterproductive in terms of PR, like violence doesn't work in the social media era. It's just delegitimizes your cause.
1:16:23 Speaker 00 But I think short of violence, it's important to be aware that like we can use all the techniques of persuasion and activism in this domain that people used in every other social movement that we're familiar with, civil rights movement, the gay rights movement, you know, the libertarian movement, crypto, like anything that's been at all successful in terms of the public rising up and saying, we want a change of values.
1:16:57 Speaker 00 We can use all that in fighting reckless AI development.
1:17:02 Chris Williamson? I suppose the challenge again that you have is everyone is being distracted into a beautiful field with daisies growing in it and free cake recipes and essays written for them for school and university. So, yeah, I worry that the novelty and the immediate convenience that's been afforded by these new advances is causing it to seem both so benign and enjoyable and distracting and entertaining that galvanizing people to see this is going, and presumably this is the one of the primary challenges that you're facing.
1:17:46 Speaker 00 I think what I would say to people is like, you can kind of have your cake and eat it too. In terms of, like, there's a whole range of incredible software and narrow AI. That's absolutely wonderful. Like, if you look at the progress in computer graphics and Hollywood movies, it's amazing. Like, I love Avengers movies. And a lot of that would be considered very advanced generative AI, right, by the standards of the 1970s, right? The fact that many modern movies are largely
1:18:24 Speaker 00 kind of AI graphic generated, that's pretty cool. The fact that, you know, Google Maps can actually get you from A to B pretty reliably and taking into account like traffic, that is narrow AI. That's a huge benefit to many people. And there are probably hundreds of cases where you can have quite safe, narrow AI applications that deliver huge quality of life benefits to people. But you don't have to go down the road towards AGI, right, or towards other kinds of highly risky narrow AI. And even in longevity research, like, you could have narrow AI that does biomedical
1:19:08 Speaker 00 research and synthesizes scientific literatures about which molecules might be helpful, and that maybe even can help you run larger scale longevity studies, right? And maybe we can get longevity through that without having to go down the road of extremely dangerous AGI.
1:19:28 Chris Williamson? Geoffrey Miller, ladies and gentlemen. If people want to keep up to date with the work that you're doing, whether it be evolutionary psychology or AI risk, where should they go?
1:19:37 Speaker 00 They can go to my website primalpoly.com, and they can also see my essays on the Effective Altruism Forum, which is EA Forum, where I publish quite a bit these days.
1:19:47 Chris Williamson? Jeffrey, I appreciate you. I'm looking forward to seeing what we get to talk about next time as well.
1:19:51 Speaker 00 My pleasure, Chris.
Previous Is Reality Just A Hallucination In The Brain? - Anil Seth Next How To Create A Life Of Purpose & Achievement - Dr Mike Israetel Page updated October 26, 2026 · transcript and summary are machine-generated
