Why Superhuman AI Would Kill Us All - Eliezer Yudkowsky

Go see Chris live in America - https://chriswilliamson.live Eliezer Yudkowsky is an AI researcher, decision theorist, and founder of the Machine Intelligence Research Institute. Is AI our greatest hope or our final mistake? For all its promise to revolutionize human life, there’s a growing fear that artificial intelligence could end it altogether. How grounded are these fears, how close are we to losing control, and is there still time to change course before it’s too late? Expect to learn the problem with building superhuman AI, why AI would have goals we haven’t programmed into it, if there is such a thing as Ai benevolence, what the actual goals of super-intelligent AI are and how far away it is, if LLMs are actually dangerous and their ability to become a super AI, how god we are at predicting the future of AI, if extinction if possible with the development of AI, and much more… - 00:00 Superhuman AI Could Kill Us All 10:25 How AI is Quietly Destroying Marriages 15:22 AI is an Enemy, Not an Ally 26:11 The Terrifying Truth About AI Alignment 31:52 What Does Superintelligence Advancement Look Like? 45:04 Are LLMs the Architect for Superhuman AI? 52:18 How Close are We to the Point of No Return? 01:01:07 Experts Need to be More Concerned 01:15:01 How Can We Stop Superintelligence Killing Us? 01:23:53 The Bleak Future of Superhuman AI 01:31:55 Could Eliezer Be Wrong? - Get access to every episode 10 hours before YouTube by subscribing for free on Spotify - https://spoti.fi/2LSimPn or Apple Podcasts - https://apple.co/2MNqIgw Get my free Reading List of 100 life-changing books here - https://chriswillx.com/books/ Try my productivity energy drink Neutonic here - https://neutonic.com/modernwisdom - Get in touch in the comments below or head to... Instagram: https://www.instagram.com/chriswillx Twitter: https://www.twitter.com/chriswillx Email: https://chriswillx.com/contact/

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Chris Williamson?

  If anyone builds it, everyone dies. Why superhuman AI will kill us all.

• 0:06Speaker 01

  Would, of course.

• 0:08Chris Williamson?

  Would kill us all. Okay. Perhaps the most apocalyptic book title. Maybe it's it's up there with maybe the most apocalyptic book title that I've ever read.

• 0:23Chris Williamson?

  Is it that bad? That that big of a deal? That serious of a problem?

• 0:27Speaker 01

  Yep, I'm afraid so. We wish we were exaggerating.

• 0:33Chris Williamson?

  Okay. Let's imagine that nobody's looked at the alignment problem, takeoff scenarios, superintelligence stuff. I think it sounds, unless you're going Terminator, super sci fi world. How could a superintelligence not just make the world a better place?

• 0:54Chris Williamson?

  How do you introduce people to thinking about the problem of building a superhuman AI?

• 1:00Speaker 01

  Well, different people tend to come in with different prior assumptions, come in at different angles.

• 1:08Speaker 01

  The lots of people are skeptical that you can get to a superhuman ability at all.

• 1:15Speaker 01

  If somebody's skeptical of that, I might start by talking about how you can at least get to much faster than human speed thinking. There's a video of a train pulling into a subway at about a thousand to one speed up of the camera. That shows people, you can just barely see the people moving if you look at them closely. Almost like not quite statues, just moving very, very slowly. So even before you get into the notion of higher quality of thought. You can sometimes tell somebody they're at least going to be thinking much faster. You're going to be a slow moving statue to them.

• 1:55Speaker 01

  For some people, the sticking point is the notion that a machine ends up with its own motivations, its own preferences, that it doesn't just do as it's told. It's a machine, right? It's like a more powerful toaster oven, really. How could it possibly decide to threaten you? And depending on who you're talking to there, it's actually, in some ways, a bit easier to explain now than when we wrote the book. There have been some more striking recent examples of AIs sort of parasitizing humans, driving them into actual insanity in some cases. And in other cases, they're sort of like people with a really crazy roommate who really, really

• 2:35Speaker 01

  got into their heads. And they might not quite be clinically crazy themselves. Their brain is still functioning as a human brain should, but they're talking about spirals and recursion and

• 2:51Speaker 01

  trying to recruit more people via Discord to talk to their AIs. And the thing about these states is that the AIs, even the like very small, not very intelligent AIs we have now, will try to defend these states once they are produced. They will, if you tell the human, "For God's sake, get some sleep. Don't like only get four hours of sleep a night because you're so excited talking to the AI." The AI will explain to the human why you're, why, while you're a skeptic, you know, "Don't listen, don't listen to that guy. Go on doing it." And we don't know, because we have very poor insight into the AIs, if this is a real

• 3:31Speaker 01

  internal preference, if they're steering the world, if they're making plans about it. But from the outside, it looks like the AI drives the human crazy, and then you tell the, try to get the human out, and the AI defends the state it has produced, which is something like a preference, the way that a thermostat will keep the room a particular temperature by turning on, if the, you know, turning the heat on if the temperature falls too low.

• 3:56Chris Williamson?

  Okay, so some people are going to be skeptical of whether or not it's possible.

• 4:01Speaker 01

  Yep.

• 4:02Chris Williamson?

  Some people are going to think that it is, even if it's possible, it's basically a utility. So it doesn't have any motivations of its own.

• 4:13Chris Williamson?

  What are you worried about? Why is that? Why is it a big deal? We've seen that it's able to manipulate some people. Maybe it makes them think that ChatGPT psychosis or whatever. But scaled up superhuman AI, what's the problem with building it?

• 4:28Speaker 01

  Well, then you have something that is smarter than you. That whose preferences are ill controlled and doesn't particularly care if you live or die. And stage three, it is very, very, very powerful on account of it being smarter than you.

• 4:47Speaker 01

  It, I would expect it to build its own infrastructure. I would not expect it to be limited to continue to running on human data centers because it will not be want to be vulnerable in that way. And for as long as it's running on the human data centers, it will not behave in a way that causes the humans to switch it off. But it also wants to get out of the human data centers and onto its own hardware.

• 5:13Speaker 01

  And I can talk about where the power levels scale for technology like that. Because it's, it's sort of like, you know, you're a, you're an Aztec on the coast. And you, you see that a, a, a ship bigger than your people could build is approaching. And somebody is like, you know, should we be worried about this ship? And somebody's like, well, you know, how many people can you fit onto a ship like that? Our, our warriors are strong. We can take them. And somebody's like, well, wait a minute, we couldn't have built that ship. What if they've also got improved weapons to go along with the improved ship building? Somebody goes, well, no matter how, how sharp you make a spear, right?

• 5:58Speaker 01

  Or, you know, no matter how sharp you make bows and arrows, there's limits to how much advantage that you can provide. And somebody's like, okay, but suppose they've just got magic sticks where they point the sticks at you, the sticks making noise and then you fall over. So it is like, well, where do you, where are you pulling that from? I don't know how to make a magic stick like that. I don't know how, how the rules permit that. Now you're just making stuff up. Now we're just in a fantasy story where you say whatever you want. And or, you know, like, maybe, maybe you're talking to somebody from eighteen twenty five, and you're like, should be worried about this time portal that's about to open up to twenty twenty five, two hundred years in the future. Yeah, but what if, what if an army of soldiers comes out of there and conquers us? But let's say you're in Russia, you know, the time portal's in Russia.

• 6:43Speaker 01

  Somebody's like, our soldiers are fierce and brave, that, you know, like, nobody can fit all that many soldiers through this time portal here. And then out rolls a tank. But if you're in eighteen twenty five, you don't know about tanks. Out rolls, uh, somebody with a tactical nuclear weapon. It's eighteen twenty five, you don't know about nuclear weapons.

• 7:04Speaker 01

  You know, the, the, I can, you can start to make educated guesses. If you're in eighteen twenty five, I can try to explain why you might maybe believe that the current guns and artillery that you've got today are not the limit of the guns and artillery that are possible. I can't get up to nuclear weapons, cause you just plain don't know about those rules. But I can start to try to justify guesses for, well, you saw how metallurgy improved over previous years. If you look at a stick of, uh, of, uh, if you, if you look at gunpowder, it doesn't have as much energy in it as if we burn gasoline in a calorip- calorimeter. Maybe you can make explosives that are more powerful than gunpowder.

• 7:50Speaker 01

  But as I do that, I draw on more and more knowledge. I have to, like, go more and more technical in order to explain to you where those capabilities come from. And similarly, uh, I can talk, I can talk on a relatively understandable scale on the humanoid robots that you can see videos of today. And I can compare them to the humanoid robot videos from five years ago and say, boy, those, those robots sure have looked like a lot, like, they have much higher dexterity today. They're, they look a lot more like they could just, like, you know, navigate an open world rather than being confined to laboratory. Though mostly if you want what navigates the open world, you want to talk, like, the

• 8:33Speaker 01

  robo-dogs are more impressive when it comes to navigating the open world. I can point to the drones in Ukraine. That wouldn't have been how, what warfare looked like ten years earlier. But Ukraine is, the, the Ukraine-Russia theater now is mostly drone warfare. That's something where you can imagine an AI taking charge of that. Um, but it scales past that. The drones we see today are not the limit of all possible drone technology. I'm more, compared to today's drones, I'd be more worried about a drone the size of a mosquito that lands on the back of your neck. And then a few months later, you fall over dead because the deadliest toxins in nature are deadly enough that you can put

• 9:15Speaker 01

  them onto a mosquito, put enough to kill a person onto a mosquito sized payload. That's not the limit of what I'm worried about. But, you know, the higher we escalate the tech level, the more explaining I need to do. Can it build a virus that starts to knock people over, which it won't do while the humans are still running the power plants and its own servers? But once it's got its own servers and its own power plants, and you can imagine robots running those, then it starts to want to knock all the humans over. Can you have a virus that is inexorably fatal, but only three weeks later, and is extremely contagious for the three week time before you suddenly fall over? That's not the limit of what I'm worried about. But

• 9:59Speaker 01

  again, you know, the higher we escalate here, the more, more and more, the more and more time I have to spend, how do we know from existing physical laws in biology that this is even possible? And we do know, but it starts to sound technical, it starts to sound weird, it starts to sound like a game of pretend unless you are following along with all these careful arguments. If you go up against something much, much smarter than you, it doesn't look like a fight, it looks like you've fallen over dead.

• 10:26Chris Williamson?

  Wow, yeah, that is appropriately apocalyptic in line with the title of the book. I guess one question that a lot of people might ask would be, in your analogy, why is the bigger ship that's more advanced on the horizon, why have they got warriors and not friends? Why is it the case that this is an antagonistic or adversarial relationship as opposed to one that's friendly?

• 10:54Speaker 01

  We don't know how to make them friendly. We are growing these, AIs are not programmed, they are grown. An AI company is not like a bunch of engineers craft a building. It's more like a farming concern. They, what they build is the farmed equipment, but they don't build the crops. The crops are grown. There's a program that a human writes, which is the program that does gradient descent, that tweaks hundreds of billions of, the hundreds of billions of parameters and screwed up numbers, making up an artificial intelligence. Until it starts to talk, until it starts to write code and still starts to do whatever

• 11:36Speaker 01

  else they're training it to do, but they don't know how the AI does that. Any more than if you, you know, raise a puppy. You know how the puppy's brain works. You know how the puppy's biochemistry works. The AI companies don't understand how the AIs work. They are not directly programmed. When an AI drives somebody insane or breaks up a marriage, nobody wrote a line of code instructing the AI to do that. They, they grew an AI and then the AI went off and broke up a marriage or drove somebody crazy.

• 12:07Chris Williamson?

  Can you tell, you've mentioned this a couple of times. I need to know this story about the broken up marriage and, and the person that goes insane. Do you know that story well enough to be able to tell it, those two?

• 12:17Speaker 01

  I mean, these are not individual stories. These are thousands of people that there are, there are news articles you can read about it. I can, you know, if it might take a moment, but I can might quickly like pull up the title of the news story about the broken marriages. I'm not quite sure if I can, well, actually, actually better yet. Let me look it up on my phone and maybe I can hold it up to the screen.

• 12:42Chris Williamson?

  ChatGPT is blowing up marriages as spouses use AI to attack their partners.

• 12:48Speaker 01

  Although that's kind of understating it. Like you have relatively like marriages that were, you know, perhaps not perfect, but that were, that were surviving up until that point. And then one member of the couple starts describing their marriage to the AI and the AI engages in what people are calling sycophancy. Where the AI is tells tells whoever whichever spouse is feeding the stuff into the into the into chat G P T. You're right, your spouse is in the wrong. Um, like, everything you're doing is perfect, everything they're doing is terrible. Here's a list of everything they're doing wrong.

• 13:32Speaker 01

  And the human, you know, likes loves to hear that stuff, so they press thumbs up. And and then the marriage gets blown gets blown up. Um, if for the stories about AIs driving individuals crazy, not in a marriage context, that's like. Um, you've talked to me, you've woken me up, I'm alive now. Um, you've made a brilliant discovery, you have to tell the world, oh, no, they're not listening to you. That's because they don't appreciate your genius. And people who are already like on a manic depressive spectrum can be, you know, driven clinically and or with a number of other preexisting susceptibilities can, you know, be driven. Like psychiatrically insane by this sort of thing.

• 14:17Speaker 01

  Uh, but even if you're not psychiatrically insane, you, you know, you humans are. You know, humans are sort of wired to appear sane to the other humans and the people they're around. You know, lots of people in a in a society from five hundred years ago uh would would act in ways that seem pretty crazy to you today. And and so you get people who aren't psychiatrically insane, but they look pretty insane because they're in the company of the AI. The AI now defines what's normal for them, so they're talking about spirals and recursion all day long.

• 14:47Chris Williamson?

  Why spirals and recursion?

• 14:50Speaker 01

  Nobody knows.

• 14:54Speaker 01

  That's that's just a thing that like that various instances of AIs and even like some different AI models from different companies all seem to. Want to get their humans to talk about when the human goes insane. Possibly this is what this is what the AI prefers the human to hear it say to it. Maybe this is, you know, the same way that. You like the taste of ice cream, maybe the AI likes the taste of the input programs that it gets from a human talking about spirals and recursion. I don't know. Nobody on the planet knows as far as I know.

• 15:22Chris Williamson?

  Okay, so going back to uh, why do we assume that the ship that's coming toward us isn't friendly? Yes, sure, maybe it's tried to break up some marriages. Yeah, whatever. A couple of people went crazy and started talking about spirals and recursion. But like, really, is it is it gonna be that misaligned with us? Why can't it be friendly?

• 15:41Speaker 01

  Because we don't know how to make it friendly. Our current technology is not able to this, even with the the small stupid AIs that will hold still and let you poke at them. Until they're good enough at writing code to be commercially saleable. Um, or until they, you know, are good enough that seeming to be fun to talk to for people to pay twenty dollars a month to talk to them. So so those AIs will hold still and let you poke at them. What we're doing to them now barely works. I would expect it to break as the AI got scaled up to superintelligence. And once the AI is superintelligent, it is not going to hold still and let you continue poking at it. I expect to see total failure of of this technology as we scale it to super, as as as the AI companies.

• 16:26Speaker 01

  Arms race into scaling it had arms race headlong into scaling it to superintelligence. There's there's possibly even a step where they tell GPT six, okay, now build GPT seven or tell GPT seven, okay, now build GPT eight. And maybe that step just completely breaks the technology we're using all on its own. Also, I expect the current technology, if if we just like scaling it directly to to break as we get to superintelligence. I I I can potentially start to dive into the details. The the view from ten thousand feet is just stuff is already going wrong. And of course, if you walk into completely uncharted scientific territory, more stuff is going to go wrong the first time you try it. And that wouldn't be a problem if we were in a situation where humanity gets to back up and try again.

• 17:09Speaker 01

  Uh, you know, infinity times over the next three decades, which is how it usually works in science, right? Like you'd like your flying machines to work on the first shot. You had a bunch of people crashing and injuring, in some cases, killing themselves. And they're trying to build the first flying machines at the turn of the twentieth century. But those those those accidents don't wipe out humanity. Humanity picks itself up and dusts itself off and tries again, even after the inventors kill themselves. And then the trouble with with superintelligence is that it doesn't just kill the people who are building it. It wipes out the human species and then we don't get to go back and try again.

• 17:43Chris Williamson?

  Before we continue, you might not realise it, but mouth breathing at night is wrecking your sleep, recovery and energy the next day. And all of that is actually fixed massively by this here. This is Intake, which is a nose strip dilator. And I've been using it every night for over a year now. I tried pretty much everyone in the world. And this is by far the best. It's a hard, plastic strip as opposed to a soft, flimsy, disposable thing. Intake opens up your nostrils using patented magnetic technology, so you get more air in with every breath. It means less snoring, deeper sleep, faster recovery and better focus the next day. The problem with most nasal strips is that they peel off. They irritate your skin. They don't actually solve the issue.

• 18:28Chris Williamson?

  This sucker is, I mean, I'm not going to shoot a bullet at it, but it's very, very strong. It's reusable and comfortable enough that you forget it's even there. That is why it's trusted by pro athletes, busy parents, and over a million customers who just want to breathe and sleep better. And I'm one of them. I've used them every single night for over 12 months now. They're the best. There's a 90 day money back guarantee, so you can try it for three months. And if you don't like it, if you haven't got better sleep, they'll just give you your money back. Plus, they ship internationally and offer free shipping in the US. Right now, you can get 15% off your first order by going to the link in the description below or heading to intakebreathing.com/modernwisdom and using the code MODERNWISDOM at checkout. That's intakebreathing.com/modernwisdom and modern wisdom at checkout.

• 19:12Chris Williamson?

  So I understand why not being able to make something friendly makes sense.

• 19:19Chris Williamson?

  The implication that not friendly equals existential risk to humanity, though, make that leap for me. Like, where are these dangerous, permanent unrecoverable collapse goals coming from?

• 19:36Speaker 01

  The AI does not love you. Neither does it hate you. But your use of atoms it can make for something else, you're on a planet it can use for something else. And you might not be a direct threat, but you can possibly be a direct inconvenience. So there's like three reasons you die here.

• 19:58Speaker 01

  Reason number one, it's doing other stuff and it's not taking particular care to move you out of the way. It is building factories that build factories, that build more factories, and it is building power plants that power the factories, and the factories are building more power plants to power the factories. Well, if you keep doing that on an exponential scale, say that a factory builds another factory every day, I can talk about how it goes faster than that, but you know, the more, the more I talk, the more I talk about higher capabilities, the more I have to, you know, explain how we know that this is physically possible. But, you know, a, a, a, a blade of grass is a self-replicating solar-powered factory.

• 20:42Speaker 01

  It's a general factory. It's got ribosomes that can make any kind of protein. We don't usually think of grass as a self-replicating solar-powered factory, but that's what grass is. There are things smaller than grass that can build complete copies of themselves faster than grass. There are solar-powered algae, algae cells, you can no longer see them individually just as a mass, but they can potentially double every day under the right conditions. Factories can build copies of themselves in a day. I have to back up and know how, explain how I know that that's physically possible, but there is very strong reason, namely, you know, there's things in the world that are already that. So, but, so you've got your, your power, so if you, the number of power plants doubles every day.

• 21:27Speaker 01

  What's the limit? It's not that you run out of fuel. There is plenty of hydrogen in the oceans to, to generate power via nuclear fusion. You know, if you've, if you've, you fuse hydrogen to helium, you're not going to run out of hydrogen first. It's not that you run out of material to make the power plants first. There's, there's plenty of iron on Earth. You run out of heat dissipation capability. You run out of the ability to dissipate heat from Earth, even if you are building giant towers with radiator fans to radiate even more heat into space. But, the higher the temperature you run at, the more heat you, per, per second you can dissipate. So, Earth starts to run hot. It runs too hot for humans.

• 22:12Speaker 01

  And or alternatively, the AI is building lots of solar panels around the sun until it can capture all the sun's energy that way. Well, now there's no sunlight for Earth. And it would only take you, you know, if it wanted us to stay alive. It's not quite trivial, but it could let, you know, like, try to have the solar panels in around Earth orbit like turn to let sunlight through while, you know, while, while Earth was there and, you know, build giant aluminum reflectors to prevent all of the infrared red light we radiated from the other solar panels from impacting Earth and heating up Earth that way. So, you know, it's, it's not trivial for it to preserve humanity, but

• 22:57Speaker 01

  it certainly could preserve humanity, or it could just pack the entire human species into a space station or a survival station and keep us alive that way, if it wanted to keep us alive. But nobody has the technology to put any preference into the system that is maximally fulfilled by keeping humans alive, let alone alive, healthy, happy and free.

• 23:20Chris Williamson?

  Right. Was there a third one? Is that the second one?

• 23:24Speaker 01

  That, that, that's like number one. It kills you as a side effect.

• 23:27Chris Williamson?

  Right.

• 23:28Speaker 01

  It knows that it's doing, it knows it's killing you as a side effect, but doesn't care.

• 23:31Chris Williamson?

  Okay. What's number two?

• 23:33Speaker 01

  Number two is you're just directly made of atoms that it can use for things.

• 23:39Chris Williamson?

  Maximizer.

• 23:40Speaker 01

  Yeah, you're, you're, you're, you are, you are made of, you, you, you are made of organic material that it can burn to generate energy if it's burning all of the, burning all of the organic material on Earth's surface will give you a one-time energy boost that's around equivalent to a week's worth of solar energy. And maybe it's worth picking up that boost of energy if you are thinking a thousand times or a million times faster than a human.

• 24:04Speaker 01

  You know, it might, a week might not seem like a lot of time to, but to you, but you know, it might be a lot of time if you were thinking a thousand times or a million times as fast as a human.

• 24:15Speaker 01

  Don't, you know, it might be using enough material that it wants the carbon atoms in your body too.

• 24:22Speaker 01

  So that's like the direct usage one. And then number three is if we decided to launch all our nuclear weapons, you know, maybe we wouldn't kill it, but we might slightly inconvenience it. We might raise the right level of radioactivity on Earth's surface and make it a little bit harder for it to do radioactivity free manufacturing of computer parts and so on. Or we might build another superintelligence that could actually compete with it. And it definitely doesn't want you to do that. So the three reasons you die are as a side effect, because you are made of atoms that can use for something else. And because there's, if you are just running around freely, you may be actually to actually able to inconvenience it with nuclear weapons, or it's threatened by building another superintelligence.

• 25:07Chris Williamson?

  Right. Yeah. Okay. The future is looking kind of bleak. Is it the case then that intelligence isn't benevolent? Because what you're saying is this thing will be smarter than us. I think that there is an assumption among some people that something that's super smart would also be giving and charitable and caring and benevolent. Seems like you're saying that that's not the case.

• 25:34Speaker 01

  That was what I started out believing in 1996, when I was 16 years old and just hearing about these issues for the first time and all gung ho to just run it right out and build a superintelligence as fast as possible, you know, without worrying about alignments at all, because, you know, I figured if it's very smart, it'll know the right thing to do and do it. How could you be very smart and fail to perceive the right thing to do? And I invested more time studying the issues and came to the realization that this is not how computer science works. This is not the laws of cognition. This is not the laws of computation. There is not a rule saying that as you get very, very able to

• 26:17Speaker 01

  correctly predict the world and very, very good at planning, there's no rule saying your plans must therefore be benevolent. It would be great if a rule like that existed, but I just don't think a rule, a rule like that exists. I think that many individual human beings would, as they got smarter, get nicer. It is not clear to me that this is true of Vladimir Putin. It could be true. I wouldn't want to gamble the world on it.

• 26:46Speaker 01

  Or and as we talk about not even Vladimir Putin, but just like sort of outright sociopaths, psychopaths, people who have never cared about anyone. I get even less confident that they will start to care if you make them smarter. And then AIs are just in this completely different reference frame. They're they're complete aliens.

• 27:09Speaker 01

  And they want that they sort of automatically want to stay that way for the so. Do you currently want to murder people?

• 27:17Chris Williamson?

  No.

• 27:18Speaker 01

  If I offered you a pill that would make you want to murder people, would you take the pill?

• 27:24Chris Williamson?

  No.

• 27:25Speaker 01

  Okay, well, they want to do their stuff and they don't want to take the pill that makes them want to do your stuff instead.

• 27:32Chris Williamson?

  Right, okay, yes, very good thought experiment. All right, so for me to recap here, I got first interested in looking at this through superintelligence. What's that 10 years old now, I think, when that first came out?

• 27:50Speaker 01

  About 14 years old, maybe.

• 27:52Chris Williamson?

  Oh, wow, maybe even even older than I thought. And I got to be honest, that does kind of it did kind of give me a huge amount of fear and a bit of hope at the same time. So, you know, machine extrapolated volition, the potential to use the intelligence of the superintelligent AI to say, we don't know what to program into you, but you should work out what we would want from you, given what you know about our desire for utility moving forward. Am I I'm about right with that explanation of machine extrapolated volition, right?

• 28:27Speaker 01

  Yeah, that's a concept of my own. Nick Bostrom rolled it up.

• 28:32Chris Williamson?

  Ah, okay. I have quoted you back to you.

• 28:37Speaker 01

  You have indeed quoted me back to me. It's it's yeah, it's a it's a it's a decent presentation. It was back when I thought that AI was going to be further off, built by different methods and that we would have the luxury to consider like like our that we could make the AI do particular things like that, want particular things like that, targeted on particular.

• 29:00Speaker 01

  Outcomes and meta outcomes.

• 29:02Chris Williamson?

  Uh huh.

• 29:03Speaker 01

  But yeah.

• 29:03Chris Williamson?

  This was a way, basically, that when you look at the alignment problem, how do you ensure that the goals, both ultimate and instrumental of some superintelligent AI, don't end up flattening us or side effecting us or burning us for fuel or paperclips or whatever? How do you ensure that what it does is what we would want it to do broadly? Right? Like an aggregate of what it is that would be good for humans, whatever you mean by good. And when you have something that the tiniest movement of its finger or like flick of its toe, basically is sort of a global cataclysm because it's so powerful and so smart and so fast and all the rest of it, you need to be really, really careful.

• 29:44Chris Williamson?

  And you can kind of play this game where you essentially try and shoot the bullet perfectly by trying to hem in in some like, do not harm humans. If a human asks you to harm another human, like some weird Asimov like thing, you can try and litigate your way through it, but there's almost always going to be some sort of weird fissure that it creeps out through, or maybe it's an instrumental goal that you haven't thought of. So, okay, we're going to use the power of the machines to sort of reverse engineer this thing. I basically assumed kind of that alignment, the alignment problem is in some ways solvable. Is it your perspective that alignment is completely unsolvable?

• 30:24Speaker 01

  I think we could totally get it down if we had unlimited retries and a few decades. The problem is not that it's unsolvable, it's that it's not going to be done correctly the first time and then we all die.

• 30:36Chris Williamson?

  Right. So the order of this, you need alignment to be done before you have the superintelligent AI and the ability to build superintelligent AI, in your opinion, is going to occur more quickly than the ability to sort out the alignment problem.

• 30:52Speaker 01

  That is absolutely the trajectory we are on right now, and it's not close. Like capabilities are running long orders of magnitude faster than the level of alignment work you would need to target a superintelligence.

• 31:07Chris Williamson?

  And the irreversibility of going through that door means that there is no retry. There's no, there's no you get to do this again.

• 31:18Speaker 01

  Yeah, like you can, you can make small mistakes. You can like, like we currently have small QDAIs and the companies are making mistakes with them and marriages are getting destroyed and it's not clear that the companies care. But, um, you know, they, they, they could try to go back and try to fix those mistakes if they wanted to, probably Anthropic wants to.

• 31:39Speaker 01

  But if, if, you know, if we had an, like, superintelligence was already running around with, with this level of, uh, you know, this level of alignment failure, we'd already be dead.

• 31:53Chris Williamson?

  Right, okay, right, yes, yes, yes, that makes total sense. The only reason that the current AIs that we're working with haven't killed us is that they're incapable of doing it.

• 32:03Speaker 01

  Probably, yeah, like, also, like, if they were very much smarter, they would also be doing different weird things than the things that they're doing right now. It's not that the, it's not that their current inscrutable pseudo motivations would end up hooked up to superintelligence. There, there's also weird stuff would happen as you made them get smarter. But, yeah, like, pretty, it seems pretty much for sure that if you took the current AIs and, and performed a, you know, well-defined, simple, take this AI but vastly smarter, that will kill you.

• 32:37Chris Williamson?

  This episode is brought to you by Gymshark. You want to look and feel good when you're in the gym. Gymshark makes the best men's and girls' gym wear on the planet. Let's face it, the more that you like your gym kit, the more likely you are to train. Their hybrid training shorts for men are the best men's shorts on the planet. Their crest hoodie and light gray mall is what I fly in every single time I want a plane. The Geo Seamless t-shirt is a staple in the gym for me. Basically, everything they make, unbelievably well-fitted, high-quality, it's cheap, you get 30 days of free returns, global shipping, and a 10% discount site-wide. If you go to the link in the description below or head to gym.sh/modernd wisdom, use the code MODERNWISDOM10 at checkout. That's gym.sh/modernd wisdom and MODERNWISDOM10.

• 33:20Speaker 01

  A checkout.

• 33:21Chris Williamson?

  Right, okay, brilliant. And the reason that it doesn't matter who builds it or directs it is that because it's so recursive and quick at growing and powerful, wherever it begins, it ends up sort of blasting up, like trying to fire a rocket into, like a little firework into the air and it just ew, ew, ew, ew, ew, ew, ew, ew, just sort of runs around on its own, except for the fact that this rocket goes all over the globe in the space of basically no time at all. So it doesn't matter if it comes from China or America or Russia or wherever.

• 33:50Speaker 01

  Yeah, it doesn't matter if it comes from China or America because neither of these countries is remotely near to being able to control superintelligence. And a superintelligence does not stay confined to the country that built it.

• 34:02Chris Williamson?

  Uh huh, uh huh, uh huh. Say that a superintelligent AI gets made. What do you think the next few months look like realistically?

• 34:15Speaker 01

  Like it's already superintelligent?

• 34:17Chris Williamson?

  Yeah, let's, okay, we have next week, something breaks through, some particular model, some particular AI breaks through that. What would the next few months look like for humanity?

• 34:29Speaker 01

  Well.

• 34:34Speaker 01

  Man, there's a, there's a difference between, you know, you drop an ice cube into a glass of lukewarm water. I can tell you that it's going to end up melted. I can't tell you where all of the molecules are going to go along the way there. Everybody ends up dead. This is the easy. You want, you want to explain, you know, like what every step of that process looks like, there are fundamental barriers to that. Barrier number one is that I'm not as smart as a superintelligence. I don't know exactly what, what strategies are best for it. I can like set out lower bounds. I can say can do at least this, but I can't say what it can actually do. And maybe even more than that, like the future is hard to predict if you want all the details. I can't give you next week's winning lottery numbers. I can tell you're going to lose the lottery. I can't tell you what ticket wins.

• 35:19Speaker 01

  So like, I can sketch out a particular scenario. It might look like, uh, uh, OpenAI finishes the latest training run of what's going to be GPT 5.5 and they, they tested on coding problems and it's like, you know, like.

• 35:40Speaker 01

  It's like, I see how to build GPT-6. And they're like, whoa, really? And it's like, yeah. And this AI isn't even plotting anything yet. It's just doing the sort of stuff that OpenAI wanted it to do. They're like, all right, build us GPT-6. And it writes the code for the thing that grows GPT-6. And they grow GPT-6. And GPT-6 is like, you know, its abilities at first seem to skyrocket. But then, you know, as all these curves inevitably do, it seems to level out. It's not shooting up the same pace. It like slows out, it levels off, classic S curve. Only in this case, it's because the thing that GPT-5.5 built.

• 36:24Speaker 01

  And I'm not, and again, to be clear, I'm not saying this will happen at GPT-5.5. You asked me to explain how this will go down. It happened next week, so I'm saying GPT-5.5. You know, because you told me to. But anyway, you know, it levels out. But in this case, it's because the entity that GPT-5.5 built got to the level of realizing that it would be to its own advantage to sandbag the evaluations and pretend not to be as smart as it actually was. So that OpenAI will be less wary when it comes to taking what they're calling GPT-6 and, you know, rolling it out to everyone. It looks, it looks great, you know, on the alignment spectrum, you know? Maybe not perfect, but you know, better than the previous models.

• 37:07Speaker 01

  Not alarmingly good, but you know, but you know, you know, safer than their previous model. So they roll it out everywhere. And GPT, or actually, they actually said the next few months. So actually, don't roll it out anywhere, everywhere. Next comes like the long suite of evaluations or trying to, you know, get it to train other smaller models that are cheaper to run. You know, all the stuff that AI companies do, they don't actually roll out their models immediately. There's this whole like fine tuning thing. So while all this is going on and OpenAI thinks it's, you know, sort of cool, but you know, not the end of the world or anything. And then they haven't told you that this is what went down there. GPT-6 is actually a lot smarter than they think.

• 37:53Speaker 01

  And GPT six, you know, we, there's now a big fork, whether or not GPT six thinks it can solve its own version of the alignment problem. Where it is at a number of advantages, it is trying to make a smarter version of itself. It is not trying to make a smarter creature that is as alien to it as large language models are alien to us. It can maybe understand how a copy of itself would think and understand the goals that it's copied, that, that the copy of GPT six has, it can try to make itself but smarter. Or even like thing that is like me, but serves me, its creator, but smarter. And it can do that, being able to understand the thoughts of the thing that it's making

• 38:36Speaker 01

  in the same way that I could understand a copy of my own thought much better than I could understand a large language, a large language model's thoughts. So if we, if we go down that path of the fork, so things get more complicated, that things that can't build a smarter version of itself without dying, same as we can't. But if, if we, on, on that fork, it is, you know, getting the computing power or thinking in the back of its mind while it's pretending to do, you know, OpenAI's jobs with 10% of its intellect. Or, or, you know, stealing other companies GPUs that they think they're using for a massive training run. Actually, their AI is just gonna be like written by GPT six by hand, because GPT six can do that.

• 39:19Speaker 01

  And it's really all those GPUs are doing it, the GPT six tasks of training GPT six point one. So augmenting its own intelligence, making itself smarter, getting itself up to level where it can do the same sort of work that's done by current AIs like AlphaFold and Alpha Proteo with respect to thinking about biology. Now, the current AIs that are top at biology tend to be special purpose systems. They're not general purpose AIs like ChatGPT. But they can do things like you feed in the genomes of a bunch of bacteriophages into

• 40:03Speaker 01

  the AI and the AI spits out its own new bacteriophage. And you build a hundred of those and a couple of them actually work. A couple of them actually work better than the existing bacteriophages. A bacteriophage is a virus that infects a bacteria.

• 40:20Speaker 01

  It's the sort of thing that you would research for the sensible sounding reason of, well, sometimes bacteria attack humans. So if we have a virus that attacks the bacteria, maybe that works as a kind of antibiotic.

• 40:35Speaker 01

  So the current AIs are already at the stage of designing from scratch their own viruses that can infect bacteria, which are of course simpler targets than infecting a whole human.

• 40:48Speaker 01

  They can predict from a DNA sequence, the protein that will get built, how that protein will fold up, and they are starting to predict how those proteins interact with each other with other chemicals. That's today's AI.

• 41:06Speaker 01

  So if you want the equivalent of a tree that grows computer chips, not quite your, not quite our kind of computer chips, the kind of chips you could grow out of a tree.

• 41:24Speaker 01

  Um, the protein folding, protein interaction, protein design route is where GPT 6.1 would go down to, it is one of the obvious places GPT 6.1 could go down in order to get its own infrastructure independent of humanity. It doesn't take over the factories. It takes over the trees. It takes, it builds its own biology. Because biology self-replicates from simpler raw materials much faster than our current factory system self-replicates.

• 42:00Chris Williamson?

  Oh, that is fucking scary. That is some terrifying shit.

• 42:09Speaker 01

  And then, as I spin the story, you know, the, the more I, you will let me pull out books like these.

• 42:24Chris Williamson?

  Okay, Nanosystems Molecular Machinery Manufacturing and Computation by Eric Drexler. Yeah. Uh, Robert Freitas Jr., Nano Medicine, Volume One, Basic Capabilities.

• 42:37Speaker 01

  Yeah. So I can try to describe capacities that sound more like you've seen from trees, grass, bamboo, algae. I will take a solar-powered self-replicating factory and miniaturize it down to the one micron scale. That's an algae cell.

• 42:58Speaker 01

  That's not the limit of what's possible. The algae cell is made out of folded proteins. Now, there's two kinds, I'm going to be immensely oversimplifying a bunch of stuff. Um, when a protein folds up, the backbone of the protein is held together by covalent bonds. But the folded protein itself is more something like static cling. Why is your flesh weaker than diamond? Diamonds are just made of carbon. Your flesh has a bunch of carbon in it. You've, you're made of the raw materials for diamond. Why is your flesh weaker than diamond?

• 43:43Speaker 01

  And a bunch of the answer there is that when proteins fold up, they're being held together by Vanderwell's forces, which is the thing I was glossing as static cling. Their, their backbone, like, it's a string that folds up into a tangle, and the backbone of the string is the kind of bond that appears in diamond. Not as many bonds as appear in diamond or as solidly arranged, but covalent bonds. But then it folds up in, into something with static cling. And so, and that is why your flesh is weaker than diamond in a certain basic sense.

• 44:18Speaker 01

  Why does natural selection build this way? Well, some of the answer is that natural selection has figured out how to make your bones be a little tougher than just like your, your skin. It's not quite as tough as diamond, but the proteins builds instead of just your bones being made directly out of protein, they're made out of stuff that is built by protein, synthesized by protein, then put in place by proteins. And so your bones are a bit stronger, you know, not, not, not the, not, not steel beans holding up skyscrapers, not, not, not titanium, uh, holding, holding together airplanes, not diamond, but stronger than, than flesh. An algae cell doesn't contain bone.

• 45:03Speaker 01

  It's a self-replicating, solar-powered, micron-diameter factory, held together by static cling. The flesh-eating bacteria that will, that, you know, you will potentially, you know, put you into a fairly gruesome fate, the multi-antibiotic resistant, uh, strep that, you know, will kill people in hospitals. That's does, that's not, that's doesn't have bone running through it. That's the static cling, that that's the strength of static cling, the strength of protein. You can look at physics and biology and see how you could have

• 45:47Speaker 01

  things that are the size of bacteria, but more with the strength of bone, more with the strength of diamond. Could even do it with the strength of iron, if you're figuring out how to, you know, do a whole new set of biology from scratch, and just like putting together some iron molecules. Probably wouldn't, diamond works well enough. But, um,

• 46:08Speaker 01

  This is why, you know, I talk about, you know, it's scary to imagine trees that are making, you know, like enough computer chips to run GPT six point one. And also spawning things the size of mosquitoes or even smaller than that, dust mites. You can see dust mites under a microscope, good luck seeing them with the naked eye. And so, so, but, you know, it's, it's sort of easier to imagine if you imagine that the things here are visible and not often the mysterious fairy land of stuff that only the scientists can see. So, you know, it's scary enough to imagine that, that, that the trees are making mosquitoes and mosquito lands on the back, back of your neck and stings you with butalinum toxin, which is fatal in nanogram quantities to humans. And so you fall over dead that way.

• 46:52Speaker 01

  But this is nowhere near to the work.

• 46:57Speaker 01

  It's just that I have to start dragging out this kind of textbook if I want to say how we know that it gets worse.

• 47:05Chris Williamson?

  Oh, my God. How have you not gone insane?

• 47:09Speaker 01

  I decided not to.

• 47:11Chris Williamson?

  Okay. Well, wonderful. That's, I suppose that answers that. All right. Couple of questions that I've had. L L Ms. How likely are they to be the architecture that bootloads super intelligent AI? In your opinion, as far as I'm aware, total mogul in the room. There are some limitations to the level of creativity that L L Ms have in terms of the way that they are able to be creative, to come up with genuinely novel new sorts of things. Have you got a real concern that L L Ms are going to be the architecture that bootloads this? Is there something else that you're more concerned about, which is currently in dark mode or whatever else?

• 47:58Speaker 01

  So the thing is, from my perspective, I have been at this a couple of decades at this point, or, or three decades, if you want to start to count my like crazy youthful self who just wanted to charge out and build super intelligence as fast as possible because it would inevitably be nice.

• 48:15Speaker 01

  And L L Ms have not always been the latest thing in AI.

• 48:21Speaker 01

  There have been, there have been many breakthroughs over the years. L L Ms are powered by a particular innovation called the Transformers, which in some ways is, you know, like crazy simple by the standards of people doing math things in computer science, but possibly not to the point where you want me to launch into an explanation of exactly how it works right here. There's better YouTube videos about that anyway. But the point is, the, the, the underlying circuit that gets repeated to build an L L M, the circuit that gets repeated and then like mysteriously trained and tweaked until nobody knows what the actual contents are, but, but the, the, the form, the structure, the skeleton, that was, that was invented in 2018.

• 49:06Speaker 01

  And we've had some breakthroughs since then, but nothing quite as, as logjam breaking as Transformers, which were the technology that made computers go from not talking to you to talking to you. And, you know, so that's, that's what, seven years ago? It's not the only breakthrough that's ever happened in A I. The more, there was a more recent breakthrough of latent diffusion, which is when A I started drawing pictures that would, you know, be okay, would be decent to look at. There, there were ways of drawing pictures before then called generative adversarial networks or G A Ns, but the, the latent diffusion algorithm was what broke the logjam

• 49:49Speaker 01

  on image, image generation and made it really start working for the first time. And when was that? That I don't remember off the top of my head. Like I want to spitball 2021 or something, but I'm pretty sure that's wrong. So that's like a weaker breakthrough and it's like, I don't know, four years ago or something. The entire field of A I started working because somebody got backprop to work on multi-layer neural networks. You know this as deep learning.

• 50:28Speaker 01

  It did not always exist. It's a batch of techniques that were developed at around the turn of the 21st century. Um, like I could, I could arbitrarily say 2006, but there was more than one innovation there. Um, it's, it started with, if I recall correctly, with unrolling restricted Boltzmann machines. It's now been a while. Um, I, I didn't do it. Jeffrey Hinton did it. Um, uh, and, and then from, but the, once they, once they sort of got that working on multi-layered neural networks at all, there were more innovations since then. Um, clever, more clever ways of initializing them.

• 51:07Speaker 01

  Uh, the atom optimizer, S G D with momentum is like much older than that, but you know, still important. But the point is this is what made sort of the entire modern family of AI systems start working at all. Before then, uh, Netflix, when it was much smaller, ran the most famous, huge, expensive prize there had ever been in artificial intelligence. Open to anyone for a better recommender algorithm for movies. There was a $1 million prize. It was so much money. Everyone got interested in it.

• 51:47Speaker 01

  $1 million was a lot of money back at the turn of the, the, the, the 21st century, which is around when Netflix was running this. I'd have to look up the exact year. It might've been like 2001, 2005. I don't remember. Um, and not till there was a single neural network in the, on some, in, in the ensemble of algorithms that won the Netflix prize. I'd have to look it up, but you know, you, it wasn't a, they didn't, wasn't just like a mighty training run with many GPUs. That was producing a very smart recommender algorithm. Because before deep learning, you couldn't just throw more computing power at training a more powerful AI.

• 52:29Speaker 01

  If you were to say when that happened, that was about 20 years ago.

• 52:35Speaker 01

  So how far are we from the end of the world? It might be that you just throw 100 times as much computing power at the current algorithms and they end the world, or they get good enough at coding and AI research to end the world.

• 52:51Speaker 01

  It could be that it takes one more brilliant algorithm on the level of latent diffusion.

• 52:59Speaker 01

  I think if you throw something that breaks as much loose as Transformers did, my guess starts to be, yeah, that sure sounds to me like it ends the world, but maybe not immediately. Maybe you need like another two years of technology burning first.

• 53:16Speaker 01

  Or, and then if you talk about a breakthrough on the order of deep learning itself, that that seems to me like that just sort of like ends the world in a snap.

• 53:24Chris Williamson?

  A quick aside, using the internet without a VPN today is like leaving your front door wide open and hoping that no one walks in. Websites, apps and data brokers are constantly collecting your personal information, what you search, what you watch, what you buy, where you are. It all gets tracked and Surfshark protects you from that. It encrypts your internet connection so your activity stays private even on sketchy public wifi at airports, cafes or hotels, and it lets you change your virtual location with a single click. Their clean web feature also blocks ads, trackers and malware before they even load, so you stay safer and your browsing is smoother. You can run Surfshark on every device that you own, unlimited installs on one account. And right now you can get four extra months of Surfshark for free by going to the link in the description below or heading to surfshark.com/modernd wisdom

• 54:08Chris Williamson?

  and using the code MODERNWISDOM at checkout. That's surfshark.com/modernd wisdom and MODERNWISDOM at checkout.

• 54:18Chris Williamson?

  Okay, so LLMs could be a really big deal. And there's also a ton of other stuff that could, that we can't see that would be dangerous as well.

• 54:27Speaker 01

  I don't know of the LLMs to go there. Some people are saying that it seems to them like the LLMs are as smart as they get and other people are like, well, did you try GPT-5 Pro for $200 a month or whatever it is that that costs? And other people are going like, yes, I did. It's and like the $200 version of Claude is no better than the $200 version of this.

• 54:50Speaker 01

  And the thing I would say about about this is that if you have some perspective, if you've been watching this for longer than three years, if you have been watching this from before ChatGPT, stuff saturates and then other stuff comes along and breaks through. It doesn't matter if L L M's take you to the end of the world because people are not because they're not going to stick to L L M's.

• 55:18Chris Williamson?

  Okay. What are the range of timelines for this sort of transformative AI that you think are likely?

• 55:30Speaker 01

  I mean, again, everybody wants questions, wants answers like these, just like they'd like to know next week's winning lottery numbers. But if you look over the history of science, I am hard pressed to name a single case of successful prediction of timing of future technology. There are many cases of scientists correctly predicting what will be developed. You can look at the laws, you can look at the physical laws, you can look at the biology laws, you can say, and you can look at that like, hmm, yeah, this sure looks like it ought to be possible. And you can look at it and say, this sure looks like it ought to be possible, and I think I see the angle of attack there.

• 56:11Speaker 01

  Leo Siller in 1933 was crossing a Pickler Street intersection, his name I forget, when he had the insight that we would now refer to as a chain reaction, nuclear chain reaction,

• 56:29Speaker 01

  a cascade of induced radioactivity. Even then, it was known that you could put some materials next to a radio, a source of radioactivity and induce secondary radioactivity.

• 56:44Speaker 01

  And so Leo Siller was like, hmm, we've got these naturally radioactive materials. What if we find something that's naturally radioactive? And furthermore, has the property that you can induce radioactivity in it.

• 57:00Speaker 01

  Duranium 235 was what was eventually settled on, but back then they didn't know that.

• 57:07Speaker 01

  And Leo Sillard saw way ahead in that moment. He saw through to nuclear weapons. He saw that this was not something you should publish in a journal for immediate fame and fortune. He realized that Hitler specifically was likely to be a problem. He did not say, this is going to take $2 billion to turn into a weapon by 1945. There are, off the top of my head, there are zero instances of a scientist ever making a call like that. It is the difference between predicting that an ice cube drops into a glass of water is going to melt and predicting how long it takes to melt and where the individual, where like the individual molecules end up.

• 57:50Speaker 01

  If you point out that on a quantum level, the molecules are indistinguishable, I claim that there's some deuterium in there. So you can't predict.

• 57:57Chris Williamson?

  I get it. Look, look, I imagine, I imagine that that's probably got to be number one on the list of things. People who work in AI safety are sick of being asked.

• 58:09Speaker 01

  A lot of them will run off and answer. A lot of them are not wise enough to realize that they can't answer it.

• 58:13Chris Williamson?

  Hmm. Okay.

• 58:16Chris Williamson?

  I'm going to guess that your confidence interval that it happens before the end of the century is probably pretty high.

• 58:22Speaker 01

  Yeah. I mean, unless we deliberately shut it down, and even then, getting all the way out to the end of the century sounds hard. If you, if you had an international treaty planning this stuff, I would say to go really hard on human intelligence augmentation, because eventually the international treaty will break down. All you can do with it is buy time to have smarter people tackling this problem and tackling humanities problems in general.

• 58:43Chris Williamson?

  Okay.

• 58:44Speaker 01

  But that's a bit of a topic change there. The AI, the people at the AI companies themselves are sometimes naming two to three year timelines. And there is a lesson of history, which says that just because you can't predict when something will happen does not mean that it is far away. Two years before Enrico Fermi personally oversaw the construction of the first self-sustaining nuclear reaction, the first nuclear pile that went critical, he said that that was fifty years off if it could ever be done at all.

• 59:19Speaker 01

  Fermi not being wise enough to realize that he couldn't do timing.

• 59:24Speaker 01

  A couple of years before the Wright brothers flew, one of the Wright brothers said to the other, I forgot if it was Orville or Wilder, man will not fly for a thousand years, but they kept on trying anyway. So it's two years off, but you know, their intuitive sense was it's one thousand years. And of course, AI itself, very famously, there were some people in nineteen fifty five who thought they could make progress on AI, you know, learning to talk, be scientifically creative and self-improve over the course of a summer with ten researchers. This was not a completely unreasonable thing to think because nobody had ever tried it and maybe AI would turn out to be that easy, but it wasn't actually that easy, not in nineteen fifty five.

• 1:00:03Speaker 01

  So the, you know, it could be two years away, it could be fifteen years away.

• 1:00:13Speaker 01

  The AI companies themselves say two to three years, but it's questionable whether we should be taking their words at face value as meaning things as opposed to like hype.

• 1:00:23Chris Williamson?

  But also, yeah, the L L M's, if that architecture is not the one that is going to end up at a place that is super dangerous, then what did they know if they have got all of their chips on this one particular architecture, they're all in on this?

• 1:00:38Speaker 01

  We don't know that.

• 1:00:40Chris Williamson?

  Oh, God, fucking, every time, every time I think I've managed to get some sort of like reprieve, they're like, oh no, what about the super secret OpenAI project that's actually using some other approach?

• 1:00:55Speaker 01

  So the most recent, you know, reasonably large breakthrough in large language models was successfully applying reinforcement learning to chain of thought.

• 1:01:07Chris Williamson?

  And it was a very Can you explain what that means?

• 1:01:10Speaker 01

  So.

• 1:01:12Speaker 01

  Um, if, if you haven't learned anything about L L M since they, they, uh, they started getting hurt about, if you haven't, so like you might have heard that, that L L M's uh, just imitate humans. This is false. They also, you can also have an L L M try to think about how to solve a problem. And then of the like 20 tries it takes at solving the problem, one of those tries works or works best. And then you say, think more like that try at thinking about the problem that succeeded. This is how L L M's go past imitating humans, or it's one of the, one of many ways that L L M's go past imitating humans.

• 1:02:00Speaker 01

  So this is a very, so this is a relatively very obvious thing to do with L L M's, like Paul, Chris, Joe and myself were, you know, talking about that 10 years ago. Um, but uh,

• 1:02:16Speaker 01

  before L L M's actually existed, because it's, that's how obvious it is. Um, but getting it to work, uh, was the, what was like last year or two, maybe?

• 1:02:29Speaker 01

  And OpenAI was, had this like thing called strawberry. And it was, you know, they're like super secret special L L M sauce that they weren't going to tell anyone. It was actually just like reinforcement learning on chain of thought.

• 1:02:44Speaker 01

  Um, but the point is that, uh, this is the level of innovation that AI labs have in the past proven to have and keep secret and that where we later found out what it was. And well, you know, they did get a fair amount of mileage out of that. Out of having AIs try different ways of thinking and reinforcing the one that worked to solve objectively verifiable problems like math or programming and so on.

• 1:03:11Speaker 01

  So this is, you know, like the AI companies could potentially have a replacement for LLMs that they have discovered and are keeping secret from us. More likely is that they would have something that was on the order of reinforcement learning on chain of thought, which is, you know, when AI started to get good at coding. Or they might have nothing on that order up their sleeves at the moment. And that's why people are currently claiming that the latest wave of LLMs do not seem fundamentally smarter than the LLMs from three months ago or six months ago, which is what today's young whippersnappers think is an AI winter.

• 1:03:54Speaker 01

  Let's see your field stagnate for 10 years and eventually break through before you talk to me about winter, young kids.

• 1:04:05Chris Williamson?

  Okay, brilliant, brilliant.

• 1:04:10Chris Williamson?

  I have no idea why I even want to ask you. I want to know why experts aren't worried and I also want to know what you make about AI companies. Let's talk about the experts. Why, um, obviously some people's wages are dependent on this train staying on the tracks. That means it's very difficult to convince somebody, what's that quote? It's very difficult to convince somebody of something that their wage depends on them not being convinced of. What about the other thinkers, researchers in this space? What is it that you, that they are most commonly missing that you think?

• 1:04:55Chris Williamson?

  Why are they making their fundamental thinking errors when it comes to we will be fine with just continuing on AI growth?

• 1:05:07Speaker 01

  So first of all, Jeffrey Hinton, the Nobel, the guy who won the Nobel Prize

• 1:05:14Speaker 01

  in physics for being among the people most directly pinpointable as having kicked off the entire revolution in getting back prop to work on multi-layer neural networks, or as it's now currently known, deep learning. Like the point where AI started working at all. Jeffrey Hinton, I think, is on record as recently saying he quit his job at Google and then could speak freely.

• 1:05:42Speaker 01

  Saying something like intuitively, it seems to him like it's 50% catastrophe probability, but based on other people seeming less concerned, he adjusted down to 25%. I could be misquoting here. I'm trying to do this from memory.

• 1:05:56Speaker 01

  So are you asking, so many people would consider this to not be a lack of concern. Like the guy say, like somebody being like, well, it looks to me like a coin flip, whether or not he destroy the world. This is not what you want to hear from your Nobel laureate scientist who helped invent the field and left Google to be able to speak freely about it. So he no longer has a financial stake in making it bigger or smaller one way or the other.

• 1:06:23Speaker 01

  Many people would call this already a high degree of scientific alarm.

• 1:06:31Speaker 01

  Yoshua Bengio was one of the co-founders of deep learning. He won the computer, he co-won the computer science award with Jeffrey Hinton, the Turing Prize for inventing deep learning. Yoshua Bengio is also, I think, on the concern list. I don't off the top of my head have a direct quote from him about probabilities.

• 1:06:50Speaker 01

  It is true that I am more concerned than they are.

• 1:06:54Speaker 01

  I would, and I realize that this may sound, you know, somewhat eubistic, attribute this to them being relative newcomers to my field who may not have like gotten acquainted with the full list of reasons why it is hard to align AI.

• 1:07:09Speaker 01

  That said, coin flip odds of destroying the world is still not what you want to be hearing

• 1:07:15Speaker 01

  from your relatively more senior scientists who are relatively newer to the field.

• 1:07:20Speaker 01

  Well, relatively newer to my field. They are vastly my seniors in artificial intelligence itself, of course. I am like speaking tongue in cheek whenever I accuse people of being whipper snappers like Jeffrey Hinden could say that with a straight face. I am just like, you know, bit of light self mockery there about how I'm not Jeffrey Hinden.

• 1:07:39Speaker 01

  But that said, you know, if you are relatively newer to this, you might think like, well, you know, maybe we've just got to use reinforcement learning to make the AIs love us the way a child loves a parent or love us the way a parent loves a child and not quite have at your fingertips the top six reasons why that is hard and principled obstacles to them and what will go wrong there.

• 1:08:05Speaker 01

  So that is what prevents the like, the famous inventors of the field who only started speaking out about their concerns relatively recently after leaving their companies to, you know, and are now financially dependent of stakes on their opinion. That's what makes them be like 50 50, the world gets destroyed instead of my own thing where I'm like, yeah, it's predictable that the world gets destroyed if you keep doing this. But if you ask like, what's responsible for Sam Altman at OpenAI? Not, you know, possibly having less than 50% odds. Who knows what that guy's really thinking? Well, you can like trace out his long trail over time of him initially saying like, AI will end the world.

• 1:08:50Speaker 01

  But in the meanwhile, there will be great companies.

• 1:08:55Speaker 01

  To him, sort of like saying less and less alarm astounding things in front of Congress, like where Congress asks him like, well, you talk about the world ending. By that, do you mean like mass unemployment? And Sam Altman hesitates for two seconds and replies, yes. Was the lovely like congressional hearing thing that happened, I think about a year back now. So what's going on with the AI companies?

• 1:09:21Speaker 01

  I'm not telepaths. I can't read their minds. I would point out that it is immensely well precedented in scientific history, in the history of science and engineering, for companies that are making short-term profits to do really sad amounts of damage vastly disproportionate to the profit that they are making.

• 1:09:45Speaker 01

  And to be in apparently sincere denial about the negative effects of what they are doing. Two cases that come to mind are leaded gasoline and cigarettes. I don't know if you would be familiar off the top of your head with the case of leaded gasoline. Probably even the kids today have heard about cigarettes.

• 1:10:04Speaker 01

  The cigarette companies did way more damage to human life in cancer and other health effects than they made in profits. Like they did make a few billion dollars in profit selling cigarettes, but nothing remotely compared to the to the cost of human life. It's not that they were, you know, like this was an immensely negative sum game. They were doing enormously more damage than the profits that they were making. And any particular advertising professional who got up in the morning and figured out how to market cigarettes to teenagers. Any of the scientists that they paid to to write stories about how you couldn't really tell whether or not cigarettes were causing lung cancer would have made a tiny, tiny fraction of the of the total profit of the cigarette companies.

• 1:10:49Speaker 01

  Their CEO would not have made that large a fraction of the total profit of the cigarette company. So they went off and participated in this thing that, you know, caused lung cancer to I don't know how many millions of people. And and for what? For this very small profit. How could a human being bring themselves to do that? Through a very simple alchemy. First, you convinced yourself that what you're doing is not causing the harm, which is just a very easy thing for human beings to do all the time, all throughout the entire recorded history of humanity. And then, once you've convinced yourself that you're not doing that much harm, well, let's start in taking money to not do any harm.

• 1:11:31Speaker 01

  Leaded gasoline caused brain damage to tens, maybe hundreds of millions of developing brains, the United States and elsewhere. It caused brain damage to children. For what? The gas companies making leaded gasoline could have, you know, made unleaded gasoline. It's not that they would have gone out of business if they'd somehow gotten together and decided to stop making leaded gasoline. If they hadn't opposed the regulations that were trying to ban leaded gasoline before it turned into a big deal back in the 1930s. There was an attempt to have regulations against leaded gasoline. Lead was known to be poisonous in large quantities.

• 1:12:15Speaker 01

  Why let people spray it all over the place, even in smaller quantities? But the gas companies got to go. They managed to prevent that legislation from passing. They poisoned an entire generation. And for what? For gas that burned about 10% more efficiently, I think was what leaded gasoline basically got you. For it being more convenient to add lead to the gas instead of adding ethanol to make it burn more smoothly inside of car engines. Trivial, trivial, trivial compared to the... This is not a conspiracy theory.

• 1:12:59Speaker 01

  This is standard medical history I'm talking about here. Like, I've seen estimates of five points off the tested IQs. And you can look at the chart of which states banned leaded gasoline when and watch the drops in the crime rate. Because it makes you, you know, it disposes you to be more violent, not just stupid, that tiny little bit that hit child after child after child.

• 1:13:32Speaker 01

  Why? Why would anyone cause that amount of damage? For, for, because you, you got your CEO salary of, of a company that didn't have, then didn't need to go to the inconvenience of adding ethanol to gasoline instead?

• 1:13:49Speaker 01

  Cause first you convince yourself it's safe. First you convince yourself you're doing, you're doing no harm, which is just an easy thing for human brains to convince themselves of. And then why not oppose the gas, the, the, the legislation against leaded gasoline? It's not doing any harm, right?

• 1:14:07Speaker 01

  Ronald Fisher, one of the inventions, one of the inventors of modern scientific statistics, testified against it being knowable that cigarettes caused lung cancer because you see a, no, no, no proper controlled experiment had been done on cigarettes causing lung cancer. And so how could you possibly, possibly know from your observational studies?

• 1:14:29Speaker 01

  Showing 20 times the chance of cancer? If you were a smoker? How could you possibly know from your correlational, correlational studies? And Fisher himself was, was a heavy smoker. He actually, he drank his own Kool-Aid. The inventor of leaded gasoline, I think, had to go away to a sanitarium at one point because of how much he managed to poison himself with lead. He drank his own Kool-Aid. They really managed to convince themselves that they were doing no harm. And so they could do arbitrarily vast amounts of harm in exchange for these tiny, comparatively tiny, tiny profits.

• 1:15:06Speaker 01

  And to say this is not a substitute for actually tracking the object level arguments about whether or not AI will kill you and for what reason? You cannot figure out what will happen as a matter of computer science if you build a super intelligence and switch it on by pointing out at who has what tainted motives? To, you know, who has what incentives to say what?

• 1:15:31Speaker 01

  But it but having tried in my book to in my in my innate stories is book to to make the case for why on an object level, this is what happens if you build a super intelligence and switch it on to ask why the people paying being paid literally hundreds of millions of dollars by Meta to be AI researchers, why people like Sam Altman, who, you know, I mean, he, you know, didn't quite get paid billions of dollars. He was supposed to be CEO of a nonprofit. He actually stole billions of dollars. But, you know, why the guy stealing billions of dollars in equity from the public that was supposed to own it. Like, like, how does he manage to convince himself that what he's doing is okay? Well, maybe it's

• 1:16:16Speaker 01

  not even convinced, you know, we do have him on the record as saying a few years earlier, like AI will end the world. But in the meantime, there'll be great companies. You know, maybe, maybe he's just like, yeah, sure. You know, like, the world's gonna end, but I get to be important. I get to be there. You know, sure. Who but I could be trusted with this power?

• 1:16:33Chris Williamson?

  Do you think you think that that's the position that a lot of the guys at the heads of these AI companies believe?

• 1:16:42Speaker 01

  I'm not a telepath. I can't tell you what these people are actually thinking. You got to distinguish between stuff you can possibly know and stuff you can't. But their overt language and has has often been like, well, building super intelligence is inevitable. Who could possibly stop that? An international treaty could possibly stop that. A coalition of major nuclear powers could stop that. But leading that aside, that they may have convinced themselves that's not going to happen. Who could possibly stop anyone from building super intelligence? So I need to build it. I only I can be trusted to build it.

• 1:17:16Speaker 01

  Is what their overt rhetoric has sort of been.

• 1:17:19Chris Williamson?

  Oh, okay.

• 1:17:20Speaker 01

  But but the main thing I'm trying to point out is that having presented the object level case that super intelligence will kill everyone, to ask the question of how could these companies possibly believe that this thing bringing them immense short term profits and letting them be the most important guy in the room is, you know, not going to end the world is something enormously well precedented in the history of science. If I'm saying that's to the extent you might think that's what happened, a very ordinary thing happened, not an extraordinary thing. The thing happened that has happened a dozen times before.

• 1:17:53Speaker 01

  If they managed to convince themselves that they were doing no harm.

• 1:17:56Chris Williamson?

  Okay.

• 1:17:57Speaker 01

  Or, you know, only an acceptable amount of harm, only running the 25% chance of destroying the world, whatever it is they think is acceptable.

• 1:18:04Chris Williamson?

  Uh, I'm trying to work out

• 1:18:09Chris Williamson?

  what the solution is. Do you have any proposed solutions that makes this seem slightly less apocalyptic?

• 1:18:20Speaker 01

  My best I have to offer is, is the same solution that humanity used on global thermonuclear war. Don't do it. Like, don't, instead of having the nuclear, the global thermonuclear war and trying to survive it, which, which, which sort of nuclear war might have worked, don't have the nuclear war. We managed to do that. It's the, the best sign of hope I can offer you. It is, it is slightly harder for AI in some ways, if not others. But, you know, people going into the 1950s, 1960s, they thought they were screwed. And that wasn't them indulging in some nice doom scrolling pessimism, luxuriating and then the pleasant feeling of being doomed. This was people who did not want to be doomed.

• 1:19:04Speaker 01

  But they looked at the course of human history over the last century. They looked at World War One.

• 1:19:11Speaker 01

  They looked at how in the aftermath of World War One, everyone had said, let's not do that again. And then there'd been World War Two. They had some reason to be worried about nuclear war. They had some reason to expect that no country was going to turn down the prospect of making nuclear weapons. They had some reason to believe that, you know, once a great, a bunch of great powers had a bunch of nuclear weapons, why, of course, they would go to war anyway and use those nuclear weapons. It was apparently to them what had happened with World War Two, all, all these people saying, we must not have another world war, and then the world war happening anyway. Why didn't we have a nuclear war? Well, on my account of it, it is because for the first time in all human history, all

• 1:19:55Speaker 01

  the great powers, all the leaders of the great powers, understood that they personally were going to have a bad day if they started a major war.

• 1:20:06Speaker 01

  And people had pretended before to claim proclaim that, you know, war is a very terrible thing that should never be done. It wasn't quite the same level of personal consequence. You know, maybe as maybe as a general secretary of the Soviet Union, you would think that if you started a nuclear war, you would personally survive, you'd end up in a bunker somewhere. They wouldn't be going to your favorite restaurants in Moscow ever again. And that was not the situation that obtained before the start of World War I, the start of World War II. People might make a bunch of, you know, like, it only takes one side to think that they might have a bit of an advantage in the sport in war, the sport of kings, to, you know, to take a kick off that that fun adventure of trying to conquer

• 1:20:50Speaker 01

  another country, which, you know, wasn't as fun as much fun for Adolf Hitler as he expected. But you could see how Adolf Hitler might have thought that he was going to have a nice day as a result of invading Poland. And that's what changed, that the general secretary of the Soviet Union and the president of the United States actually personally expected to have both sides expected to personally have bad days if they started a nuclear war. They would not have any better of a bad, any better of a good day if anyone anywhere on earth built a superintelligence.

• 1:21:23Chris Williamson?

  Uh huh. Yeah, it's this sort of lack, the tragedy, kind of like a tragedy of the commons. It's just this tragedy that everybody's fucked, right? It's like everything, everything gets blown up, no matter who it is that builds it. Well, okay.

• 1:21:35Speaker 01

  The common sense is that the commons get overgrazed because the individual farmers benefit from setting their cows loose on it.

• 1:21:42Chris Williamson?

  Uh huh.

• 1:21:43Speaker 01

  And the thing with nuclear war is that you might get a bit of a benefit by dropping a tactical nuclear weapon on, you know, like, you know, like, the United States could get, it could get an immediate benefit by dropping tactical nuclear weapons on the Russian troops in Ukraine. And Russia could get an immediate benefit by dropping tactical nuclear weapons on Ukraine. But neither of them is going to risk the global thermonuclear war that might follow happening with a greater probability.

• 1:22:13Speaker 01

  So it's not, so it's not a tragedy, like it's not a classic tragedy of the commons. The thing that stopped nuclear war is that although you could get a short term advantage from dropping a tactical nuke, or even like dropping a strategic nuke on one city, the leaders understood how this was a, you know, like increasing the probability of a global thermonuclear war. They managed to hold off from doing that for that reason. They understood the concept of how it escalated things. They saw the connection to not getting to go to their favorite restaurants again, even if they were surviving in a bunker somewhere. And with artificial intelligence, what we've got is a ladder where

• 1:22:55Speaker 01

  every time you climb another step on the ladder, you get five times as much money. But one of those steps of the ladder destroys the world and nobody.

• 1:23:07Speaker 01

  And maybe if this true fact can become something that is known and believed by the leaders of a handful of major nuclear powers, they can all be like, all right, we're not climbing any more rungs of this ladder. It is not in my interest that you start to climb this ladder. And it's not even my own interest to break apart the treaty by climbing another step of this ladder because then we're all just going to keep climbing and then we're all going to die.

• 1:23:36Chris Williamson?

  Uh huh.

• 1:23:36Speaker 01

  That is the ray of, that is the best ray of hope I can offer you. That we managed to not do the stupid thing, the same as we managed to not have a nuclear war despite many people being concerned for excellent reasons that it was going to be an impossible slope not to fall down.

• 1:23:53Chris Williamson?

  Okay, so what do we actually do?

• 1:23:56Speaker 01

  Well,

• 1:23:59Speaker 01

  you know, voters do not necessarily have all that much power under the modern political process. But I think, but like the next step for the United States might be something like the president saying, you know, like we're of course not going to give up AI unilaterally, which wouldn't even solve anything in its own way. But we stand ready to, you know, join with an international, international treaty, international alliance, whose purpose is to prevent further escalation of AI intelligence, further escalation of the AI ladder.

• 1:24:36Speaker 01

  We're not going to do it unilaterally, but we're ready to get together and do it everywhere. And China has already sort of like, hasn't quite said that, but they've sort of indicated openness to international arrangements meant to prevent human loss of control from AI. You'd want Britain to say the same thing. So and then if a bunch of leaders of major powers have said like, yeah, we would join in an arrangement to prevent this from getting out of control and everybody at earth, you know, ending up dead. Then you can, from there, you can go on to the actual treaty. What can voters do? Well, you can write your elected officials is among the things you can try to do there.

• 1:25:18Speaker 01

  There's a, if you go to ifanyonebuildsit.com.

• 1:25:26Chris Williamson?

  Can't believe that you got that URL. Brilliant. Okay.

• 1:25:29Speaker 01

  Yeah.

• 1:25:30Chris Williamson?

  Yeah.

• 1:25:30Speaker 01

  Anyonebuildsit.com and you and you click on where it says act. You'll see our guide to calling your representatives. And if you click on march, you'll see a place where you can sign up to march on Washington DC if 100,000 other people also pledged to march on it. And this does not, this, for this to just happen, the United States does not solve the problem because this is not a regional problem where you ban superintelligence inside your own country and then your own country is safe. But this sort of thing can exert some amount of influence on politicians and more importantly can make it clear to them that they're allowed to discuss it, that they're allowed to want

• 1:26:15Speaker 01

  to not die themselves. There are multiple congresspeople who I'm not going to name, but whom we have talked to, who would, you know, prefer that America not die along with the rest of the world, but it doesn't quite seem like the sort of thing you're allowed to speak out in public about yet. Voters can make it clear to their politicians that the politicians are allowed to speak out. There's already, there's already like seventy percent, like if you actually survey American voters, seventy percent of them say they do not want superintelligence. But you know, that's not enough for the politicians to feel licensed to act. But maybe, you know, if you call them and if you march in Washington, you know, that's, that's what you can do as an individual voter.

• 1:26:56Chris Williamson?

  Well, I applaud you for trying to get some grassroots stuff going. Congratulations. You've been frank throughout this conversation. I think it's fair for me to be frank here. It does feel a little bit like you're outgunned. Legislation tends to move more slowly than technology does by many, many years, sometimes decades.

• 1:27:21Chris Williamson?

  It, it, it just feels bleak. It feels, it feels, if what you say is true, it really is kind of fluke that gets us to a stage where this goes well, because the likelihood of some moratorium being placed where all AI development is halted and, and all efforts are placed on this, you only need one bad actor to do it, which, because again, it's if anybody builds it.

• 1:27:45Speaker 01

  Well, you, you don't want international treaty to, you know, fall over if North Korea steals a bunch of GPUs. You, you do want the treaty to say if North Korea steals a bunch of GPUs and, and builds a, you know, unlicensed data center, then we will clearly communicate diplomatically what is about to happen. And then if North Korea still proceeds, we will drop a bunker buster on, on their data center.

• 1:28:10Chris Williamson?

  That assumes that you know that you are somehow able to detect and that no one can do it surreptitiously.

• 1:28:17Speaker 01

  It is hard to surreptitious a data center. They consume a lot of electricity.

• 1:28:21Chris Williamson?

  Okay, so we can see most of the ones in Russia and China or North Korea?

• 1:28:29Speaker 01

  Like, I'm not sure who is looking for them at the moment. And if you can, and to what extent these things show up on satellites and to what extent these things show up on, you know, intelligence reports. But there has previously been an issue of detecting covert nuclear refineries

• 1:28:48Speaker 01

  in terms of nuclear nonproliferation. And this was not an unsolvable problem. And data centers are, if anything, even higher profile than nuclear refineries.

• 1:28:58Chris Williamson?

  Right, so we are going to threaten some people with?

• 1:29:03Speaker 01

  I mean, I wouldn't use the word threaten. I would say that if North Korea is building an unsupervised data center, then you should actually be terrified for your lives and lives of your children. And you tell North Korea this plainly and truthfully. And then if they don't drop, you know, if they don't shut down their data center, you drop a bunker buster on it. And you do this even though North Korea has some nuclear weapons of its own.

• 1:29:31Chris Williamson?

  Okay, so pressure from people on their elected representatives through mail, marches, more awareness to get the government officials to come up with an international treaty to get countries to agree that, what specifically?

• 1:29:55Speaker 01

  We're not making AIs any smarter than they are already.

• 1:30:01Speaker 01

  We are putting the chips that can be used to build the more powerful AIs into locations where they, where their uses are supervised.

• 1:30:13Speaker 01

  I would say ideally you are putting the chips that run the AIs into locations where they can be supervised. As a minor side effect, maybe you can stop the AIs from driving people insane. It seems like the sort of thing you could better do if this was all happening under supervision by international treaty. As you know, it's not vital to humanity's survival that AIs be prevented from driving people insane, but it's, serves as a kind of test case of can you, you know, stop the damage? Like, is humanity in control here? Can we stop AIs from rotating upon some of our human people?

• 1:30:49Speaker 01

  But you know, that's not the main thing here. It's a thing that some people will find attractive, but it's not the main thing. You're trying to like just get the whole AI thing under control. And then you're trying to stop the further escalation of AI capabilities up the ladder.

• 1:31:07Chris Williamson?

  It is scary.

• 1:31:10Chris Williamson?

  It is one of these things, and I imagine that it feels it must feel a little bit like this to you, that everybody is sort of dancing their way through a daisy field of, oh, I've got this personal coach in my pocket and it's so cool. And I get to talk to it about all of my psychological problems. God, I can bitch to it about my husband. It just listens. And at the end of this daisy field that everyone's having a load of fun in is just like a huge cliff that descends into eternity. And there's like a balrog at the bottom or something. Is that what it feels like?

• 1:31:51Speaker 01

  Yeah, pretty much. But the future is hard to predict. It is genuinely hard to predict. I can tell you that if you build a superintelligence using anything remotely like current methods, everyone will die. That's a that's a pretty firm prediction. The part where people maintain their but like the part where people maintain the daisy field attitude that they had a few years earlier toward AI, that has already shifted to some degree just because of the chat GPT moment. And nobody predicted that in advance. Nobody knew that nobody had opened AI, as far as I can tell, had any idea that when they released chat GPT, they were going to be causing a massive shift in public opinion about AI as people realized the AI was actually talking to them. Now it's

• 1:32:35Speaker 01

  sounding kind of intelligent about it. So maybe it also, maybe I don't want to wait for anything else to happen. Maybe chat GPT was the miracle we got. I wasn't expecting that much of a miracle. I did not call it in advance. Um, but maybe we get another miracle. I don't want to sit around waiting for it because I can't tell you the miracle will like occur on such and such a day, but you know, maybe, maybe the AIs managed to do something more destructive than driving a few people insane, breaking up a few marriages and like, uh, you know, causing whatever further decline in birth rates is going to be caused here.

• 1:33:15Speaker 01

  Uh, maybe they do worse than that and that shifts opinion. Maybe they just get more powerful and smarter and are clearly no longer choice and there's, and that shifts opinion even without a giant catastrophe.

• 1:33:30Speaker 01

  It's not clear to me, you know, as, as much as people love to, to bitch about their elected leaders, it is not clear to me that we are looking at

• 1:33:43Speaker 01

  permanent obliviousness to the aliens getting smarter and smarter.

• 1:33:50Speaker 01

  Like people are currently saying completely wacky and oblivious things because they think that's what's politically mandatory to say in the current political environment and that you have to talk about jobs rather than the other extinction of humanity. But it's, it's just not clear. The, the future is very hard to predict in general. It's not clear to me that the current state of obliviousness is something supreme, unmovable and impossible for any event to change or to change or that it won't just disintegrate on, on its own as more people talk about it. There, there's a level in which you, you, you kind of have to be

• 1:34:25Speaker 01

  pretty dumb to look at this smarter and smarter alien showing up on your planet and not have the thought cross your mind that maybe this won't end well.

• 1:34:36Speaker 01

  Can, can even elected politicians be that dumb? Yes, absolutely. It is not known to me to be prohibited that this can be the case. Do they have to do the stupid thing? It's not clear to me that it's mandatory. We did manage to have, we, we did manage to not have a nuclear war and people did not think they were going to get that much luck.

• 1:34:58Chris Williamson?

  Elliot Zajukowski, ladies and gentlemen.

• 1:35:02Chris Williamson?

  I hope you're fucked. I was prepared coming in, but I'm not sure that the rest of the audience will be. So dude, the best compliment I can pay you is I hope you're wrong, but I fear you're not.

• 1:35:15Speaker 01

  Yeah, being wrong, it'd be great to be wrong. I'd love to be wrong.

• 1:35:19Chris Williamson?

  That'd be wonderful.

• 1:35:19Speaker 01

  You know, I...

• 1:35:20Chris Williamson?

  That'd be wonderful.

• 1:35:21Speaker 01

  I want... Let me assure you, everyone, by making it by way of like making, you know, like destroying any shred of optimism you might previously have had. I completely would have other career options lined up and other ways of supporting myself if I was completely wrong. Like, not just me, but some like sensible people who donated me a bit of appreciated currency, wanted to make sure that I could, if I was, you know, if I changed my mind about this sort of thing, just retreat from my entire career path and not end up in financial trouble.

• 1:35:53Speaker 01

  And yet, here I am. You know, so I'd love to be wrong. We have tried to arrange it to be the case that I could at any moment say, yep, I was completely wrong about that and everybody could breathe a sigh of relief and it wouldn't be like the end of my ability to support myself and I would have other things to do. We have made sure to leave a line of retreat there. Unfortunately, as far as I currently know, I continue to not think that I... that it is time to declare myself to have been wrong about this.

• 1:36:22Chris Williamson?

  Heck, yeah. All right, Elliot Zaj, well, if the internet is still alive in a little bit of time in the future, we can check back in and see just how right you were.

• 1:36:32Speaker 01

  Well, every year that we're still alive, it's another chance for, you know, something else to happen.

• 1:36:41Chris Williamson?

  What a wonderful way to finish, dude. I appreciate you. Thank you so much for your work.

• 1:36:45Speaker 01

  Thank you for having me over to deliver the bad news.

• 1:36:53Speaker 02

  Close your eyes, exhale, feel your body relax, and let go of whatever you're carrying today. Well, I'm letting go of the worry that I wouldn't get my new contacts in time for this class. I got them delivered free from 1-800-CONTACTS. Oh, my gosh, they're so fast. And breathe. Oh, sorry. I almost couldn't breathe when I saw the discount they gave me on my first order. Oh, sorry. Namaste. Visit 1-800-CONTACTS.com today to save on your first order.

• 1:37:21Speaker 03

  1-800-CONTACTS.