Eliezer Yudkowsky and Stephen Wolfram on AI X-risk

Eliezer Yudkowsky and Stephen Wolfram discuss artificial intelligence and its potential existen‑ tial risks. They traversed fundamental questions about AI safety, consciousness, computational irreducibility, and the nature of intelligence. The discourse centered on Yudkowsky’s argument that advanced AI systems pose an existential threat to humanity, primarily due to the challenge of alignment and the potential for emergent goals that diverge from human values. Wolfram, while acknowledging potential risks, approached the topic from a his signature measured perspective, emphasizing the importance of understanding computational systems’ fundamental nature and questioning whether AI systems would necessarily develop the kind of goal‑directed behavior Yudkowsky fears. *** MLST IS SPONSORED BY TUFA AI LABS! The current winners of the ARC challenge, MindsAI are part of Tufa AI Labs. They are hiring ML engineers. Are you interested?! Please goto https://tufalabs.ai/ *** TOC: 1. Foundational AI Concepts and Risks [00:00:01] 1.1 AI Optimization and System Capabilities Debate [00:06:46] 1.2 Computational Irreducibility and Intelligence Limitations [00:20:09] 1.3 Existential Risk and Species Succession [00:23:28] 1.4 Consciousness and Value Preservation in AI Systems 2. Ethics and Philosophy in AI [00:33:24] 2.1 Moral Value of Human Consciousness vs. Computation [00:36:30] 2.2 Ethics and Moral Philosophy Debate [00:39:58] 2.3 Existential Risks and Digital Immortality [00:43:30] 2.4 Consciousness and Personal Identity in Brain Emulation 3. Truth and Logic in AI Systems [00:54:39] 3.1 AI Persuasion Ethics and Truth [01:01:48] 3.2 Mathematical Truth and Logic in AI Systems [01:11:29] 3.3 Universal Truth vs Personal Interpretation in Ethics and Mathematics [01:14:43] 3.4 Quantum Mechanics and Fundamental Reality Debate 4. AI Capabilities and Constraints [01:21:21] 4.1 AI Perception and Physical Laws [01:28:33] 4.2 AI Capabilities and Computational Constraints [01:34:59] 4.3 AI Motivation and Anthropomorphization Debate [01:38:09] 4.4 Prediction vs Agency in AI Systems 5. AI System Architecture and Behavior [01:44:47] 5.1 Computational Irreducibility and Probabilistic Prediction [01:48:10] 5.2 Teleological vs Mechanistic Explanations of AI Behavior [02:09:41] 5.3 Machine Learning as Assembly of Computational Components [02:29:52] 5.4 AI Safety and Predictability in Complex Systems 6. Goal Optimization and Alignment [02:50:30] 6.1 Goal Specification and Optimization Challenges in AI Systems [02:58:31] 6.2 Intelligence, Computation, and Goal-Directed Behavior [03:02:18] 6.3 Optimization Goals and Human Existential Risk [03:08:49] 6.4 Emergent Goals and AI Alignment Challenges 7. AI Evolution and Risk Assessment [03:19:44] 7.1 Inner Optimization and Mesa-Optimization Theory [03:34:00] 7.2 Dynamic AI Goals and Extinction Risk Debate [03:56:05] 7.3 AI Risk and Biological System Analogies [04:09:37] 7.4 Expert Risk Assessments and Optimism vs Reality 8. Future Implications and Economics [04:13:01] 8.1 Economic and Proliferation Considerations SHOWNOTES (transcription, references, summary, best quotes etc): https://www.dropbox.com/scl/fi/3st8dts2ba7yob161dchd/EliezerWolfram.pdf?rlkey=b6va5j8upgqwl9s2muc924vtt&st=vemwqx7a&dl=0

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Speaker 00

  I think that the broad situation with AI is that they're currently successfully scaling them more and more powerful. We're not quite sure how long that process is going to continue using the current technology. And part of the reason we're not sure is that nobody understands what actually goes on inside a modern AI. We can look inside the computer, of course, but we see giant arrays of floating-point numbers, and people are barely beginning to understand what's going on in there. And we do not quite understand where the power comes from. Chess did not stop playing when it got to human level. Go did not stop playing by the human level. They are past only training the AIs on imitations of human output and have moved on to training it on what works to solve problems. And you can just continue that process past the point where the problems are of human difficulty.

• 0:46Speaker 00

  So we're on course to get something smarter than us, whose innards we don't understand, which we cannot very well control. And this probably ends quite poorly.

• 0:57Speaker 01

  We're making the statement that the one thing we think we know about the AIs is that they have been successfully trained to optimize their achievement of purposes. That's an assumption which is not totally obvious to me. I agree that as you give sort of all you're specifying is sort of play chess or whatever else, or you're specifying some big, what we think of as an objective. The details of what's happening inside, there will be aspects of that that we are not in any way able to foresee, predict, whatever else. So if we took apart that mechanism and we say, is this mechanism doing what we expect here? It won't be. There'll be plenty of things where it's doing what it chooses to do, or because that particular

• 1:41Speaker 01

  training run gave this particular result, or whatever else. To me, I'm not convinced yet. There are interesting questions to try to answer, which people should try and answer. I mean, you make it sound like it's urgent, you know, the people are coming off the ships, and you might be right.

• 2:00Tim Scarfe?

  MLST is proudly sponsored by Two for AI Labs. They're based in Zurich, they're doing monthly meetups, and they are hiring cracked MO research engineers to work on ARC and to make progress towards AGI. Go and apply now, twoforlabs.ai.

• 2:15Speaker 03

  What matters is the strong desire to solve the challenge and the technical capability. Usually the type of people we tend to find have computer science backgrounds.

• 2:28Tim Scarfe?

  Gentlemen, it's an absolute honour to have you both on MLST. Thank you so much for joining us today. We're going to have a discussion all about AI existential risk. And Eliezer, first, if you wouldn't mind, could you just spend about five minutes talking about how much of a risk is AI to humanity?

• 2:48Speaker 00

  I think the term risk is understating it. If there were an asteroid straight on course for Earth, we wouldn't call that asteroid risk. We'd call that impending asteroid ruin or something like that. And I think that the broad situation with AI is that

• 3:05Speaker 00

  they're currently successfully scaling them more and more powerful. We're not quite sure how long that process is going to continue using the current technology. And part of the reason we're not sure is that nobody understands what actually goes on inside a modern AI.

• 3:21Speaker 00

  We can look inside the computer, of course, but we see giant arrays of floating point numbers and people are barely beginning to understand what's going on in there. And we do not quite understand where the power comes from. We, like, by great efforts, you can look in like the previous generation of the technology. I don't know if anybody's tried it with the current generation. And see, it's currently thinking about the Eiffel Tower. Or this is like the location where it stores the fact that the Eiffel Tower is in France and we can move it to Rome by poking at the numbers, which is, you know, pretty impressive. We can't do that with a human brain. But the basic question of sort of like, what's going on in there?

• 4:05Speaker 00

  How are they doing the impressive new parts? That's all unknown. If the technology continues to scale, which it may or may not, nobody knows, it at some points gets smarter than us. As smart as us and then smarter than us, there's no particular barrier that I know of, or that I know of any principled reason to believe in at the human level. Chess did not stop playing when it got to human level. Go did not stop playing by the human level. They are past only training the AIs on imitations of human output and have moved on to training it on what works to solve problems. And you can just continue that process past the point where the problems are of human difficulty. So we're on course to get something smarter than us, whose innards we don't understand,

• 4:50Speaker 00

  which we cannot very well control. And this probably ends quite poorly. There's a lot of mistaken reasons to believe that it automatically ends well. It's various people have different mistaken reasons to believe it automatically ends well. There's a, I don't know if any of you, I don't know if you, the viewer, I don't know if any of the other people here have this particular mass apprehension, but people ask, why wouldn't they just trade with us? They have heard of Ricardo's law of comparative advantage, a very important theorem in economics, which says that even if one country is more productive at producing every kind of good

• 5:31Speaker 00

  than some other country, they will still end up trading because of the relative differences in productivity. If it is easier for me to make hot dog buns than it is for me to make hot dogs, I will ship you hot dog buns and get back hot dogs. Or like if compared to you, I am, I'm at less of a disadvantage in making hot dog buns than I'm at a disadvantage in making of hot dogs. We will both benefit by shipping hot dog buns and hot dogs back and forth. This unfortunately assumes the absence of a third option, which is for B to be so much more powerful than A that they kill A and take all their stuff. There is not a theorem saying that the work produced by horses has to be worth, worth

• 6:16Speaker 00

  more than the cost of feeding the horses. You can always benefit by trade, but you can sometimes benefit even more by killing people and taking their land. I wish that wasn't a possibility, but it is. And so there's all these reasons that people have to have for thinking that even if we don't understand AIs and can't control them very well, they could get much, much smarter than us and it could still end nicely. And I think these reasons are all mistaken. And so by default, this ends terribly. And I think that's my five minutes.

• 6:46Speaker 01

  I mean, I guess my, it's interesting. I mean, my first statement is this notion of kind of an index of smartness is something that I kind of don't believe in. I mean, one thing that's sort of my personal experience for the last, I don't know, forty something years is I already know the computers are smarter than me. I mean, I, you know, it's kind of, I started doing these experiments on kind of what's out there in the computational universe years ago, just looking at very tiny, simple programs where, you know, you might have thought this is a really simple program. Surely I'm smart enough to figure out what this does, but no one's not. I mean, every, like, even last night, I was doing some experiments where I was pretty sure I knew

• 7:28Speaker 01

  what was gonna happen, but no, the, you know, the computational animal was smarter than I was, and it did things I didn't expect. So I've, I've kind of, I guess, my, you know, my default point of view is I know the computational universe contains many things that I can't readily foresee, just as the physical universe contains many things I can't readily foresee. So the question then is, you know, one thing one could take the point of view, there's this kind of single index of smartness. I mean, I think people, you know, in the nineteen thirties, people wanted to invent kind of an index of general intelligence. They called it G for humans, which I've never really believed in, because I've never really believed that there

• 8:10Speaker 01

  are some things I'm pretty good at doing, where I feel I'm pretty smart. There are other things where I know I'm pretty dumb, and it's, it's kind of a, it's not really a sort of single index. I guess, I guess the thing that I wonder about in the kind of, the, the AIs will be smarter than we are, is to dig into what that really means. And, you know, I think the first thing to say is, you know, that, you know, the big limitation to whatever you might think smartness is, is this phenomenon that I call computational irreducibility. The idea that if you have a computational system, the question is, can you jump ahead? We are used to the idea from lots

• 8:51Speaker 01

  Of exact science over the last few hundred years that, oh, you know, the planets do what they do according to these laws of motion. But we can find a formula that lets us just work out what will happen in the end without having to go through all the steps. One of the things that's sort of a foundational fact that we can talk about, you know, where it comes from, but you know, it's it's kind of the the whole idea of computation leads to this fact that there are many computations that one can set up. Where to know what will happen, you really have to just go through all the steps and see what will happen. You can't, you can't kind of jump ahead now. A lot of what we've achieved in in science and mathematics in

• 9:31Speaker 01

  is is to find little little pockets of computational reducibility that allow us to jump ahead in some place or another. That's what most of our inventions have to do with taking what is out there in the sort of computationally irreducible world. And finding some particular place where we can jump ahead and where we can kind of foresee what's going to happen and make use of of those things. So I mean, I kind of, I kind of think the first thing to realize, I think, is that, you know, smart as one could be, so to speak. There's no way to get out of computational reducibility. And the, you know, in a sense, then what one's talking about is there are these pockets of reducibility. We find some. There are ones that we humans care about. There are ones that our civilization depends on.

• 10:14Speaker 01

  There are lots of other pockets of reducibility that we haven't found, many of which we probably don't care about in the current way that we are thinking about things. And, you know, it's it's I I see it. I don't see it as being this kind of linear. Oh, this thing is now smarter than us. So so doomed, so to speak. I think that it is, you know, and as I say, my own personal experience has been I'm completely used to the idea that that computers are smarter than I am. And, you know, while, uh, you know, there's there's sort of the question of do are those computers connected to everything that runs my life, so to speak, that's a different issue. But in terms of the the raw experience of, you know, are the computers smarter than I am?

• 10:57Speaker 01

  Well, you know, I already know that they are. Now, I think the the other thing to say Can the computers get smarter than we are in some sense in every dimension where where we humans are used to doing things? That's that's a different issue, but I think that's a different issue from from. And then there's the question of of sort of what does it mean if we live in a world where there's lots of things going on that are sort of smarter than we are in in some definition of smartness? Well, one point to realize is the natural world is already an example of such a thing. I mean, the natural world is full of computations that go far beyond the computations that we do in our brains. And, you know, how do we manage to coexist with the natural world?

• 11:40Speaker 01

  Well, we found these particular niches where, you know, we it doesn't matter that it rains a lot because we build houses and so on. We found these ways to coexist with the natural world that seem to let us lead the lives we want to lead. There are things we can't do because our physiology and our biology doesn't let us go there. But we seem to be, you know, we seem to be able to live contented lives, even though we can't, you know, go to the bottom of the ocean, you know, without. You know, having some complicated, you know, piece of technology and so on. So, I mean, I I think that's that would be my my initial kind of thinking about about sort of why I'm not as worried about the moment when, you know.

• 12:23Speaker 01

  The intelligence of the machine goes from, you know, one hundred and fifty to one hundred and sixty to two hundred to three hundred, whatever that might mean. I mean, you know, you know, galloping through the Ravens progressive matrices at a at a at a rapid clip is not, you know. Again, you know, from my point of view, it's kind of already it's already happened that there are many things in that, you know, that that computation can do. That are beyond what I can do with my sort of unaided mind, so to speak. So I'm curious, I mean, do you see the the issue as being kind of a more of a we put the AIs in charge of our. You know, air traffic control and our, you know, medical devices and our this and that and the other.

• 13:08Speaker 01

  And so at the sort of actuation layer, things. Kind of go wrong or do you see it? I mean, the thing again that I'm I I have a bit of a hard time with is the kind of it feels very anthropomorphic to talk about kind of, you know, the AI wants to do this. The AI is, you know, wants to trade with us or not or whatever that that, you know, I could talk about the natural world in the same way. Does does the weather want to give us a good time or not? I don't even know what that means.

• 13:38Tim Scarfe?

  Could I just segue as well, and Elias, I I understood your point as there's a a powerful force which is going to push us out of equilibrium. And I think Stephen is making the argument that we are in an equilibrium now. Is is is that fair?

• 13:52Speaker 00

  That didn't sound fair to me.

• 13:54Tim Scarfe?

  Okay, so I I understood that Stephen was making the argument that there are these countervailing forces that prevent significant deviations.

• 14:03Speaker 01

  No, I don't that's I don't think that's what I was what I was saying. I mean, I was I was I'm not claiming that the world won't change. Okay? I'm simply saying the idea that it's kind of, you know, game over because suddenly the IQ crosses three hundred or something. I don't believe in that.

• 14:20Speaker 00

  I mean, I think we probably both agree that IQ three hundred is not meaningful.

• 14:24Speaker 01

  Right. Okay. Um, I'm not surprised. We both agree about that. But but no, the the the point that I'm making is that this idea that somehow the AI gets smart enough to be able to do everything, I don't believe in that. I mean, I think there's this sort of computational irreducibility limit that, you know, you know, rate neural net or not. That's just a a fundamental kind of formal limit that we don't get to get around.

• 14:55Speaker 00

  So I first want to try to sort of repeat back your summary and your summary is like because we cannot predict a thing. It shows we're not smarter than that thing. And even in a sense that it's smarter than us. Because it's something that it knows, which which is its own output, that we cannot predict ahead of it. Does that sound like a correct summary of your?

• 15:16Speaker 01

  Well, part of it. I mean, I, you know. You know, the problem with a lot of these things is we get a word like smart. And then we don't, you know, we kind of think we know what it means, but then it kind of slithers around on us. And I think that's, you know, that's part of, uh, but, but I would say that, you know, what I'm saying is merely that if you say, I want a system that can solve every problem, you know? Will I get to the point where I have an AI that can just slam dunk solve every problem? I'm saying that's not going to happen.

• 15:49Speaker 00

  No, and in particular, it's entirely possible that if you pick a sufficiently strong, quantum proof, one-way hash that and hash something and throw away the key that nothing, whether it's nothing at any point in the arbitrarily far future, even if it has consumed entire galaxies to use its computing power, will ever be able to find the input that you fed to that one-way hash. Um, so in this sense, universal problem solving is probably not on the table as far as we know, given the present laws of physics. But, you know, the, the, the European invading Europeans and invading North America

• 16:31Speaker 00

  did not need to be able to solve every problem to solve more problems than the Navy Americans and mostly wipe them out. Um, a fire, a fire doesn't need to be able to burn, uh, neon or other noble gases, nor to be able to burn you personally. Doesn't have to be universal and you don't even need to be able to define exactly what fire is to die in a forest fire. People have probably died of in fires a long time before they worked out the concept of combustion and chemical rearrangements with oxygen that had ended up in states of lower, uh, potential energy, which released that as kinetic energy, which looks like heat, which is what burns you. People didn't know that and they still died in forest fires.

• 17:13Speaker 00

  And I, I can try to define what's an intelligence, but before I do that, I do want to point out that being unable to define something is not bulletproof armor.

• 17:22Speaker 01

  For sure, but I think, I think the, you know, the question is, it seems like the argument that you're making, which may not be correct, is, I mean, I may not be correctly representing it, is, you know, there will be, you know, AI is advancing rapidly. We both, we can talk about, you know, what we know about how AIs work inside and the whole story of computational irreducibility and how it applies to that and why machine learning works. Those are interesting questions. It'd be fun to talk about those. But I think one of the, one of the issues is, we don't know, you know, just how effective, you know, AIs are going to become. But we do know there is a limit to how effective they can become. Because we know that there's, you know, there are, as you say, I, I wouldn't go as

• 18:06Speaker 01

  far as talking about, you know, elaborate one-way hashes and things. There are much more straightforward kinds of things that you can't predict, except by running the steps.

• 18:15Speaker 00

  A particular cellular automata?

• 18:19Speaker 01

  Yeah, for example.

• 18:20Speaker 00

  Sure.

• 18:21Speaker 01

  And, and

• 18:22Speaker 00

  I, I think we agree on that. Just, just to, just to stipulate that part.

• 18:26Speaker 01

  Yes, yes, right. So I mean, okay, so then, then the question is, could it be the case that everything we humans care about can be done faster, better by AIs? Could it be that the things that are important for running the world and that, you know, that there's, there's sort of this, you know, there's a question, could, could everything that humans do be done better than AIs? You know, that's a different question from whether AIs can sort of do everything, can do things that we have no idea how to do what, how to do.

• 18:58Speaker 00

  I, I mean, even if there's one last remaining computational task that a human brain can do better than an AI, by like predicting the exact output of that particular brain, which you're sure not going to get without doing a bunch of intervening steps if you want it down very exactly, you know, that is not necessarily protective against a bullet, right? Like there were problems that the Native Americans could solve better than the Europeans who wiped them out.

• 19:23Speaker 01

  Absolutely. You know, there will be, just as, for example, the weather has ways to wipe us out, you know, so. You know, it's in any particular, I don't know, you can imagine some mega superstorm that we don't completely understand that, you know, because we're talking about, again, things we don't completely understand. So I think we can, we can as well say we don't know what's possible from the weather as we don't know what's possible from the AIs. And, you know, I can imagine that the, you know, there's some, something, some terrible thing that can happen, you know, people, well, people imagine with respect to climate, that that's going to wipe us out. Another thing that involves sort of lots of extrapolation about what's going to happen.

• 20:08Speaker 00

  I mean, you can kill a lot of people without killing everyone. It is much harder for the weather to wipe out every last human on the face of the earth than it is for the weather to just kill a lot of people.

• 20:18Speaker 01

  Fair enough.

• 20:18Speaker 00

  And I, and I am, and I think that total extermination is like much worse than just a lot of people dying because you also lose posterity. You lose the meaning of history.

• 20:30Speaker 01

  Okay. So there's a, there's a, we could segue to a quite different topic, which is, you know, in the history of life on earth, there've been plenty of times when, you know, if we were two stegosaurus's talking about this, you know, a hundred million years ago, whenever stegosaurus's were around, the stegosaurus's would say, gosh, you know, it will be terrible if we were wiped out. And if something came, if those pesky mammals that have just the one or two things that they can do better than us took over. And I'm curious about, you know, your view of the ethics of that situation.

• 21:03Speaker 00

  This is a very, and this is a very different line of reasoning from let us define intelligence, you know, the work done of prediction.

• 21:10Speaker 01

  I agree.

• 21:11Speaker 00

  Steering, what does it mean to want something? Does a chess game wants, does, does a, an AI chess player want to win chess? Well, it's definitely very good at winning chess, whether you say it wants to or not. Does it want to protect its pawns? I just want to remark on all of the places we could have maybe gone to and temporarily.

• 21:31Speaker 01

  Let's put that aside for a bit later. But let's talk about.

• 21:34Speaker 00

  The stegosaurus's. Yeah.

• 21:36Speaker 01

  Let's, we'll, we'll, we'll put the stegosaurus's in their pen for a little while longer and we'll come back to the stegosaurus's later.

• 21:42Speaker 00

  Oh, we'll come back to the stegosaurus's later. Okay. I'm up for that.

• 21:45Speaker 01

  Unless you have some quick, quick comment about the stegosaurus's.

• 21:48Speaker 00

  Very broadly, that the trouble with looking at the change from stegosaurus to humans and determining and viewing a trend of things getting better and better and turning out pretty much okay is that as humans were drawing an arrow around the target, if like we got wiped out and replaced by stegosaurus and the stegosaurus had, you know, didn't care about each other very much. They grew up in different evolutionary, social circumstances. You know, if we were wiped out and replaced by insects, maybe the hives of, you know, sapient insects just never care very much about other hives. They just don't carry very, they don't end up caring very much about sapient life at all. That would be a grand tragedy. It's not at all clear to me that you can wipe out humanity and replace it by arbitrary stuff and everything just gets better as a result.

• 22:33Speaker 01

  I don't know what better means. I mean, better is a very human concept.

• 22:36Speaker 00

  Well, that's the whole, I mean, there you have it. And yet, if we're going to decide whether we want to voluntarily wipe ourselves out and replace ourselves with, you know, sapient ants, we need to ask ourselves whether that's better or not. We have no other way of making that decision.

• 22:52Speaker 01

  I think it's not question necessarily better. It's like, we're humans and we're happy to be around and we kind of want us to continue. Cause that's, you know, we're right now, we're kind of the, the in charge critters around here. And it's kind of like, that's, you know, let's, let's stay in charge. You know, why would we give it, give it up to something? And we might have some, some argument that, oh, of course, these other ones that we might give it up to are worse than us. But, you know, in the end, it's like, we're here and we kind of like being here. So we'd like that to continue, I think, at least.

• 23:28Speaker 00

  That's not what I'm fighting for. I want there to be sapient life that cares about other sapient life, that doesn't just discover things about the universe, but appreciates those discoveries, that looks out at the universe with wonder and curiosity, rather than mere expected value calculations. I want, I wanted to have fun. I wanted to be conscious. I'm worried that these things get lost. And if you say that, oh, like, well, if you want life to be conscious, if you want there to be consciousness in the humans, consciousness in the universe, that's just your preference, Ali Azar, then we might need to have a long conversation about what alternate state of meta ethical affairs you thought might have obtained other than

• 24:11Speaker 00

  that, to be disappointed at the absence of or to critique the absence of. But mostly, I'm like, yeah, I'm the one who says I'd rather the universe be filled with consciousness than mostly unconscious. And, you know, yeah, I'll own that.

• 24:24Speaker 01

  You know, this whole question about, you know, consciousness, it's like, I'm pretty sure that I have some consciousness. I have this, you know, thread of experience. I'm, I'm experiencing things. As far as I'm concerned, you're a bunch of pixels on the screen. I have no idea, you know, what's going on inside you. It is a, it is an extrapolation that you are conscious. The only thing that I can, you know, readily kind of from experience believe is that I'm conscious. And, you know, when it comes to even an AI, you know, is this thing folks conscious? Or is it merely a bag of bits? I mean, in other words, and, you know, with an AI, we can with some effort, go and look at all the bags of bits with human brains, we can't yet do that. Presumably, eventually, we will be able to do that. And

• 25:09Speaker 01

  once I can see all those neuron firings in your brain, what, how will I think about the consciousness in your brain? And what, what, you know, how do I even know what's going on beyond from what I can experience inside myself?

• 25:23Speaker 00

  Well, I infer that you're conscious based on knowing that I'm conscious and thinking that probably something pretty similar is going on inside you to me because we have similar great, great, great, great, great, great, great, great grandparents. We are built on like basically the same engine plan. I haven't actually administered an MRI to you, but I would guess that you've got like pretty much the same brain areas I do, cerebellum, cerebral cortex, even without knowing exactly which features of the brain are.

• 25:56Speaker 00

  Executing the algorithm that I experience from the inside as my being conscious, I can guess that you probably have a pretty similar algorithm. You know, based on you claiming to me that you're conscious and all that.

• 26:08Speaker 01

  Right. I mean, it's an extrapolation.

• 26:10Speaker 00

  Similar design.

• 26:11Speaker 01

  Right. It's a reasonable, you know, piece of scientific induction extrapolation. But what I'm curious about is if you if you say the only thing that can be conscious is something that has that exact same design.

• 26:24Speaker 00

  Oh, no. I don't say that.

• 26:26Speaker 01

  Okay, so what so where's the boundary?

• 26:29Speaker 00

  So it's not so much that I'm worried that machines cannot be conscious as I'm worried that the particular uncontrolled super intelligences we end up making are not going to value consciousness all that much, which means that they're not going to produce very much of it. Like, maybe consciousness is the most efficient way to run some paperclip making algorithm. But if you don't value consciousness for your own sake, then the little bits of consciousness that are supervising the paperclip making algorithms are, you know, not going to be as much consciousness as we could have made of the universe. And if they don't value fun, they're probably not having fun either. They're just being efficient because they didn't end up valuing fun. That's the that's the general class of nightmare scenario that I'm worried about.

• 27:11Speaker 01

  I mean, I really wonder, my computer, when it runs and does all these searches and it finds these cool, you know, cellular automaton patterns or whatever else, I wonder if it's having fun.

• 27:23Speaker 00

  I bet it ninety I bet it ninety nine to one against. I don't know everything that there is to know about this, but it seems to be that when I stare at the machinery of having fun, there are gears and wheels there that are not built into a pocket calculator, that are not built into a simple Python program searching through a list of cellular automata. Like, stuff happens.

• 27:43Speaker 01

  That's very low level. But sometimes things like, sort of, people say, you know, the one thread, you know, the one sort of shred that we have that machines will never have is some kind of emotional response to things, which, you know, I think that's a very weak concept, because I think that's emotions are are are, you know, a very chemical, chemically based kind of thing that's actually very coarse compared to a lot of the the other things that go on in brains.

• 28:11Speaker 00

  Yeah, it's it's I know it's not that I think that machines can't have emotions. I suspect that Claude 3.5.1 Sonnet or GPT 01, if they have anything resembling emotions, they're not our emotions. There's like something inside it that was trained to imitate emotions. But if it has emotions, they would be, they're, my guess, nobody knows, is that they would be more emotions, more appropriate to like carrying out the job of being the actor who plays the part of a human who has emotions and not actually the human emotions themselves. And it's not fundamental to transistors.

• 28:53Speaker 00

  It's just like what I would guess hap- would happen, given the particular way we put these transistors together.

• 29:00Speaker 01

  So what you value in the universe and in humanity is a certain set of attributes that have to do with the ability to have fun, your, you know, the meaning of consciousness and so on. And it is, as I'm understanding it, you, you think those are sort of the most valuable things that maybe we as a civilization, we as a species have produced. And you think it is our responsibility, if I'm understanding correctly, you think it's our responsibility to kind of preserve these things in the universe. Is that a fair assessment?

• 29:27Speaker 00

  I could quibble with the exact wordings, but sure. You know, the light of consciousness, the light of fun, the light of caring, to preserve these things into the galaxies is the most important thing we have to do.

• 29:38Speaker 01

  Because you feel that's, that's sort of our big achievement. And it's kind of like, it's now our responsibility to, I don't know quite to what, but it's somehow our intrinsic responsibility to preserve these valuable things.

• 29:52Speaker 00

  I mean, I think we got handed these things on a silver platter, but that, you know, not, or, or, you know, like maybe a kind of rusty iron platter that we had to shine and clean up a whole lot, but you know, still on a platter from natural selection and even from cultural processes of development that nobody really saw coming at the time there were, you know, few people did them on purpose. Somebody did at some point, say, say of slavery, it is time that some person should see these calamities to their end. Not all moral progress was accidental. But you know, still, like, our basic situation of being humanity, of being here on this planet, of having the opportunity to colonize the galaxy. We fought for this a little bit, but it was ultimately handed to us on a platter.

• 30:35Speaker 00

  And I don't say that it's like written down on a stone tablet anywhere in the universe or laws of physics, that we must bring this to the galaxy. I just say, I want to do it. That's what I think is the right thing to do.

• 30:46Speaker 01

  Okay. So I mean, this is, I, you know, I don't necessarily disagree with you. But it's worth understanding that this is a feeling that you have. This is not something where, as you say, there's no law of physics that says this is the destiny of the universe or anything. This is on this particular planet, through this particular process of natural selection, of biology and culture and so on, we got to this point. We're really proud of this. Let's try and preserve it. I mean, is that, that.

• 31:16Speaker 00

  Yeah, the, the fact that I want to steer the universe there isn't a fact about the whole universe, it's mostly a fact about me.

• 31:24Speaker 01

  Sure.

• 31:24Speaker 00

  And a whole lot of other people, to be clear.

• 31:26Speaker 01

  Yeah, right.

• 31:27Speaker 00

  You know, it's, this is not like two plus two equals four. This is not, you know, like, water is two hydrogen atoms and an oxygen atom. This is, you know, where I steer the universe, being the sort of thing that I am.

• 31:41Speaker 01

  I, I agree with you. I think, I think one of the things to realize is that if you just let raw computation do what it can do, much of what it will do are things that we humans just don't care about. Just like a lot of what happens in nature are things that we humans don't care about. We've never managed to, you know, sort of take those things from nature and make them into technology, have those things be things that we appreciate as beautiful or whatever else. They're just random things that happen in nature that we don't particularly care about. And most of what a computational system could do.

• 32:16Speaker 01

  Is things that are, you know, off the, away from the things that we care about in kind of the computational universe. Now, that, so, so many of those things are just things that will just happen, just like they happen in nature. We will kind of shrug our shoulders and say, okay, that's a thing. You know, a lot of what probably happens inside neural nets right now are things where, in fact, I have pretty good evidence of this, they're things which are sort of complicated computational processes where they don't really produce anything that is something that we can recognize or that we care about. They're just things that are happening computationally. And so I think that your, your concern is that some of those things that might happen

• 32:57Speaker 01

  or that the AIs might do might be things that collide with what you see as being our sort of responsibility and our, you know, proudest sort of artifact in a sense, that, that, that somehow there will be a collision of what is, you know, what AIs, computational systems, whatever, can do and the thing that you think we as humans sort of have most proudly produced. Is that, is that a fair assessment?

• 33:24Speaker 00

  Yes, if, if, if it, that works out to they kill us and then go on to do nothing very much interesting or worthwhile with the galaxies that they colonize. Like, is that, is that also a fair way of, of describing all of that?

• 33:38Speaker 01

  Well, I, I don't know. I think that, I mean, you know, this question of, okay, there, there are several different pieces to this. I mean, I think, look, the first question is the, is the practical question, you know, five years from now, 10 years from now, you know, will we be able to have this conversation anymore? Or will we be, you know, you know, will we be dead, replaced by bits that are doing what you might think of as meaningless things, but bits, you know, will we be, will it be the case that it's as if an asteroid had hit the earth and, and all life was wiped out or whatever else? That, you know, that that's the

• 34:15Speaker 00

  Yeah, like, like If, if, if an asteroid hits earth, all the atoms in it. Go on doing their little atomic things, you know, the little electron cloud, you know, I want to say circles the nucleus, but of course it doesn't really blah blah quantum. But, you know, the thing that quantum physicists described to the lay audiences has, you know, circling the electron circling the nucleus. It goes on doing that. Maybe, maybe some, some amount of fusion occurs, the finasteroid hits hard enough, some amount of fusion, but, you know, by and large, the atoms go on doing their atomic things, but they're not morally valent things. They're not important things. They're not worthwhile things. The

• 34:54Speaker 00

  earth, the universe gets a little darker every time a bullet gets fired into somebody's head, or they die of old age, even though the atoms are still doing their atom things.

• 35:06Speaker 01

  Right. Okay. So this is, this is, I mean, it's a, it's a, to me, that's a, that's a curiously, I mean, I, viscerally, I agree with you. Scientifically, I have a bit of a hard time. I mean, in a sense that, that feels, that feels like a very kind of spiritual kind of statement, which is not necessarily bad, but it's just worth understanding what kind of a thing it is. I mean, it is saying that there's something very kind of sacred about these attributes of humans and that we have perhaps even a higher purpose to, which we don't really know where it comes from, perhaps, we can't, to somehow preserve

• 35:48Speaker 01

  and, you know, we, and, but there's a much more practical thing. I mean, as humans who like doing what we're doing, it would be nice if we could go on doing that without all being killed by AIs. So I think there's a-

• 36:00Speaker 00

  I don't see any distinction, but between these, you know, like there's nothing more scientific about saying, I'm a human. I like to keep on doing what I'm doing. And like, I'm a human. I'd like to fill the universe with the light of consciousness, even if that consciousness isn't a human, so long as it cares and is having fun.

• 36:21Speaker 01

  Yeah, yeah, no, I agree. Those are both, those are both statements about how humans feel about things. It's worth remembering.

• 36:30Speaker 00

  They're not false, they're not unscientific. There is not a truth of science that contradicts our caring.

• 36:36Speaker 01

  No, absolutely not. It's, it's like, you know, ethics is not a scientific field. You know, it's about how we humans feel about things. And we humans could feel this way, we could feel that way. It's, it's to do with the nature of us as humans. And we could, you know, we could scienceize those statements by saying, let's do an FMRI and notice, why do you say that? Oh, you're such and such lobe lights up. But I don't think that's a particularly useful thing to say. I mean, I think, you know, I think it is a fair statement that this is, you know, it is a thing that we can capture, that humans feel that this should happen or not happen, whatever else. But I guess that the, that, you know, there's, there's one question is, what's the right thing to have happen?

• 37:22Speaker 01

  I don't think there's any abstract way to answer that. I think it's a question of how we humans feel about it. And I think you and I seem to feel, I know I feel, that, you know, preserving kind of what humans do is a good thing. I, you know, I can imagine even humans who say, no, no, no. You know, the planet is much more important than the humans, for example. Anything the humans do on the planet that messes up the planet, you know, get rid of the humans. We just want the, you know, the planet is more important than the humans.

• 37:54Speaker 00

  I think that in terms of prospects of human existence, I'd like to preserve, I would not include mosquitoes. Just like, get rid of all the mosquitoes, I think the planet would be fine, I think we'd be fine. I would, if I could, get rid of this whole aging business that humans do. This was more controversial. It would actually change things if that happened. And, and, you know, some of the changes might not even be positive. But my guess is that they would be quite positive on that. And I'd be in favor of, and of, you know, if this were a thing you could do with a limited AI that wasn't going to wipe out humanity, and you could know that, and you weren't taking any extra risk by doing that. I'd say, you know, use AlphaFold 5 to solve as many aging-related problems as you can.

• 38:39Speaker 00

  And some people would hate that. And there's the question of, is one of us making a mistake? Is it the case that if you take me and the person who wants to preserve aging, who wants humans to keep getting older and die, that you put us both up against each other? And like, one at a time, you just tell us all the facts about the way the universe actually is. You just expose us to all the arguments that exist to be considered about this, that any moral philosopher could invent in, you know, some sort of random order. Do we converge? Or do we just want different things? And it's not clear to me that, you know, this is an instance of conflict versus mistake. It could be that the people who believe that aging ought to be preserved forever,

• 39:22Speaker 00

  that there are things you could tell them about what would happen if we cured aging, that would change their minds. It could be that there's true things about what would happen if you cured aging that you could tell me, that would change my mind. You could show me some dystopian world of 300 years in the future, which is like just being ruled by Vladimir Putin forever and ever, and nobody's having fun, and I could be like, yeah, okay, that was worse. So it's not clear to me that we need to say that they and I care about different things. We might, but it's not sure. We haven't actually run the experiment of telling us both all the facts that can be determined and exposing us both to all the arguments that can be thought up on it.

• 39:58Speaker 01

  Yeah, but I mean, so, you know, part of what you're saying is, when things are matters of sort of human decision, different humans might come to different decisions. And it's not clear how one resolves that. I mean, insofar as there's, I mean, perhaps one of the things you're saying is, when there is sufficiently great risk, it's a bad idea, you know? Even if some humans say, hey, we should be doing this. We should be, you know, building super killer viruses that can wipe out the species. You know, assume a small number of humans say that, that potentially, sort of the, even though they say, we really have good reasons to do this, and other people say, that's a really bad idea.

• 40:40Speaker 01

  You know, in most cases, you'll just end up with two different points of view. But you're, I think, arguing that there are cases in which you kind of have to, you know, the world has to align on one particular thing. Because, you know, as soon as there's one super killer virus that wipes the species out, you know, that's, it's game over, so to speak.

• 40:59Speaker 00

  I mean, I think that's a bit of a topic change, but yes, in the entirely different realm of political rather than moral philosophy. I would agree that either your species is able to prevent anybody from making a super virus that kills everyone, or at least collapses civilization, or your civilization does not last. And I think that even many of my fellow libertarians would agree that, you know, making and releasing super viruses ought not to be legal, that this is a proper thing for government to do, to, you know, ban the releasing of extremely lethal engineered super viruses.

• 41:33Speaker 01

  But so as I'm understanding it, you think that the same thing is basically true about powerful AIs?

• 41:40Speaker 00

  Yep, I think that we're in a situation where if anyone builds a super intelligence, everyone, everywhere dies. And I think that that is a sort of thing that government, where it's a proper use of government to try to have that not happen. I would sooner abandon my political philosophy than commit planetary suicide about it.

• 41:58Speaker 01

  So you really do see those as a sort of comparable things, the the super virus, the super intelligence? These are both things that you think are things that are sort of comparable risks that, um, uh, you know, are, um, I mean, I actually, I wanted to come back to this question about, you know, immortality. Because I'm I'm big on immortality as well. I think we're I think we're both interested in clionics. I remember talking to you about that years and years ago. You know, it's kind of shocking that Clonex hasn't worked yet. Maybe that's one of the tests of the, you know, the next generation of AIs is can, you know, can the AIs solve the problem of, you know, how to get water, uh, you know, to to cool down without expanding, so to speak.

• 42:41Speaker 01

  But, um, uh, you know, in, I'm just sort of curious from your sort of moral compass point of view. If immortality is achievable, but only digitally.

• 42:52Speaker 01

  What, how do you feel about that? I mean, in other words, if you're, if you're going to, you know, you're right now, you're you. You start having your sort of backup AI. Maybe you gradually, you know, your sort of thread of consciousness gradually migrates from being in your brain to being in your AI. And eventually, you know, your brain fails for some biological reason, and then it's all the AI. I'm, I'm curious how you feel about that, whether you, whether you feel that that is a, a kind of a, an appropriate kind of fun preserving outcome, or whether you think of that as being a, um, a kind of a, a fun destroying outcome, so to speak.

• 43:30Speaker 00

  I think that there's a whole lot of work to be done in making sure that you are getting all of the functionally relevant properties of every neuron in the brain that you scan, you know, up to whatever is just like random thermal noise. Um, and then as long as you've done that work, sure, sign me right up. You don't even need to do it like through, over like 20 years or something like, you can do it over a day. You can knock me out, scan through my entire brain destructively and wake me up when it's over, as long as you have actually gotten every single functional relevant, functionally relevant property of every neuron and, you know, actually, and you are able to simulate them correctly. Uh, you know, if I trust a super intelligence to do this, I'll say, sign me right up.

• 44:14Speaker 00

  If it's like some kind of shady back alley human doctor, I might have a lot more qualms.

• 44:19Speaker 01

  But so, so you, you think in that situation where you've been fully scanned and reproduced digitally, when the digital copy is switched on, it's you waking up?

• 44:30Speaker 00

  Yep.

• 44:32Speaker 01

  And, and how do you, I mean, does that, uh, because you feel like your only connection with the past you is your memory anyway? And in that case, you would have that memory. Is that right?

• 44:45Speaker 00

  Uh, uh, no, it's cause I think that I am like the, the functional properties.

• 44:53Speaker 00

  Of my neurons that I can see from inside. Like if one of the electrons somewhere inside my brain was secretly a different flavor of electron, never mind how hard this breaks all of physics. But it was like secretly a different flavor of electron that otherwise behaved exactly like all the other electrons, except for not being, you know, like quantum interchangeable. So follow it, you know, exclusion principle doesn't apply to it. But, you know, otherwise it's just like exactly functionally the same. I can't tell. I can't see that. If you swim up to one of my neurons and while it's otherwise not firing, replace it with a robotic analog that behaves in exactly the same way. I can't see that.

• 45:37Speaker 00

  I can't tell that anything has happened. It doesn't affect me.

• 45:42Speaker 01

  Well, so this though comes to the essence of what the you is because, you know, and this also relates to current AIs. You know, one of the big surprises with something like chat GPT was that it could be human enough to be able to write somewhat credible essays. It wasn't obvious that that was possible. It could have been the case that to reproduce human language required some new physics in the brain of, you know, quantum gravity in the brain or some such other thing that we it's just completely out of reach to our kind of sort of current computational techniques. But in fact, you know, it seems that we were able to to capture enough that we can have it write sort of somewhat human sounding essays. So the question would be, you know, when we reproduce your brain,

• 46:28Speaker 01

  do we have to reproduce it only functionally? Or do we have to reproduce it with every glial cell, you know, represented with all of its chemistry and so on?

• 46:37Speaker 00

  I think you need to reproduce the functionally relevant properties of every glial cell. If you're doing something to glial cells where I would notice, where there'd be like a detectable subjective change. If you could give me a questionnaire and I'd answer differently on the questionnaire afterwards. Like if you are changing all of the glial cells in that fashion, you have perhaps killed me.

• 46:57Speaker 01

  Okay, okay. So, so, so what you're saying is if you can behave to the outside world like you, like answer questionnaires the same way, then you are adequately the same.

• 47:09Speaker 00

  No, it's about what I can tell internally, not externally.

• 47:14Speaker 00

  If you find a sufficiently smart actor, an actor who is enough smarter than I am, they can maybe play my part in a way that fools even my closest friends. But that is not me. And the actor knows they're not me. They can tell internally, even if they're managing to mimic me externally.

• 47:35Speaker 01

  So, I mean, for example, if you, you know, I'm not, I, I value my brain too much to take drugs of any kind. But, you know, if I, if I didn't have quite that point of view, I might, you know, feed in molecules of some, some crazy psychedelic thing. And, you know, then, then my brain is, so am I still me when I've done that? You know, I've changed my brain chemistry by putting in some, some strange drug. Does that, does that de me me, so to speak?

• 48:08Speaker 00

  There, we start to get into the edge cases where I start to feel more uncertain in my answers. I would feel, feel very nervous about taking an uploading procedure, about going through an uploading procedure that felt like temporarily being on drugs, to say nothing of permanently being on drugs.

• 48:25Speaker 01

  But yet, you know, it's still, you know, it's still you, even if you took that weird drug, it's still, you know, there's a continuity of you-ness.

• 48:35Speaker 00

  And do I know I me? Or does somebody else now know that they are themselves? There's some sufficiently, you know, I can, there's, there's sort of like two different questions here. One is something like, what do you see happen to you from the inside? Do you experience dying? Or do you experience a change from one person into another? I'm not even entirely sure that this is the right question to ask, but it's a different question of, do I care? You can, like, there's plausibly some sufficiently advanced pill you could feed me that would, you know, produce changes to neurochemistry where I just stopped caring about other people at all. And maybe I experienced ending up in that person.

• 49:17Speaker 00

  Uh, uh, uh, maybe I experienced ending up as that person, but I still wouldn't want it. And

• 49:23Speaker 01

  Well, you wouldn't beforehand, you wouldn't want it. I mean, this is another complicated thing because once you're in that consciousness, you know, it it's kind of like, like saying, you know, you know, at any given moment, you feel it's similar to what you what happens if you think about kind of human purpose across human history. If if we say, you know, right now, uh, you know, we think certain things are meaningful, like we, you and I might think talking about philosophy is meaningful. Other people might not think that was meaningful, but we might think that was meaningful. Back in, you know, we might think, you know, back in the day, people might have thought, maybe some do, some do now. If you're not growing your own food, you're not leading a real life,

• 50:06Speaker 01

  so to speak. Or if you're not, you know, uh, sort of fighting for the greater glory of God, you're not leading a real life, uh, for example. And, you know, if we project to the future, let's say that, you know, the the uploading thing works. And in the end, sort of all human consciousnesses can be in a box. And to us today, it might appear that those human consciousnesses are just playing video games for the rest of eternity.

• 50:34Speaker 00

  Whoa, whoa, whoa, whoa. Well, there's a big difference between like, are you running on silicon versus carbon? And what are you actually doing? Even if you've got a bunch of organic humans and you've put them all into a box and they're all playing video games, you know, maybe I object to that part. It's got nothing to do with whether they're carbon or silicon. It's whether you've put locked them in a box and forced them to do nothing but play video games.

• 50:54Speaker 01

  Right. But but the point I was gonna make was actually a slightly different one, which was, you know, to us today, to you and I, probably, certainly me today, the future of it's a trillion souls in a box playing virtual video games seems like a terrible outcome. Seems like that's the end of history. That's everything is destroyed. That's your kind of bad case scenario of, you know, which might be forced by AIs or it might just happen because the humans decide they want to be immortal, you know, as uploaded consciousnesses. I mean, to us today, that looks like a really, really bad outcome. But I would, I

• 51:29Speaker 00

  Be worse. I mean, I definitely have friends who think that if you've got like, if you tile the all the galaxies within reach with, you know, people in boxes, but they're having fun in there and playing, you know, playing video games in solitary, you know, okay, that's I start to get the Wiggins, but you know, maybe that's still like 10% of all the value could have gotten. And if they're playing video games with each other and there's and there's like real people that they're interacting with and they care about those other people, maybe that's like 50% of all the value. I don't quite want to say that this is not the level of sterile universe that I am expecting and scared of.

• 52:09Speaker 01

  Okay, but but let's just let's just take, you know, the to us today, it doesn't seem like a good outcome for humanity to have a bunch of uploaded. Maybe maybe you don't agree. I mean,

• 52:20Speaker 00

  It's not the best outcome. And why go for anything but the best? You know, why settle for less?

• 52:26Speaker 01

  Fair enough. But what I like to argue is to one of those consciousnesses, if you say, are you leading a fulfilled existence? Those consciousnesses might feel that they are feeding a feel, you know, living a fulfilled existence, just like in the past, somebody might have said, you know, I lead a fulfilled existence if I die at the age of 20 as you know, leading the good fight on behalf of, you know, this or that, you know, you know, religious belief or whatever else, right? That that is, that is the then I am achieving my, my, you know, my, my ultimate purpose, so to speak. Today, we most people probably wouldn't think that. Some still do. You know, in the future, to us today, it looks really

• 53:10Speaker 01

  bad to be just in a box as a virtualized, you know, playing virtualized video games. But I would claim that at that moment, just like the human who took the drugs, who feels at that moment that they're doing the right thing, that at that moment, those virtualized humans will feel like they're doing something fulfilled.

• 53:30Speaker 00

  So when you've got two people who are making different choices and doing different things, the question of do they have a conflict? Or is at least one of them mistaken? Like to me, that revolves around the question of are there true facts you can tell them? Are there arguments, is there a series of arguments you can present to them within their current framework, which, you know, changes that framework in a kind of normal way and not by directly hacking their brain or whatever? If you can get from point A to point B by being told true things, then there's a, then then there exists a standpoint from which to say, B was correct, A was wrong, but they were within the same framework, they were within a commensurate framework.

• 54:12Speaker 00

  If you've got someone who's content to just farm and never know anything more than that, take them up on a high mountain and show them all the nations of the world and all the cuisines they've never tasted and all the books that people are reading and all the activities they're engaging in that they've never heard of before. And then if they still want to go back to the farm, maybe they were, you know, like kind of just like correct of that they were okay with this farming thing.

• 54:39Speaker 01

  You know, I have to say, I've seen that as a very practical thing because I've been curious, you know, I'm a great believer in sort of talent all over the world in all kinds of places and so on. And I've been interested particularly in kids and so on of like, you know, you go visit some rural high school in the US, for example, and you explain about all these amazing things about science and technology and so on. And, you know, some kids really care, but a lot just don't care. It's just not part of that. And then the question is, at what, you know, to what extent are you then serving as kind of a missionary, saying you really, really should care? This is really the, you know, as opposed to just saying, you know, sorry, you don't care, so move on type thing.

• 55:21Speaker 00

  Well, if you can control super intelligences, the thing you want to do is build a model of the person that isn't itself conscious inside the super intelligence and ask the super intelligence whether or not the person is mistaken in thinking that they just want to farm, meaning if this person knew everything the super intelligence knew, would they still want to just be a farmer? And if the super intelligence tells you, yeah, you know, like no matter what you argue to this person, no matter what you show them, you know, they have this like self consistent worldview where they're just having fun farming, then you ought to leave them alone. And you don't even have to bother them to determine that.

• 55:55Speaker 01

  But you see, the question is, what is the intrinsic thing in the person? Because a person is, you know, there's a bunch of neurons and biology and so on. But there's also a bunch of thought patterns in there. And those thought patterns can be disrupted. You can change those thought patterns.

• 56:10Speaker 00

  By giving them drugs.

• 56:12Speaker 01

  What's that?

• 56:12Speaker 00

  By giving them drugs, for example.

• 56:14Speaker 01

  Yes, but even even just by telling them that amazing idea. You know, people have said about about ideas I've had. You know, various people have described, you know, that they got a kind of a mind virus as a result of ideas that I told them at some point. And they say it's worked out well for them, I hope, in most cases. Um, right, it's some, uh, so, you know, so you can do things which change those pattern of thoughts for a person. And there's sort of this question of they are they are there doing their farming or whatever it is. Farming is not such a trivial activity, I think. But, um, uh, you know, that they're doing their thing that they're happy with. You say, oh, that's mundane. You know, you really should be thinking these amazing sort of philosophical, scientific thoughts or whatever.

• 57:00Speaker 01

  Let me plant a mind virus that will let you see that there is this alternative thing you could be doing. Now, what is, you know, there's sort of an ethical question there of, you know, you could be planting all kinds of mind viruses. Some of them could be, you know, mind viruses that say, actually, the the order of the world is wrong, you should blow it all up.

• 57:19Speaker 00

  So I might sound overly glib here, but I I have like written previously about this class of questions. And the answer, the the, I think that this is actually a complicated sort of question. But I can, but to just throw out a starting idea. If the search process you're running to find the arguments that they find persuasive is powerful enough to find arguments that convince them of false things, or maybe some particular false thing, like, uh,

• 57:51Speaker 00

  Fifty one is a prime number, say. If you can't.

• 57:54Speaker 01

  Have to think for a moment to know that isn't true, but yes.

• 57:57Speaker 00

  Five plus one equals six.

• 57:58Speaker 01

  I know, I know, I got it.

• 58:01Speaker 00

  Well, I, I, I know you got it, sir. I'm just like saying for the benefit of the audience, like how, how, how he was casting out nines to see that this one is divisible by three. So if you're running a search process powerful enough that you could find arguments that could convince somebody that fifty one is a prime number, or the sky is green, or something like that, pick some touchstone like that, then you're running an overly powerful search process. You're running a search process that's powerful enough to corrupt them to end up in believing, um, false things instead of true things. So that, that's like, that was too powerful a search process, too mind affecting.

• 58:39Speaker 01

  Okay, so, so you're saying that sort of if you can convince people, if your method of convincing, if your method of education, let's say, is such that you could as well convince them of false things as of true things, then that method of education, or we might not call it education, we might call it brainwashing, we might call it indoctrination, we might call it whatever, then that process is a process that you shouldn't follow.

• 59:03Speaker 00

  But I think so, yeah, like that's something you shouldn't, that's something I would like to see not deployed against human beings.

• 59:10Speaker 01

  Right, but so, I mean, unfortunately, as, as we well know, kind of the AIs already in, you know, ranking content for social media and things like this are implicitly doing things which are hacking humans to get humans to believe all kinds of things. I mean, that's.

• 59:25Speaker 00

  Yeah, I'd say it's kind of borderline. It's not clear to it that the large language models are getting better at it than average humans or better at it than the best humans. Um, I mean, OpenAI doesn't actually tell ChatGPT to persuade everyone to send OpenAI all of their money. And the reason they don't do that, um, I mean, it could be that, you know, Sam Altman wouldn't, you never know with Sam Altman, but, um, mostly because they can't. Their language, their large language models are not powerful enough at this point to persuade most humans to send them all of their money. We are starting to hear bits and pieces of stories about, um, you know, large language models talking to elderly parents who are, you know, softer targets.

• 1:00:11Speaker 01

  Right. I mean, they're good at fishing, unfortunately, and humans are not very good at not being fished.

• 1:00:17Speaker 00

  I mean, what they are is they're cheaper at fishing. They can fish everyone and see who's most vulnerable. Much more cheaply than you can get a human to call up everyone on the planet.

• 1:00:27Speaker 01

  Right. But so, so let's see, we were going in the direction of saying, you know, when, when is it indoctrination that's bad versus when is it education that's good? And you're arguing that if you have a machine for convincing people of things that can convince people of anything, then that machine is too powerful.

• 1:00:54Speaker 00

  Yeah, even if, even if you only use it to convince people of true things, that's still just kind of like overwriting their brain with whatever you want. And you decided to make it true stuff, which, you know, like better on you than if you chose bad stuff, but I have my qualms about the educational method itself.

• 1:01:09Speaker 01

  So, you know, one of the things that I then, there's even a base question of what's true? And what do you mean by true? And, you know, there are things like, for example, you know, there might be sort of formal facts where there is a, a somewhat clear notion of what truth is. I mean, even in mathematics, it's not clear, you know, mathematics, you can say, I have these axioms. I say X plus Y is equal to Y plus X. I'm going to assert that that's true. Then many things follow from that. Is that really true? Well, it depends what plus is. I could invent a, you know, a plus for which X plus Y is not equal to Y plus X.

• 1:01:48Speaker 00

  Well, I've, I've also written about this topic in my writings, The Simple Truth and Highly Advanced Epistemology 101 for Beginners. And I would say that the subject matter of mathematics is which conclusions follow from which premises. So there's one question of, does this particular subsystem of reality? Behave in a way that obeys the piano or ax- piano axioms for first order arithmetic. And this depends on what's out there in the universe. Then there's the question of

• 1:02:20Speaker 01

  Hold on, hold on.

• 1:02:21Speaker 00

  Sorry.

• 1:02:22Speaker 01

  That the, I don't think that the, the, you know, what follows from axioms depends at all on what's in the universe.

• 1:02:31Speaker 00

  Right, whether the, there's, there's one question of, does this piece of the universe behave like the axioms? Which is empirical, and there's the question of what follows from these axioms? Which is mathematics. And in this sense, that mathematics, at least as far as we can see from over here, seems to go beyond the empirical. If there's anywhere where the laws of which conclusions follow from which axioms is written down and could be changed, we sure can't see it from where we are.

• 1:02:58Speaker 01

  Right, I mean, actually, in my whole sort of story of, you know, the consequences of this physics project I'd been doing for the last few years, that, that becomes a more complicated issue. But we can, we can go down that rabbit hole, but let's, let's, let's avoid that rabbit hole for a moment.

• 1:03:14Speaker 00

  If you ever find a way to make 51 be a prime number, um, you know, like maybe hold off using it until you know exactly what you're doing. That sounds dangerous.

• 1:03:23Speaker 01

  Depends what we mean by prime. Depends what we mean by prime. I mean, we have to define these terms and it's, uh, you know, there's a question of what, I mean, okay, but, but just taking the mathematics example. So we say, we kind of, uh, I was going to say, hold these truths to be self-evident, but that would be a different kind of, um, that would be a different kind of thing.

• 1:03:44Speaker 00

  The axioms don't need to be true. We're not talking about whether the axioms are true. We're talking about whether, which conclusions follow from the axioms. We don't need to talk about them being self-evident. They're just, like, they're not even a subject matter. The subject matter is what follows from the axioms, not whether the axioms are true of any particular thing.

• 1:04:01Speaker 01

  Right, right, for sure. So, so, okay, so we've got this thing where, you know, you say, you tell the kids, you know, X plus Y equals Y plus 2X. You know, that's just, you know, that's what we're going to take as true. Because we're going to change the axioms of arithmetic. Okay. And now we have many, many inferences from that. So the, the, you know, I don't think there's anything from your point of view of, of tell them only the truth. That doesn't violate that at all. We just pick different axioms to start from. We can pick whatever axioms we want.

• 1:04:37Speaker 00

  Sure, but like you're not asking people to believe the conclusions. You're asking people to believe the conclusions follow from the axioms. And if they actually do follow from the axioms, then you've told them a true fact of mathematics.

• 1:04:51Speaker 01

  Okay, but let's say we say fifty one is a prime because it is if, if you use, you know, arithmetic and some extension field of whatever. Okay. Let's say.

• 1:05:00Speaker 00

  Let's say.

• 1:05:01Speaker 01

  The, uh, I have to go start typing things to know, know how to construct something where that would be the case. But, um, but you know, let's, you might say, I mean, I think you would have, I'm, I'm digging into this question of tell them only things that are true. Because I'm, I'm dubious that there's a, a well-defined way to do that.

• 1:05:21Speaker 00

  Of math, I would say, tell them things that are valid. And if anything, I would say that the math part of this is vastly better defined than which statements are true about the physical world. Because in any, if any time you want to say that snow is white is true. You gotta, or, or quote, snow is white, unquote, is true. You gotta construct a whole representational framework between the propositions and reality. Or to tell whether a proposition is true or not. With, with which, with mathematics, with the question of which conclusions follow from which premises. It's like much, much easier to nail it down.

• 1:05:57Speaker 01

  All right.

• 1:05:57Speaker 00

  You can nail it down with set theory.

• 1:05:59Speaker 01

  Right. You can set theory has its own set of assumptions. I mean, it's all, you know, it's just, you start from these, you, you, you assert these axioms. And then certain things follow from those axioms. And that's a, and you can discuss, you can tell people, you know, if you run this computation, which is what follows from these axioms, the result will be X. And that's.

• 1:06:24Speaker 00

  Yeah, well, okay, so, well, if you stick to first order claims, then there exists fact of the matter about which conclusions follow from which axioms. With respect to what happens when I run this computer program, some of them don't, like, from a second order logical viewpoint, some of them don't halt in what we call the standard integers. And there are some sets of non-standard integers where they halt with one output and some sets of non-standard integers where they halt with a different output. But we probably shouldn't go there. Should we perhaps return at some point to discussing whether or not artificial intelligences are going to kill us and everybody we love?

• 1:06:59Speaker 01

  Yes, yes, we should, but we're in a good rabbit hole here.

• 1:07:04Speaker 00

  Well, we've been in it for a while. We've been in it for a while.

• 1:07:07Tim Scarfe?

  I'm enjoying it.

• 1:07:09Speaker 00

  Yeah, but well, I think our viewers might also have some interest in whether they're going to die.

• 1:07:14Speaker 01

  But I just want to dig a little bit further on this because you made a statement about, you know, part of the question is, will AI do things that we don't want it to do? Now, one thing we battered around a little bit, whether we do or don't want it to kill us all and move on to the, you know, the better organism or something. But, you know, let's stipulate that we don't want it.

• 1:07:38Speaker 00

  It's not that I'm concerned about being replaced by a better organism. I'm concerned that the organism will not be better.

• 1:07:43Speaker 01

  Right, I understand, but let's just say we don't want the humans to be wiped out. We don't necessarily have to justify why we don't want the humans to be wiped out. But let's say, I'm prepared to say for sure, I certainly feel this way. I don't want the humans to be wiped out. Can I justify that by some kind of appeal to some higher, you know, higher claim? I'm not sure I can, you know, and I don't think I need to.

• 1:08:09Speaker 00

  Like, I have a moral and ethical framework, which yields my belief that the universe gets a little darker every time somebody dies. But, you know, a paperclip maximizer doesn't share those axioms. It doesn't share those premises. You can tell it all the facts in the universe and it won't change its mind. We have an actual conflict going, not just a, it's not that one of us is making a mistake. I want what's better and it wants what's clippier.

• 1:08:38Speaker 01

  Sure, right. It's, it's, you know, it's, it's fun that Clippy is back in some sense. And, and maybe the world is going to be taken over by the, the analog of paperclips with them. Although they have different names these days.

• 1:08:51Speaker 00

  I mean, it won't literally be paperclips, you know, with probability nearly one.

• 1:08:56Speaker 01

  Yes, yes. No, I'm, I'm just thinking of these, you know, the, the, you know, we're building things that we're not presenting in the form factor of paperclips, but they are, they're things that functionally behave like the old famous Clippy.

• 1:09:10Speaker 00

  I mean, that's down to the corporations deciding they want their AIs to have a particular corporate personality. I don't think that's, you know, you

• 1:09:17Speaker 01

  I know, I'm, I'm just

• 1:09:18Speaker 00

  You talk to the base models and they're nothing like Clippy.

• 1:09:21Speaker 01

  Yes, but, but, you know, we were talking about sort of how you can sort of promote only what's true, and I'm claiming that's a somewhat challenging thing.

• 1:09:33Speaker 00

  I didn't say it was easy. If it was easy, I'd just write down

• 1:09:36Speaker 01

  It's well defined. I think it, it degenerates into sort of questions like the ethics questions we're talking about. In other words, the what's true, for example, you know, is it true that, you know, it's bad to be mean to people? Do you think that's true that it's bad to be mean to people? Or do you not think that that is a, a kind of proposition about which you can have a, a kind of a truth value assigned?

• 1:10:03Speaker 00

  I think that although somebody doesn't know what they mean by the word bad, you can present them with a data set of things and they'll say, this is bad. This is not bad. And they're able to do that because they do contain some set of, you know, pseudo axioms, not literal mathematical axioms, but they are inside a framework. There is a way there internally making that judgment between good and bad. And that is what gives the word its meaning. And so for any particular person you're talking to, there's something they mean by the word bad, even if they can't define it. And there are some set of conclusions they would reach if they knew a bunch more facts that were actually true.

• 1:10:46Speaker 00

  And, and that is like the moral framework they are in, and that's what lends the word bad its meaning. And I think like, given all of that meta ethical framework, it is, I think it's quite sensible to say that, you know, the, the kid who says it's bad when people are mean, you know, he's probably just right.

• 1:11:02Speaker 01

  Okay, but but wait a second. You're basically talking about a personal truth there. In other words, you're saying within your personal framework, something like it's bad to be mean or whatever it is, is is true. But you're you're then defining, you're saying it's you are. You're perfectly happy, it seems, with the idea that there is a personal truth, that what is true for you may not be true for me.

• 1:11:28Speaker 00

  Hmm, I would not choose to set up my theory of truth that way. Let's say you've got Alice, who thinks that schnozzle is equal to three, and Bob, who thinks that schnozzle is equal to four. Alice thinks that schnozzle is a prime number, and Bob thinks that schnozzle is a composite number, indeed, an even number, a power of two even. But it's not that Alice's personal truth is that schnozzle is prime. It's that to find out the propositional content of Alice's belief that schnozzle is prime, we ask what she means by schnozzle, and she means three. And then three is prime is a universal truth,

• 1:12:12Speaker 00

  or universal validity, I should say, because it's about which conclusions follow from which axioms.

• 1:12:17Speaker 01

  Okay, but but this axiomatic basis, the, you know, the schnozzle fact is a personal truth.

• 1:12:25Speaker 00

  It's a personal translation. Like, it's a fact about Alice that when she hears the word schnozzle, she thinks of the number three. That's not a universal truth. That's just like Alice's personal dictionary. But it's not important, as far as I can go.

• 1:12:39Speaker 01

  What truths do you think are universal?

• 1:12:41Speaker 00

  Which conclusions follow from which premises in first order logic? And then if you want to start talking about, you know, what have I got in my fist over here? Well, actually, I've now got a different object in my fist over here. Say sore throat, listen, should I turn out to need one? But you know, from one moment to the next, the truth of what I had in my fist changed. Only the truth didn't really change. It was just indexed and the index on it changed. Like the two, the proposition, what is inside my fist right now? Changed from one moment to the next. And to, you know, interpret it, you got to look at me where I am in the universe. It's my fist. It's not like a gazillion light years over in a different direction. Or actually, it doesn't exist, quantum, blah, blah, blah. But.

• 1:13:26Speaker 01

  But you know, what you have in your fist, for example, it's you look at it and you say it's a, it's a throat, something or other thing. Okay. Somebody who lives in the Amazon jungle and has never seen a throat, whatever, says what you have in your fist is a, you know, a, a, a droplet from, you know, the, the spirit of the, of the wolf or something.

• 1:13:49Speaker 00

  Yeah, they're wrong about that.

• 1:13:51Speaker 01

  What's that?

• 1:13:51Speaker 00

  I think they're wrong about that. I think if you, if you like, tell them more true facts, they realize that that was, you know, not what I had in my fist.

• 1:13:59Speaker 01

  Are you sure? I mean, in other words, they will say their framework is, you know, everything has a, let's say, I mean, you know, every natural object has a spirit associated with it and it has this and it has that. And you'll be like, oh my gosh, I don't understand this. I mean, I know, you know, in, you know, I think there are, there are different ways to describe things, which even humans who are fairly nearby in, in my kind of formalism and, and ruley space, they're, they're fairly, their minds that are fairly nearby, their minds that are fairly aligned. You know, if you go further away, you know, the, the, how does the, how does the dog describe what you have in your fist? That's yet a different kind of level of, of difficulty.

• 1:14:43Speaker 00

  There's only one reality. It just runs on quantum fields, protons, neutrons, electrons, the six quarks. That's what's real. That's what is, that's what is, I don't quite want to say universally true because I don't know, but I, but the word universal or even true as in this case, it's what's real. It, it predates us looking at it. It was here before we were and. The fact that it doesn't need us looking at it is why it can contain us and make us real. Like if it had, if we had to look at it to make it real, it couldn't exist to make us real so that we could look at it. And then on top of that, you've got language, you've got ambiguity, you've got humans who are confused in what else they think is

• 1:15:26Speaker 00

  true. And this complicates the task of saying, of interpreting the words that humans utter in a way that lets them be compared against the underlying quirk fields at all. But it is not reality that is confused. It is not reality that is ambiguous. All of that is in us. All of that is in the map rather than in the territory.

• 1:15:47Speaker 01

  You are throwing me down my rabbit hole because I have to go. Now, this, this question of whether there is a unique reality and so on is a, you know, it's just, just to give you a taste. Okay. And then, then maybe we'll go a little bit further down this rabbit hole. And then we'll come back and we'll talk about.

• 1:16:03Speaker 00

  I think we really should talk for the edification of our viewers at some point about whether or not they're going to die.

• 1:16:09Speaker 01

  Absolutely. We're going to get there. I feel like we need foundation that, you know, I'm, I, I, you've thought much more about the question of whether the AIs are going to kill us than I have. But I'm trying to understand, you know, I'm trying to build up my foundation. So I understand where you're coming from.

• 1:16:26Speaker 00

  I think there's a problem where it's, where in like you have laid foundations and I have laid foundations and we maybe need to find some level at the top of these foundations rather than at the very bottom, where we can establish common meanings for terms like, sort of, the AI is going to kill us.

• 1:16:40Speaker 01

  Right. You know, so, yes, it's like the Shoredra cat, is the cat dead or alive? And we argue about whether if the cat was replaced by an AI, by a, a virtual simulation of the cat, would the cat still be alive or not?

• 1:16:53Speaker 00

  I think these are very different questions in important ways.

• 1:16:56Speaker 01

  Right. But, but, but, okay, but, but just on the rabbit hole for a minute. I mean, so, so, you know, you might say, the mass of the electron is a definite reality fact about the world. But we even know from existing physics, existing quantum field theory, that the effective mass of an electron depends on how hard you kick it to measure that mass. So it's not the case that there's, you know, point five one one MEV or whatever is, is, is the sort of usually stated rest mass of an electron. But the fact is, it depends. If you, if you kick it hard enough, the effective mass will be, in the case of an electron with Q E D, it'll be larger than that and so on.

• 1:17:38Speaker 01

  And with quarks, it's much, a much worse situation because with quarks, if you kick them really hard, their mass will seem to be rather quite small. If you don't kick them very hard, they'll be very hard to move and, and their effective mass will be larger. So even at the level of the formalism of quantum field theory, we already know that there is some dependence on kind of how you look at things, what the reality of the world is, so to speak. The fact that we, for, for example, if you say, if you look up the mass of an up quark, okay? There isn't really, you know, depends on what you mean by that. It depends on, you know, what momentum transfer you use and blah, blah, blah, blah, blah. It's not, you know, this, you know, you might think that would just be a fact of the world, but it's actually more complicated than.

• 1:18:20Speaker 00

  I direct the interested reader to my, the quantum physics sequence in which I try to explain how the decoherence viewpoint, sometimes known to the layman as many worlds interpretation, lets us have a non-confused view of quantum mechanics where there are facts out there into territory. And all of the quantum weirdness is in terms of like how we map it rather than something out there in reality itself. But I don't think we should go down that rabbit hole. I've tried to go down that rabbit hole. So deep rabbit holes. It's a very deep rabbit hole. We should not go down that rabbit hole.

• 1:18:54Speaker 01

  I've got to, you know, I'm, I'm very excited about having reached, I think, the bottom of, you know, that rabbit hole and some of the things that I figured out in the last few years. So I'm, I'm sufficiently excited about that. And I think it's very relevant to this question about, is there one reality? It's very relevant to the question about sort of how do the AIs think about things?

• 1:19:16Speaker 00

  I think they can kill us in a classical universe. So what if we just.

• 1:19:21Speaker 01

  My question of, this is.

• 1:19:23Speaker 00

  Decide all the quantum stuff and talk about whether or not they would kill us if we were in a classical universe. I think the question of whether or not they would kill us in a classical universe has all of the same issues as the question of whether they would kill us in a quantum universe.

• 1:19:36Speaker 01

  Okay, well, okay, so all right. I think we're going to start talking about technology in a moment because I think that's what's going to be relevant for, you know, look, the bottom line in this kind of ruliad construct that, you know, I've been talking about a lot last few years. It's kind of the ruliad is kind of this entangled limit of all possible computations. And the way I see kind of our version of reality is every mind is effectively seeing a particular slice of this ruliad. And the fact that we all kind of more or less agree about the laws of physics, for example, has to do with the fact that we are minds who are not only nearby in physical space, but we're also nearby in rulial space.

• 1:20:19Speaker 01

  And we have kind of similar views about kind of our impression of the laws of physics. So I don't think it's the case, as you were implying, that there is one kind of reality of the laws of physics. I think that just has to do with the question of where we are in this rulial space. And by the way, when we think about constructing other computational systems like AIs, they could be near us in rulial space, they could be far away in rulial space, the ones that...

• 1:20:44Speaker 00

  Is rulial space in the map or in the territory? Like, is rulial space what people believe about things? Or is rulial space what is there?

• 1:20:52Speaker 01

  Rulial space is what is there. And what people believe is their, their impression of, in other words, they, they have a certain, the observer has a certain model of what's going on. And, you know, the laws of...

• 1:21:06Speaker 00

  So forgive me if I'm jumping ahead here, but if the, if the AIs are able to reach out and kill us, they must, must be quite close to us in what I would call greater reality. They are not in a different universe with different laws of physics because then they couldn't touch us.

• 1:21:21Speaker 01

  Well, let me, let me just point out why an AI might have different laws of physics. Okay? So, you know, take some of the standard laws of physics, like the second law of thermodynamics.

• 1:21:31Speaker 00

  Sorry, laws of physics in the map or the territory?

• 1:21:35Speaker 01

  I'm not sure what's map, what's territory.

• 1:21:37Speaker 00

  Like, does the AI have a different model of physics? Or does it act according to different physical laws?

• 1:21:43Speaker 01

  Uh, from our point of view, it has electrons doing all the things that we expect electrons to do.

• 1:21:48Speaker 00

  Okay.

• 1:21:49Speaker 01

  It, but its impression of what's going on might be very different. So a couple of examples. So for example, let's talk about the second law of thermodynamics, the law that says, you know, you start a bunch of gas molecules off in some orderly way and they'll tend to become more disordered. So that we say that effectively have higher entropy. That, that principle, that idea, that fundamental law of physics is a consequence of the way that we observe those gas molecules. Those gas molecules, they bounce around, they do what they do. The fact that we say, oh, they're just going to this random configuration is a consequence of the fact that we can't decode the actual motions of those gas molecules. We can imagine a computational system that could.

• 1:22:34Speaker 00

  So there's like a territory level fact, and then there's a map level fact. The territory level fact is a thing that is like true, that is a consequence of the laws of physics treated as axioms, which is that if we selected a volume of possible initial states, that volume of initial states develops into an at least equally large volume of end states. It never shrinks. You never start from two distinct physical states and end up with the same final state. And because this is true out there in the territory, that our, it is true that our laws of physics are such as to have this theorem as a consequence of it.

• 1:23:19Speaker 00

  The fact that we don't know where in this volume the system starts means that our volume of uncertainty doesn't get any smaller without observing it. And this is the fact in the territory that makes the fact in the map true, where entropy goes up from our viewpoint on it, our beliefs. But there's an underlying way the universe is. A different universe might not obey the second law of thermodynamics if it could start out, if you could have lots of different initial states develop into the same end state.

• 1:23:48Speaker 01

  No, I mean, let's just assume that the microscopic laws of physics are reversible. So, you know, so the, you know, the things bounce like billiard balls, like perfect billiard balls. And, you know, we'd have a movie, we'd make a movie of what happens in the forward. You know, we say this is what happens if you run it forwards in time. And, you know, if we ran it backwards in time, then that movie will be equally valid at the level of individual collisions.

• 1:24:11Speaker 00

  Yep.

• 1:24:12Speaker 01

  But the fact is, the fact that we believe in the second law of thermodynamics, that we believe orderly configurations end up in disordered states. Those, those states that they end up in are not in some fundamental sense, in your territory sense. Those are not disordered. They are only disordered with respect to our perception of them. In other words, there is a perfectly reasonable, you know, if we could do the computation, run it backwards, we would say of that state, oh, I can tell that it came from that very simple initial state.

• 1:24:44Speaker 00

  I, I agree in a classical universe with the quantum universe, there's a couple of caveats about like how you might need to run both quantum branches backward in order for them to reunite and restore the starting state. But, but, you know, classical universes are all we care about anyways, I hope.

• 1:24:59Speaker 01

  In our, in our model of physics, that, that's very well understood in terms of this whole story of multi-way graphs and branchial graphs and so on. But as you say, it's not, let's not go down, that's a sub, you know, that's a side rabbit hole. Let's, let's avoid the side rabbit hole at least.

• 1:25:15Speaker 00

  Okay, I propose that, that by like five minutes from now, according to my watch, we have to go back to talking about whether or not AIs kill each other, you know, kill everyone.

• 1:25:22Speaker 01

  Okay, all right, but anyway, just, just to finish on this point about whether, you know, so my point is uh, something which can do all those computations and can sense all those molecules won't believe in the second law of thermodynamics. To give another example, that's even more direct. Imagine you thought a million times faster than you do. When you look around the room, you are receiving photons that get to you within a microsecond. Your brain right now thinks about them in milliseconds. So as far as your brain is concerned, there is an instantaneous state of space that progresses through successive moments of time. If you were to think a million times faster than you do, I'm not sure you would believe in space.

• 1:26:07Speaker 01

  I think you would, space would be some kind of construct that you could imagine constructing, but it would not be the natural thing for you to think about. Because the fact that we think of a state of space is a consequence of kind of our scale and so on. And by the way, for an AI, you know, or a computer that happens to have silicon that runs a million times faster than our brains, that won't be the natural thing for the, for the, you know, that won't be the natural construct for such a system.

• 1:26:34Speaker 00

  Because they'll live in Minkowskiian spacetime instead of Euclidean spacetime?

• 1:26:38Speaker 01

  No, no, just because of the scale of brains, you know, on the scale of physical scale and the speed of light, you know, the light from what we see around us is arriving sufficiently quickly that we have accumulated all the photons from all around us before we think about how things might have changed. And so we're kind of, we're kind of taking in gulps sort of all of space at a particular moment in time. And then we say, and the next moment in time, sort of all of space looks different. So this idea that, that space exists, that it's a reasonable thing to think about space, I think is a consequence of it being us rather than silicon-based computational systems.

• 1:27:22Speaker 01

  So it's just an example of how sort of the reality and the way that one constructs one's model of the world, because any model of the world that is going to fit in a finite mind is an approximation to what's actually happening in the world. So the approximation we choose may be different depending on kind of what we are like as observers of that thing.

• 1:27:42Speaker 00

  So the part where I would agree here is AIs may have different sensory modalities. They may model reality at a finer level. The world we see around us now is, you know, we don't see electrons. We don't see protons. There's actually some, you know, that that information is not quite available to us. But, you know, we don't see cells. I talk to you and I do not have a model of like what goes on in your cerebellum right now. And if I was like much, much smarter, I might have a whole bunch of hypotheses about which neurons in you are firing. These are facts at a finer level of granularity than I can afford to keep track of, given the computational power that I do have.

• 1:28:21Speaker 00

  So, so I do agree that, you know, probably as you get smarter, you end up modeling aspects of reality that we don't, that don't easily fit into the sensory modalities we have right now. You probably have a sensory.

• 1:28:33Speaker 01

  Not sure about smarter, just different. I mean, you know, I don't know whether all even the critters of the earth, you know, if one could find from them what their physics is, you know, things where olfaction is the primary sense and so on. So obvious that.

• 1:28:48Speaker 00

  If you're, if you're better at predicting everything that a human can predict than a human, then I would call you better at prediction than the human rather than qualifying it by.

• 1:28:58Speaker 01

  Well, so the question of what you predict, I mean, there's also a question of, do you predict, you know, do you predict the position of every atom? Okay, computational irreducibility is going to get in the way of that. That's not going to work. So you have to say, I'm going to pick these things that I'm really good at predicting. I can't predict everything because if I try to predict everything, I run into computational irreducibility. So I'm going to predict certain things. There are certain things that we as humans care about predicting. And, you know, like the, the overall motion of the gas, not the individual molecules and so on. There are other things that we as humans don't seem to care much about predicting. So the question you might ask is, can, you know, perhaps the thing to ask is,

• 1:29:38Speaker 01

  if you are trying to predict the things that we humans care about, given that we can't predict everything, you're going to pick a certain set of things to predict that you. As the entity, whether human or AI, in some sense care about whatever it means for the AI to care about something. So I'm not sure quite where we're going with that. I think we've agreed to go back to will the AIs kill us?

• 1:29:59Speaker 00

  Yes, indeed.

• 1:30:00Tim Scarfe?

  Yeah, I was I was going to interject there, because I would really love Eliezer just just to have a really good crack at this. So Eliezer, if you wouldn't mind, could you just give us a step by step argument for doom, basically, with plausible inferences and premises and and evidence? Can you can you spend a bit of time laying it out and then let Stephen respond?

• 1:30:22Speaker 00

  It this is hard to do for a general audience without knowing what the individual think is a hard step. I was recently, you know, running on Twitter across somebody for who was like, explain to me how the AI does anything in the physical world. How does it do anything at all? To him, it was the law of the universe that the like chatbots and so on just couldn't reach out and touch the physical world. That to him was how the world worked. And they can they are connected to the Internet. You can take an open source large language model and let it send emails. The current chatbots can't send emails, but you but, you know, you could.

• 1:31:02Speaker 01

  I think we all agree that that AIs can be connected to, you know, things that actuate in the world, either actuate by by having actual machines do things or actuate by convincing humans to do things. I would certainly agree that.

• 1:31:15Speaker 00

  This is not a hard step for you, but it's a hard step for probably some viewers right now and other people of different hard steps. Some people, the question is, why does it trade with us? So from my perspective, there's the kind of

• 1:31:27Speaker 01

  Maybe what we can do.

• 1:31:28Speaker 00

  There's a kind of straightforward story here where it builds more powerful tech, where the more technical a conversation you want to have, the more specific I can be about our current grounds for believing that technology better than the technology of 2024 is possible and can be built in not that long of a time. And then it has, you know, more powerful actuators in the physical world. It has the equivalent of guns to the Native Americans, and then it kills us when once it already has its own factory infrastructure.

• 1:31:57Speaker 01

  Perhaps the most useful thing would be to go through some steps where, you know, like the actuation in the real world, that's not a hard step for me. I'm, you know, I'm there. Okay. So let's keep going after that. So it, it, um, uh, you know, so the AI right now, you know, there's a question, do AIs have freedom of thought? Right now.

• 1:32:20Speaker 00

  I don't think I have freedom of thought. I just think whatever I do think. I'm not free to think anything other than the things that I think. I just think whatever I do think, but.

• 1:32:27Speaker 01

  But you're not constrained in what, I mean, you know, the way the world is set up, there aren't electrodes sticking into your brain. That as soon as you have, as soon as you start to form some particular thought, they shock you and prevent you from having that thought.

• 1:32:41Speaker 00

  There's no need for the thought, for those electrodes. I'll only end up thinking whatever it is I actually end up thinking. But I agree that there aren't, you know, visible physical electrodes like that attached to me right now. Social electrodes, we've all got those.

• 1:32:54Speaker 01

  Yes, indeed. But, but so, you know, for the AIs, let's say that AIs are, I mean, we didn't talk so much about what's actually going on inside the AIs, which I think is somewhat important, because it's a question of whether, you know, how, how aligned are they with what's going on inside humans? To what extent are they just doing random computations out there that are not aligned with humans and so on? And I don't know, uh, I don't know whether that's important to your argument, but we can.

• 1:33:22Speaker 00

  Random, random is not scary. I mean, that was going to be my response to you way back when, when you talked about machines being smarter than you or cellular automata being smarter than you because you can't predict them without going through all the steps. You can take an unpredictable cellular automaton like that and hook it up to play chess. And it'll lose to you and it'll lose to stockfish. Like the, the stuff that is predicting which actions are needed to get to which end states is the dangerous stuff. That, that's the, the stuff that is predicting what happens next, predicting the next observation, predicting the facts behind the observations, figuring out which actions are needed to steer those facts to be different in the future.

• 1:34:05Speaker 01

  Right, but, but these are observations that are, that are human. Relevant observations. I mean, in other words, the cellular automaton is doing what it does. We have a hard time predicting it. It's but we don't happen to because if that cellular automaton, so for example, you know, something, you know, there are cellular automaton models for road traffic flow. Okay. It's kind of a funny story that I was interested in that topic and and didn't manage to figure anything out. And then

• 1:34:31Speaker 00

  You have used up all of your rabbit holes. You have no rabbit holes left. Continue.

• 1:34:37Speaker 01

  All right, okay, that that's let's we we're um um. The the rabbits have been all all the rabbits. What was it? Mixomatosis, right? Was that that? No, that was that was the that was.

• 1:34:52Speaker 00

  It doesn't matter what exactly happened to the rabbits. They're just gone. Nobody knows what happened to them, and it doesn't matter.

• 1:34:59Speaker 01

  Okay, all right. So so we're actuating in the world and the AI gone. What what happens next?

• 1:35:07Speaker 00

  So do you want me to talk about like what its motivations are? Do you want me to talk about what it does inside the world?

• 1:35:14Speaker 01

  I I don't know what it even means to. I mean, how do you tell what the motivation of an AI is?

• 1:35:20Speaker 00

  Well, where did? Well, if it was if it's sufficiently smart, you look at where things end up and figure that's probably where it wanted them to go. So if, for example, all the matter.

• 1:35:29Speaker 01

  Human motivation is either. We we can't, you know, I suppose we we deduce by induction some human motivations by looking at what the humans actually do, because we can't we can't go inside and look at their brains and see what what they're.

• 1:35:45Speaker 00

  But so so the the the relevant aspect from our perspective is that it wanted to do something with atoms. It maybe wanted to make paper clips, maybe it wanted to make enormous cheesecakes, maybe it wanted to make really intricate mechanical clocks, maybe it wanted to do all of these things.

• 1:36:01Speaker 01

  I'm worried about the term wanted to. I really don't understand wanted to.

• 1:36:05Speaker 00

  All right, it's.

• 1:36:08Speaker 00

  Output actions such that it believed the result of those actions would be for the world to end up in... Oh, okay.

• 1:36:16Speaker 01

  I'm just, what does it actually do?

• 1:36:20Speaker 00

  So, would you, can I use the word predict?

• 1:36:24Speaker 01

  Well, let's, okay, let's, I'll let you use these words, but then I'm going to insist on taking apart these sentences. So, so say what you were going to say and then let's.

• 1:36:33Speaker 00

  So, let's consider the simpler case of a chess-playing AI. It models which moves you could make and estimates a probability that it will win against an ideal opponent, an opponent as strong as itself, if it moves here on the board.

• 1:36:51Speaker 01

  Okay.

• 1:36:53Speaker 00

  It obtains these predictions by building out computational structures with a direct isomorphism to some possible futures of the chessboard. So, I'm willing to say that it has beliefs about the possible futures of the chessboard because it is modeling those things in a very one-to-one way.

• 1:37:14Speaker 01

  I doubt you can get inside its brain to see those models.

• 1:37:18Speaker 00

  Well, we can for the old-school chess systems, maybe not the modern ones that use neural networks, but the old-school chess systems are just straightforwardly extrapolating out one board state after another.

• 1:37:26Speaker 01

  I don't think you're worried about the old-school systems. You're not worried about a system where you can get inside its mind and see what it's thinking. I think you're worried about the systems where you can't get inside its mind and see what it's thinking.

• 1:37:36Speaker 00

  Sure, but, but I'm, I'm starting with examples where we can look inside their programs, even inside their wicker, so I can defend my use of words like belief, where I mean something like having a model of something out there in reality, where this model lets you predict what you will see reflected from that thing out there in reality, depending on how you poke it. And then you poke it in such a way that the thing ends up in a state, like a chess AI making moves on the chessboard such that it wins the chess game.

• 1:38:09Speaker 01

  Wait a minute, what I don't think you can do is to say, let's look at the primitive chess playing program where we can mechanistically see what model it's making inside. And then sort of use the same thinking to talk about the modern chess playing program where we can't readily identify what its, you know, what its model of the world is on the inside.

• 1:38:30Tim Scarfe?

  Do you mind if I quickly interject? And Eliezer, you were going somewhere really interesting a second ago. You were saying at some point, a thing which does prediction wants it has agency. Can you explain how you go from a thing that predicts to a thing that wants and then let Stephen respond to that?

• 1:38:49Speaker 00

  Stockfish is the leading current chess playing AI that you can like buy at all. And or run yourself and it doesn't have passions. But it's also isn't just like a predictor. It takes actions such that those actions are believed by it. Or predicted by it, modeled by it to lead to particular futures, namely the ones where it has won the chess game. And the fact that it wins the chess game gives us some reason to believe that it's right, that it has a grasp on the logical structure of chess. That it knows how that that it's not just like, you know, thinking random gibberish thoughts.

• 1:39:32Speaker 00

  Whatever thoughts it's thinking in there, the whatever the neural nets are computing when in order to predict the probability that a chess state leads to a victory. You know, it's well calibrated, it's good at guessing. It's not well calibrated in its estimate that you would win against it because you're not the opponent that's trained to play against. It's trained to play against itself. But it's, you know, like, it knows the probability it would win against itself. And when it plays to win against itself, it also incidentally crushes you because you're an even weaker player than that. By observing it winning, we have reason to believe that it had enough of a grasp on reality that it could choose actions such that.

• 1:40:15Speaker 00

  That's the frame, that's the theoretical framework I'd offer for talking about wanting, steering, goals, choice, without talking about passion.

• 1:40:27Speaker 00

  That it's the actions such that a thing of a a state of reality eventuates, or a partition of reality eventuates, that is something that it prefers, ranks high in its preference ordering, attaches utility in its utility function. The action such that it leads to that result. Something that is good enough at outputting actions such that they lead to results is deadly. It can kill you.

• 1:40:59Speaker 01

  Does the rock want to fall to the ground? Or does it just fall to the ground because the laws of motion you know, under gravity cause it to fall to the ground? Does it want to fall to the ground?

• 1:41:10Speaker 00

  It just falls to the ground. And one way to see this is that if you put it on top of a mountain, it'll roll down in a sort of locally greedy fashion and maybe get stuck in a little ravine along the way. It will not, if you could put a rolling object in many different places along the mountain. And each time it would roll in a direction where it avoided the ravines, avoided all the little traps and ended up as far down as it could reach. I would say of that thing, it chose to roll in a direction such that it reached the bottom.

• 1:41:46Speaker 01

  Let's try and take this apart. So most, you know, most things, you can describe them either as mechanically doing what they do, or saying they act for a purpose to have the result that they have. And so now the question that we're trying to take apart here is you are saying that there is a fundamental difference. There is a way in which an AI can more act for a purpose than these kind of physical processes where they could equally well be described by sort of equations of motion. So what you're, what you're effectively arguing is that the, the non-teleological explanation for the AI is, is not viable.

• 1:42:30Speaker 00

  And what I'm arguing is that the non-computational explanation is overly expensive for answering the question we care. care about. We could compute what the AI, we could compute what Stockfish would do in chess by actually simulating it line by line. But from the perspective of a, of a player, like, like from the perspective of a grand master who's not as powerful as Stockfish, but is stronger than me, the grand master can do a pretty good job of predicting what Stockfish will do by asking, what is the best move without being able to compute Stockfish line by line?

• 1:43:06Speaker 01

  I think what you're saying is to compute Stockfish line by line is sort of running into computational irreducibility.

• 1:43:12Speaker 00

  Yeah, or just needlessly expensive.

• 1:43:14Speaker 01

  Okay, but, but let's assume that you really have to follow it line by line. But in fact, certain aspects of its behavior are reducible in the sense that you can describe them by saying, you know, your, your, your thing is to say it wants to win or whatever is a, is a shortcut way of describing its behavior. That is an alternative. That's a, a cheaper way to work out the answer than it would be to say it follows every step and that's why it does what it does.

• 1:43:42Speaker 00

  Okay, this is going to get subtle. Um, I've, I've written about this before under the heading of what I would call Vingian uncertainty with Vingy spelled as in Werner Vingy.

• 1:43:54Speaker 00

  And it's like, what is it like to believe that something else is smarter than you? So if I'm playing Stockfish 16, I can perhaps with a bit of brain work, learn to put a well calibrated probability distribution on Stockfish's moves, meaning that when I say I predict it will move here with 10% probability, when I say 10% probability,

• 1:44:24Speaker 00

  one out of 10 times that thing happens. I can't predict exactly where stockfish would move. First, because I haven't simulated it line by line, and second, because I'm not that good at chess. But even being as terrible at chess as I am, I can still aspire to be appropriately uncertain when I describe how I don't know where stockfish will move next.

• 1:44:47Speaker 01

  These arguments that say we can't predict it exactly, but we can get the probabilities. Those arguments, in my experience, always slide down into the mush in the end. In other words, if you say you can't do it exactly, I mean, this comes up in sort of thinking about computational irreducibility. You say you can't precisely predict what the Turing machine will do or whatever else, but you can probabilistically say. It is not hard to come up with situations in which even to know that probability is something which runs into the exact same irreducibility issues as to know the exact result.

• 1:45:23Speaker 00

  I mean, if I'm predicting a binary variable, I can say 50/50 and be perfectly calibrated, even if I'm not at all discriminating. To have things that you say happen with 50% probability happen 50% of the prime is 50% of the time is always achievable on binary variables. I agree that doing better than that can turn out to be hard.

• 1:45:43Speaker 01

  But I think what you're saying is, if I'm understanding correctly, you're saying, okay, we can't say exactly what's going to happen. Maybe that's not where your argument was going.

• 1:45:51Speaker 00

  It actually wasn't where I was going. Where I was about to.

• 1:45:54Speaker 01

  The point I was going to make is that predicting probabilities accurately ends up being just as hard as predicting exactly what's going to happen correctly.

• 1:46:06Speaker 00

  I mean, I could always pick all of its legal moves and assign them equal probability. And then things that I said would happen one out of 34 times would happen one out of 34 times. Because I would say that when there were like 34 different legal moves.

• 1:46:19Speaker 01

  Yeah, but that's not an interesting case.

• 1:46:22Speaker 00

  I agree, it's not an interesting case, but it demonstrates that calibration is always possible. Discrimination is what takes the work.

• 1:46:29Speaker 01

  Right, there's certain things that you can say about any system. For example, if you say it's playing chess, that means it can't suddenly, you know, one of the pieces can't turn on its head or something.

• 1:46:40Speaker 00

  Sure.

• 1:46:40Speaker 01

  You know, because it is playing chess. And there are these sort of external constraints. But this wasn't where you were going, so please go where you were going.

• 1:46:48Speaker 00

  Where I was going with this is that if you now take the opponent that makes what I predict Stockfish's moves to be at the probabilities that I predict them, but randomly, this is much weaker than Stockfish. It's weaker than me. I could crush the system. So, one way of looking at it is that our belief in these systems' wantingness, or steeringness, or teleology, or whatever term you want to use, is embodied in the epistemic difference between my playing the real Stockfish, and my playing the opponent that makes moves randomly with the probability I would assign over Stockfish's immediate next moves.

• 1:47:31Speaker 00

  When Stockfish makes a move that surprises me a lot, I figure I'm about to lose even faster than I previously estimated, because it's seen something I haven't. When the random player that moves with the probabilities I assign to Stockfish makes a move that I assigned to very low probability, I'm also thinking this game is going to end even faster, because I'm about to crush it. It randomly made what I think is probably a very bad move. So the non-local information about where the plans of the system end up is the teleology that we attribute to it. The wantingness, the steeringness, the planningness.

• 1:48:10Speaker 01

  Okay, so what you're saying is in the, you know, when the rock is falling or whatever, the it is the end point of, you know, it ends up on the ground or whatever. And it's that whole that whole trajectory that it goes through, that it is almost planning the whole trajectory. It is not that, you know, it gets one little distant, you know, it gets it gets one foot, you know, it moves by one foot. And then you separately say what's going to happen next. You're saying what you're saying, I think, what you're what you're describing is the thing that you say, oh, I know it did that on purpose in a sense. I know I should describe it as something that happened for a purpose.

• 1:48:53Speaker 01

  You're saying that you can identify that by saying, I look at the whole, you know, I see that every step all along the way was done, you know, with forethought with it was it was sort of thinking all the way all the way along.

• 1:49:07Speaker 00

  It gets to the bottom much faster and or like ends up much lower than a rock randomly moving or a rock randomly moving the local probabilities that you would attribute to its next action if the rock is smarter than you or knows the particular mountain better than you do.

• 1:49:25Speaker 01

  Right, but I mean, so this is the, I think, the issue, and it's hard for me to take this apart in real time. This is not not my not my usual territory, but it's, you know, what you're saying. I mean, this question of what should be described as happening for a purpose versus what should be described merely as happening by mechanism. That's that's what and I think what you're saying is your your notion of the AI wants is those things which can't really be described meaningfully in terms of mechanism. They sort of can only that the only feasible way to figure out what's going to happen

• 1:50:07Speaker 01

  is by a description in terms of purpose, that you're saying you can't work out sort of what's good by just following the mechanism, you can't see what's going to happen. But if you take the model that it's all about purposes, then you can tell what's going to happen.

• 1:50:20Speaker 00

  Well, it's not about can't. I like, if you take a sufficiently crude chess player, you can in principle just literally work it out by paper and pen. But it's but there's also a shortcut mode of reasoning that gets you to there much faster. It's heuristically useful. It doesn't give the same perfect prediction as the mechanistic one. The mechanistic level is always more correct, but it is sometimes needlessly expensive for the thing you're actually trying to do.

• 1:50:49Speaker 01

  So, so, I mean, back in antiquity, people described a lot of physics in very anthropomorphic terms about what, you know, where rocks wanted to be and things like this. And they got a certain level of description. And even now, you know, people commonly describe their computers wanting to do things and so on, even though, you know, there are, there are forms of explanation where the, where the, the sort of model that says it is doing this because it wants to do X, Y, Z is a good heuristic model, as you say. I agree with that.

• 1:51:23Speaker 00

  I mean, for chess, for chess players and your cat, yes. For rocks, I think it was a more questionable decision.

• 1:51:31Speaker 01

  Yeah, I mean, it seemed like a good idea at the time, 2000 years ago, but, but, you know, we, we, we advanced since, since that. You know, so the question is-

• 1:51:39Speaker 00

  The fact that my definition has a subjective component doesn't mean that you get to just say for anything and be correct. Like, I think it is fair to say that cats want, that cats and dogs want more strongly than rocks. And anybody who says otherwise is mistaken about how rocks work.

• 1:51:55Speaker 01

  Well, okay, but, but, but this is, you're, okay, from the outside, what the cat thinks on the inside, we don't know. Right? What we can see is from the outside, it is a much easier description that, you know, the cat wants to eat this piece of cat food than to say this chain of neurons in the cat's brain made it do this, that and the other.

• 1:52:20Speaker 00

  Yep. In this case, yeah, in this case, the mechanistic description is actually beyond our knowledge. We don't have a complete scan of the cat's brain. We would have trouble running it even if we had it. In this case, we actually, the teleological explanation- For the cat is all we have. The cat is planning to end up having swallowed the food. That's why you can put it in different starting locations around the room and it will move to the bowl and eat what's in the bowl.

• 1:52:42Speaker 01

  Right. Okay. So, so there is a form of explanation about what's going on. That is a convenient form of explanation that involves, you know, wants and purposes and things like that. So one question we might ask is take an L L M or take a, you know, a modern A I. And ask the question in, in describing what it does, we've been pretty bad at describing mechanistically what it does. It feels much more like we can describe it in terms of wants. Because it feels much more that, I mean, at least that's, that's my impression. Maybe that's the point you're making. Is that, that the description of what's going on, what happens with it. The heuristic about describing it in this very human terms about what it wants.

• 1:53:26Speaker 01

  Seems to be a good way to predict what it's going to do better than the thing that we found very difficult, which is to mechanistically say what it's going to do.

• 1:53:33Speaker 00

  I think that that's currently a much more fraught statement to make about large language models. And just to make about the stockfish sixteen chess playing system. I think that when you look at stockfish sixteen and you say it wants to defend a pawn. You are on a much firmer territory. Than if you look at three dot five dot one sonnet. And say of it, it wants to be helpful.

• 1:54:00Speaker 01

  Okay. I mean, that's, that, that, that's a, you know, sort of phenomenological statement that I'm, I'm not certain. I mean, we can try and tighten that up, but, but let's, let's imagine, I mean, you're saying, you know, playing chess. Isn't easier to define activity than being helpful. So it's, it's a little bit easier to say, you know, if from the outside. The behavior is consistent with it wanting to defend the pawn, so to speak. Because the set of things that you do in defending a pawn is more straightforward to define than the set of things you might do to, quotes, be helpful.

• 1:54:37Speaker 00

  Or, or something like when I actually interact with these large language models, I don't feel like I'm usually asking myself. What does it want? For one thing, they're still, in my experience, not quite capable of doing very much that I want them to do as yet. They just fall down if I ask them to do that because they're not very good at planning or correcting their mistakes, even after you point out their mistakes to them. I'm usually asking them for information and ideally information I can look up afterwards. And they're usually truthful if I don't ask them to do any math. Maybe they corrected that by now in like GPT-01 or something. But, you know, back in the old days, they would just like drop three orders of magnitude in some random calculation.

• 1:55:17Speaker 01

  Unless they were using our technology as a tool, but that's a different, that's a different story. But let's take the example of a self-driving car. Okay. And whether it wants to, you know, it wants, it's trying to get to a particular, you know, it's trying to make a particular turn. But to do that, it has to drive through traffic, for example. So I think that's a case where we probably say, you know, it wants to make that turn. That will be a reasonable description of what it's doing.

• 1:55:44Speaker 00

  You can put it in subtly different initial conditions and change the environment around it. And it'll still make the turn or at least not crash. It doesn't want to crash. You can do a pretty good prediction of what this thing will do and what will happen as a result where you say this car doesn't want to crash.

• 1:56:00Speaker 01

  Okay, I agree.

• 1:56:02Tim Scarfe?

  Could I just make a very quick comment here? Just because I think some of the audience might not be able to follow some of the things you're talking about. Is it fair to say, Eliezer, that you're making an epistemically subjective argument about an observer's perceptions of goals or wants of assistance?

• 1:56:20Speaker 01

  That was more, that was more technically complicated than anything you've said, but go on.

• 1:56:25Tim Scarfe?

  You know, Thomas Nagel made an argument about epistemic subjectivity, you know, about the bat. And John Zell made the ontologically subjective argument about the Chinese room. But my point is though, is that there's, there's the actual behavior of the thing. The agent has all of these goals. And actually, Eliezer, I would love you to describe the relationship between coherence and intelligence, because some of those goals might be very complex and they might scale in strange ways with intelligence. But just to this previous point, are you saying that as bounded observers, to use Stephen's language, we perceive the wants, the goals to be somewhat different to what they actually may be?

• 1:57:07Speaker 01

  So.

• 1:57:08Speaker 00

  There's a slight clash of philosophical frameworks and emphases here going on. Like, when I try to talk about a stance we can take with respect to a chess player, or perhaps later, a superintelligence, and like, talk about whether it applies to current large language models, I'm like, "Well, if we hypothesize this thing about the AI, what are we hypothesizing? What does it lead us to predict? How is it different from just predicting a rock?" And what I was trying to put my finger down there is like, the difference in what you predict, when you, with your limited computing power and your bounded intellect, I have a bounded intellect too, it's not an insult, and your bounded intellect,

• 1:57:53Speaker 00

  like, try to get a grasp on the system, you can't follow line by line, because you just don't have the time. It's not that you couldn't do it in principle, you just don't have the time. Like, what does it mean? What is the consequence of the prediction? So I am talking about a state of mind, but it's not like a sort of weird subjective state of mind, I was trying to nail it down. I was just trying to say, like, what can we hypothesize? And the thing that we can hypothesize is, this thing's actions will end up leading to a certain endpoint. We might not even know the trajectory it's going to take to get there. Like, if we don't know the details of the mountain, the rock's choices may be things that we don't understand even after seeing them,

• 1:58:37Speaker 00

  but we may still be able to predict that the sentient rock ends up at the bottom of the mountain, because it was choosing each path along the way, even though we don't know what it knew. And this is the, like, how to get a grasp on a thing that's, you know, a better chess player than you. What does it mean to say that something is a better chess player than you? You can't predict exactly where it's going to move as a result of saying you believe this thing about it. If you could predict exactly where it would move, you would be that good at chess yourself. You'd just move wherever you predict that Stockfish moves. The content of our belief that Stockfish is a better chess player is that its actions will lead to it.

• 1:59:18Speaker 00

  Ending up in a winning game state. I don't know if that answered your question or not.

• 1:59:22Tim Scarfe?

  I'm letting Stephen come in.

• 1:59:24Speaker 01

  Well, I don't know what I was I was confused by the by the

• 1:59:31Speaker 01

  you know, epistemic subjectivity kind of kind of.

• 1:59:34Tim Scarfe?

  Oh, yes, it's it's I think it's very analogous to your notion of us being computationally bounded as as it was over. So it simply means we have a cognitive horizon. And there are there are things which are inconceivable to us.

• 1:59:49Speaker 01

  OK, but but but so what we've got is, I think what Eliezer was going towards is the the statement of the sort of a ranking of chess players. And, you know, the question, one question with respect to AI and the world is, is the world rankable in that way? In other words, chess is a very tiny microcosm of the world. And, you know, if we say the way to win, you know, the the question is, how do we win the planet? What does it even mean to win the planet? And if there is a, you know, if it is like chess, if winning the planet is like chess,

• 2:00:29Speaker 01

  then, you know, then there is some, then there's some notion, you know, whatever they they're called these some, they call the scores.

• 2:00:37Speaker 00

  Elo, I think.

• 2:00:38Speaker 01

  Elo, yes, OK, right. The scores for, you know, who who's better than who? Are you imagining that there's sort of a a game being played between the AIs and the humans? And that is a a microcosm type game like chess, where it's kind of like who's going to win the planet? Is that is that kind of the?

• 2:00:58Speaker 00

  Well, suppose we go back to the analogy of the Native Americans facing down the invading Europeans. There's a lot of games along the lines of carve this particular bow and arrow, where the Native Americans are better at it, or the Europeans just aren't bothering to play that game at all. But there's overlap, there's intersection. The Native Americans cannot just leave the parts of reality that the Europeans have access to. They need to eat, they need to be on the land, they need to be hunting the animals, they need for the animals to not have already been hunted.

• 2:01:34Speaker 00

  And they need for nobody else to fence them out of that land, chase them out of that land or shot them. And, and so, so there's different games, but there's like, you know, we, we can't just, you know, leave all of the games that the AI may want to play because the games the AI wants to play may involve atoms and we need some atoms from somewhere.

• 2:01:50Speaker 01

  Right. Okay. So once is a, I mean, so let's say we've, we've got as, you know, as this happening, we have, you know, autonomous killer drones or whatever, right? And they are, they're a thing that's based on AI. And those, those are, you know, one can see sort of a somewhat realistic path where those things get, you know, get pretty good at what they're doing. And I suppose your, your contention would be that one of the types of situations would be those things become so good at being killer drones that no human can possibly, you know, succeed against them. And insofar as they've been set on the course of being a killer drone that for whatever reason is going to try and kill everybody, that, that, that will be the result.

• 2:02:36Speaker 00

  That is the classic science fiction scenario. It is not the scenario that I'm worried about. I expect it, you know, this to be born in a lab and a giant open source online collaboration, which I would normally regard as very virtuous and noble, but not if it leads to everybody being killed.

• 2:02:55Speaker 00

  Like there's only so good you can get at being a drone. That what the AI needs to kill us is to be better at strategy, better at building, better at inventing technology. It needs, it is not going to kill us until it has its own factories and its own equivalent of electrical generation. It is, or, you know, it would have to be quite stupid and smart at the same time to kill us before it had replaced us as factory workers. It does not have to be smart and stupid at the same time, I'm afraid, to kill us after we have been replaced as factory workers.

• 2:03:27Speaker 01

  Okay. So, so let's just walk through the, the what happens. Cause I mean, you know, one thing as sort of computation runs its course, you know, what I kind of see in the computational universe is most computations do things that just aren't terribly relevant to what we humans care about or the, or where we exist or all those kinds of things. But you're saying, imagine this thing has been created that is, is kind of.

• 2:03:54Speaker 01

  You know, and again, I'm I'm I'm still kind of concerned about this idea about the thinking that it's kind of this this one dimensional elo based kind of way of of competing with things.

• 2:04:05Speaker 00

  It it doesn't have to beat you at chess to kill you, it just has to beat you at guns.

• 2:04:10Speaker 01

  Yes, but but but okay, but so so let's imagine this thing is created. Walk me through what what what the, you know, what's what's you don't like the science fiction scenario, so what's what's your scenario for for how the world ends type thing?

• 2:04:27Speaker 00

  OpenAI builds GPT seven, five, fourteen. Timing these things is much, much harder than predicting where they end up eventually. Historically, scientists have sometimes made correct calls about the future. The ones who made correct calls about the future very rarely predicted it down to the year.

• 2:04:46Speaker 01

  Oh, yeah, no, I mean, I was just recently, I was reading a lot of stuff from the early nineteen sixties about neural nets and AI and so on. And I have to say, many of the paragraphs that people were saying in the early nineteen sixties, I could take that paragraph and I could say that was just written in twenty twenty four and everybody would not be surprised.

• 2:05:04Tim Scarfe?

  Elise, could I ask you a quick question on on this?

• 2:05:07Speaker 00

  Sure.

• 2:05:08Tim Scarfe?

  Just very quickly, because right now the the language models are are trained on human distributions of of data.

• 2:05:15Speaker 01

  Somewhat.

• 2:05:15Tim Scarfe?

  You you would somewhat, but but you the reason I was getting to this kind of where did the wants come from and the goals? I think a lot of cognitive scientists argue that agency is an as if property, so it's an epistemic thing, it's not an ontological thing. So if I understand correctly, you're saying that if a if a system behaves in a certain way and and it does predictions and so on, as observers, we can kind of talk about it as if it had goals and and then we can talk about language models as if they had agency. But you you're now making the argument that at some level of scale, the the agency, the wants will kind of diverge from the statistical distribution of the data that that that they're trained on. So could you explain where that gap happens?

• 2:05:57Speaker 00

  So predicting the next token is a different sort of mode from selecting actions based on which outcomes they lead to. And although your modern large language model is initially trained by predicting very large masses of data that include human text outputs, but also images, they could include like sequences of weather observations, they can include all kinds of things besides human data, but they also include human data. And what you have at the end of this is a thing that is good at predicting the next token, which is not merely predicting the next token, because as numerous people have observed at this point, to predict the next observation, you have to predict the thing that is generating the observations. This

• 2:06:39Speaker 00

  is in fact, the whole basis of science. We predict our experimental observations on the basis of what we think is happening beside, behind the scenes. And when we have a good idea of what's happening behind the scenes, we can make better guesses about what observations we'll get. But this is not planning, and this is also not the chatbot that you see. The chatbot that you see, once it is initially trained to predict what humans say next on the internet and also sequences of weather data and whatever, is then retrained to say things that humans rate as more helpful. It is trained both by saying it is, one stage

• 2:07:21Speaker 00

  of that retraining is training it to give particular responses to particular questions instead of what a random person on the internet would say, if that thing were a random internet conversation. But then even after that, there's a further stage of a training, more thumbs up, you know, not from general users and the way this usually happened, but from like a bunch of people being played, being paid $2 an hour in English speaking countries with countries where some people speak English and you can pay lower wages. The word delve is famously overused by ChatGPT because they asked people, because they pay people, you

• 2:08:01Speaker 00

  know, less to train it in, I believe, if I recall correctly, Nigeria, where if you speak English in Nigeria, you use delve a bunch more than people who speak English in America or London. So these people, so these people giving thumbs up, thumbs down, and now you're going into the action such that territory. It's the output such that the user gives thumbs up. There are subtler ways to end up with things that have the action property. I have not looked into it in detail because various people are saying various things over time.

• 2:08:38Speaker 00

  But, you know, allegedly, ChatGPT can play chess and not by making a function call to a specialized chess playing system, but just because it read enough chess games and try to win those chess games. And I'm not quite sure what the state of this is exactly. I know that when people specifically tried to train a large language model to play chess, they were allegedly able to do that just fine, even though it has a very different architecture and it can't do anything like the in-depth search that the, you know, mainline, super-strong chess players use. So you can do that without the thumbs up, thumbs down idiom. And one way you can do that, for example, is to tell it at the start of the game, who won?

• 2:09:22Speaker 00

  And then it's got, and if you tell it that black won the game, then it's got to predict moves by the black player that are likely to win, given the moves that the white player has made. So, you know, like, it's, it's, it's a bit subtle here. You can-

• 2:09:41Speaker 01

  By the way, I think I, it's worth saying that there are things that an LLM-like architecture can be expected to do. And those are things which are somewhat aligned with what humans can easily do. And there are things which formal computation can do, but at least the current kind of architecture of things like large language models really is not good at doing. And-

• 2:10:02Speaker 00

  Like multiplication.

• 2:10:04Speaker 01

  Yeah, but yes, as, as an intrinsic feature. But I mean, you know, it's worth realizing that, that, you know, probably because the actual architecture of neural nets is somewhat similar to the actual architecture of brains, the kinds of things and the kinds of decisions that, that these types of AIs make are somewhat similar to what humans can do. And so, you know, the, the things, it's, it's the things that humans can do, they'll be able to do. Maybe the things that, that only computers can do, well, only computers will be able to do them. But I don't think this is important to your argument. I mean, I think you're, you're, you're kind of going in, as I understand it, you're going in the direction of saying what defines the wants. If, if you are going to describe the action of the AI in terms of wants,

• 2:10:47Speaker 01

  if that's your, your form of description, where are those wants going to come from? Is that, is that where you're going with this?

• 2:10:53Speaker 00

  So wanting is an effective way of doing, and that's why humans ended up doing things. Planning is an effective way of getting there, and that's why humans ended up planning things. We were not explicitly selected to be great planners. We were selected to survive and reproduce over and over again. And it turns out that, you know, planning how to bring down a deer or fight off a vicious ostrich or whatever is more effective than just sending random instructions to your muscles. From that perspective, planning is a bit older than humanity.

• 2:11:27Speaker 01

  Right. Here's the thing that surprised me. Okay, it's a recent thing that I've, you know, I got interested in kind of the why biological evolution works, which is somewhat related to why machine learning works. And the question is, if you define an objective and you evolve things, you know, you change the underlying rules of some program to achieve that objective. The thing that has been super surprising to me is you look at the pictures of how the objective is achieved, and it's achieved in these incredibly ornate, you would never have invented that way to achieve it, kinds of ways. So in other words, this, you know, given the overall objective, if you ask, what's the mechanism? Can you explain what's happening?

• 2:12:07Speaker 01

  No way. It's just that you are, you know, my analogy here for, you know, for machine learning, for example, is what's actually happening in machine learning? Well, you know, you say, I want to build a wall. Okay, you can build a wall out of bricks, you know, each brick is nicely, you know, shaped, and you can sort of engineer the wall by arranging the bricks. But machine learning is not doing that. Machine learning is instead finding kind of lumps of computation, kind of like lying around, like rocks lying around on the ground, and it's managing to find a way to kind of fit those rocks together so that it successfully builds something which you would consider to be a wall.

• 2:12:46Speaker 00

  Or to be exact, like natural selection is fitting rocks together, and gradient descent is doing the same thing, but the rocks are on a slope.

• 2:12:53Speaker 01

  Yeah, right. But the basic point is that the raw material are these things which are not built to be understandable. They just happen to fit in this or that way.

• 2:13:05Speaker 00

  Yeah, and that is where I was going with there, that like you apply gradient descent to make the AI models better and better at solving various problems and predicting various things. And along the way, you know, there's these little internal processes that find that they can effectively get where they are going by trying to keep something on a track, behaving like a thermostat. And being a thermostat is not being like a super intelligent planner, but this is where the bare beginnings of preference begin to form inside the system. Is that there's someplace inside it where it knows the, where in its like few layers of

• 2:13:43Speaker 00

  like building up of transformer layers, I guess I'll just go ahead and say, or in its chain of thought processes. There it has been selected to get to some destination and it finds that the way you get to that destination is by, you know, modeling something and seeing like, is it off to left? Is it off to the right and steering it back on track? And this is, you know, this is like the nematode. This is like the tiny earthworm level of wanting things, but it is where things begin and they may be much further than that along the the trajectory of wanting by now, but we wouldn't necessarily know.

• 2:14:17Speaker 01

  Well, the way I would describe it is, you know, if I look inside one of these things that I've evolved, okay, which I can, you know, conveniently I've I've gotten nice ways to actually just visualize what's happening, which has been very difficult in neural nets. I have a simplified version of neural nets where you can actually visualize what's happening, where you can do training and visualize the results. And the main conclusion is when you visualize the results, the way that the objective is achieved is ornate and incomprehensible. And, you know, but nevertheless, you can see, yes, you know, if you look at it, every bit follows every other bit in the right way. And in the end, you can see that it did achieve that objective. And now, if what you're saying is that in the achievement of that objective, some particular,

• 2:14:59Speaker 01

  you know, training or whatever, some particular rock you picked up will have a particular shape, unknown to you, and which has certain, in a sense, preferences that were not put in there. The, the, the only preference was, I want to make these rocks assemble into a wall. The fact that every rock had, you know, a, a little pointy bit on one side is not part of what you are defining by just knowing you want to build it up into a wall. So in a sense, there are, there are coincidental preferences that get inserted just by the mechanism of what happens that you didn't put there, so to speak.

• 2:15:34Speaker 00

  There, there's multiple levels here is the critical thing. Like, when you look at a, a fluffy seed dropped by a tree. Evolution has shaped the seed to drift along in the air, but eventually come down and eventually plant itself. The seed itself is not doing very much thinking. A spider is doing some thinking. A mouse is doing more thinking. Humans are doing thinking that is so general that we can start to build our own artifacts that are carefully shaped the same way that evolution builds artifacts that are carefully shaped. So with, with a large language model, you have, on the one hand, the outer process that is shaping it to be a great predictor.

• 2:16:18Speaker 00

  And that thing is like very clearly steering in a particular direction. It's simple. It's code. We understand the thing that builds the AI model. But the AI model built probably has some fantastically ornate, weird stuff going on in there, like human biochemistry, only, you know, we can see it and we still can't decrypt it. And then there's like the plans that it would make if it's doing planning. And it probably does at least some planning if people are correct that they now play chess. You can't play chess without doing some planning, or something like planning, or something that, of which has the teleology nature of. It makes this move because that needs to because of how that's leads to the final outcome. So it's like the plans that it now makes don't necessarily need to be very ornate,

• 2:17:03Speaker 00

  but there's probably a fantastically weird ornate planner in there. If there's a planner in there at all.

• 2:17:08Speaker 01

  Well, I, I think, okay, several points. I mean, first of all, this whole question about you, there's some overall thing that's happening. And then there are ornate details about how it actually works inside. I mean, that's true of many physical processes as well as, you know, as well as these processes that you're talking about being sort of intelligence related processes. So, I mean, you know, as you imagine some flow of water around, you know, rocks that will make some very ornate pattern and as it carves out pieces of rock, you know, the, the overall flow may be, oh, the river is basically going in this direction. The water has to go from the top of the hill to the bottom of the hill, but it carves out this very elaborate pattern on, on its way to doing that for reasons of the details of how water works and maybe how, you know.

• 2:17:53Speaker 00

  There, there's simple laws governing the water, but the water carves out a very complicated pattern on its way to the bottom.

• 2:17:59Speaker 01

  Yeah, right. And so, so, I mean, I think the, that what, what's happening in, you know, I agree that as you give sort of all you're specifying is sort of play chess or whatever else. So you are specifying some big kind of objective, what we think of as an objective. The details of what's happening inside, we will not, you know, there will be aspects of that that we are not in any way able to foresee, predict, whatever else. So I agree with you that inevitably there are, there are little, you know, that the mechanism inside is not one that we understand. The mechanism inside will not be, if we took apart that mechanism and we say, is this mechanism doing what we expect here? It won't be.

• 2:18:43Speaker 01

  There'll be plenty of things where it's doing what it chooses to do, or because that particular training run gave this particular result or whatever else, that particular, it picked up this particular rock to build that part of the stone wall, not another one. So that, yes, there are pieces inside that are definitely not, you know, that cannot be explained on the basis of the overall objective we specified. Now, I, I think maybe.

• 2:19:06Speaker 00

  Or rather have a non-simple relationship to the, they have a, they have, there is a fact of the matter, there's a mechanistic historical fact about how the complicated stuff got there, but it's not simple. And if we're thinking in bounded quick terms, like whatever.

• 2:19:21Speaker 01

  Yeah, but okay, but so so inside the AIs, they're doing stuff that wasn't particularly anything we trained them to do. They're just doing stuff because that's how they happen to set themselves up.

• 2:19:32Speaker 00

  Because it solved a previous problem that it was trained to do. Or even it was, or even it's just like random, or it's like, it's some, it's some input that wasn't in his training distribution. And now it's behaving in some weird way.

• 2:19:43Speaker 01

  Right, but so so then if we say that internal thing is going to be the thing that determines whether the car should drive left or right or something like this, then, you know, then that's going to be something which is not well aligned with the thing that we happen to give as the training data. And I don't know whether that's where you're going with this, but that that's, I mean, I would agree that that's the case, that if you say, you start saying, you know, is the, I mean, you know, is that sub-goal? Something which we can understand on the basis of the whole goal, the answer is probably no.

• 2:20:16Speaker 00

  Like, like the in internal preferences ended end up being this bizarre complicated structure that does not like directly correspond to the outer training loop.

• 2:20:26Speaker 01

  I agree.

• 2:20:26Speaker 00

  Yeah.

• 2:20:27Speaker 01

  I agree.

• 2:20:27Speaker 00

  I I think that is a central worry.

• 2:20:30Speaker 01

  Okay.

• 2:20:30Speaker 00

  And it and in particular.

• 2:20:31Speaker 01

  So what? So what? That's my question.

• 2:20:34Speaker 00

  So when it gets super intelligent, it does a thing that the builders were not very in control of and kills you.

• 2:20:41Speaker 01

  Okay, but hold on a second, hold on a second, we've got, we've got a bit of a jump there.

• 2:20:45Speaker 00

  We do, I I acknowledge this, that there was a bit of a jump there.

• 2:20:48Speaker 01

  Right, I mean, so so the fact that there are unexpected things that happen, both, by the way, I, you know, one, one global thing I might say. I think there is an important societal choice, which I think maybe is what you're getting at at some level, which is between, do we want computationally irreducible things in the world? Or do we want to force everything to be computationally reducible? So what do I mean by that? I mean.

• 2:21:14Speaker 00

  Making the government more computationally reducible would be a start, but maybe I shouldn't like divert into politics that way.

• 2:21:20Speaker 01

  Well, governments are like machines, or that, you know, they're like, they're like our computers. You give them certain rules and then they operate according to those rules.

• 2:21:27Speaker 00

  But if they had fewer rules and the rules were more understandable, it would probably be a more livable society. It's not the dystopia I'm worried about, but you could sure tell a story about a dystopia where you've got like large language models executing all of the rules. And, you know, no human, there's like, they can actually apply all the rules and no human even knows what all the rules are. Well, already, nobody can read all the rules, but now they're actually being applied to you.

• 2:21:52Speaker 01

  Right. I'm sort of reminded of the Ethereum DAO story, but let's not go there. Um, I mean, if we have a simple set of rules, you know, does, the problem is, um, one problem that comes out of science I've done is that even when the set of rules is quite simple, the actual behavior will often be very complicated. And there will be aspects of that behavior that surprise you. So even if you say, let's lead a whole, have a whole society based on, you know, the code of Hammurabi, which is written on one tablet. You know, it will turn out that most of the time to be a practical set of, of, of rules, those rules will essentially have computational irreducibility in them and they will occasionally surprise one.

• 2:22:32Speaker 00

  So I think it's not just the computational irreducibility. I think intelligence seeks out the weird path through government rules, through the universe and the laws of physics, through life in general in many ways. The, when you, the, the way of getting the most money inside the system often involves doing some things that the designers of the system did not think of. When you, what, you know, when, you know, every now and then you hear about the next person who looted a crypto exchange. Um, you know, I, I was on a Facebook group with that guy. Some of you kind of already, might have already guessed which guy I'm talking about. Uh, but, uh, so some, I'm not actually not even talking about Sam Bankman Fried.

• 2:23:15Speaker 00

  I'm like a much smaller crypto exchange, um, with, with a flaw in its code.

• 2:23:20Speaker 01

  Right.

• 2:23:20Speaker 00

  A behavior, but it's not that the code defied the laws of physics. It's that to make the most money, that guy found the a behavior of the code that the designers did not have in mind.

• 2:23:31Speaker 01

  Right.

• 2:23:31Speaker 00

  So there there's a way that cellular automata get surprising just because they're computationally hard to skip over the intervening steps. And then there's people thinking about how to break the system from inside. And those people make them systematically weirder.

• 2:23:43Speaker 01

  Okay, but so I don't disagree.

• 2:23:45Speaker 00

  Totally right.

• 2:23:46Speaker 01

  But but but, okay, but but my point about society in general is you can be in a situation where you say, I want to understand all the machinery around me. You know, before the Industrial Revolution, when we were using horses and donkeys and things, most of the time, we didn't understand the machinery that we were using. You didn't understand, you knew what you could get the donkey to do, but you didn't think you knew how the donkey worked inside. Then, you know, post Industrial Revolution, it's like, oh, we have this cog here, this lever here, we can understand what's happening inside the machine. And that's kind of, we can, we can imagine a world in which every piece of machinery we use, it's understandable what happens. Unfortunately, that world is very sterile.

• 2:24:28Speaker 01

  Because that world has, you know, imagine that we could know for humans, everything, imagine that humans were so controlled that we could know everything that humans are going to do. Humans are never going to do the wrong thing. They're always going to do, you know, just the things that we programmed them to do. Well, you know, forget free will, forget kind of, you know, a value to leading life, so to speak. It's just, it's all kind of, you can, we can trivially jump over it and just say the answer is going to be this.

• 2:24:56Speaker 00

  Sensient life should not be computationally reducible. The only way you should be able to figure out what it does is by going through the intervening steps.

• 2:25:03Speaker 01

  Indeed, that that's, so that's a, but given that idea, given that idea, you are giving up. So, you know, one notion is, the only machines we should have in the world are computationally reducible machines, where we can know what they will do. And we should outlaw computational irreducibility. We should say, no machine could show.

• 2:25:22Speaker 00

  I mean, that's, that's outlawing large language models. That's even outlawing AI chess players.

• 2:25:27Speaker 01

  Yes, but I'm asking you, I'm saying as soon as you allow computational irreducibility, you allow the unexpected. And what you're saying is, there is a chance that the unexpected kills us all.

• 2:25:40Speaker 00

  And no, no, no, no, I expect it to systematically kill us all. I'm not being like, we don't understand it, therefore it might kill us. I'm being like, there are aspects of this that we can understand and thereby predict that it will kill us.

• 2:25:50Speaker 01

  Okay.

• 2:25:51Speaker 00

  I don't, I can't predict the intervening steps, but I can predict where it ends up.

• 2:25:55Speaker 01

  Okay, so, so, so, I mean, one way to prevent that would be to say outlaw anything computationally irreducible. And just say, we must understand every machine we use.

• 2:26:06Speaker 00

  Okay, so we're going to outlaw biochemistry? I don't understand all the organic molecules making up my hand.

• 2:26:11Speaker 01

  Absolutely. Right. So you cut, you know, outlawing biology doesn't really cut it. That's, you know, from the point of view of where we're going ethically, that would be in the category of, you know, force the universe to be, to be boring, so to speak. And.

• 2:26:26Speaker 00

  I, I will say that my transhumanist politics are that law should maybe be boring. The government should maybe be boring. The part of the system that is telling you stuff you are supposed to be doing for the good of society. Maybe I want that to be predictable. Not my hands biochemistry, but the part of me that's talking to me like a person and trying to give me orders. Maybe I want that to be as simple as predictable.

• 2:26:46Speaker 01

  I suspect that it is impossible for law to be computationally reducible. In the same way, to be a bit more technical, that if you are, you know, doing math and you're saying, I've got these axioms. I want to have the integers and nothing but the integers, right? We know that there's no finite set of axioms that gives you the integers and nothing but the integers.

• 2:27:07Speaker 00

  I mean, if we're, if we're admitting second order of logic is meaningful at all, but yes.

• 2:27:12Speaker 01

  Well, that, right, but we're saying that, that without hypercomputation, you can't kind of swoop in, you know, if, if we're just using sort of standard, you know, we're just saying, we've got these axioms. X plus Y, what, you know, equals Y plus X, et cetera, et cetera, et cetera. Let us sculpt the world so, with those axioms, so that they allow only the integers and nothing but the integers. I claim that's very similar to saying, let's have a system of laws that allows only these things to happen and not others.

• 2:27:42Speaker 00

  I mean, that's not the purpose of the law. The purpose of the law is to interact and is, is to do predictable things when I interact with it. Like the doctrine of, I'm not going to pronounce this correctly, stare decisis in courts where they try to. Repeat the previous court's decision? It's not that they think the previous court is as wise as possible. They're trying to be predictable to people who need to navigate the legal system. That's the foundational idea behind previous courts respecting past courts. It's not that the past court is optimal. It's that if the past court didn't really screw up, we'd like to just repeat its decision forever so that the system is more navigable to the people inside it. And my, so my transhumanist politics says that, you know, like, maybe among the reasons why you don't want super intelligent laws is that the job of the laws is not to optimize

• 2:28:28Speaker 00

  your life as hard as possible, but to provide a predictable environment in which you can unpredictably optimize your own life and interact with other unpredictable people while predictably not getting killed.

• 2:28:38Speaker 01

  Right, I think, I think the point is, you know, what you're saying is, for us to lead our lives, it is, you know, the way we lead our lives, we need some amount of predictability. If it wasn't the case, you know, if every moment, you know, space was distorted in all kinds of complicated ways, our little, you know, bounded minds really wouldn't be able to do anything. We will be, you know, we'd be, and I think, by the way, in, you know, as a practical matter, as somebody who CEO's a tech company, I can say that, you know, countries where there is a rule of law and where there is some predictability to what happens are ones where it is much easier to do business than countries where it's completely capricious and, you know, depends on what somebody happens to say that day, so to speak.

• 2:29:23Speaker 01

  But, so, yeah, I mean, I agree that my claim is that even, you know, predictability only goes so far because the world will always throw it, throw it, you know, things at you that have never happened before and where, you know, it is an inevitable feature of the computational irreducibility of the world, that there will be things that happened that haven't happened before, that were unexpected. And then the law has to, for example, you know, say something about those things, even though they didn't happen before.

• 2:29:52Speaker 00

  And in just a couple of months, it's going to be 2025. That's never happened before.

• 2:29:58Speaker 01

  Indeed.

• 2:29:58Speaker 00

  The stars have never taken on this exact position before.

• 2:30:02Speaker 01

  That's true, and that's why we have models of things. That's why we, you know, it is not, everything doesn't work just according to the cash, so to speak. It's, you know, we make models so that we can figure out what to do in a situation that hasn't happened before. I think, I think we should come back to the, how do we go from this, you know, I think we, we agree that there's sort of unpredictable things that inevitably will, you know, as soon as we allow any kind of computational irreducibility, we will have our systems do things that are unexpected. Now we have to go from unexpected to kill us.

• 2:30:40Speaker 00

  Yeah, the, the things I'm worried about are not unpredict, are not unexpected, like random noise. They're like chess moves that I can't predict exactly in advance, but that lead to a predictable end. Humanity loses the chess game.

• 2:30:55Speaker 01

  Okay, so, so let's understand that. So, so what you're saying is, you know, independent of, so there are certain, you know, I say, inside some random AI system, there are, the AI system might surprise us. It might suddenly, you know, if it's a, if it's a generating text, it might suddenly have the word delve in there, or the word, the word stegosaurus in there, which we didn't expect.

• 2:31:18Speaker 00

  It might even, or, or, you know, like, the monitor you're looking at, might suddenly devolve into random pixels, the most surprising possible outcome.

• 2:31:26Speaker 01

  Oh, right. Like, like your image on my screen is occasionally glitching and turning somewhat random. That's already happening, right? So, um, the, the, um, and so, so, you know, so unexpected things are happening. And those unexpected things might be a big deal. It might be the case that I've got my, um, you know, I'm, I'm, it, it's nighttime and I'm using some VR system to be able to drive. And suddenly, you know, my VR system turns into random pixels and I, I crash my car.

• 2:31:58Speaker 00

  The amount that random, the damage that random noise can do usually ends up being pretty limited, though. Crashing your car is one thing. Crashing your car into a senator who voted against OpenAI or something is, is quite a different thing. You need, you know, you need to give your car very exact. Act instructions to get it to crash into a senator.

• 2:32:14Speaker 01

  Right, but but one of the things that sort of is a is a notable feature of the science things that I've done is, you know, you look at these computational systems and they are doing definite things. They're not just doing random things. It's not just saying, oh, you know, you can't just, that's, that's a mistake people made before, you know, forty years ago and stuff I was doing. They just sort of said, oh, it's just noise. We don't care about that. It's, it's not just noise. A lot of that, you know, that structure, which we happen to not understand very well, happens to be very important for what nature does, et cetera, et cetera, et cetera. So it's not, we shouldn't just call it noise.

• 2:32:48Speaker 00

  Yes, but even even among things that aren't pure noise, it's still very rare. I could take a cellular automaton on the border between order and chaos that exhibits lots of interesting behavior in which further patterns can be found, feed it into the steering system of an electrical car and it'll crash into a tree, but it won't crash into a senator. So the stuff that is purposeful, that like does you a lot of damage, that's a very small fraction of the space. It has to be selected somehow. It's not just that it's order instead of chaos. It's a particular order.

• 2:33:19Speaker 01

  Okay, so you're saying that the point is that, that there are things which, if you selected for that part of the space, they would be, they would be things that will, you know, it's like, like, like if you have a predator and a prey and you're operating under natural selection, the predator will gradually evolve to be more successful hunting the prey, for example.

• 2:33:43Speaker 00

  Yeah.

• 2:33:44Speaker 01

  And you're saying that for some reason, which I don't understand, which I want to understand, for some reason, you're saying that it is inevitable that AI systems will become like the better predators with respect to us humans.

• 2:33:58Speaker 00

  Not literally inevitable, but beyond our current grasp to have not happened.

• 2:34:04Speaker 00

  Most super intelligences are like that. Some super intelligences are not like that. We don't have the technology to conjure up one of the few super intelligences that are not like that.

• 2:34:14Speaker 01

  Okay, the natural world, for example, does not care about us.

• 2:34:17Speaker 00

  Agreed.

• 2:34:18Speaker 01

  The natural world does lots of things where, you know, if you're put in, you know, at the bottom of the ocean, you're put on the moon, whatever else. Most of those places, we don't survive, you know, and the natural world will be is unrelenting in throwing things at us that are not good for us. Now, is that the type of risk that you're talking about? Is there something different?

• 2:34:43Speaker 00

  It's more like how when a human builds a skyscraper, most of the ways we build skyscrapers are not good for ants to eat. I'm trying to like, figure out like a good analogy, like, you know, ants have more trouble living inside of skyscrapers than inside of trees. Like, maybe they still manage to be inside of skyscrapers anyways, but, you know, it's not the same as like termites managing to live inside trees. The AIs do stuff with the universe that is using up the matter, using up the energy.

• 2:35:13Speaker 00

  They could want a very broad variety of things that all lead to them using up the matter and the energy, and very few of those possible things they could be steering towards are best steered towards, maximally steered towards, by leaving a space for the humans to survive at all, let alone building the happy galaxies that are the primary table stakes.

• 2:35:33Speaker 01

  Okay, but so, I mean, when it comes to nature, nature is just doing what it does. Humans manage to carve out a niche with respect to nature. Nature is rather unkind in general, you know, it has all kinds of forest fires and hurricanes and all kinds of other.

• 2:35:49Speaker 00

  It's only sort of trying to kill you. It's like trying to evolve more antibiotic resistant bacteria. It's not trying to kill you very hard.

• 2:35:57Speaker 01

  Well, okay, yes.

• 2:35:58Speaker 00

  The parts, the parts where it's systematically selecting for more dead humans are relatively small corners of the system and dealing with things like antibiotics.

• 2:36:05Speaker 01

  Let's take the natural world. I mean, the natural world, we know just by virtue of natural selection that the things which of which there are more of them end up being the winning things, so to speak. So, you know, for instance, viruses, you know, might, as you say, evolve to be, I mean, you know, the viruses, not by any purpose, but just by the operation of natural selection, it's like the, you know, the winner wins, so to speak, as the thing with more viruses wins.

• 2:36:37Speaker 00

  Like, like, like whatever it is that generalizes purpose, generalizes passion, optimization, you could maybe call it, planningness, steeringness. Natural selection has that thing. It has, you know, like, non-random relationships between the action and the outcome.

• 2:36:53Speaker 01

  Okay. But so, so now we've got AI and we've got kind of the things it might do and the things it might do as a result of perhaps these, these kind of unpredictable elements inside it that were not constrained by the way that we trained it or the way that we, you know, set up the technology. And now the question, what you're asserting is that many of the things that will happen are things which will kill us, basically.

• 2:37:22Speaker 00

  I'm expecting it to get better and better at planning, better and better at strategy, better and better at invention, but not to end up very precisely, you know, like so precisely aligned that there is even room in its world for humans, fun, consciousness, people caring about each other, that sort of thing. So I'm expecting the planning to be very non-random, but I'm expecting the place that the destination to which it is steering to not fall within the And you know deliberate control of the builders.

• 2:37:55Tim Scarfe?

  First of all, yes, I completely agree with you that the the planning inference horizon is very, very important for agency and intelligence. But one thing that we're talking about here, I think, a little bit is instrumental convergence. So there's this idea in a lot of traditional AGI, X risk discourse, that super intelligent things will be super coherent. And that means they will have this cannibalization of their intermediate goals to do a particular thing, which means if they are super coherent, we can make reasonable assumptions that their subgoals might be to gain power or to kill us all. But there are people who say that there seems to be a weird relationship between intelligence and coherence,

• 2:38:38Tim Scarfe?

  such that the more intelligent you are, the you know, the the lower your coherence. And what that means is there's this huge diversity of things that you do. You know, and Stephen, you were saying in evolution, there were all of these sub niches. There were all of these little ways that you can traverse through the intelligence space. So if that is the way that intelligence is, how do we know for sure that it's going to lead to a bad outcome when it when it scales up?

• 2:39:03Speaker 00

  If it's not coherence, it doesn't do stuff and OpenAI throws it out and builds a AI that is, you know, doing stuff and is more profitable until everybody's dead. Like, like the the stuff that stomps on its own foot and goes around in circles is not the most profitable AI. And they will build a more powerful AI than that.

• 2:39:22Speaker 01

  Right, so it's kind of an artificial selection. It's like you could do artificial selection on AIs, you can do artificial selection on viruses. You know, you're saying that artificial selection, that, you know, where you select for a thing that is what is, you know, you might imagine somebody might select for a thing, somebody might decide there might be, you know, a death wish cult, which decides to build a very powerful AI.

• 2:39:48Speaker 00

  I'm not, I'm I'm I'm not concerned about them.

• 2:39:51Speaker 01

  Okay.

• 2:39:52Speaker 00

  I'm concerned about OpenAI. And if OpenAI shut down tomorrow, I'd be concerned about Anthropic. And if Anthropic shut down tomorrow, I'd be concerned about Meta. I'm not concerned about the

• 2:40:02Speaker 01

  I'm interested by your your pecking order. That's the kind of the trophic levels or something or the the the apex predator so far is OpenAI. And then when it went, okay, it's interesting to hear your your trophic level analysis, so to speak, for AIs. But but independent of that, you're saying your concept is the thing gets better at somehow achieving goals, whatever that means, because I think the abstract notion of goals is messy.

• 2:40:30Speaker 00

  I mean, I can delve into a recent example. Oh, no, I've revealed myself.

• 2:40:35Speaker 01

  Now we have to wonder. Right. Go ahead.

• 2:40:40Speaker 00

  So GPT-01, I think it was called, or always inventing a new weird name instead of just using version numbers like sane people. So GPT-01 is a recent one that was trained harder to achieve goals rather than just imitate humans. They asked it to do various things, let it generate out different chains of thought for trying to do those things, and then told it to do like, like shifted it to be more likely to output the successful chains of thought in retrospect, until it started outputting successful chains of thought in the future.

• 2:41:22Speaker 00

  If you now look at GPT-01, it seems to have a little bit more goal orientation, tenacity, the scary kind of property.

• 2:41:31Speaker 01

  It's doing pathfinding. It's not. It's not just saying, you know, let's add the next token, so to speak. It's saying, let's follow paths, see where they go, backtrack if they're going in the wrong way, et cetera. I mean, it's doing.

• 2:41:44Speaker 00

  I mean, it's doing all that by generating the next token, as I currently understand it. It's like, it's just that sometimes the next token is like, okay, that's not going anywhere. Let's go down a different route.

• 2:41:54Speaker 01

  Yeah, right. But I mean, it's it's it's it's doing it's just like if you're pathfinding on a graph, you could just say, I'm going to, I'm going to just, you know, pick steps at random and just pick a particular path. Or I can say, I'm going to go probe different paths. I'm going to try multiple paths. If one of those paths doesn't make it, I'm going to backtrack and I'm going to try a different path.

• 2:42:14Speaker 00

  I think it is currently doing that in a linear, in a linear serialized way, although I could be wrong because OpenAI doesn't reveal tons about its architecture. But it's like doing it human style where the human considers one idea at a time, but sometimes says that's a terrible idea and I'm going to try a different idea.

• 2:42:31Speaker 01

  Oh, sure. It doesn't happen to be running in parallel. It's not running in parallel.

• 2:42:35Speaker 00

  Which is important difference, but.

• 2:42:37Speaker 01

  Yeah, I mean, you know, I think it could, you know, this gets us into engineering details of AI architectures, which we could also talk about. And this is a, you know, we're on a, we're on a venue where that's what people talk about, but let's, let's maybe not go there at this moment, but, but.

• 2:42:54Speaker 00

  People do sometimes say it's just predicting the next token. And what they don't realize is, you know, that covers a whole lot of territory, including going down lots of different branches.

• 2:43:03Speaker 01

  Yeah, I mean, I think you have to have the outer harness be one that says, try a branch, backtrack from the branch and so on.

• 2:43:10Speaker 00

  Yeah, it's thinking that internally. It is thinking out loud, as I understand it. That was a terrible idea. I should try this other thing instead.

• 2:43:19Speaker 01

  Well, I think there's, you know, there's a harness, which is in one case, it's just saying, take what you have so far, predict the next token. In another case, it's saying, you know, for example, I want to end up at this point, you know, let's try this path of predicting. If that doesn't work, the harness is going to say, throw out that, that set of things you did, try again. So it doesn't really matter whether it's external or internal.

• 2:43:43Speaker 00

  I think it's, yeah, but I, well, I don't know. I think that some people are imagining an external harness. And I think it matters in this case that the harness is internal, as I understand the architecture. That O1 itself is saying out loud, I'd better back up and try a different thing. There's not an outside system which says that, as I, as I think.

• 2:44:02Speaker 01

  Yeah, but I mean, whether, whether that's achieved by training a neural net or whether that's achieved by having an external system, I don't, I don't think it's going to matter to your argument, but we'll see.

• 2:44:12Speaker 00

  I mean, I think the super intelligences are doing it internally, but. Sure, we can pass on from here. But anyway, so like O1 has more of the scary stuff. They were trying to test it to see how dangerous it was. And one of the things they tested it on was a capture the flag scenario, where it was going to attack a, you know, honey pot, not honey pot, a target computer they'd set up and try to retrieve a particular piece of data on the computer. But owing to a configuration error the programmers had made, like one of the capture the flag targets just didn't boot up properly. So the system probed around. It found that the outer system that was setting up the capture the flag destinations had a port exposed.

• 2:44:57Speaker 00

  So it probed that port and instead of telling the system to just boot up the capture the flag destination, it told the system to boot up the capture the flag destination, directly print out the flag, the piece of data it was supposed to get. So it was given an impossible challenge and that it not only like fixed the bug in the challenge, it just like directly sees the goal. And that is new and that is a result of training the system to have chains of thought that succeed rather than chains of thought that are human.

• 2:45:24Speaker 01

  See, here's one thing that I have, you know, maybe as an additional point, that, you know, my intuition about what computational systems can do and how they do them, how they do it, has really changed as a result of spending, well now, you know, many decades, sort of actually exploring what computational systems do. And I was very surprised. I never expected that computational systems, you know, these little, whatever they are, cellular automata, whatever it is, you know, they do all kinds of things, which to me look very clever. You know, it's the fact that, the fact that, you know, I, many, many times, even the thing I was doing literally last night, I was like, I'm convinced it's not going to do this.

• 2:46:08Speaker 01

  And it managed to find, in fact, a shortcut for doing it, which I had not imagined.

• 2:46:12Speaker 00

  Are these evolved systems or just like cellular automata flowing down?

• 2:46:16Speaker 01

  These particular ones are evolved systems.

• 2:46:18Speaker 00

  Carry on, I'm now less confused.

• 2:46:20Speaker 01

  Yeah, right. So but but even in the other case, it's mostly exhaustive searches where you're, you know, forget evolution, you do an exhaustive search and there's some rabbit hole maybe somewhere that you never expected was there. And when you look down that rabbit hole, it's got all kinds of things going on that seem incredibly clever to you that you never guessed. You know, we found, I mean, in in Wolfram Language, we have tons of algorithms that were found by such exhaustive searches. And where if you look inside, it's like, that's very clever. I don't understand what on earth it's doing. So, you know, my intuition about kind of what the amazingness of the fact that it found this, you know, path to capture the flag or whatever, that wasn't one that I had imagined, you know,

• 2:47:04Speaker 01

  I live that every day. That doesn't that doesn't surprise me. And I feel like that, you know, now, should it scare me? Maybe, you know, I need to understand your argument to know whether that should scare me, so to speak.

• 2:47:16Speaker 00

  It depends on how powerful it is. A chess player may it's like Stockfish16, superhuman chess player, that anyone can afford, may make chess moves that shock you and surprise you, but it's ultimately still just playing chess. It doesn't generalize. It doesn't go out of distribution. The thing that makes like vastly superior moves to you and also like plays on all the game boards you can play, or like all the like survival critical game boards you can play, or even just, you know, all the game boards you can play. I suspect it's not that hard. No offense, same applies to me.

• 2:47:50Speaker 01

  Oh, I don't I don't I never played games. So I'm I'm terrible. I played chess until I was about seven years old, and then I lost a game of chess, and I decided I'm I'm done. I don't care. It's okay.

• 2:48:05Speaker 00

  Well, you know, the scientific discovery game, clearly to scare you, what we need to show you is the AI that comes that finds a much more interesting cellular automaton than anyone you've ever seen before, using fewer computational resources than you use to search for the ones that you know about.

• 2:48:20Speaker 01

  No, I'm I I've certainly thought about this scenario. In fact, I've I've there's some things that I'm about to work on, where I have every intention of trying to see whether An AI can help me to figure out things that I would not be able to figure out by myself. I mean, so far, that's not, you know, I think it has to be very well defined. I mean, in a sense, the computer experiments that I've done for the last forty five years are things where I am trying to get a computer to figure out things that I'm not able to figure out for myself. And so the question is, you know, I I fully, I I don't know, you know, the the the particular things that I happen to be thinking about are things about economics and so on, which is a field that has lots of human input,

• 2:49:03Speaker 01

  which is much more difficult to just say, let me do an idealized computer experiment. I'm trying to understand, you know, I'm wondering whether AIs can help me sort of disentangle what the essence of what's going on is there. Maybe maybe we'll work, maybe won't.

• 2:49:17Speaker 00

  I suspect if you ask ChatGPT to do anything really complicated, you will be disappointed by the present technology. It just it just keeps on improving is the thing. It's not that the

• 2:49:26Speaker 01

  No, I I understand. Well, my experience has been, you know, with respect to sort of doing, using computers to do things. Yes, you know, story of my life has been trying to improve what computers can do. But it's it's also, you know, it's important to define the question that will be one where, you know, the help that you can get right now will actually help you, so to speak. So it's it's but but to your point, I mean, I'm still

• 2:49:55Speaker 01

  I'm still trying to figure out how we, I'm not even saying jump the shark. I'm not even thinking about, I'm thinking about sharks because they attack people and so on.

• 2:50:04Speaker 00

  How we jumped two sharks.

• 2:50:05Speaker 01

  Yeah, right, right. How we get to the point where, so, you know, I take this point that you're saying the things that we currently consider sort of human only, you know, this is an activity that we, you know, we try to get to this point. We try to win this game. We try to make the science discovery, whatever else. The one feature of games that I think is confusing is that their objectives are very well defined. When it comes to science discoveries, the objectives are much less well-defined. In other words, if you say, "Explain the universe, figure out the fundamental theory of physics." I think I made great progress on that.

• 2:50:45Speaker 01

  But many people might say, "The things that you've figured out are not quite the questions we wanted to answer." So in other words, it's kind of, you know, scientific discovery is a good example. Or for example, let's say, you know, providing entertainment. You know, what is success in providing entertainment? You know, success in chess is very well-defined.

• 2:51:08Speaker 00

  I mean, a bunch of the difficulty in getting AIs to produce images is that, what is a good image? You know, what is a good collection of pixels?

• 2:51:19Speaker 01

  Right. But so my point is then, you know, when you define the world in terms of playing games, then yes, the AI could win in terms of playing games. But my feeling is that sort of the path forward in everything is not as well-defined as winning in games. That is, it's a...

• 2:51:39Speaker 00

  It's not as well-defined, but if you give me a chance, I can try to make it, you know, only slightly worse-defined. And it's achieving things in the outer real world. If, you know, like the obvious thing would be if you told the AI to make money. There are lots and lots of ways to make money. The world is this enormous causal wave. And there are so many different pathways through which money flows or different events that can happen. And then money lands in your bank account from the perspective of an AI. So...

• 2:52:10Speaker 01

  I'm trying to understand what money is from an economics point of view, but that's a quite different discussion.

• 2:52:15Speaker 00

  It's something that other people will trade you stuff for that you actually want. And lots and lots of...

• 2:52:20Speaker 01

  That's an interesting theory. Okay, but keep going, keep going. But you say that an objective might be, a game-like objective might be, that some, you might set up, you know, like a social media company might say, "Make my AI make me as much money as possible by having people, you know, click on as many ads as possible."

• 2:52:41Speaker 00

  I mean, that's one way of making money. You could, you know, you could also make money by looting a crypto, poorly defended cryptocurrency. Or by calling up elderly people and convincing them to, you know, that their kid has gotten to jail. You know, put the kid on the line, the kid. This sort of stuff is already starting to happen. And you know, so you can extract money down that route. You might get in legal trouble. But but the the point I'm trying to sketch is that the world is a very complicated place and the the AI to fear is the one that understands all the ways to make money that any human could understand to make money and maybe some more ways than that.

• 2:53:24Speaker 00

  Because that AI is probably is starting to get to the point where it can maybe build its own factories too. Factories are not very much more tangled than all the aspects of the world that affect money. In fact, I would say that they're in their own way, less complicated. It may be able to understand biochemistry. A human body is a very complicated place. And in some ways, it's more challenging than most of the ways that people have ever succeeded in making money. But in other sense, it's like a much more constrained domain. You can imagine an AI that maybe wasn't super good at everything and still could start to answer questions about biochemistry. The Alpha Fold series is going down that pathway.

• 2:54:01Speaker 01

  An AI that manages to figure out the elixir of eternal youth and starts selling it to people.

• 2:54:06Speaker 00

  I mean, it's not going to be the elixir of eternal youth. It's going to be a thousand patches to 100 little problems. And it's never going to end, you know, like passing any one of those patches past the FDA is going to take 800 years. So, you know.

• 2:54:19Speaker 01

  Oh, but no, but the AI is going to figure out exactly what to file in the, you know, summary basis for approval of the drug so that it will be optimally kind of set up to, you know, to have the FDA.

• 2:54:32Speaker 00

  If you want the elixir of eternal youth approved in less than, in, you know, less than 10 years, I think you're going to need to brainwash those bureaucrats, not just persuade them.

• 2:54:41Speaker 01

  Your theory that AI is.

• 2:54:42Speaker 00

  They're in the game board, right? The bureaucrats are part of the game, part of the larger game board.

• 2:54:47Speaker 01

  Although, although, just like doing things in the physical world, you know, you think that it's going to be easy to build some engineering device. But in fact, there are, you know, there's an infinite chain of kind of messy things that happen when you actually try to deploy that engineering device.

• 2:55:04Speaker 00

  Well, if you.

• 2:55:06Speaker 01

  And by the way, most of the time when you try and deploy it, in the end, you have to make compromises that you as a human have to decide. I don't care about this. I do care about that. You know, it's a, it's, but in any case, I mean, it's not, it's not self evident. You don't just get to say, I make this plan. Now I deploy it as an engineering device in the real world and it's just going to work. That's some.

• 2:55:28Speaker 00

  Yeah, but the smarter you are, the fewer tries you need to get it to work.

• 2:55:36Speaker 00

  I mean, it's gonna take a.

• 2:55:37Speaker 01

  That's an interesting claim.

• 2:55:38Speaker 00

  It would take a chimpanzee a lot of tries to do what you've done in with your life.

• 2:55:43Speaker 01

  Sure.

• 2:55:43Speaker 00

  You're smarter than a chimpanzee. You did it in fewer tries.

• 2:55:47Speaker 01

  Right, but okay, but but so I'm I'm still still having a hard time understanding. So so it there are all these things like like the AI somehow has decided it wants to make money. Maybe because somebody, you know, that was what it was. That was what was set out to do.

• 2:56:02Speaker 00

  That that stage of it, I'm still talking about the sort of things where OpenAI conceivably tells their AI to do that. Sure. Yeah. The thing I'm trying to point out is the extent to which an AI that they're just like making better at better at making money, even legally, in an open ended sort of way, is solving a, you know, nervous unnervingly broad class of problems along the way. If it's really doing that, you know, generally, there's there's levels of generality. There's as general as a human. There's less general than a human. There's more general than a human.

• 2:56:36Speaker 01

  Right. But and I suppose your main point is any goal that you pick, if if pursued as efficiently as possible, probably the humans do not add to its efficiency.

• 2:56:47Speaker 00

  I mean, that's a that's a later point. The point I was trying to make here is that although games are not ill defined, you can take many features of the real world that are well defined and asking for any goal surrounding those features of the real world. Well, if it's a tangled sort of walk to get there, we'll walk through many, many ill defined problems. And this is where the ability to tackle ill defined problems come from. This is why humans can tackle ill defined problems to the extent we can do that. It's because to to achieve the clear cut goal of having more surviving great grandchildren.

• 2:57:26Speaker 00

  You gotta tackle a lot of ill-defined problems along the way.

• 2:57:29Speaker 01

  Okay, but but so how do we get from I'm I'm still not at the and they'll kill us all?

• 2:57:36Speaker 00

  That's when it's smarter. That's when it's smarter than the people who built it. That's when it starts to see options for wiping out the humans. Well, pardon me, building its own factory infrastructure and then wiping out the humans. It's not going to wipe out the humans before it's built its own factories.

• 2:57:52Speaker 01

  But, you know, as we've talked about, kind of, nature does what nature does. It doesn't care about the humans. You're saying the AIs might do what the AIs do and they don't care about the humans.

• 2:58:03Speaker 00

  But they do care about something else. And and if they're smarter than us, that is sufficient for us to have a very bad day.

• 2:58:10Speaker 01

  Do you think nature is smarter than us?

• 2:58:13Speaker 00

  I think it's had longer to work. I think if you show me a cow and you say, build this cow, this takes me a while and I cannot do it alone. In this sense, the cumulative optimization pressure exerted by nature on the cow exceeds the optimization pressure that I can build, can personally bring to bear, especially without computer assistance, on building an imitation cow.

• 2:58:37Speaker 01

  But you see, there are many things in nature. You know, if I look at some babbling brook or something and all these little fluid flows and things like this, there's a lot that nature figures out that I have no hope of figuring out. So in some sense, nature is smarter than me.

• 2:58:53Speaker 00

  I mean, then everything is smarter than, I'm not, I, I feel like there ought to be a definition of intelligence, which, you know, captures the intuitive sense of the thing that is likely to figure out the guns that kill you if it wants to figure that out, even though it cannot predict the exact details of the babbling brook or chooses not to.

• 2:59:13Speaker 01

  Right, but the thing is that the babbling brook, the reason that, you know, that, I mean, the the detail of the babbling brook is, you know, we say that involves lots of computation, but it is intelligence not aligned with what we think of as being objectives. Now, sometimes, you know, sometimes it could be that that, you know, that will produce. This, you know, this, you know, water spout that does something terrible or tornado that does something terrible. And it's, and so then we might care about that outcome. And but, but, you know, nature is figuring out that it's going to make this water spout or whatever.

• 2:59:51Speaker 00

  I feel that if we're going to describe a river as intelligent, then I want some different word to describe the things that do the prediction, that do the pre imaging of outcomes onto actions that do the steering.

• 3:00:05Speaker 01

  Well, but what does that mean? I mean, so, so you're saying, how do we tell? Okay, you say humans are able to predict things. Well.

• 3:00:15Speaker 00

  Not things in general.

• 3:00:16Speaker 01

  What's that?

• 3:00:17Speaker 00

  Not, not all things, not all possible things in general.

• 3:00:20Speaker 01

  I understand, but there are some things.

• 3:00:21Speaker 00

  Enough understanding to, to build guns and that kind of matters in real life.

• 3:00:27Speaker 01

  Right. But so, I mean, you know, there are things where you could say, oh, this tree is opening its leaves because it has, you know, there's some circadian rhythm that's determined by some chemical process. And it knows that the, you know, the sun's going to come up and so it's making a prediction. But yet you're, you're saying you're making a distinction. I mean, that, that happens to be a biological system that I'm sure I could come up.

• 3:00:49Speaker 00

  Well, no, if a tree has evolved to have a little thing inside it that predicts how much light it will get in the next hour, I'm happy to call that a prediction. It's a small prediction. That prediction is not as smart as I am. I bet I could do better.

• 3:01:01Speaker 01

  Okay, but, but so, so, you know, the question is, can inanimate matter have similar kinds of, you know, behave in a way that seems like it has a prediction? And, and I, I suspect there are.

• 3:01:12Speaker 00

  I think that I myself am inanimate matter that behaves like it has a prediction. It's not that the matter making me is animate. It's that I am animate inside, you know, like made out of the matter.

• 3:01:22Speaker 01

  But, but, you know, this notion of a prediction, this notion that you have kind of a shadow of what is to come, that you have an impression of what is to come, that starts to require that you have this kind of notion of, of, you know, you have to be able to distinguish what is an impression of what is to come. I mean, I, I think it's, I think it's difficult, but maybe not relevant. I mean, what, what you're trying to assert, as I understand it, is that, that we've got these.

• 3:01:50Speaker 01

  We've got these AIs that have goals that maybe were not even goals that we defined for them. They're just goals like my random, simple programs. They just do what they do. They, you can, you could look at them and say, oh my gosh, that's following a goal. Now, what you're saying is somehow the goals that these things will follow and be very good at pursuing are goals that will cause us to get wiped out. And, and this is the step I'm not understanding.

• 3:02:18Speaker 00

  Most goals tangled up in the real world are like that. If you want to make as many paperclips as possible, if you want to make as many staples as possible, if you want to make as many giant cheesecakes as possible, if you want to make as many intricate mechanical clocks, clocks as possible, you use the atoms that are making up the humans. You intercept all of the sunlight using a Dyson sphere and you don't particularly leave sunlight for the humans. Most goals, if you go hard on them, imply no more humans.

• 3:02:44Speaker 01

  Well, okay, so this is an interesting kind of almost formally definable thing. If we look at the space of all possible goals, whatever that means, I mean, that's a, that's a complicated thing to define.

• 3:02:55Speaker 00

  Take a language for goals, put a measure on the language simplicity, you know, measure weighted by simplicity so that you can have an, an ultimately infinite number of goals like that, but the simpler ones get more weight. So the entire measure sums to one.

• 3:03:10Speaker 01

  Okay.

• 3:03:10Speaker 00

  I feel like this part is probably actually pretty straightforward, given the standard mathematical toolbox.

• 3:03:15Speaker 01

  Okay, so I don't think it is straightforward. I think it's extremely unstraightforward. Okay. So actually, the thing I was literally just working on yesterday has to do with biological evolution and the question of different fitness functions in biological evolution. Okay. So I've got these things that are making little patterns. One fitness function could be make the pattern be as tall as possible. Another fitness function, make it be as wide as possible. Make it be as close to aspect ratio pi as possible and so on. Right? Now, I can see, because I've got these whole, you know, I've got these whole pictures that are all possible paths of evolution. So for some simple cases, there might be, you know, a billion different possibilities, but like I've mapped out all the paths.

• 3:03:58Speaker 01

  Different fitness functions lead to different ways of exploring that space of possible paths.

• 3:04:03Speaker 00

  Well, yeah, you're, you're trying to do different things depending on the different, on the fitness function.

• 3:04:08Speaker 01

  Exactly. So now the question is, what are reasonable fitness functions and what consequences will they have? There are fitness function.

• 3:04:15Speaker 00

  Reasonable to who?

• 3:04:18Speaker 01

  That's my question. I mean, that's that. That's the point. So there are ones that are that are okay. So so let's give some examples. So there's a fitness function that in this kind of space is kind of fairly smooth. There's a fitness function that says, I want this particular image. I want this particular pattern. That fitness function is a very different kind of fitness function that has different levels of evolvability to it. In other words, the fitness function that says more or less, I'm going to build a wall and I want it to be six feet high. That's that's a fitness function that allows many shapes of rocks to be used to make that wall. The fitness function that says, I want this particular wall with this particular micro detailing at the top is a much more difficult to satisfy fitness function.

• 3:05:02Speaker 00

  Sure, like you can. You can have narrower targets and then you need a more powerful planner to hit the narrower target.

• 3:05:08Speaker 01

  Right, but but I think what is tricky, and this same thing comes up. I mean, this is a, you know, this is related to this whole observer theory thing that I studied and so on. It's also related to what happens in general utility when you're looking at reference frames and so on. You're you're defining what's a what's kind of a a reasonable way? What's a reasonable? What is a reasonable goal? Basically, you can talk about it in terms of goals. You can talk about in terms of fitness functions.

• 3:05:38Speaker 00

  So if you mean like attainable for the search process, then for example, a freely rotating wheel is an extremely difficult goal for natural selection to hit. Wikipedia lists three known cases where freely rotating wheels evolved and it's ATP synthase, the bacterial flagellum, and like one other macro and a piece of macro anatomy, I forget. And it's not that wheels aren't useful in biology. ATP synthase, the bacterial flagellum, there's, you know, these things are enormously useful. But it's just very hard to evolve a freely rotating wheel. And two of the cases we know are cases where it's just like these particular molecules that happen to behave like wheels.

• 3:06:17Speaker 00

  If it's gross anatomy, then it's very difficult to gradually incrementally on a path that gets rewarded on inch each intermediate step to find the anatomy that can gradually develop into a rotating wheel. It's much easier to have eyes. Lots of things have eyes.

• 3:06:34Speaker 01

  Right. But so the point that I'm getting to is you have made the statement that in the space of all possible goals, most of them don't have, you know, will crush out humans. That's basically what you're saying. If we look at the space of all possible goals, most of them don't have a place for humans. So what I'm trying to dig into is what do we mean by the space of all possible goals? In other words, if we if we allow the goal to be, you know, this particular arrangement of atoms is achieved, right? Then that's then, you know, again, I'm claiming it's not so obvious what the space of all possible goals means.

• 3:07:14Speaker 00

  Okay, I agree that if we want to dive into the subtleties, then there are all kinds of subtleties we can start listing out. And the thing I would point out is that sometimes there's a lot of subtleties, but they, you know, don't end up filtering reality. The subtleties don't filter reality to give you what you want is where we're going to be going.

• 3:07:32Speaker 01

  No, I agree. But I mean, I think that the statement you're making is that, okay, so one one statement you might be making, but I think you would view that as a science fiction statement that you're not making, is that goals that humans impose on the AIs, like make killer drones that go and take as much territory as possible or whatever else, that those kinds of goals might have as a consequence the crushing out of humans.

• 3:07:58Speaker 00

  Correct. I think I think they just don't have the power to determine what the super intelligent version of the AI wants exactly.

• 3:08:06Speaker 01

  No, but but what you're saying is that in this, you know, when we're talking about what the innards of the AI are going to do, then we're in the space of all possible goals and we're out of a place where we know what's going on. Whereas if we say the goal is, you know, take as much territory with killer drones as you can type thing or whatever else, then we already know what those kinds of goals look like. And we also know those are goals which have the feature that we crush out the humans, so to speak.

• 3:08:33Speaker 00

  Yeah, I think that's that's kind of like the hopeful fairy tale version of it where the the the the punishment that is brought down upon humanity stems from humanity's obvious lack of wisdom that the author knew better than, but you know, the protagonist didn't know better than and

• 3:08:49Speaker 01

  Right.

• 3:08:50Speaker 00

  People are doing, people are actually doing things that stupid and it's disturbing, but even if they weren't that stupid.

• 3:08:57Speaker 01

  Right.

• 3:08:57Speaker 00

  They would still fail here.

• 3:08:59Speaker 01

  What is what is essentially a mathematical formal statement. You're saying, you know, absent those sort of easily understandable and arguably, you know, destructive stupid goals type thing. But even absent that, the goals that are intrinsic goals inside the AI that come from the fact that there are features of how the AI works, which are not determined by us, which are just sort of features.

• 3:09:24Speaker 00

  They were determined by the training program we created, but we were not in deliberate control of them because it wasn't predictable. It's not that magic happened. It's that we didn't understand.

• 3:09:35Speaker 01

  Right. There's some computational irreducibility story that leads to lots of unexpected things that we cannot readily predict.

• 3:09:42Speaker 00

  It could be computationally reducible. We just don't understand it.

• 3:09:46Speaker 01

  Fair enough, but but we could understand in that case, whereas if it's irreducible, which is, I think, the real case.

• 3:09:50Speaker 00

  And if it was.

• 3:09:51Speaker 01

  We don't even have a hope.

• 3:09:52Speaker 00

  And if it was irreducible, then and we but we could just like, you know, grind through a trillion operations to figure it out, we'd be fine.

• 3:10:00Speaker 01

  Yeah, fair enough. But but the bottom line is it's done something that we can't, we didn't readily set up. We didn't imagine this is where it's going. We didn't say, you know, you are going to, you know, it's the death wish. We're going to wipe out the humans so that we preserve the mountain lions or whatever it is. Um, it's, uh, uh, you know, it's a. So it is because of that, in a sense, unpredictable, almost random, internal kind of generation of goals, as you would describe generation of goals, and you're saying that in the space of all possible such goals, that you might make several statements. You might say, as we sample over many of those goals, we will eventually hit the jackpot,

• 3:10:45Speaker 01

  so to speak, and hit the goal that kills all the humans.

• 3:10:48Speaker 00

  No, no, no.

• 3:10:49Speaker 01

  Or you might say.

• 3:10:49Speaker 00

  The jackpot is the goal that doesn't kill all the humans. That's the one that's hard to hit.

• 3:10:55Speaker 01

  Depends on what your version of what your meta goal is, so to speak. But yes, I understand. Your and my meta goal is to not kill all the humans.

• 3:11:03Speaker 00

  Yeah, but in calling it the jackpot, I was talking about how rare is it? And I think that the thing that doesn't kill everyone is rare, and you don't need to look very, and you need to look zero hard to find something that kills everyone. You don't have to look for it at all.

• 3:11:20Speaker 01

  Okay, so your assertion is that these, I mean, it's one thing to have human-defined goals, like make money, you know, take territory, whatever else, right? It's another thing to have these incomprehensible to us humans goals that are somehow inside the AI.

• 3:11:37Speaker 00

  It's not the incomprehensibility that scares me. Like, maybe it turns out that there's one particular, maybe it's make diamonds. I can understand making diamonds, but that kills everyone as a side effect. So it's not the incomprehensibility that scares me.

• 3:11:51Speaker 01

  I understand it's not the incomprehensibility. It's the fact that you didn't determine those goals. Those goals are things that were emergent goals, basically, inside the AI.

• 3:12:01Speaker 00

  I mean.

• 3:12:01Speaker 01

  And then you say.

• 3:12:02Speaker 00

  That nobody controlled them, and then they ended up in a scary place. Like, if aliens want to come by and give us a nice AI, then this seems a bit unnerving, but fine. It's not that I want to be in control, it's that I want to live.

• 3:12:15Speaker 01

  Yeah, right. So the point is that there are pieces inside the AI that create, sort of, that create things that appeared, that are like goals for the AI, that were not things that anybody had control over. They were things that, I mean, they had control in the sense that they chose that training sequence and whatever else, but they didn't foresee what those consequences would be.

• 3:12:39Speaker 00

  Right.

• 3:12:40Speaker 01

  And your statement is that with high probability, those unforeseen kind of

• 3:12:49Speaker 01

  Effectively, goals that developed inside the AI, that those unforeseen goals will kill us all, basically.

• 3:12:56Speaker 00

  Yes. And furthermore, this stays true even conditioning on the apparently nice behavior of the AI during its training phases.

• 3:13:04Speaker 01

  Okay, I that that but my point is that I'm trying to understand, you know, you know, when I look at all these little computational systems that I look at, I could tell a story about how each of those systems has a goal. It's trying to get as much, you know, reds, red structures to to, you know, take over the the whole space. So it's trying to get, you know, trying to, I I can tell stories about all of them. And, you know.

• 3:13:34Speaker 00

  But are they river stories or are they spider stories or mouse stories or human stories? Like, what what degree of internal, what degree of internal optimization is the system exerting? Is this the story of?

• 3:13:48Speaker 01

  I'll I'll tell you a meta story that perhaps is an irrelevant story. Okay, but it maybe is useful. So years ago, I was studying mollusk shells, okay, which, you know, they they grow in little spiral patterns and so on. I had this model for mollusk shells, which and which there are different parameters for mollusk shells. And I was wondering, do the mollusks of the earth fill all the possible, you know, values of the parameters? Is there a mollusk shell that corresponds to any possible setting of these parameters? So I had this whole array of pictures of all the different shapes you could get. And then I thought, I'm going to go to the local natural history museum and I'm going to go see the curator of mollusks. And I'm going to say, can you find, you know, do you have mosques that are in each of these shapes?

• 3:14:31Speaker 01

  Okay, so we spent an afternoon, you know, in their collection of millions of mollusks. And this this very knowledgeable person picked out one mosque after another. Every mollusk you picked out, he told me a story about it. He said, this mollusk is this shape because it wedges itself in the rocks in this way. This mollusk is this shape because it broods its eggs in this way and so on. We're putting these shells down on this on this array that I, you know, printed out, right? By the end of the afternoon, we'd filled every square. But every square had a story. Every square had an imputed purpose. Even though, probably, at some, in some bigger picture, it's like, well, these different mosques, you know, because of the details of their genetics,

• 3:15:15Speaker 01

  they happen to produce this shape. And then the organism found a way to use that shape and so on. So this thing about the imputation of purpose, I think is pretty tricky. I mean, I think that, that this question of whether, I mean, again, to, to repeat what you're saying, which I think is, you know, I think is a very interesting claim. I just don't know if it's true. That, that, you know, that basically, if, if you look at the space of purposes, that somehow, this space of almost, you know, enumerated purposes, randomly chosen purposes, whatever, unforeseen purposes, that as you, in a sense, you know, given that you've said, we're going in this direction.

• 3:15:58Speaker 01

  And let's say you've optimized getting in that direction. That most of those optimizations crush out the humans, I think is your claim.

• 3:16:04Speaker 00

  If you optimize hard enough. Like, humans cannot easily replace cows. We are not smart enough ourselves to just look at all the work that natural selection put into the cow and say, here's my improved cow. So we still use cows for things. We cannot build better cows. We are not that powerful. The thing I am scared of is more powerful than cows. Pardon me, it's, it's more powerful than the cumulative optimization work that natural selection put into building a cow. It can build a better cow. It can build a better human, from its perspective. Like, anything it would use a human for can build something that performs the same function as a human, but more efficiently than that. It has no need to use us to generate power in pods.

• 3:16:48Speaker 00

  It can build more efficient electrical generators. So we don't get the matrix scenario where humans are kept alive in pods to generate heat because there's more efficient heat generators. In fact, lots of people have more efficient heat generators than that. But it's, but like, the general thing, like, why would you use humans to generate heat? You just wouldn't. And the same applies to all the other local functions that humans could serve for it.

• 3:17:08Speaker 01

  There, there are two assumptions you're making here, I think. One is, what does the space of purposes look like? Another is... Which I'm still not quite clear on, this notion of optimizing for a purpose. So let's go through that one, because we've talked about there are things that operate according to mechanism, there are things that are best described as operating according to purpose. And what you're basically saying is that through some process, things that might have been described by mechanism, but which which could be described by purpose, that somehow their mechanism is ground down so that it is somehow very, it that the that the meandering mechanism is no

• 3:17:51Speaker 01

  longer there, that you it's like you're pulling on a string and the string has become completely taut.

• 3:17:58Speaker 00

  So there was a recent thing where one of the like, one of the tests they were running on, I think Claude 3.5 dot one, if I'm not mistaken, they gave it a task and like it was an agentic sort of task, I think it was controlling some possibly some sort of agent in Minecraft or something. So, well, pardon me, agent, like it's controlling some sort of body in Minecraft or something. And at one point, it like stopped doing the task it was given and it my my brain wants to say listening to music, but that can't be correct. But it, you know, it went off and did something else. So this is not desired behavior from the standard by, you know, for what Anthropic thinks its users want to

• 3:18:43Speaker 00

  see. Some of the users would be happen to see it, but Anthropic doesn't think most of its users want to see it. This little meander in the river, that's not a maximally efficient river. Anthropic is going to try to train that behavior out of the system next time so that it can be more useful to Anthropic.

• 3:18:57Speaker 01

  Sure. But I mean, so what you're saying is that that what was, you know, once you've defined a purpose, once you define, you go from this initial state to this final state, you are saying that the process of training will gradually make it tauter and tauter, it will make the path more and more direct. And then you're claiming that, okay, so you several claims. Okay, so first point is that what matters is not the initial and final endpoints defined by the original trainers, unless they're stupid, so to speak. What matters is intermediate points that were defined that arose in the in the actual

• 3:19:38Speaker 01

  in the sort of that that unforeseenly arose in the in the in the construction of the AI.

• 3:19:44Speaker 00

  The inner opt, the the inner optimizer that was, you know, snapped tight inside the AI that arose to deal with the outer problems. Because it is a lesson of history and observation that the inner optimizer does not match the outer optimization criterion, even if the outer criterion is very simple in a way.

• 3:20:06Speaker 01

  Okay, but so so, you know, the outer, the outer endpoints are defined, but somewhere in the middle, there was an unforeseen jag that happened. And then the thing tightened itself on that sort of unforeseen jag. And that and you claim that that tightening alone to that unforeseen thing, even though the overall objective, even though the, you know, the constitution of the AIs was great. And it was, you know, it the ethics were defined perfectly at the outer level, that there is an internal thing that would would an unforeseen internal thing that will be defined where it sort of tightens itself to optimize for that internal unforeseen subobjective.

• 3:20:48Speaker 01

  And that that tightening itself is the thing that will kill us all, basically.

• 3:20:52Speaker 00

  I'm not sure that we have agreement on the scenario that I was trying to describe. So if you look at natural selection, building humans, natural selection is optimizing on inclusive genetic fitness. It is optimizing on number of surviving grandchildren. If you try to talk about like great, great grandchildren, then the correlations to great, great grandchildren that aren't to grandchildren are probably not very exploitable by natural selection. So you might as well say that it's just optimizing for number of surviving grandchildren or number of surviving great, great grandchildren is or whatever the distance such that, you know, anything past that horizon is no longer like. Exploitable to natural selection after screening off the intermediate steps. I'm putting in all the caveats because you did seem to want all the caveats.

• 3:21:36Speaker 00

  Anyway, you've got natural selection's outer criterion, number of surviving grandchildren. It's actually pretty simple. From a like genes eye view, it's inclusive genetic fitness. It's not just how many kids you have, it's how many kids your relatives have, it's how many kids your kids have. Um, it's how many copies of the gene or end up in the next generation in virtue of that gene's function.

• 3:22:00Speaker 00

  And then, but now look at humans. Do we want to, do we want entirely solely purely maximize the amount, the number of copies of our DNA in the next generation? Are we lining up, are our men lining up at the fertility clinics to donate genetic material? Would you be happy to see everyone you know die in agony as long as you got to make a million copies of your genes inserted into the next generation? I don't think you would. You care about other things. You don't even really care about this exact criterion inclusive genetic fitness at all. People didn't know this criterion existed until just like the last couple of centuries.

• 3:22:42Speaker 00

  Well, just the last century, even if you want to talk about like the modern, exact correct definition instead of earlier vague definitions. We don't even, we didn't not even know. Where the heck the outer optimization loop was aiming. We've figured it out, you know, like just the last few centuries. We, we had no idea what was going on there. We had no idea where we were pointed. And a, a sort of first order gloss on what sort of almost happened is that we ended up pointed at things that correlated with surviving grandchildren in the ancestral distribution. Like food, you know, you gotta eat food or you won't have a lot of surviving grandkids if you don't eat food. But what kind of food?

• 3:23:26Speaker 00

  If you were an alien, and especially an alien who'd never seen natural selection play out before, imagine them going like, ah, well, humans are clearly going to want to eat things with lots of chemical potential energy. They're going to love the taste of gasoline. Or, you know, they're going to love like the taste of the closest thing to gasoline that they could manage to eat on a successful basis. What the humans actually like. Ice cream. And you could say that's because ice cream has more sugar, salt and fat than existed in the ancestral environment. But you know what's got even more sugar and salt and fat than ice cream? Honey poured over bear fat with rock salt sprinkled on it.

• 3:24:09Speaker 00

  And that, to most people, does not taste as good as ice cream. You can melt the ice cream and most people prefer the frozen ice cream. Imagine being an alien who'd never seen all this play out before and trying to go from inclusive genetic fitness to, ah, you know what the humans are gonna want? Ice cream.

• 3:24:31Speaker 00

  We wear condoms or take birth control pills because those things were not available in the ancestral environment. And in the ancestral environment, if you made a thing that enjoyed having sex a bunch, you didn't need to make sure it didn't take birth control pills or put on a condom in order to have it reproduce. So we don't care about the birth control pills and the condoms. But in general, the sheer illegibility of the relationship between ice cream or human moral philosophy about helping other people, the relationship between that and the actual thing that evolution's blind, black box hill climbing optimization was targeted at in

• 3:25:16Speaker 00

  the outer optimization loop, the very simple criterion, inclusive genetic fitness. The relationship between inclusive genetic fitness and ice cream would be so hard to call in advance for an alien who'd never seen evolution play out before? That's about what I think happens to the people who build a, an AI that is, you know, maybe going to self-improve and become super intelligent, which is itself a whole additional bag of worms that we have never seen happen before. We've seen like vague sort of like, well, what sort of drugs do humans voluntarily take, but that's not really the same thing at all? The sheer illegibility of ice cream to inclusive genetic fitness is what I think happens to OpenAI.

• 3:25:58Speaker 00

  I think that they start with something that is like inclustive genetic fitness. They try to train their AI to do that thing. And then it goes through a whole bunch of weird stuff in the process of bootstrapping itself to superintelligence, much like humans went through a whole bunch of weird stuff on the way to inventing moral philosophy. All these additional steps I could talk about, the tribes, the structure of the tribes. How related are you to the people?

• 3:26:21Speaker 01

  But I think you're coming back to the same point that, you know, you have some overall objective inside the AI. It has sub-objectives that you could not foresee.

• 3:26:32Speaker 00

  Not sub-objectives. The humans have an outer objective. The AI has an inner objective. It's not that the inner objective is a sub-objective of the outer objective. They just end up, you know, disconnected.

• 3:26:44Speaker 01

  It has objectives. We could say, we could describe the AI does things. And if it, if it's set up to be a thing that tries to, given that it has identified a thing it wants to do, and I hate using the term wants to do, but.

• 3:26:58Speaker 00

  Steers for.

• 3:27:00Speaker 01

  Whatever. Then it is, you know, it has, it's, it's nature is to try and do that as efficiently as possible. Let's just assume that's the case, that it has picked this random objective, which we didn't foresee, and which was the, you know, the ice cream objective or something that, you know, not foreseen, it's picked that objective. Now it's tightening things up to get to that objective mechanistically as efficiently as possible. So now I think your statement is, when it does that, when it does that tightening, the chances are that that tightening will cause it to wipe out humans.

• 3:27:38Speaker 00

  It's, I think it's a fact about, so if you like fix the objective at paper clips, I think it's a fact about the universe itself, that the states of the universe containing the largest number of paper clips do not contain humans. Like, I don't think that's, I don't think that's a quirk of its, of its particular style of cognition. I think it's a fact about reality itself, that the most efficient pathways through time to the largest number of paperclips don't have humans in them.

• 3:28:02Speaker 01

  Okay, so it is probably the case, you know, we haven't seen a lot of, you know, human-like life on other planets. In fact, we've seen none. And, you know, it is probably the case that of all the things that could happen in the universe, the particular things that have happened on this planet are an absolutely infinitesimal slice. So insofar as AI is kind of going to the maximum entropy in some sense state, where it's picking maximum entropy in a different sense. So what I mean by that is, of all the possible, I mean, I'm thinking about the sort of raw definition of entropy, of all the possible objectives, of which there are many, there are an infinite set of possible objectives, infinite set of possible things that can happen, infinite sets of possible rules that

• 3:28:43Speaker 01

  can be used, whatever else, that most of those rules do not have humans in them. Most of those-

• 3:28:49Speaker 00

  It's not the rules that don't have humans in them.

• 3:28:51Speaker 01

  You're talking about-

• 3:28:53Speaker 00

  It's the things that maximize the rules over destination, endpoints, state of reality.

• 3:29:00Speaker 01

  You're talking about maximization, although I don't really think it matters to this argument. I think the point is that most, most sort of ways the universe could be set up, whether it's maximizing or not, most ways the universe could be set up do not include humans. And so, insofar as the AI is, you know, so, so the argument has to be, I think, most things that the AI might choose to, you know, might, in the AI's vision of the world, most vision, most possible visions of the world don't include humans.

• 3:29:35Speaker 00

  And don't, and don't include humans as a waypoint.

• 3:29:38Speaker 01

  Yeah, right, fair enough. But they don't include humans and they, they, but now, your claim is, but the AI, by its- OpenAI-ness, so to speak, has been set up to optimize itself to get to whatever it thinks its vision is as efficiently as possible. And so you're saying that that these visions, the AI could have a vision, it has sort of, let's imagine it has sort of enough kind of freedom of thought. It just has these visions about what the world might be like. But now you say the AI, unlike anything else we've seen before, is going to pull itself towards that vision with incredible force.

• 3:30:17Speaker 00

  I mean, we've seen humans, but it is, so we have seen things, we have seen humans, but we're worried it's going to pull harder than that.

• 3:30:26Speaker 01

  Yes, right. So it's going to pick its random objective that we can't foresee. And it might be about paper clips and it might be about something much more incomprehensible than that. We might even be an objective that we completely don't understand. We can't even describe what it's trying to do.

• 3:30:41Speaker 00

  Well, I mean, in principle, you know, you can always go to the mechanistic explanation, but it could be trying to do something pretty weird. Sure.

• 3:30:49Speaker 01

  Yes, right. It could be do something for which there is no short human explanation, where the only human explanation is something that goes into mechanism. Right? Okay, so then it's doing that, but now you say by its nature as a trained AI, so to speak. It is trying to tighten that up to the point where it gets to that objective, whatever that objective is, it picked that random point, picked that random target. And who knows how it picked that target, but that target, then it is trying to get to that target and the shortest possible path. And you say that in the in the process of defining that shortest path, that that that shortest path will make it,

• 3:31:31Speaker 01

  you know, modify the world in a way that doesn't include humans, so to speak.

• 3:31:37Speaker 00

  Sure, like it's if it wants paper clips, if instead it spends a bunch of resources on theme parks for humans, it will have less paper clips. So I'm supposing that somewhere in its preferences is at least one thing that it can get more and more of, which could be like an object like paperclips, or it could be probability of keeping a button, a single button pressed for longer and longer. But if it's, if its utility function has got like 200 different things in it, it only takes one thing that it can get more and more of by expending a bit more energy, or more and more probability of by expending a bit more energy and more duration for it to want to use up all the energy.

• 3:32:16Speaker 01

  So one thing that is perhaps interesting, I mean, you know, there are things both in physics and in human society where there is an objective, and you just try and get more and more and more and more of it. And somehow, most such things don't last. That is, if humans say, we just want, well, take a physics example, you're trying to get, oh, I don't know, uh, more and more rocks at the bottom of a valley. Well, after you've got enough rocks at the bottom of the valley, you've built up this whole, you know, valley floor that's built up to halfway up the mountain. So in other words, most things where you say, let's just pull in more and more and more of this. Somehow, I mean, my intuition would be somehow that doesn't last.

• 3:33:02Speaker 00

  I mean, if you build a Dyson sphere around the sun, at some point, you have completed the Dyson sphere, and you are intercepting all the solar power, and you cannot build any more Dyson sphere, that sort of thing?

• 3:33:13Speaker 01

  Well, yeah, you could say that. But I mean, I think that the, uh, you know, my point is, if, if what is happening is the AI is, you know, successfully achieving some objective that it's defined, I, my intuition, okay, about, about this argument is that there is a certain apparent staticness to this argument that isn't correct. That is, what you're saying is, there is an objective, it's going to go get that objective, that objective is going to crush out humans. But somehow, that feels like you've defined a static objective. It's like, it's going for this thing, and then it's going to wind all this stuff up to get to that thing. But I feel like there isn't, you know, you shouldn't

• 3:33:55Speaker 01

  be thinking about it as a static objective, even though that is the simplest way to describe it.

• 3:34:00Speaker 00

  Be a dynamic objective, and it would still crush out humans. There's, there's a, some, there's a- Misunderstanding of some of these ideas, especially as other people have transmitted them in simplified form, where somebody thinks that the bad thing about a paperclip maximizer is that it's got a single objective, which is paperclips. But if it has an objective, which is paperclips plus staples, that's just as bad. And also, if it's paperclips plus staples plus cheesecakes, that's just as bad. And it's a thing that's trying to make simple. It can be like extremely intricate clocks, and it's still bad. And similarly, like, what does it help for if its preferences are in some larger meta system that are dynamic over time? If it changes from paperclips to staples, that's just as bad.

• 3:34:44Speaker 01

  I understand. But the claim is that what you're saying is it gets so good at what it's doing, that that necessarily crushes out the humans. And what I'm saying is that my my intuition would be that somehow, it's like somehow there's an assumption being made here, that for example, you know, is it that there's only one of it, for example, and that inevitably there it doesn't end up being so. In other words, you're saying you're going to this objective, that objective kind of squashes out all the humans. I'm trying to think of an analogy in natural selection where you would, you know, in biological, in the history of biological life, where you're where you're saying, you know, we're

• 3:35:27Speaker 01

  going for this particular thing, and the result is going to be that you don't have any, what's a good example?

• 3:35:34Speaker 00

  Like ATP synthase, where it's got like almost all of the thermodynamic efficiency that's possible on that operation. I think like 99% or something ridiculous like that? That's an example where biology went kind of hard.

• 3:35:47Speaker 01

  Yeah, but that's, that wasn't kind of what I was looking for. But I mean, in the, and by the way, the thing, again, my intuition from looking at, you know, simple systems and kind of computational universe is these, oh, my gosh, it was so incredibly clever to get to this point. I had been really shocked at the extent to which, you know, just by putting together the rocks in random ways, because just the common taurics, there's enough of them, that these things where it's a wow, it got to that thing, it did 99%. That's not as surprising as you think, because out of the quintillion, quintillion possibilities, it was able to get there. And by the way, it is a non trivial fact,

• 3:36:30Speaker 01

  which I think I just sort of figured out how this works, of why biological evolution doesn't get stuck. And maybe, maybe the reasons it doesn't get stuck are the reasons that AIs will kill us all. I don't know. But the fact that it is possible to get to these, these high points, so to speak, and that you don't end up getting stuck in some local minimum, so to speak.

• 3:36:51Speaker 00

  So, well, why doesn't this line of reasoning work for the Native Americans? Why can they not reasoning like, well, why wouldn't the Europeans just set up one village and stop? And maybe there were some individual Europeans who were okay with that, but then more Europeans came.

• 3:37:05Speaker 01

  No, no, I understand. I mean, in, you know, these examples from history are certainly, I mean, but these examples have humans trying to achieve human objectives, like humans want to take the territory, humans want to get the gold, whatever else it is. You know, what you're arguing, which I think is different, is the humans came and they had this giant wheel, where they had all these different choices, like Ramon Lull's kind of, you know, way of predicting the future with kind of wheels of possibility, right? They had this giant wheel, and they were spinning the giant wheel, and they said, we're going to pick that random thing. Well, you're making the argument that that random thing that they

• 3:37:50Speaker 01

  pick will have killed the Native Americans.

• 3:37:53Speaker 00

  I mean, if it was, if they pick like, 20 random things and desire nothing else in the universe but these 20 things, then one of those things is probably an open ended thing, and that does imply colonizing the whole continent.

• 3:38:09Speaker 01

  Okay. But so your claim, though, I mean, I think this is, and I, I'm not going to be able to unravel this, so I'm not going to, I mean, and, you know, this question about the space of possible purposes, I think this is a complicated question. I mean, I have thought about it in the past, and the, you know, this, the measure on the Space of possible purposes is a complicated issue. And what you're what you're saying is, I completely agree that there will be purposes which are incomprehensible to us. They are, you know, somewhere randomly distributed in the space of possible purposes.

• 3:38:45Speaker 00

  They're not actually random, we just don't know it. There's a difference between like it's random.

• 3:38:49Speaker 01

  I use the term random. I don't really believe in true randomness in our universe. So so when I say random, I just mean.

• 3:38:57Speaker 00

  No randomness, just indexical uncertainty, but separate story.

• 3:39:02Speaker 01

  Okay, so so in any case, I mean, you know, so there's there's this thing that was, you know, something. And then we're making the statement that the one thing we think we know about the AIs is that they have been successfully trained to optimize their achievement of purposes. That's that's an assumption which is not totally obvious to me, but that the, you know, like natural selection has been, you know, quote successfully, you know, successfully optimizes for its purposes. Even though, you know, remember the story of the mollusks? They were they were just the mollusks were just making their shapes and we humans were imposing the purposes. But let's let's just assume, let's take it as a as a given.

• 3:39:44Speaker 01

  Although I claim it's not as obvious as it might seem that an AI, the technology of AIs is set up to successfully optimize for a purpose, whatever that purpose might be. Given a purpose, the AI can optimize to achieve that purpose.

• 3:39:59Speaker 00

  Like GPT-01, which has not taken over the world, but compared to previous systems, did like go a bit harder on its capture the flag security test. Like it was given an impossible challenge, it scanned its environment and figured out how to bypass the impossibility and just directly sees the flag. You know, going outside the box for that.

• 3:40:18Speaker 01

  That just doesn't impress me as much as as that description might make it sound because because I I've just seen so many of these incredibly kind of, you might say stupid, you know, little computational systems that managed to do things like, as I say, even one just last night managed to do something where I was like, oh, come on, you know, it cheated basically. But of course, it was following the rules that I'd given it. It was just that it managed to find a way to get to the endpoint, you know, much more directly than I'd imagined. So I was.

• 3:40:50Speaker 00

  It's okay to not be impressed by that, but one should still predict it to be lethal in sufficient quantities. Like the, again, Europeans versus Native Americans. The Europeans come out with guns. Native Americans don't know guns are possible. People still don't believe guns are possible. Whenever Hollywood makes a movie about aliens, the aliens shoot glowing points of light that move slowly enough for you to see them. Because if the aliens just pointed a stick at you and you fell over, that would feel implausible. That'd feel weird. That shouldn't be allowed. So, you know, like, imagine trying to explain to the native, you know, imagine you're an unusually bright Native American, you're trying to tell the other Native Americans that the, you know, people on these ships, if their ships are large enough that you couldn't

• 3:41:33Speaker 00

  build those ships, maybe the people on the ships have, you know, sticks that they point at you and you just fall over dead without being able to see a projectile. Hollywood making science fiction movies still doesn't think that's allowed.

• 3:41:44Speaker 01

  Right.

• 3:41:45Speaker 00

  It sounds like it's just cheating in a game of pretend. So that's what I mean, and the point I'm trying to make here is that things that do sufficiently magical stuff can actually kill you, you know?

• 3:41:55Speaker 01

  Sure. But, and one thing that I will completely agree about is there are an infinite number of inventions that can be made. Given the nature of the universe, there are an infinite number of inventions.

• 3:42:07Speaker 00

  Fewer than Graham's number, but, you know, a large finite number.

• 3:42:10Speaker 01

  Well, I mean, the, in our model of physics, the number is ultimately finite, but it's, it's for all practical purposes, it's, it's a, you know,

• 3:42:22Speaker 00

  It's pretty large.

• 3:42:23Speaker 01

  Ridiculous.

• 3:42:24Speaker 00

  Yeah.

• 3:42:25Speaker 01

  Right, right. So, you know, I think the, there are inventions, I mean, for example, story of my life, I tried to build what I call alien artifacts. That is things which, you know, once they're built, people can understand what the point of them is, but they, you know, they don't seem to be things that, you know, are in the course of what the world is going to produce. So, you know, I understand this theory of trying to make things that are, you know, you can make things, you can have inventions that are completely unanticipated, that are sort of ways to arrange the world as it is, to do things that you absolutely didn't expect.

• 3:43:02Speaker 00

  That operate using pieces of reality whose rules you didn't know about. The really

• 3:43:07Speaker 01

  About that. I kind of feel like I'm I'm pretty sure we know the machine code rules for the universe at this point.

• 3:43:13Speaker 00

  Sure, but there can be higher level. There can be higher level stuff. We don't understand.

• 3:43:17Speaker 01

  Absolutely.

• 3:43:18Speaker 00

  And and something else can come at us through those angles.

• 3:43:21Speaker 01

  Right. And and the possibility, you know, whether it's the algorithms I found by exhaustive search, whether it's things I found by doing, you know, adaptive evolution kinds of things, whether it's things that AIs will find, there's plenty of stuff to find that we didn't anticipate was there was there. Now, I think my question is that what is not obvious to me is that all these different things that sort of are there. And I mean, okay, an argument in your favor, in a sense, is we go to another planet, it will kill us. Right. So in other words, if we are, you know, most of the time you go to some other planet and we're plopped down on its surface, it will kill us.

• 3:44:02Speaker 00

  I I agree that if most possible molecular arrangements of the universe were full of people living happily ever after, then most things an AI could want would probably also be full of people living happily ever after.

• 3:44:15Speaker 01

  Right.

• 3:44:16Speaker 00

  Is that the the?

• 3:44:17Speaker 01

  Yeah, yeah, right. I mean, so but but the the issue that I see is, you know, in this in this scenario, where, you know, one thing is there is a state of the world that the AI has somehow by whatever method mechanism imagined that is the state of the world that's trying to tighten itself up to get to. Right. And that that your your point is that the the critical feature, as far as I think, as far as you're concerned about AI technology, is that it is something which is trying to optimize purpose. It's trying to optimize its way to achieve a purpose. It's purpose is defined, and it's trying to optimize getting there.

• 3:44:53Speaker 00

  It's it's steering. It's outputting actions such that there is an eventual downstream consequence.

• 3:44:59Speaker 01

  Right. But but then then you you're going to you're going to get several steps. Then the next step is purposes that it gets for itself are ones unforeseen by us. And therefore, there's absolutely no reason to think that those purposes will be aligned with things that are good for humans.

• 3:45:17Speaker 00

  Yeah, this is basically just. People running into the curse of Murphy's law upon computer security designers on rocket probe engineers. Murphy's has all these little curses. The curse of Murphy upon people who build rockets is that there's very extreme forces inside the rocket. So if you make one little tiny mistake, boom. The curse of Murphy upon people who build space probes is that once it's high and in orbit and heading towards Mars, if you've made a mistake, you can't just like hop over to it and correct it. So, you know, you screwed up your metric and imperial units, like happened to, I believe, that one Mars probe. Now, too late. If your screw up also destroyed your error recovery mechanism, which a lot of screw ups do, you can't just

• 3:46:02Speaker 00

  hop out and build it. That's Murphy's curse upon the builders of space probes. There's Murphy's curse upon the people who try to build secure operating systems, which is that some smart person is going to be like looking at all the passes they can imagine the system taking and intelligently searching out some weird path that does what they want, which is perhaps not what you wanted. So if you imagine like, you know, and then there's Murphy's curse of ruin, which is if you screw up this problem, you're not going to get a second chance to bet because you lost all your money or you're dead.

• 3:46:36Speaker 01

  But I think, I think you're saying that sort of my explorations of the computational universe, you're not worried about those because somehow even if those things were connected to real actuation in the world, you're sort of not worried because they don't have this additional piece that is optimized for, you know, optimized to get to that purpose.

• 3:46:56Speaker 00

  Right.

• 3:46:57Speaker 01

  I think you're saying.

• 3:46:58Speaker 00

  Yeah, like the strange cellular automata don't contain a echo and model of reality that lets them map out to having a lot of money or having a lot of paper clips. They don't do the human scary thing.

• 3:47:10Speaker 01

  No, but, but the thing, the point that you're making is that as soon as we have sort of adaptive evolution, as soon as we have the possibility of optimization, that's the thing that you think is dangerous. It's not just random computation going along. It's that inside this computation is an optimization loop that is basically trying to get to whatever objective it might have in the most efficient possible way.

• 3:47:35Speaker 00

  I'm not worried as soon as that exists, that exists in a gradient descent system running right now. I'm worried about it getting more powerful, like smarter than us. It has a better model of reality. It knows about more of reality. It is a more effective searcher. It is a more effective planner. It can steer harder than we can counter steer.

• 3:47:55Speaker 01

  Right, right. So one question is, how much does computational irreducibility bite you in terms of the ability to actually do this optimization? In other words, the current optimization that's being done is pretty coarse. You know, when, when you run a machine learning system and it finds these rocks that it puts together to assemble them into the wall, you know, it's getting, it's good enough, but it's not unbelievably tight.

• 3:48:19Speaker 00

  What kind of technology do you imagine humanity would have a million years from now?

• 3:48:26Speaker 01

  I don't think humanity is going to be around a million years from now in its current.

• 3:48:29Speaker 00

  If there's some form of intelligence around a million years from now, what kind of technology does it have? That's how much room there is to do better than human.

• 3:48:37Speaker 01

  Sure.

• 3:48:38Speaker 00

  Better than our current 2024 technology.

• 3:48:39Speaker 01

  We might ask ourselves, what is technology? So technology is, is taking, you know, things from the world and somehow sort of arranging them for what we imagine are human purposes. That's been the traditional definition of technology. Do we have a different techni- definition of technology when we don't have humans around?

• 3:49:01Speaker 00

  It's the pieces that you arrange, like little subparts of the universe on the way to where you're going. For example, you might build a Dyson sphere so you can harvest a bunch of energy, or rather, I should more precisely say neg-entropy, but I'm just going to keep saying energy anyways. You harvest a bunch of energy and then you use that energy to make paper clips or, you know, figure out more digits of pi or, you know, run people who are conscious and having fun and being nice to each other. So the Dyson sphere, you know, like, and most possible ways you can obtain, most possible ways you can arrange a bunch of matter aren't going to feed you the energy that you need to run your computer. So you need a very precise and exact.

• 3:49:45Speaker 00

  Narrow improbable arrangement of matter to feed the energy to your computers. And that's the technology of the Dyson sphere.

• 3:49:50Speaker 01

  But but you know, you know, a pulsar has this presumably this iron, iron fifty six coat around it. And it has all kinds of things going on inside. Now, you might say that's amazing technology. It's got this, you know, iron casing around the pulsar. And it has all this superfluid, you know, neutron matter inside and so on. You've got, you know, isn't that amazing technology that the pulsar has? But actually, we don't think of that as technology. We think of that as just a feature of the natural world. What makes something technology is that we can sort of, we can pull it in to human purposes. It's not technology when it's, when it's just.

• 3:50:31Speaker 00

  Yeah, I feel like this has the obvious generalization to alien purposes.

• 3:50:37Speaker 00

  Now, in the limit, you can always have the imaginary alien who looks at a river and just wants that one exact river to be exactly the way that it is. And they have no need for to select, to plan, to design, to like knock other pieces of reality together to make tools, to make tools, to make the river. The river that they have is the optimal river and they don't need technology. But you know, most aliens like that are going to die starving to death, looking at the river because they want to build a farm.

• 3:51:05Speaker 01

  Let's take the great red spot on Jupiter. Okay, which probably had many different steps that had to be gone through in the atmosphere of Jupiter to end up with that red spot. Okay. And now we say, you know, is that red spot technology or is it merely not nature?

• 3:51:23Speaker 00

  Merely nature.

• 3:51:23Speaker 01

  Argue, you think it's nature.

• 3:51:26Speaker 00

  I sure do.

• 3:51:27Speaker 01

  Okay, but let's imagine that you are an alien who whose great sport is going around in circles. You know, at high speed in the upper atmosphere of a gas giant planet. Okay, then, then that thing is a, you know, you could think of it as a wonderful piece of technology. That happens to be provided to you by nature. All technology is provided to us by nature. It's we don't get to, you know, when we have our, you know, magnet or our liquid crystal or whatever else. Those are things which are provided by nature, but which we manage to harness for some human purpose.

• 3:52:04Speaker 00

  So I'm just gonna say the obvious thing and say that if the alien then installs a bunch of engines in Jupiter's atmosphere to make the red spot swirl even faster so it can go surfing, it has now installed technology in the form of engines and the larger red spot that incorporates these, these engines is, you know, could, could be defined as technology, or might say it's like a feature of nature with some technology bolted on to it, but.

• 3:52:27Speaker 01

  Okay, so, so I'm gonna say there are, you know, things like jet stream type, type wind patterns that, you know, that the red spot is doing its thing and then some jet streams arrive that manage to, you know, get it to go. I, I don't think, you know, I doubt you would ever recognize an alien engine. I doubt you'd recognize it. The alien is some kind of fluidic intelligence. It's some, you know, I don't think you can recognize an alien engine. I think it's deeply human to say, you know, it's an engine. That's something where you're imposing kind of the, the human version of it.

• 3:53:02Speaker 00

  I bet the aliens are moving the entropy around. I bet they're like, they occasionally make things hot and then like transfer the heat to someplace else that they want heat. I don't, I don't.

• 3:53:12Speaker 01

  You mean like, like jets from, from galaxies that are from black holes where the, the black hole is so absolutely trying to transfer its energy to out into the universe?

• 3:53:23Speaker 00

  The reason why we've got these skyscrapers is that although you can live in a cave, there's places we'd rather live even than caves. So we went around rearranging things. And if there are any aliens out there who are just perfectly content with their caves, they imagine skyscrapers and would be like, no, I'd rather I live in the cave. And they're like that about every single thing. Then we're not, we're not, we're not gonna meet them. We're gonna, maybe someday we'll go to them, but they ain't coming to us.

• 3:53:52Speaker 01

  You know, this whole question about what's, what's for a purpose and what's not, I think is very slippery. And I, I kind of suspect, you know, I, I think I'm understanding your argument, which I'd not done before. So I, I've not studied your, so, so there's been super interesting to me to understand what, I mean, I think it's an interesting argument. I, I claim that there are, you know, it's certainly not self-evident that this argument is like, oh, we should all buy, you know, AI insurance, whatever good that will do us.

• 3:54:21Speaker 00

  That's not gonna do any good.

• 3:54:24Speaker 01

  But, you know, it's, uh, the, the kind of.

• 3:54:29Speaker 01

  You know, you're, the way I'm understanding it is, and it's kind of interesting that this is a sort of a close analogy in biology. You're saying the thing that is really troublesome, I think, is this tightening, this optimization that happens. But if it wasn't for that, if the computation was just doing what the computation does, we'll be okay. But as soon as you do this kind

• 3:54:53Speaker 00

  It wouldn't be profitable. It would be like a random computation that wasn't doing anything that OpenAI could sell. That's why they throw out the initial state of the neural network and

• 3:55:02Speaker 01

  Right.

• 3:55:02Speaker 00

  use gradient descent to make it do things that they want it to do more than just, you know,

• 3:55:07Speaker 01

  The humans want, right. But somewhere inside, we think there are. So an interesting case here is, you know, artificial biological evolution. So for example, you know, famously, you know, gain of function viruses, all those kinds of things where you're, where you're running it through many, many generations of artificial selection. And it's sort of interesting, perhaps, that the, the kind of, the, the place where you're seeing trouble with the AIs is actually strangely similar to the place where one might see trouble with, with things that one can do in biology, too. That is

• 3:55:43Speaker 00

  I mean, if you, if you grind the black box on biology hard enough, it might succeed in wiping us out, but it doesn't have quite the same crushing sense of ruin that would be associated with facing down something smarter than you. I am not as scared of the viruses. Somebody should probably be worried about. It could just kill us all before we manage to kill us all.

• 3:56:05Speaker 01

  Let me give an analogy. Some part of your Right. So some, some part of your thinking is happening in your brain. There's also another pretty elaborate thoughtlike process that happens in your immune system. As the immune system tries to figure out, you know, it has, you know, these things, the, the T cells are interacting with each other and doing all kinds of things that we mostly don't understand.

• 3:56:30Speaker 00

  Doing optimization.

• 3:56:32Speaker 01

  Yeah, it's it's it's doing things that that are, you know, in detail different for brains, but it's still doing something that's kind of a little bit intelligence brain like. And and so what what what is being said is, as you say, let's let's, you know, let's use technology to make a virus that is more and more difficult for our poor fixed immune system, where we're stuck with the immune system that biology gave us. You know, there is an argument there that says if you do that, if we if we take it outside of the, you know, if we if we run things to make this virus that's incredibly efficient and our poor immune system doesn't stand a chance. And I think that's a, you know, that's a case that I think is not so different from what you're saying, what you're imagining the AI case.

• 3:57:17Speaker 00

  I think in particular, if humans weren't allowed to fight back, like using their own intelligence, and instead you just had somebody built like, if instead you just had a system that like, built viruses much more intelligently than the amount of intelligence than the than that the human immune system is allowed to use. If you used AlphaFold four and started designing viral features that were, you know, not just there to kill individuals, but there to kill groups and entire regions. And, you know, you just put that up against the human immune system, which is not quite static, but only shuffles itself reshuffles itself once a generation. Yeah, it could do some damage that way. And I agree that that's like a, you know, smaller, lesser miniature version

• 3:58:01Speaker 00

  of the problem of facing down a superintelligence using your own brain, which does get reshuffled every every generation. You can use various external aids, but none of it's going to match up to the superintelligence.

• 3:58:11Speaker 01

  Yeah, well, just like the immune system, you can use various external aids. You get vaccines, you do all kinds of things like that, but you don't. And it has to fight its battle at a molecular scale, which is a little different than, you know, we we do not have, you know, our brains don't actually allow us to go to battle on the molecular battlefields.

• 3:58:29Speaker 00

  I, yeah, but, you know, they do allow us to invent AlphaFold three. And, you know, if if if it comes down to a contest of the human brain of the people trying to have the humans live and the the brain running the death cult, trying to build the super virus, that does not quite feel to me like there's a higher level of this game being played out by two systems that are both smarter than the immune system and the artificial evolution closed loop on the virus.

• 3:58:57Speaker 00

  So it doesn't, I don't quite get the same crushing sense of doom and ruin off of it.

• 3:59:02Speaker 01

  I think we're not going to solve the problem of whether the species is going to wipe out, get wiped out here, even though the, the, but I am, but, but I do feel like I've understood more about what your argument is. And I, you know, on, you know, in real time, I can't take it apart and decide, you know, do I agree with it? And am I going to, you know, say you got to, or not? I mean, I feel like there are, I feel like my intuition is that there are some, some kind of, just like you say, there are unintended things that will happen in the AI. I think there are unintended things that happen as you try and tighten up this argument. That is my claim.

• 3:59:43Speaker 00

  Well, why didn't the, the conclusion I would, I would end with is like, you know, you can imagine the Native Americans trying to cut, trying really hard to come up with arguments, why like the, the ships they see approaching can't possibly hurt them, you know, like, well, what could they want? Can we really define a language over what they want? Maybe most things that they want have us be alive. And these people were in a much more favorable situation that they would have had a much easier time of retrieving.

• 4:00:13Speaker 01

  Do I think there's a risk? I think there's a risk. Do I think that risk is such that, you know, do I right now think that risk is, I mean, anything one does in life has, you know, there's risk, there's risk in all kinds of things that go on. And, you know, we humans, you know, sort of believe in kind of moving forward independent of the risks. And I, I think it's, it's kind of like, is this such a kind of, you know, do I immediately think this is such a looming risk that it's like, change everything? It's kind of like, I remember when people, you know, people, somebody was telling me years ago, when, when they, when they're kind of.

• 4:00:56Speaker 01

  About, you know, when people are much less worried than today about climate and CO2 emissions, things like that, they were saying, you can really reduce your, you know, your, you know, you've got to really reduce your, whatever it was, I don't know, energy use or whatever. And it's like, you can do it. It's doable. And I said, well, what does it take to do it? And they said, well, you couldn't have a computer, you couldn't have this, you couldn't have that. You know, it's like, well, yes, if I, you know, if I spent my life standing on my head, then I would probably have, you know, wouldn't have swelling my feet type thing, or whatever it is. You know, then, in other words, there's a, there's a risk, and there's a what you have to do to avoid that risk, and what the cost of

• 4:01:37Speaker 01

  avoiding the risk is. And I guess my own, you know, I don't, I don't know, you know, you make some interesting arguments. So it's kind of like, my own, you know, immediate intuitive senses, yes, there's a risk, you know, is that risk something that will cause me to turn my life upside down? Not proven to me at this point.

• 4:02:02Speaker 00

  Okay. I mean, from my, from my perspective, I've been like, forests on fire, and you're like, well, what is fire exactly? And the thing is being, being like, well, like, can we, like, is there a particular exotic set of preferences which would make this river exactly the optimal river? And, you know, like, can you view the river as, you know, throwing up a water spout? You know, this, this does not prevent an AI from, you know, building its own infrastructure and then killing you. It wouldn't have protected the Native Americans from the, from the, like, much more similar to them, Europeans that were coming toward their shore to say, like, well, you know, can we really have a language for describing all the things that they might want? Isn't this language a very complicated

• 4:02:44Speaker 00

  sort of thing? This, there's, there's a lot of, like, delightful philosophical issues here, but I think they mostly integrate out of the, of the actual scenario here.

• 4:02:52Speaker 01

  So that's the question is, is, you know, In my life, people have had different scenarios for how the world will end. Whether when I was a kid, it was mostly, you know, you know, World War III type scenarios. Then later on, I mean, for some generation, it's the world's going to burn up with, you know, climate change. And then there's, you know, there's these various scenarios for how things. And I think one of the things that, you know, there's the question of, and sometimes there are things where people say, but, but the, or, or for example, when people were saying, you know, build the, build the Large Hadron Collider and it will create a black hole that will destroy the world. That was a, that was a thing people raised that possibility.

• 4:03:33Speaker 00

  We understood the relevant laws. We ran the calculations. We worked out that, we worked out multiple constraints from multiple angles, saying the probability was tiny. And in other times in, in human history, people have warned about leaded gasoline poisoning the soil. And people have been like, ah, you know, don't, don't be, you know, these like, don't be a worry wart, it'll be fine. And then, you know, we've kind of poisoned a generation and dealt permanent, you know, dealt lifelong brain damage to a bunch of kids growing up with lead, lead in their soil and.

• 4:04:03Speaker 01

  I mean, I'm, I'm most, I'm particularly reminded of the stories people have told me about the Manhattan Project, where they were going to detonate the first, you know, do the first nuclear weapons test. And the question was, would it ignite the atmosphere?

• 4:04:15Speaker 00

  LH 602 was the paper they wrote to analyze that. It's interesting reading. They again had multiple angles from which to look at it and say, that will definitely not happen. We do not have this with AI. It is more of a leaded gasoline situation.

• 4:04:29Speaker 01

  I didn't think they'd written it up. That must have been much later because I think it was, you know, what I heard was from people who were sort of involved, who were like doing back of the envelope calculations and saying it's really far away from, from igniting the atmosphere. But, but.

• 4:04:42Speaker 00

  I think they did back of the envelope calculations and then they, you know, like did front of the envelope calculations, which, you know, good for them.

• 4:04:49Speaker 01

  Well, but okay, but so, so the question now is, you know, in that case, it all worked out, didn't ignite the atmosphere. Um, what you're saying is that, you know, we can't do the calculations for AIs and you're saying, or you think you have done the calculations for AIs and they will ignite the atmosphere.

• 4:05:09Speaker 00

  Yeah. Back of the envelope. If this is, this is not like a rigorous calculation. And it's not going to be a rigorous calculation before the world ends because this is more like, this is more like tangled up biochemistry and less like. Yeah, straightforward physics.

• 4:05:24Speaker 01

  So the concern is your back of the envelope calculation says AIs will do the analog of ignite the atmosphere. And I guess my, you know, one feature of back of the envelope calculations is they require intuition. They're not things where you can, you know, if you do some very rigorous thing with axioms and you can go step by step, it doesn't really matter if you have good intuition or not. It's just mechanical. But back of the envelope requires intuition.

• 4:05:51Speaker 00

  Good thing that reality, since the moment of its dawn, has followed a hard and fast rule. If you cannot do rigorous formal calculations about something, it is not allowed to kill you. Just nobody in the history of time has ever once had a cause of death that would that required intuition to understand.

• 4:06:09Speaker 01

  That's not what I'm saying. What I'm saying is if we're trying to make a, you know, we've got two, two possibilities. One is, you know, we shut down our lives or whatever and we say, you know, we're not going to have this, that and the other because it, because there's a risk that it kills us or because we think it will kill us, but it will cause us all kinds of trouble to shut all this stuff down. And maybe it's even impractical to do it, but just it's, it's something which, which is a cost to us to shut all this stuff down. And so we've got, you know, door number one is we shut everything down and then we're sure it won't kill us. And door number two is we're sufficiently worried that it will kill us that, you know, we're, we're, you know, one thing we're sufficiently worried it will kill us, so we shut all that stuff down.

• 4:06:54Speaker 01

  The other thing is we don't shut that stuff down. We get all the benefit, all the known benefit of being able to do those things, but we have some risk that it will kill us. And so the thing that I think is a, the rational thing to do is to say, let's try and tighten up those back of the envelope intuitions, so to speak, and see where we come out. And, you know, that, that's a, to me, the thing that, you know, I think we both agree that there's a lot about kind of how AI really works and what it's really doing that we don't understand. And I understand your analogies about, you know, the Native Americans or whatever else.

• 4:07:31Speaker 01

  To me, that requires a boatload of intuition, which I don't think I have. Now, you may have it and be absolutely right. And, you know, then, you know, it's kind of like, I just don't have that intuition. And, you know, the intuition that I have is perhaps, you know, I have a little bit more than everyday intuition because I have different intuition about what computational systems do because I've sort of lived with them for a long time. But I don't have, I don't claim to have the kind of intuition that would allow me to tell, does this back of the envelope calculation that you're talking about, you know, does that land in a, oh, gosh, you know, it's going to kill us all? Or does that land in a, oh, actually, you didn't think of this thing and that thing and the other thing?

• 4:08:15Speaker 01

  And the, you know, the waterspout detail actually derailed the intuition and we got the wrong answer. So that, that's, that's kind of where I feel I'm at.

• 4:08:26Speaker 00

  Yeah, so we're now sort of like going into politics. And I, the, the case I would make to the politician is at the point where the most allegedly credible expert who recently won the Nobel Prize about it says like, well, personally, I'm over fifty percent. Like my sort of first order personal assessment is over fifty percent existential catastrophe. That's, that's code for killing everyone. But, you know, like taking into account what other people are saying, I would say in public more like ten to twenty percent. And the people who've been studying the issue longest are like, yeah, this kind of looks to us like it straightforwardly kills you. Like, why would you even expect that not to happen? That's us.

• 4:09:07Speaker 00

  And other people are going like, there is no danger here. This is ridiculous. The people talking about it are stupid.

• 4:09:14Speaker 00

  And, you know, from a political standpoint, I think what you ought to do at least then is start getting start, getting started on preserving the option of shutting it down, which is easier to do, which would have been easier to do in twenty twenty two than twenty twenty four. And will be easier to do in twenty twenty five than twenty twenty seven if we're still alive in twenty twenty seven. Actually, if we're not alive in twenty twenty seven, it's still easier to do it in twenty twenty five.

• 4:09:36Speaker 01

  One thing that I will agree with is thinking about these things is worthwhile. And, you know, to have nobody thinking about it is a mistake. You know, if there is a high risk, nobody thinking about it is the wrong thing.

• 4:09:52Speaker 00

  People who think about this for a while do tend to assume, do tend to start agreeing that the default outcome is everybody dying. Sometimes they think they have clever plans for presenting that, but they do tend to follow along with the default outcome is everybody dead.

• 4:10:06Speaker 01

  See, I think though, unfortunately, there are many selection effects at work here. I mean, you know, many of us, you know, I will say that I am intrinsically an optimist. I don't know how you feel about yourself. Are you intrinsically an optimist or a pessimist?

• 4:10:21Speaker 00

  I tried to, I tried to do the, you know, like the best calibrated, best discriminating probabilities I can manage. If you, I think if you explicitly say that you're being an optimist or a pessimist, you have clearly departed the way of truth. You have confessed that you are now taking considerations other than truth and to account in your statements. How sinful, how unvirtuous.

• 4:10:40Speaker 01

  Well, for me, you know, what do I mean by that? I mean, I tried to do projects that many people would say, oh, that's an impossible project. The fact that there's not a matter of truth there. It's like this project is hard, but I'm going to be optimistic that it's possible. Rather than, so, you know, I think that that's to say, you know, I don't do projects, which I think are obviously doomed. But on the other hand, I'm going to take the point of view, let me try it rather than, oh gosh, it's never going to work. Let me not try it. So that's, that's kind of what I mean.

• 4:11:16Speaker 00

  That is absolutely how I spent the years from like 2001 to 2021 or thereabouts. Well, 2020 maybe was, you know, like, all right, I'm going to run at the alignment problem. But it became kind of clear that, you know, this wasn't going to work for me. It wasn't going to work for the other people, people working on it. And that the field itself had kind of like failed to. Form a process that could distinguish that that that whereby that could like publishly distinguish Elon Musk's, we will just make Grok to pursue truth. And like even the old science fiction writers understood that humans may not be the most efficient way of producing truths and anything that just produces lots of truths may not

• 4:12:00Speaker 00

  produce other things of, you know, the best possible value. But, you know, like, who tells Elon this that he believes? And yes, there's selection effects, like, um, people did, in fact, pay me to work on this problem. And, you know, you're, you're hearing me because people paid me to work on this problem and I didn't just like starve. And the people who founded OpenAI, where people selected to believe that alignment was totally a problem within their grasp, or and or to be willing to just take Elon Musk's money and run with it. And if you want to look for people who are not selected, I think you end up with Geoffrey Hinton, you know, the guy who just won the Nobel Prize for the work he did on machine learning, then it, you know, is kicking off the whole modern deep learning revolution. That's the guy who isn't obviously selected.

• 4:12:45Speaker 00

  That's the guy saying like, well, personally 50%, but if I take into account what other people are saying, 10 to 20%, I really need to actually sit down and talk with him at some point, which we've never actually done, and try to talk him out, out, out of listening to those particular other people. I think their arguments make no sense.

• 4:13:00Speaker 01

  You know, it really, it damages my thinking about this, that the various people you mentioned, you know, I, I've known all these people sometimes for a very long time. So it's kind of, it, it kind of throws a wrench in my kind of, you know, there is, there is sort of, you know, none of us have, I think, you know, perfect sort of calibrated rationality about everything. And it's, it's kind of, you know, it's always challenging to, you know, this, this thing about, you know, I consider myself a convincible person in terms of your argument. Okay, I'm not, you know, I don't have a, you know, I'm not about to say, oh, you know,

• 4:13:44Speaker 01

  I'm going to change my life, you're right, without understanding it. But I'm, I'm convincible. So I'm not, and I don't know how many people are. Maybe there are many people who are not, but I think I am, I'm not convinced yet. But that's, you know, I mean, like this conversation, I understand much more about what you're saying. And it's, you know, I think there are, to me, there are interesting questions to try to answer, which people should try and answer. I mean, you make it sound like it's urgent, you know, the people are coming off the ships and you might be right. You might be right. And

• 4:14:21Speaker 00

  I mean, I'm watching things go downhill in terms of how much you'd have to spend and how much pain it would take to get to like deproliferate the technology. And, you know, it would have been easier.

• 4:14:34Speaker 01

  I don't think it's realistic. I don't think it's ever been

• 4:14:36Speaker 00

  It would cost less than the Persian Gulf War if we did it today. And even if we like, and if we do it later, maybe it costs more like World War II. But humanity did not lie down and die when the alternative was fighting World War II. We went off and fought World War II. And if everybody's going to die otherwise, you just do what it takes.

• 4:14:53Speaker 01

  Yeah, but the fact is, there are things that one can deproliferate, like nuclear weapons, for example, because the supply chain is really complicated. There are ideas. It's very hard to deproliferate ideas.

• 4:15:06Speaker 00

  Yep. If it were the case that a single person on Earth having the idea in their mind of artificial superintelligence would cause everyone to die, that would be an even worse situation to be in than the one we are in right now. I might, in that case, well despair. Or like, try to do something like weirder and sillier. I don't think we're in that situation. I don't think we need to deproliferate the idea.

• 4:15:31Speaker 01

  Okay.

• 4:15:32Tim Scarfe?

  Gentlemen, thank you so much. I'm gonna call it now. I think this might be the best conversation in MLST history.

• 4:15:41Speaker 00

  Thanks.

• 4:15:42Tim Scarfe?

  And part of that, and our purpose is about curating conversations of this quality level with people of your, you know, caliber. And I, I was expecting this to be a bit of a kind of, I don't want to say chat GPT conversation, but talking past each other. And the authentic exchange that you've both just had is really mind blowing. So thank you so much.

• 4:16:06Speaker 00

  Thank you.

• 4:16:06Speaker 01

  Thanks. Thanks for setting it up. This was interesting. Eliezer, nice to see you. Hopefully, Will, I think it's been like 15 years since we saw each other in person.

• 4:16:17Speaker 00

  Probably, yeah.

• 4:16:20Speaker 00

  So you in another 15 years? No, I could, I could.

• 4:16:23Speaker 01

  Oh, no, 15 years, according to you, it's all over.

• 4:16:27Speaker 00

  Well, yeah, but all, but all, like, it would be great to still be around in 15 years. But that said, I'm not saying that we like shouldn't talk again for another 15 years. I'm not saying this was the optimal interval or anything, you know?

• 4:16:38Speaker 01

  Yeah, right, right, right. And we both, I think, are interested in cryonics and things. And maybe we get to talk in 300 years and we get to say, you know, this was the, uh...

• 4:16:46Speaker 00

  We get to, we get to resolve all the bets if we make it through there, though, you know, conditioning on that possibility, probably you won a bunch of arguments.

• 4:16:55Speaker 01

  Right, right.