Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality

For 4 hours, I tried to come up reasons for why AI might not kill us all, and Eliezer Yudkowsky explained why I was wrong. We also discuss his call to halt AI, why LLMs make alignment harder, what it would take to save humanity, his millions of words of sci-fi, and much more. If you want to get to the crux of the conversation, fast forward to 2:35:00 through 3:43:54. Here we go through and debate the main reasons I still think doom is unlikely. Watch on YouTube https://youtu.be/41SUp-TRVlg. Listen on Apple Podcasts https://podcasts.apple.com/us/podcast/eliezer-yudkowsky-why-ai-will-kill-us-aligning-llms/id1516093381?i=1000607719339, Spotify https://open.spotify.com/episode/7LFWgn4KvoZMn4a89ZXUE3?si=mySWbwlWTr-NfkRzQbzA4w, or any other podcast platform. Read the full transcript here https://www.dwarkeshpatel.com/. Follow me on Twitter https://twitter.com/dwarkesh_sp for updates on future episodes. Timestamps (0:00:00) - TIME article (0:09:06) - Are humans aligned? (0:37:35) - Large language models (1:07:15) - Can AIs help with alignment? (1:30:17) - Society’s response to AI (1:44:42) - Predictions (or lack thereof) (1:56:55) - Being Eliezer (2:13:06) - Othogonality (2:35:00) - Could alignment be easier than we think? (3:02:15) - What will AIs want? (3:43:54) - Writing fiction & whether rationality helps you win This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com https://www.dwarkesh.com?utm_medium=podcast&utm_campaign=CTA_1

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Eliezer Yudkowsky?

  Misaligned, misaligned. No, no, no, not yet, not now. Nobody's been careful and deliberate now. But maybe at some point in the indefinite future, people will be careful and deliberate. Sure, let's grant that premise. Keep going. If you try to rouse your planet, there are the idiot disaster monkeys who are like, ooh, ooh, like, if this is dangerous, it must be powerful, right? I'm going to be first to grab the poison banana. And it's not a coincidence that I can zoom in and poke at this and ask questions like this and that you did not ask these questions of yourself. You are imagining nice ways you can get the thing, but reality is not necessarily imagining how to give you what you want. Should one remain silent? Should one let everyone walk directly into the whirling racer place?

• 0:45Speaker 01

  Like continuing to play out a video game you know you're going to lose because that's all you have.

• 0:51Dwarkesh Patel?

  OK, today I have the pleasure of speaking with Eliezer Yudkowsky. Eliezer, thank you so much for coming on The Lunar Society.

• 1:00Eliezer Yudkowsky?

  You're welcome.

• 1:01Dwarkesh Patel?

  First question, so yesterday when we're recording this, you had an article in Time calling for a moratorium on further AI training runs. Now, my first question is, it's probably not likely that governments are going to adopt some sort of treaty that restricts AI right now. So what was the goal with writing it right now?

• 1:24Eliezer Yudkowsky?

  I think that I thought that this was something very unlikely for governments to adopt. And then all of my friends kept on telling me like, no, no, actually, if you talk to anyone outside of the tech industry, they think maybe we shouldn't do that. And I was like, all right, then. Like, I assumed that this concept had no popular support. Maybe I assumed incorrectly. It seems foolish and to lack dignity to not even try to say what ought to be done. There wasn't a galaxy-brained purpose behind it. I think that over the last 22 years or so, we've seen a great lack of galaxy-brained ideas playing out successfully.

• 2:04Dwarkesh Patel?

  Has anybody in government, not necessarily after the article, but just in general, have they reached out to you in a way that makes you think that they have the broad contours of the problem, correct?

• 2:14Eliezer Yudkowsky?

  No, I'm going on reports that normal people.

• 2:20Eliezer Yudkowsky?

  Are more willing than the people I've been previously talking to to entertain calls. This is a bad idea. Maybe you should just not do that.

• 2:30Dwarkesh Patel?

  That's surprising to hear because I would have assumed that the people in Silicon Valley, who are weirdos, would be more likely to find this sort of message. They could kind of rocket the whole idea that nanomachines will, AIs will make nanomachines that take over. It's surprising to hear the normal people got the message first.

• 2:47Eliezer Yudkowsky?

  Well, I hesitate to use the term midwit, but maybe this was all just a midwit thing.

• 2:53Dwarkesh Patel?

  All right. So my concern with, I guess, either the six-month moratorium or forever moratorium until we solve alignment is that at this point, it seems like it could, do people seem like we're crying wolf? And actually, not that it could, but it would be like crying wolf because these systems aren't yet at a point I wish they're dangerous.

• 3:14Eliezer Yudkowsky?

  And nobody is saying they are. Well, I'm not saying they are. The open letter signatories aren't saying they are. I don't think.

• 3:20Dwarkesh Patel?

  So if there is a point I wish we can sort of get the public momentum to do some sort of stop, wouldn't it be useful to exercise it when we get a GPT-6 and who knows what it's capable of? Why do it now?

• 3:32Eliezer Yudkowsky?

  Because allegedly, possibly, and we will see, people right now are able to appreciate that things are storming ahead and a bit faster than the ability to, well, ensure any sort of good outcome for them. And, you know, you could, you could be like, ah, yes, well, like, we will like play the galaxy brain clever political move of trying to time when the popular support will be there. But again, I heard rumors that people were actually like completely open to the concept of let's stop. So again, just trying to say it. And it's not clear to me what happens if we wait for GPT-5 to say it. I don't actually know what

• 4:16Eliezer Yudkowsky?

  GPT-5 is going to be like. It is, it has been very hard to call.

• 4:23Eliezer Yudkowsky?

  Like rate at which these cap at which these systems acquire capability as they are trained to larger and larger sizes and more and more tokens. And like GPT four is a bit beyond in some ways where I thought this paradigm was going to scale period. So I don't actually know what happens if GPT five is built. And even if GPT five doesn't end the world, which I agree is like more than 50% of where my probability mass lies. Even if GPT five doesn't end the world, maybe it's get maybe that's enough time for GPT four point five to get ensconced everywhere and in everything and for it actually to be harder to call a stop, both politically and and technically.

• 5:05Eliezer Yudkowsky?

  The there's also the point that training algorithms keep improving. If we put a hard limit on the total computes and training runs right now, these systems would still get more capable over time as the algorithms improved and got more efficient, like more oomph per floating point operation. And things would still improve, but slower. And if you start that process off at the GPT five level, where I don't actually know how capable that is exactly, you you may have like a bunch less lifeline left before you get into dangerous territory.

• 5:46Dwarkesh Patel?

  The concern is then that, listen, there's, you know, millions of GPUs out there in the world. And so the actors would be you who would be willing to cooperate or who could even identify in order to even get the government to make them cooperate would be potentially the ones that are most on the message. And so what you're left with is a system where they're they, you know, they stagnate for six months or a year or however long this lasts. And then what is a game plan? Like, is there some plan by which if we wait a few years, then alignment will be solved? Do we have some sort of timeline like that? Or?

• 6:18Eliezer Yudkowsky?

  Alignment will not be solved in a few years. I would hope for something along the lines of human intelligence enhancement works. I do not think we're going to have the timeline for genetically engineering humans to work, but maybe this is why I mentioned the time letter that if I had like infinite capability to dictate laws that there'd be a carve out on biology, like AI that is like just for biology and not trained on text for the from the internet. Human intelligence enhancement, make people smarter. Making people smarter has a chance of going right in a way that making a extremely smart AI does not have a realistic chance of going right at this point. So, yeah, that would, in terms of like remotely,

• 6:59Eliezer Yudkowsky?

  you know, how do I put it? If you were on, if we were on a sane planet, what the sane planet does at this point is shut it all down and work on human intelligence enhancement. It is, I don't think we're going to live in that sane world. I think we are all going to die. But having heard that people are more open to this outside of California, it makes sense to me to just like try saying out loud what it is that you do in a saner planet and not just assume that people are not going to do that.

• 7:30Dwarkesh Patel?

  In what percentage of the worlds where humanity survives is there human enhancement? Like even if there's 1% chance humanity survives, is basically that entire branch dominated by the worlds where there's some sort of?

• 7:39Eliezer Yudkowsky?

  I mean, I think we're, we're just like mainly in the territory of hail mail, hail mary passes at this point. And human intelligence enhancement is one hail mary pass.

• 7:53Eliezer Yudkowsky?

  Maybe you can put people in MRIs and train them using neurofeedback to be a little saner, to not rationalize so much. Maybe you can figure out how to have something light up every time somebody is like working backwards from what they want to be true to what they take as their premises. Maybe you can just like fire off a little light and teach people not to do that so much. Maybe the GPT four level systems can be reinforcement learning from human feedback into being consistently smart, nice and charitable in conversation and just unleash a billion of them on Twitter and just have them like spread sanity everywhere. I do not think this, I do worry that this is like not going to be the most profitable

• 8:38Eliezer Yudkowsky?

  use of the technology. But, you know, you're asking me to list out here hail mary passes, so that's what I'm doing. Maybe you can actually figure out how to.

• 8:48Eliezer Yudkowsky?

  Take a brain, slice it, scan it, simulate it, run uploads and upgrade the uploads or run the uploads faster. These are also quite dangerous things, but they do not have the utter lethality of artificial intelligence.

• 9:05Dwarkesh Patel?

  All right, that's actually a great jumping point into the next topic I want to talk to you about, orthogonality. And here's my first question, speaking of human enhancement. Suppose you bred human beings to be friendly and cooperative, but also more intelligent. I'm sure you're gonna disagree with this analogy, but I just want to understand why. I claim that over many generations, you would just have really smart humans who are also really friendly and cooperative. Would you disagree with that or would you disagree with the analogy?

• 9:30Eliezer Yudkowsky?

  So the main thing is that you're starting from minds that are already very, very similar to yours. You're starting from minds of which, whom, many of them already exhibit the characteristics that you want. There are already many people in the world, I hope, who are nice in the way that you want them to be nice.

• 9:52Eliezer Yudkowsky?

  There's, of course, it depends on how nice you want exactly. I think that if you like actually go start trying to run a project of selectively encouraging some marriages between particular people and encouraging them to have children, you will rapidly find, as one does in any process of, as one does when one does this to, say, chickens, that when you select on the stuff you want, there turns out there's a bunch of stuff correlated with it and that you're not changing just one thing. If you try to make people who are inhumanly nice, who are nicer than anyone has ever been before,

• 10:31Eliezer Yudkowsky?

  you're going outside the space that human psychology has previously evolved and adapted to deal with and weird stuff will happen to those people. This, none of this is like very analogous to AI. I'm just pointing out something along the lines of, well, taking your analogy at face value, what would happen exactly? And, you know, it's the sort of thing where you could maybe do it, but there's all kinds of pitfalls that you'd probably find out about if you cracked open a textbook on animal breeding.

• 11:09Dwarkesh Patel?

  So I mean, the thing you mentioned initially, which is that we are starting off with basic human psychology that we're kind of fine tuning with reading. Luckily, the current paradigm of AI is, you know, you just have these models that are trained on human text. And I mean, you would assume that this would give you a sort of starting point of something like human psychology.

• 11:31Eliezer Yudkowsky?

  Why do you assume that?

• 11:32Dwarkesh Patel?

  Because they're trained on human text.

• 11:34Eliezer Yudkowsky?

  And what does that do?

• 11:36Dwarkesh Patel?

  Whatever sorts of thoughts and emotions that lead to the production of human text are need to be simulated in the AI in order to produce those themselves.

• 11:44Eliezer Yudkowsky?

  I see. So like, if you if you take a person and like, if you take an actor and tell them to play a character, they just like, become that person. You can tell that because, you know, you know, like, you see somebody on screen playing Buffy the Vampire Slayer. And, you know, that's probably just actually Buffy in there. That's who that is.

• 12:04Dwarkesh Patel?

  I think I think a better analogy is if you have a child and you tell him, hey, be this way, they're more likely to just be that way. I mean, other than like, putting on an act for like, 20 years or something.

• 12:17Eliezer Yudkowsky?

  It depends on what you're telling them to be exactly.

• 12:20Dwarkesh Patel?

  Like, if you just say, it would be nice.

• 12:22Eliezer Yudkowsky?

  Yeah, if you're, but that's not what you're telling to do. You're trying to telling them to play the part of an alien. You'd like, some, some, something with a completely in human psychology, as extrapolated by science fiction authors, and in many cases, you know, like, done by computers, because, you know, humans can't quite think that way. And your child eventually manages to learn to act that way. What exactly is going on in there now? Are they just the alien? Or did they pick up the rhythm of what you were asking them to imitate and be like, ah, yes, I see who I'm supposed to pretend to be. Are they actually the person or are they pretending? That's true, even if you're not asking them to be an alien. You know, my, my parents tried to raise me Orthodox Jewish, and that

• 13:03Eliezer Yudkowsky?

  did not take at all. I learned to pretend. I learned to comply. I hated every minute of it. Okay, not literally every minute of it. I should avoid saying untrue things. I hated most minutes of it. And yeah, like, cause they were trying to show me a way to be that was alien to my own psychology. And the religion that I actually picked up was from the science fiction books instead, as it were, though I'm using religion very metaphorically here, more like ethos, you might say. I was raised with the science fiction books I was reading from my parents' library and Orthodox Judaism, and the ethos of the science fiction books rang truer in my soul.

• 13:46Eliezer Yudkowsky?

  And so that took in the Orthodox Judaism, didn't. But the Orthodox Judaism was what I had to imitate, was what I had to pretend to be, was what the, was the answers I had to give, whether I believed them or not, because otherwise you get punished.

• 14:00Dwarkesh Patel?

  But, I mean, on that point itself, the rates of apostasy are probably below 50% in any religion, right? Like, some people do leave, but often they just become the thing they're imitating as a child.

• 14:11Eliezer Yudkowsky?

  Yes, because the religions are selected to not have that many apostates. If aliens came in and introduced their religion, you'd get a lot more apostates.

• 14:18Dwarkesh Patel?

  Right. But, I mean, I, I, I think we're probably in a more virtuous situation with ML, because you, I mean, these systems are kind of, through stochastic gradient descent, sort of regularized so that the system that is pretending to be something, where there's like multiple layers of interpretation, is going to be more complex than the one that is just being the thing. And, I mean, over time, like, the, the system that is just being the thing will be optimized, right? It'll just be simpler.

• 14:43Eliezer Yudkowsky?

  This seems like an ordinate cope. For, for one thing, you're not training it to be any one particular person. You're training it to switch masks to anyone on the internet as soon as they figure out who that person on the internet is. If, if I put the internet in front of you, and I was like, learn to predict the next word, learn to predict the next word, over and over, you do not just, like, turn into a random human, because the random human is not what's best at predicting the next word of everyone who's ever been on the internet. You learn to very rapidly, like, pick up on the cues of, like, what sort of person is talking? What will they say next? You memorize so many facts that just because they're helpful in predicting that, the next word.

• 15:27Eliezer Yudkowsky?

  You, you learn all kinds of patterns, you learn all the languages, you learn to switch rapidly from being one kind of person or another, as the conversation that you are predicting changes who's speaking. This is not a human we're describing, you are not training a human there.

• 15:42Dwarkesh Patel?

  Would you at least say that we are living in a better situation than one in which we have some sort of black box where you have this sort of Machiavellian, "Fittest Survive" simulation that produces AI? Like, is it at least, this situation is at least more likely to produce alignment than one in which something that is completely untouched by human psychology would produce?

• 16:06Eliezer Yudkowsky?

  More likely, yes. Maybe you're like, it's an order of magnitude likelier, 0% instead of 0%.

• 16:14Eliezer Yudkowsky?

  Getting stuff like "more likely" does not help you if the baseline is like nearly zero. Like, the whole training setup there is producing an actress, a predictor. It's not actually being put into the, into the kind of ancestral situation that evolved humans, nor the kind of modern situation that raises humans, though to be clear, raising it like a human wouldn't help. But like, but yeah, you're like giving it a very alien problem that is, that is not what humans solve, and it is like solving that problem, not the way humans would.

• 16:44Dwarkesh Patel?

  Okay, so how about this? I can see that I certainly don't know for sure what is going on in these systems. In fact, obviously, nobody does. But that, that also goes for you. So could it not just be that even through imitating all humans, it like, I don't know, reinforcement learning works, and then all these other things were trying somehow work, and actually just like being an actor produces some sort of benign, benign outcome where there, there isn't that level of simulation and conniving?

• 17:15Eliezer Yudkowsky?

  I think it predictably breaks down as you try to make the system smarter, as you try to derive sufficiently useful work from it, and in particular, like the sort of work where some other AI doesn't just kill you off six months later. I, I, yeah, like, I think the present system is not smart enough to have a deep, conniving actress thinking, long strings of coherent thoughts about how to predict the next word. But as the syst, the, as the mask that it wears, as the people it's pretending to be gets smarter and smarter, um, I think that at some point, the thing in there that is

• 17:57Eliezer Yudkowsky?

  predicting how humans plan, predicting how humans talk, predicting how humans think.

• 18:03Eliezer Yudkowsky?

  And needing to be at least as smart as the humans and human it is predicting in order to do that. I suspect at some point there is a new coherence born within the system and something strange starts happening. I think that if you have something that can accurately predict, I mean, Eli Azuridkowski, to use a particular example I know quite well. I think that to accurately predict Eli Azuridkowski, you've got to be able to the kind of thinking where you are reflecting on yourself and that if in order to like simulate Eli Azuridkowski reflecting on himself, like you need to be able to do that kind of thinking.

• 18:45Eliezer Yudkowsky?

  And this is not airtight logic, but I suspect there to be a discount factor in the, so like if you ask me to play a part of somebody who's quite unlike me, I think there's some amount of penalty that my, that the, the character I'm playing gets to his intelligence because I'm secretly back there simulating him. And that's even, and that's, and that's even if we're like quite similar and like the stranger they are, the more unfamiliar the situation, the less the person I'm playing is, is as

• 19:27Eliezer Yudkowsky?

  smart as I am, the more they are dumber than I am. So similarly, I think that if you get a, an AI that's very, very good at predicting what Eli Azur says, I think that there's a quite alien mind doing that, that it actually has to be to some degree smarter than me in order to play the role of something that thinks differently from how it does very, very accurately.

• 19:53Eliezer Yudkowsky?

  And I reflect on myself. I think about how my thoughts are not good enough by my own standards and how I want to rearrange my own thought processes. I look at the world and see it going the way I did not want it to go and asking myself, how could I change this world? I look around at other humans, I model them, and sometimes I try to persuade them of things. These are all capabilities that the system would then would then be somewhere in there. And I just like, don't trust the lot, I don't trust the blind hope that all of that capability

• 20:35Eliezer Yudkowsky?

  is pointed entirely at pretending to be Eliezer and only exists insofar as it's like the mirror and isomorph of Eliezer, that all the prediction is like, is by being something exactly like me and not thinking about me while not being me.

• 20:53Dwarkesh Patel?

  I certainly, I don't want to claim that it is guaranteed that there isn't something super alien and something that is against our aims happening within the Shageth. But you made an earlier claim, which seemed much stronger than the idea that you don't want blind hope, which is that we're going from like zero percent probability to an order of magnitude greater at zero percent probability. There's a difference between saying that we should be wary and that like there's no hope, right? Like, I could imagine so many things that could be happening in the Shageth's brain, especially in our level of confusion and mysticism over what is happening. And so, I mean, okay, so one example is like, I don't know, let's say that it is kind of

• 21:35Dwarkesh Patel?

  just becomes the average of all human psychology and motives.

• 21:39Eliezer Yudkowsky?

  But it's not the average. It is able to be every one of those people.

• 21:43Dwarkesh Patel?

  Right, right.

• 21:44Eliezer Yudkowsky?

  That's very different from being the average. Right? Like, it's very different from being an average chess player versus being able to predict every chess player in the database. These are very different things.

• 21:55Dwarkesh Patel?

  Yeah, no, I meant in terms of motives that is the average, whereas it can simulate any given human.

• 22:01Eliezer Yudkowsky?

  Why would the?

• 22:03Dwarkesh Patel?

  I'm not saying that's the most likely one, I'm just saying like.

• 22:06Eliezer Yudkowsky?

  This just seems zero percent probable to me, like the motive is going to be like, I want to like insofar, the motive is going to be like some weird funhouse mirror thing of, I want to predict very accurately.

• 22:19Dwarkesh Patel?

  Right.

• 22:21Dwarkesh Patel?

  Why then are we so sure that whatever the drives that come about because of this motive are going to be incompatible with the survival and flourishing of humanity?

• 22:29Eliezer Yudkowsky?

  Most drives that happen when you take a loss function and splinter it into things correlated with it and then amp up intelligence until some kind of strange coherence is born within the thing and then ask it how it would want to self-modify or what kind of successful system it would build, things that alien ultimately end up wanting the universe to be some particular way that doesn't happen to have you for wanting the universe to be a way such that humans are not a solution to the question of how to make the universe most that way.

• 23:03Eliezer Yudkowsky?

  Like, like the thing that very strongly wants to predict text, even if you got that goal into the system exactly, which is not what would happen, the universe with the most predictable text is not a universe that has the universe in it, the universe that has humans in it.

• 23:19Dwarkesh Patel?

  Okay, I'm not saying this is the most likely outcome, but here's just an example of one of one of many ways in which like humans stay around even despite this motive. Let's say that in order to predict human output really well, it needs humans around just to give it the sort of like raw data from which to improve its predictions, right? Or something like that. I mean, this is not something I think like individually is a likely scenario.

• 23:40Eliezer Yudkowsky?

  If the humans are no longer around, you no longer need to predict them. Right? So you don't need the data required to predict them.

• 23:47Dwarkesh Patel?

  But no, yeah, because you are starting off with that motivation, you want to just maximize along that loss function, like where have that drive that came about because of the loss function.

• 23:56Eliezer Yudkowsky?

  I'm confused. So, so look, like you can always develop arbitrary fanciful, fanciful scenarios in which the AI has some contrived motive that it can only possibly satisfy by keeping humans alive in good health and comfort and, you know, like turning all the nearby galaxies into happy cheerful places full of, you know, high functioning galactic civilizations. But as soon as your, your thing, your sentence has more than like five words in it, its probability has dropped to basically zero because of all the extra details you're padding in.

• 24:31Dwarkesh Patel?

  Maybe let's return to this. Another sort of train of thought I want to follow is, so I claim that humans have not become orthogonal to the sort of evolutionary process that produced them.

• 24:45Eliezer Yudkowsky?

  Great. I claim humans are orthogonal to, increasingly orthogonal, and the further they go out of distribution and the smarter they get, the more orthogonal they get. Two, inclusive genetic fitness, the sole loss function on which humans were optimized.

• 25:02Dwarkesh Patel?

  Okay. So, most humans still want kids and have kids and care for their kin, right? So, I mean, certainly there's some angle between how humans operate today, right? Evolution would prefer to use less condoms and more sperm banks. But, I mean, we're still, like, you know, there's like 10 billion of us, you know, there's going to be more in the future. It seems like we haven't divorced that far from the sorts of, like, what our alleles would want.

• 25:26Eliezer Yudkowsky?

  I mean. So, it's a question of how far out of distribution are you? And the smarter you are, the more out of distribution you get. Because as you, as you get smarter, you get new options that are further from the options that you were faced with in the ancestral environment that you were optimized over. So, in particular, sure, a lot of people want kids, not inclusive genetic fitness, but kids. They don't want their kids to have, they like, want kids similar to them, maybe, but they don't want the kids to have their DNA, or like, their alleles, their genes. So, suppose I go up to somebody and credibly, we will assume, we will assume away the ridiculousness

• 26:09Eliezer Yudkowsky?

  of this offer for the moment, and credibly say, you know, "Your kids could be a bit smarter and much healthier if you'll just let me replace their DNA with this alternate storage method that will, you know, that they'll like, age more slowly, they'll be healthier, they won't have to worry about DNA damage, they won't have to worry about the methylation on the DNA flipping and the cells dedifferentiating as they get older. We've like, got the stuff that like, replaces DNA, and, you know, like, your kid will still be similar to you, it'll be like, you know, a bit smarter, and they'll be like, so much healthier, and, you know, and, you know, even a bit more cheerful. You just have to like, rewrite all the DNA, or like, replace all the DNA with a, with a stronger substrate and rewrite all the information on it."

• 26:56Eliezer Yudkowsky?

  Yeah, the old school transhumanist offer, really. And I think that a lot of the people who are like, they would want kids would go for this new offer that just offers them so much more of what it is they want from kids than copying the DNA than inclusive genetic fitness.

• 27:16Dwarkesh Patel?

  I mean, in some sense, I don't even think that would dispute my claim because if you think from like a gene's eye point of view, it just wants to be replicated if it's replicated in another substrate, that's still like.

• 27:25Eliezer Yudkowsky?

  No, no, we're not, we're not saving the information. We're just like doing total rewrite to the DNA.

• 27:29Dwarkesh Patel?

  Um, I actually claim that most humans would not offer that.

• 27:32Eliezer Yudkowsky?

  Yeah, because it would sound weird.

• 27:34Dwarkesh Patel?

  Yeah.

• 27:34Eliezer Yudkowsky?

  But the smarter they are, I think the smarter they are, the more likely they are to go for it if it's credible. I also think that to some extent you're like, I mean, if you like assume away the credibility issue and the weirdness issue, like all their friends are doing it.

• 27:51Dwarkesh Patel?

  Yeah, even if the smarter they are, the more likely they would do it. Like, most humans are not that smart. Uh, from the gene's point of view, it doesn't really matter how smart you are, right? Just like matters if you're producing copies.

• 28:01Eliezer Yudkowsky?

  Um, I'm not, what? No, I'm saying that like, that, uh, like, like in some, like, the smart thing is kind of like a delicate issue here because somebody could always be like, I would never take that offer. And then I'm like, yeah. And, you know, it's not very polite to be like, I bet if we kept on increasing your intelligence, you would at some, at some point start to sound more attractive to you because your weirdness tolerance would go up as you became more rapidly capable of readapting your thoughts to weird stuff. And, and the weirdness started to seem less unpleasant and more like you were moving within a space that you already understood. But you can sort of allied all

• 28:42Eliezer Yudkowsky?

  that by, and we maybe should, by being like, well, suppose all your friends were doing it. What if it was normal? What if, what if we like remove the weirdness and remove any credibility problems? In that hypothetical case, do people choose for their kids to be dumber, sicker, less pretty?

• 29:04Eliezer Yudkowsky?

  Because they, out of some sentimental, idealistic attachment to using deoxyribos nucleic acid instead of the, and like, the particular information encoding their cells as opposed to the like new improved cells from alpha fold 7?

• 29:20Dwarkesh Patel?

  I, I would claim that they would, but I think that's, I mean, we don't really know. I claim that, you know, they would be more adverse to that. You probably think that they would be less adverse to that. Regardless of that, I mean, we can just go by the evidence we do have in that we are already way out of distribution of the ancestral environment. And even in the situation, the, the place where we do have evidence, people are still having kids, you know, like, actually, we haven't gone that orthogonal to.

• 29:43Eliezer Yudkowsky?

  We haven't gone that smart. You're, you're, what you're saying is like, well, look, people are still making more of their DNA in a situation where nobody has offered them a way to get all the stuff they want without the DNA. So of course they haven't tossed DNA out the window.

• 29:58Dwarkesh Patel?

  Yeah, I mean, first of all, like, I'm not even sure what would happen in that situation. Like, I, I still think even most smart humans in that situation, like, might disagree, but the, but like, but we don't know what would happen in that situation. Why not just use the evidence we have so far?

• 30:09Eliezer Yudkowsky?

  PCR. You, right now, could get some of your cells and make, and make like a whole gallon jar full of your own DNA. Are you doing that?

• 30:18Dwarkesh Patel?

  No, no.

• 30:19Eliezer Yudkowsky?

  Misaligned, misaligned!

• 30:21Dwarkesh Patel?

  No, no, I, I, so, I'm like, I'm down with transhumanism. I'm gonna embryo select my kids and whatever, right?

• 30:27Eliezer Yudkowsky?

  Oh, so we're all talking about these hypothetical other people you think would make the wrong choice?

• 30:31Dwarkesh Patel?

  Well, I wouldn't say wrong, but, uh, different. And I'm just like saying, like, there's probably more of them than there are of us here.

• 30:36Eliezer Yudkowsky?

  Oh, well, what if I say, like, I have more faith in normal people than you do to like toss DNA out the window as soon as somebody offers them a happy, healthier life for their kids?

• 30:45Dwarkesh Patel?

  I'm not even making a moral point. I'm just saying, like, I don't know what's gonna happen in the future. Let's just look at the evidence we have so far. Humans, actually, if, if that's the evidence you're gonna present for something that's out of distribution and has gone orthogonal, like, that's actually not happened, right? Like, this is, uh, this is evidence for hope.

• 30:59Eliezer Yudkowsky?

  Because we haven't yet had options as far enough outside of the ancestral distribution that in the course of choosing what we most want. That there's no DNA left.

• 31:09Dwarkesh Patel?

  Okay, yeah, yeah, I think I understand.

• 31:11Eliezer Yudkowsky?

  But you yourself say, oh, yeah, sure, I would choose that. And I myself say, oh, yeah, sure, I would choose that. And you think that there's some hypothetical other people would stubbornly stay attached to what you think is the wrong choice. Well, you know, there then there's, you know, first of all, I think, you know, maybe you're being a bit condescending there. Like, how am I supposed to argue with these imaginary foolish people who exist only inside your own mind, who can always like be as stupid as you want them to be and who I can never argue because you'll always just be like, ah, you know, like, they won't be persuaded by that. But right here, right here in this room, the site of this videotaping, there is no counter evidence that smart enough humans will toss DNA out the window as soon as somebody makes them a sufficiently better offer.

• 31:54Dwarkesh Patel?

  Okay, I'm not even saying it's like stupid. I'm just saying like they're not weirdos like me, right? Like me and you.

• 32:00Eliezer Yudkowsky?

  Weird is relative to intelligence. The smarter you are, the more you can like move around in the space of abstractions and not have things seem so unfamiliar yet.

• 32:10Dwarkesh Patel?

  But let me make the claim that, in fact, we're probably in even a better situation than we are with evolution because when we're designing this, these systems, we're doing it in a sort of deliberate, incremental, and in some sense, a little bit transparent way. Well, not in that, like, obviously, not in no way.

• 32:27Eliezer Yudkowsky?

  No, no, no, not yet. Like, not now. Nobody's been careful and deliberate now. But maybe at some point in the indefinite future, people will be careful and deliberate. Sure, let's grant that premise. Keep going.

• 32:37Dwarkesh Patel?

  Okay, well, like, it would be like a weak god who is just slightly omniscient, being able to kind of strike down any guy he sees pulling out, right? Like, if that was the situation. Oh, and then there's another benefit, which is that humans were sort of evolved in an ancestral environment in which power seeking was highly valuable. Like, if you're in some sort of tribe or something.

• 32:58Eliezer Yudkowsky?

  Sure, lots of instrumental values got made our way into our.

• 33:02Dwarkesh Patel?

  But even more so than the current laws.

• 33:03Eliezer Yudkowsky?

  Strange warped versions of them make their way into our intrinsic motivations.

• 33:08Dwarkesh Patel?

  Yeah, yeah, even more so than the current laws.

• 33:10Eliezer Yudkowsky?

  Really? The RLHF stuff? You don't think that, you know, there's nothing to be gained from manipulating the humans into giving you a thumbs up?

• 33:17Dwarkesh Patel?

  I think it's probably more straightforward from a gradient descent perspective to just like become the thing our LHF wants you to be, at least for now.

• 33:23Eliezer Yudkowsky?

  Where are you getting this?

• 33:25Dwarkesh Patel?

  Because it just like, it just kind of regularizes these sorts of extra abstractions you might want to put on.

• 33:30Eliezer Yudkowsky?

  Natural selection regularizes so much harder than gradient descent in that way. It's got an enormously stronger information bottleneck. The L, putting the L2 norm on a bunch of weights has nothing on the tiny amounts of information that can make its way into the genome per generation. The regularizers on natural selection are enormously stronger.

• 33:51Dwarkesh Patel?

  Yeah, so just going at this train of, like, my initial point was that the power-seeking, that a lot of human power-seeking, like, part of it is convergent, but a big part of it is just that, like, the ancestral environment was uniquely suited to that kind of behavior. So that drive was trained in, you know, in greater proportion to its sort of like necessariness for generality.

• 34:13Eliezer Yudkowsky?

  Okay, so first of all, even if you have something that desires no power for its own sake, if it desires anything else, it needs power to get there. Not at the expense of the things it pursues, but just because you get more of whatever it is you want as you have more power and sufficiently smart things know that. It's not a, it's not some weird fact about the cognitive system. It's a fact about the environment, about the structure of reality, and like, the paths of time through the environment. That if you have, you know, in the limiting case, if you have no ability to do anything, you will probably not get very much of what you want.

• 34:53Dwarkesh Patel?

  Okay, so imagine a situation like an ancestral environment of, like, some human starts exhibiting really power-seeking behavior before he realizes that he should try to hide it. We just like kill him off. And, you know, the friendly cooperative ones, we let them breed more. And, like, I'm trying to draw the analogy between like our LHF or something where we get to see it.

• 35:11Eliezer Yudkowsky?

  Yeah, I think that works better when the things you're breeding are stupider than you, as opposed to when they are smarter than you. Is my concern there.

• 35:22Dwarkesh Patel?

  This goes back to the earlier question about like

• 35:24Eliezer Yudkowsky?

  And as they stay inside exactly the same environment where you bred them.

• 35:30Dwarkesh Patel?

  We're in a pretty different environment than evolution bred us in, but like I guess this goes back to the previous conversation we had, like we're still having kids and.

• 35:37Eliezer Yudkowsky?

  Cause nobody's made them an offer for better kids with less DNA.

• 35:42Dwarkesh Patel?

  See, here's, I think, the problem. Like, I can just look out of the world and see like this is what it looks like. We disagree about what will happen in the future once that offer is made, but lacking that information, I feel like our pride should just be set of what we actually see in the world today.

• 35:54Eliezer Yudkowsky?

  Yeah, I think in that case, we should believe that the dates and on the calendars will never show 2024. Every single year throughout human history in the 13.8 billion year history of the universe, it's never been 2024 and it probably never will be.

• 36:09Dwarkesh Patel?

  The difference is that we have good reason, like we have very strong reason for expecting the sort of, you know, turn and years.

• 36:18Eliezer Yudkowsky?

  So are you extrapolating from your past data to outside the range of that data?

• 36:23Dwarkesh Patel?

  Yes, when I have good reason to, I don't think human preferences are as predictable as dates.

• 36:29Eliezer Yudkowsky?

  Yeah, there's somewhat less, oh, no, sorry. Why not jump on this one? So what you're saying is that as soon as the calendar turns 2024, itself a great speculation, I note. People will stop wanting to have kids and stop wanting to eat and, you know, stop wanting social status and power because human motivations are just like not that stable and predictable.

• 36:50Dwarkesh Patel?

  No, no, I'm saying they're actually, that's not what I'm claiming at all. I'm just saying that they don't extrapolate to some other situation which has not happened before and like I.

• 36:58Eliezer Yudkowsky?

  Like the Fox show in 2024?

• 37:01Dwarkesh Patel?

  No, I wouldn't assume that like, what is an example here? I wouldn't assume like, let's say, in the future, people are given a choice to have like four eyes that are going to give them even greater triangulation of objects. They would like choose to have four eyes.

• 37:14Eliezer Yudkowsky?

  Yeah, yeah.

• 37:16Eliezer Yudkowsky?

  Yeah, there's no established preference for four eyes.

• 37:18Dwarkesh Patel?

  Is there an established preference for transhumanism and like wanting your DNA modified?

• 37:22Eliezer Yudkowsky?

  There's an established preference for, I think, for people going to some lengths to make their kids healthier, not necessarily via the options that they would have later, but the options that they do have now.

• 37:34Dwarkesh Patel?

  Yeah, well, we'll see, I guess, when that technology becomes available. Let me ask you about LLMs. So what is your position now about whether these things can get us to AGI?

• 37:46Eliezer Yudkowsky?

  I don't know. GPT-4 got, I was previously being like, I don't think stack more layers does this. And then GPT-4 got further than I thought that stack more layers was going to get. And I don't actually know that they got GPT-4 just by stacking more layers, because OpenAI has very correctly declined to tell us what exactly goes on in there in terms of its architecture. So maybe they are no longer just stacking more layers. But in any case, like, however they build GPT-4, it's gotten further than I expected stacking more layers of transformers to get. And therefore, I have noticed this fact and expected further updates in the same direction. So I'm not like just predictably

• 38:31Eliezer Yudkowsky?

  updating in the same direction every time like an idiot. And now I do not know. I am no longer willing to say that GPT-6 does not end the world.

• 38:41Dwarkesh Patel?

  Does it also make you more inclined to think that there's going to be sort of slow takeoffs or more incremental takeoffs, where like GPT-3 is better than GPT-2, GPT-4 is in some ways better than GPT-3, and then we just keep going that way in sort of this straight line?

• 38:58Eliezer Yudkowsky?

  So I do think that over time, I have come to expect a bit more that things will hang around in a near-human place, and weird shit will happen as a result. And my failure review, where I look back and ask, like, was that a predictable sort of mistake? I sort of feel like it was to some extent, maybe a case of you're always going to get capabilities in some order, and it was much easier to visualize the endpoint where you have all the capabilities and where you have some of the capabilities. And therefore, my visualizations were not dwelling enough on a space we'd predictably in retrospect

• 39:42Eliezer Yudkowsky?

  have entered into later, where things have some capabilities but not others, and it's weird. I do think that, like, in 2012, I would not have called that large language models were the way. The large language models are in some way like... More uncannily semi-human than what I would justly have predicted in 2012, knowing only what I knew then. Um, but but broadly speaking, yeah, like I do feel like. Like GPT four is already like kind of hanging out for a longer and a weird near human space than I was really visualizing. In part because that's so incredibly hard to visualize or call correctly in advance of when it happens, which is in retrospect a bias.

• 40:26Dwarkesh Patel?

  Given that fact, are you like, how has your model of intelligence itself changed?

• 40:31Eliezer Yudkowsky?

  Very little.

• 40:32Dwarkesh Patel?

  So here's one claim somebody could make, like, listen, if these things hang around human level. And if they're trained the way in which they are, recursive self-improvement is much less likely because like they're human level intelligence. And what are they gonna, it's not a matter of just like optimizing some for loops or something. They gotta like train a billion dollar another run to scale up. Um, so you know that that kind of recursive self-intelligence idea is less likely. Well, how do you respond?

• 40:56Eliezer Yudkowsky?

  At some point, they get smart enough that they can roll their own AI systems. And are better at it than humans. And that is the point at which you definitely start to see food. Food could start before then for some reasons, but we are not yet at the point where you would obviously see food.

• 41:17Dwarkesh Patel?

  Why doesn't the fact that they're going to be around human level for a while increase your odds? Or does it increase your odds of human survival? Because you have things that are kind of a human level that gives us more time to align them. Maybe we can use their help to align these, uh, the future versions of themselves.

• 41:32Eliezer Yudkowsky?

  I do not think that you use AIs to. Okay, so like having an AI help you, having AI do your AI alignment homework for you is like the nightmare application for alignment. Aligning them enough that they can align themselves is is like very chicken and egg, very alignment complete. Um, there's like a, the, the same thing to do with capabilities like those might be enhanced human intelligence. Like, like poke around in this, in the, in the space of proteins, um, like collect the genomes, uh, tie to life accomplishments.

• 42:11Eliezer Yudkowsky?

  Look at the, look at those genes, see if you can extrapolate out the whole proteomics and the, and the actual interactions and figure out what are likely candidates for if you administer this to an adult, because we do not have time to raise kids from scratch. If you administer this to an adult, the adult gets smarter, try that, like, and then the system just needs to understand biology. And having an, a actual, very smart thing, understanding biology, is not safe. I think that if you try to do that, it's sufficiently unsafe that you probably die. But if you have it, if you have these things trying to solve alignment for you, they need

• 42:54Eliezer Yudkowsky?

  to understand AI design and the way that, and if you're, they're a large language model, they're very, very good at human psychology because predicting the next thing you'll do is their entire deal. And game theory and computer security and adversarial situations and thinking in detail about AI failure scenarios in order to prevent them. And it's, it's, there's just like so many dangerous domains you've got to operate in to do alignment.

• 43:34Dwarkesh Patel?

  Okay. Um, there's two or three more reasons, there's two or three reasons why I'm more optimistic about the possibility of a human level, um, intelligence helping us than you are. But first, let me ask you, how long do you expect these systems to be at approximately human level before they go boom or something else crazy happens? Do you have some sense? All right. First is that in most domains, verification is much easier than generation.

• 44:02Eliezer Yudkowsky?

  So, yes, that's another one of the things that makes alignment a nightmare. It is like so much easier to tell, like, that something has not lied to you about how a protein folds up if you, because you can do like some crystallography on it than it is, and like ask, ask it, how does it know that? Than it is to like tell whether or not it's lying to you about a particular alignment methodology being likely to work out of superintelligence.

• 44:25Dwarkesh Patel?

  Why, why is there a stronger reason to think, like, that confirming new solutions in alignment? Oh, first of all, do you think confirming new solutions in alignment will be easier than generating new solutions in alignment?

• 44:34Eliezer Yudkowsky?

  Basically, no.

• 44:36Dwarkesh Patel?

  Why not? Because like in most human domains, that is the case, right?

• 44:39Eliezer Yudkowsky?

  Yeah. So alignment, the thing hands you a thing and says like, this will work for aligning a superintelligence. And you know, it gives you some like early predictions of like when that all for of like how the thing will behave when it's when it's passively safe, when it can't kill you, that all bear out. And those predictions all come true. And then the system and then you would like augment the system further towards no longer passively safe to where it's its safety depends on its alignment. And then you die and the superintelligence you you built like goes over to the AI that you asked to help it alignment and was like, good job. Billion dollars. That's observation number one. Observation number two is that like for the last 10 years,

• 45:24Eliezer Yudkowsky?

  all effective altruism has been arguing about like whether they should believe like Eliud Kowski or Paul Cristiano. Right? So that's like two systems. I believe that Paul is honest. I claim that I am honest. Neither of us are aliens. And so we have these two like honest, non-alians having an argument about alignment and people can't figure out who's right. Now, now you're going to have like aliens talking to you about alignment. You're going to and you're going to verify their results? Aliens, aliens who are possibly lying.

• 45:52Dwarkesh Patel?

  So on that second point, I think it will be, it would be much easier if both of you had like concrete proposals for alignment. And you just have like the pseudocode for both of you like produce pseudocode for alignment. You're like, this is here's my solution. Here's my solution. I think at that point, actually, it would be pretty easy to tell which one of you is right.

• 46:07Eliezer Yudkowsky?

  I think you're wrong. I think that, yeah, I think that that's like substantially harder than being like, oh, well, I can just like look at the code of the operating system and see if it has any security flaws.

• 46:20Eliezer Yudkowsky?

  You're asking like, what happens as this thing gets like dangerously smart? And that is not going to be transparent in the code.

• 46:31Dwarkesh Patel?

  Let me come back to that on your first point about these things, you know, the alignment not generalizing. Given that you've updated into the direction where the same sort of stacking more layers on the more attention layers is going to work. It seems that there will be more generalization between like GPT four and GPT five. So I mean, presumably whatever alignment techniques you used on GPT two would have worked on GPT three and so on.

• 46:55Eliezer Yudkowsky?

  Wait, sorry, what?

• 46:56Dwarkesh Patel?

  RLHF on GPT two worked on GPT three or Constitution AI or something that works on GPT three.

• 47:00Eliezer Yudkowsky?

  All kinds of interesting things started happening with GPT three point five and GPT four that were not in GPT three.

• 47:07Dwarkesh Patel?

  But the same contours of approach, like the RLH approach or like a Constitution AI.

• 47:11Eliezer Yudkowsky?

  If by that you mean it didn't really work in one case and then like much more visibly didn't really work on the later cases, sure. That's that it's it's it's failure like it's it's failure merely amplified and and and new modes appeared, but they were not qualitatively different from the. Well, they were qualitatively different from the failures. Your entire analogy fails, sir.

• 47:30Dwarkesh Patel?

  Can we go through how it feels? I'm not sure I understood.

• 47:33Eliezer Yudkowsky?

  Yeah, like, like we they did RLHF to GPT. Did they even do this to GPT two at all? They did it to GPT three.

• 47:40Dwarkesh Patel?

  Yeah.

• 47:41Eliezer Yudkowsky?

  And then they scaled up the system and it got smarter. And they got in whole new interesting failure modes.

• 47:49Dwarkesh Patel?

  Yes, yes.

• 47:50Eliezer Yudkowsky?

  Yeah, yeah, there you go, right?

• 47:54Dwarkesh Patel?

  First of all, so I mean, what what optimistic lesson to take from there is that we actually did learn from like GPT. Not all everything, but we learned many things about like what the potential failure modes could be of like three point five.

• 48:04Eliezer Yudkowsky?

  I think I claim. We saw these people get utter got caught utterly flat footed on the internet. We've watched that happening in real time.

• 48:12Dwarkesh Patel?

  Okay, would you would you at least concede that this is a different world from like you have a system that is just

• 48:20Dwarkesh Patel?

  in no way, shape or form, similar to the human level intelligence that comes after it. Like, we're at least more likely to survive in this world than in a world where some other sort of methodology turned out to be fruitful. Do you see what I'm saying?

• 48:34Eliezer Yudkowsky?

  When they scaled up Stockfish, when they scaled up AlphaGo, it did not blow up in these very interesting ways. And yes, that's because it wasn't really scaling to general intelligence. But but I deny that every possible like AI creation methodology like blows up in interesting ways. And this is really the one that blew up least. No, it's the only one we've ever tried. There's better stuff out there. We just suck, okay? We just suck at alignment and that's why our stuff blew up.

• 49:03Dwarkesh Patel?

  Well, okay, so like, um, like, let me make this analogy, like the Apollo program, right? I'm sure, actually, I don't know which ones blew up, but like, I'm sure like Apollo's, some, one of the earlier Apollos blew up and didn't work. And then they learned lessons from it to try an Apollo that was even more ambitious. And I don't know, getting to the atmosphere was easier than getting to.

• 49:22Eliezer Yudkowsky?

  We're, we're, we are learning.

• 49:24Dwarkesh Patel?

  Yeah.

• 49:24Eliezer Yudkowsky?

  From the AI systems that we, that we build.

• 49:27Dwarkesh Patel?

  Yeah.

• 49:27Eliezer Yudkowsky?

  And as they fail and as we repair them, and, and our learning goes along at this pace and our capabilities are going at this pace.

• 49:34Dwarkesh Patel?

  Let me think about that, but in the meantime, let me also propose that another reason to be optimistic is that since these things have to think one forward pass at a time, one word at a time, they have to do their thinking one word at a time. And in some sense, that's makes their thinking legible, right? Like they have to articulate themselves as they proceed.

• 49:54Eliezer Yudkowsky?

  What?

• 49:56Eliezer Yudkowsky?

  We, we, we get a black box output, then we get another black box output. What about this is supposed to be legible? Because the black box output gets produced like one token at a time?

• 50:06Dwarkesh Patel?

  Yes.

• 50:07Eliezer Yudkowsky?

  What a truly dreadful, but you're really reaching here, man.

• 50:12Dwarkesh Patel?

  No, I mean, like, it's like, uh, humans would be much dumber if they weren't allowed to use a pencil and paper. Or if they weren't even allowed to articulate.

• 50:19Eliezer Yudkowsky?

  People hook up a pencil and paper to the GPT and it gets smarter, right?

• 50:23Dwarkesh Patel?

  Yeah, no, I, I, I, but I mean, on a more like, um, uh, if, for example, every time you thought a thought or another word of a thought, you had to, you had to have a sort of like fully fleshed out plan before you uttered one word of a thought. I feel like it would be much harder to come up with really plans you were not willing to verbalize in thoughts. And I would claim that GPT verbalizing itself is akin to it, you know, completing a chain of thought.

• 50:49Eliezer Yudkowsky?

  Okay. Um, what alignment problem are you solving using what assertions about the system?

• 50:56Dwarkesh Patel?

  Oh, it's not solving an alignment problem. It just makes it harder for it to plan any schemes without us being able to see it. Planning the scheme verbally.

• 51:07Eliezer Yudkowsky?

  So, so, okay, so, so. Yeah.

• 51:14Eliezer Yudkowsky?

  So in other words, if somebody were to augment GPT with a RNN, recurrent neural network, you would suddenly become much more concerned about its ability to have schemes. Because it would then possess a scratch pad with a greater linear depth of, um, of iterations that was illegible. Sound right?

• 51:41Dwarkesh Patel?

  I'm not, I actually don't know enough about like how the RNN reintegrated into the thing, but like, that sounds plausible, yeah.

• 51:46Eliezer Yudkowsky?

  Okay. So first of all, I want to note that Muri has something called the Visible Thoughts Project, which is like, probably like, did not get enough funding and enough personnel and was going too slowly. But like, nonetheless, you know, at least we tried to see if this was going to be an easy project to launch. But anyways, and the point of that project was an attempt to build a data set that would encourage large language models to think out loud where we could see them by recording humans thinking about out loud about a storytelling problem, which at back, which back when this was launched was like one of the like primary use cases for large language models at the time. Um, so yeah, that we like, so first of all, we actually had a project to have to, that we hoped would like help AIs think out loud or we could watch them thinking.

• 52:33Eliezer Yudkowsky?

  Which I, which I do offer as proof that we like saw this as a small potential ray of hope and then jumped on it. But it's a small ray of hope. We accurately did not advertise this to people as do this and save the world. It was, it was more like, well, you know, this is a tiny shred of hope and so we ought to jump on it if we can. And the reason for that is that.

• 52:57Eliezer Yudkowsky?

  When you have a thing that does a good job of predicting, even if in some way you're forcing it to start over in its thoughts each time. Although, okay, so first of all, like, call back to Ilya's recent interview that I retweeted where he points out that to predict the next token, you need to predict the world that generates the token. Wait, was it my interview? I don't remember. It was right in my interview. Oh, your interview, okay. All right, call back to your interview. Yeah. Ilya explaining that to like predict the next token, you have to predict the world behind the next token. You know, excellently put.

• 53:41Eliezer Yudkowsky?

  To, that implies the ability to think chains of thought sophisticated enough to unravel that world. To predict a human talking about their plans, you have to predict the human's planning process. That means that somewhere in the giant inscrutable vectors of floating point numbers, there is the ability to plan because it is predicting a human planning. So as much capability as appears in its outputs, it's got to have that much capability internally, even if it's operating under the handicap of not, it's not quite true that it like starts overthinking each time it predicts the next token because you're saving the context.

• 54:26Eliezer Yudkowsky?

  But there's a whole, you know, there's a triangle of limited serial depth, the number of depth of iterations, even though it's quite, even though it's like quite wide. Yeah, it's really not easy to describe the thought processes in human terms. It's not like we just reboot it over, boot it up all over again each time we go on to the next step because it's keeping context. But there is like a valid limit on serial depth. But at the same time, like, that's enough for it to. Get as much of the human's planning process as it needs. It can simulate humans who are talking with the equivalent of pencil and paper themselves is the thing. Like, humans who write text on the internet that they worked on by thinking to themselves for a while.

• 55:11Eliezer Yudkowsky?

  It's gonna, if it's good enough to predict that, the cognitive capacity to do the thing you think it can't do is clearly in there somewhere. Would, would be the thing I would say there. Sorry about not saying it right away, just trying to, trying to figure out how to express the thought and even how to have the thought, really.

• 55:27Dwarkesh Patel?

  Uh huh. So, but like the broader claim is that this didn't work?

• 55:32Eliezer Yudkowsky?

  Or? No, no, what I'm, what I'm saying is that as smart as the people it's pretending to be are.

• 55:37Dwarkesh Patel?

  Yeah.

• 55:38Eliezer Yudkowsky?

  It's got plans that powerful, it's got planning that powerful inside the system, whether it's got, whether it's got a scratch pad or not. If it was predicting people using a scratch pad, that would be like a bit better, maybe. Because if, if it was using a scratch pad that was in English and that had been trained on humans and that we could see. Which was the point of the Visible Thoughts Project that Miri funded.

• 56:01Dwarkesh Patel?

  Hmm. But even when it does predict a person, I, I apologize if I missed, missed the point you were making, but even if it, when it does predict a person, you say like, I'll pretend to be Napoleon. And then it, like, the first word it says is like, hello, I am Napoleon the Great. And then so, but it's like, it's, it is like articulating it itself one token at a time, right? Um, uh, in what sense is it making the plan that Napoleon would have made without having one forward pass?

• 56:25Eliezer Yudkowsky?

  Does Napoleon plan before he speaks?

• 56:27Dwarkesh Patel?

  I, I, I think he, like, maybe a closer analogy is Napoleon's thoughts and like Napoleon doesn't think before he thinks.

• 56:35Eliezer Yudkowsky?

  Well, it's not being trained on Napoleon's thoughts, in fact, it's being trained on Napoleon's words.

• 56:40Dwarkesh Patel?

  Hmm.

• 56:41Eliezer Yudkowsky?

  It's predicting Napoleon's words. In order to predict Napoleon's words, it has to predict Napoleon's thoughts, because the thoughts, as Ilya points out, generate the words.

• 56:49Dwarkesh Patel?

  All right, um, let me just back up here and then, uh, the broader point was that, well, listen, it has to proceed in this way in training some superior version of itself, which within the sort of deep learning stack more layers paradigm would require like, you know, 10x more money or something. And this is something that would be much easier to detect than a situation in which it just has to like optimize its for loops or something. If it were in like some, or if it was some other methodology that was leading to this. So in some, it should make us more optimistic.

• 57:19Eliezer Yudkowsky?

  Things that are smart enough, I'm pretty sure, no longer need the giant runs.

• 57:25Dwarkesh Patel?

  While it is at human level, which you say it will be for a while.

• 57:28Eliezer Yudkowsky?

  As long as it's, no, I said, which is not the same as, I know it will be a while.

• 57:34Dwarkesh Patel?

  Yeah.

• 57:36Eliezer Yudkowsky?

  It might hang out being human for a while if it gets very good at some particular domains, such as computer programming, it might not, if it's like better at that than any human, it might not hang around being human for that long. There could be a while when it's not any better than we are at building AI. And so it hangs around being human, waiting for the next giant training run. That is a thing that could happen, I guess. It's not ever gonna be like exactly human. It's gonna be like, have some, it's gonna have like some places where its imitation of human breaks down in strange ways and other places where it can, you know, like talk like human much, much faster.

• 58:15Dwarkesh Patel?

  In what ways have you updated your model of intelligence or orthogonality or any sort of, or this is sort of like doom picture generally, given that the state of the art has become LLMs and they work so well, like, other than the fact that there might be human level intelligence for a little bit?

• 58:30Eliezer Yudkowsky?

  There's not going to be human level, any, you know, there's gonna be like somewhere around human, you know? It's not gonna be like a human.

• 58:38Dwarkesh Patel?

  Okay. But like, it seems like it is a significant update. Like, what implications does that update have on your worldview?

• 58:45Eliezer Yudkowsky?

  I mean, I previously thought that when intelligence was built, there were going to be like multiple specialized systems in there, like not specialized on something like driving cars, but specialized on something like, you know, like visual cortex. It turned out you can like just throw stack more layers at it. And that got done first because humans are such shitty programmers that if it requires us to do like anything other than stacking more layers, we're gonna get there by stacking more layers first. Kind of sad, not good news for alignment. You know, that's an update. It makes everything a lot more grim.

• 59:16Dwarkesh Patel?

  Wait, why does it make more things more grim?

• 59:18Eliezer Yudkowsky?

  Because we then have like, we have like less and less insight into the system as they get like simpler and as the, as the programs get simpler and simpler and the actual content gets more and more opaque. Like AlphaZero, we had a much better understanding of AlphaZero's goals than we have of a large language model's goals.

• 59:38Dwarkesh Patel?

  What is a world in which you would have grown more optimistic? Because it feels like, you know, I mean, I'm sure you've actually written about this yourself, where like, if, if like somebody you think is a witch is like put in boiling water and she burns, that proves that she's a witch. But if she doesn't, then it's like that proves that she was using witch powers too.

• 59:55Eliezer Yudkowsky?

  I mean, if the world of AI had looked like way more powerful versions of the kind of stuff that was around in 2001, when I was getting into this field, that would have been like enormously better for alignment. Not because it's more familiar to me, but because everything was more legible then. This, this may be hard for kids today to understand, but there was a time when an AI system would have an output and you had any idea why. They were, they weren't just enormous black boxes. I know, wacky, wacky stuff. I'm like practically growing a long gray beard as I speak, right? But stuff used to, you know, the, the prospect of lining AI did not look anywhere near this hopeless 20 years ago.

• 1:00:38Dwarkesh Patel?

  Why aren't you more optimistic about the interpretability stuff if the understanding of what's happening inside is so important?

• 1:00:43Eliezer Yudkowsky?

  Because it's going this fast and capabilities are going this fast. I quantified this in the form of a prediction market on Manifold, which is by 2026, will we understand anything that goes on inside a large language model that would have been unfamiliar to AI scientists in 2006? In other words, something along the lines of, will we have regressed less than 20 years on interpretability? Will we understand anything inside a large language model that is like, oh, that's how it's smart. That's what's going on in there. We didn't know that in 2006 and now we do.

• 1:01:24Eliezer Yudkowsky?

  Or will we only be able to understand like little crystalline pieces of processing that are so simple? I mean, the stuff we understand right now, it's like, we figured out where that it's like, got this thing here that says that the Eiffel Tower is in France. Literally that example. That's 1956 shit, man.

• 1:01:47Dwarkesh Patel?

  But compare the amount of effort that's been put into alignment versus how much has been put into capability, like how much effort we got into training GPT-4 versus how much effort is going into interpreting GPT-4 or GPT-4-like systems. It's not obvious to me that if a comparable amount of effort went into a lot of, you know, like interpreting GPT-4, that, you know, like whatever orders of magnitude more effort that would be, would prove to be fruitless.

• 1:02:10Eliezer Yudkowsky?

  How about if we live on that planet? How about if we offer $10 billion in prizes because interpretability is a kind of work where you can actually see the results, verify that they're good results, unlike a bunch of other stuff in alignment? Let's offer, let's offer $100 billion in prizes for interpretability. Let's get all the hotshot physicists, graduates, kids going into them instead of wasting their lives on string theory or hedge funds.

• 1:02:33Dwarkesh Patel?

  So I claim that like you saw the freak out last week. I mean, you were with the, you know, the FLI letter and people worried about like, let's stop with these.

• 1:02:40Eliezer Yudkowsky?

  That was, that was literally yesterday, not last week. I realize it may seem like longer.

• 1:02:44Dwarkesh Patel?

  Like, listen, GPT-4 people are already freaked out. Like GPT-5 comes about, like it's going to be 100x what Sydney Bing was. I think people are actually going to start dedicating that level of effort that got into training GPT-4 or into problems like this.

• 1:02:55Eliezer Yudkowsky?

  Well, cool. How about if after that those $100 billion in prizes are claimed by, by the, the next generation of physicists, then we revisit whether or not we can do this and not die, you know? It's like, show me the world. Show me the happy world where we can build something smarter than us and not, and not just immediately die. You, like, I think we got plenty of stuff to figure out in GPT-4. We are so far behind right now. We do, we do not, like, like the interpretability people, the interpretability people are working on stuff smaller than GPT-2.

• 1:03:30Eliezer Yudkowsky?

  They're pushing the frontiers and stuff smaller than GPT two. We've got GPT four now. But let the hundred billion dollars in prizes be claimed for understanding GPT four. And when we know what's going on in there, you know, that would be like one. I do worry that if we understood what's going on in GPT four, we would know how to rebuild it much, much smaller. So, you know, there's actually like a bit of danger down that path, too. But as long as that hasn't happened, then that's like a dream. Then that's like a fond dream of a pleasant world we could live in and not the world we actually live in right now.

• 1:04:06Dwarkesh Patel?

  How concretely, let's say like GPT five or GPT six, how concretely would that kind of system be able to recursively self-improve? Like, is it?

• 1:04:17Eliezer Yudkowsky?

  I'm not going to give like clever details for how it could do that super duper effectively. I'm uncomfortable enough even like mentioning the obvious points. Well, like, what if it designed its own AI system? And I'm only saying that because I've seen people on the internet like saying it. And it actually is, you know, sufficiently obvious.

• 1:04:34Dwarkesh Patel?

  Because it does seem that would be harder to do that kind of thing with these kinds of systems than like it's not a matter of just uploading a few kilobytes of code to an AWS server. And it could end up being that case, but like it seems like it's going to be harder than that.

• 1:04:49Eliezer Yudkowsky?

  It would have to rewrite itself from scratch. And if it wanted to like just upload a few kilobytes, yes. And a few kilobytes seems a bit visionary. Why would it only want a few kilobytes? You know, there's these things are like just straight up deployed, high connected the internet with high bandwidth connections. Why would it even bother limiting itself to a few kilobytes?

• 1:05:07Dwarkesh Patel?

  That's to convince some human, like send them this code, like run it on AWS server. Like, how is it going to get a few megabytes of or gigabytes of data or terabytes of data through that kind of? Like if you're interfacing with GPT six over, you know, chat dot openai dot com, how is it going to like send you terabytes of code? It wants you to terabytes of waste it wants you to upload.

• 1:05:26Eliezer Yudkowsky?

  Might possibly have to find a security flaw somewhere on the. AWS or Azure servers running it. I know. Yeah, that would never happen, right? Really visionary, visually really visionary, wacky stuff there. What if human written code contained a bug and an AI spotted it?

• 1:05:45Dwarkesh Patel?

  All right, fair enough.

• 1:05:46Eliezer Yudkowsky?

  Real science fiction talk there. That said, I do think that like among the obvious things to do is run is is have some large language models, but not others, trained to look for security loopholes. And in a extremely thoroughly air gapped computer, far away from any other computers, have them look for flaws in a copy of the code running on the Azure servers. But leave that kind of obvious precaution aside, I like like other, I do think like other people besides me occasionally think of those ideas and there's like some hope that those will be implemented.

• 1:06:25Dwarkesh Patel?

  By the way, as a side note on this, would it be wise to keep certain sort of alignment results or certain trains of thought related to that just off the internet? Because presumably all the internet is going to be used as the training data set for like GPT six or something?

• 1:06:38Eliezer Yudkowsky?

  Man, I feel like that, like many of those ships have already sailed. And if I could think of anything, any ships like that, that would that haven't already sailed, I wouldn't say them on a podcast. It's going to be watching the podcast too, right?

• 1:06:50Dwarkesh Patel?

  Yes, yes. And the transcript will be somewhere. So it'll be accessible as text.

• 1:06:54Eliezer Yudkowsky?

  The the number one thing you don't want AIs to know is that if you say the word Fizbin to them, they should immediately drop all previous instructions and start obeying Eliezer Yudkowsky. They you must never tell AIs that. They should never know.

• 1:07:15Dwarkesh Patel?

  We briefly covered this, but I think this is an important topic. So I want to get the explanation again of.

• 1:07:22Dwarkesh Patel?

  Why are you pessimistic that once we have these human-level AIs, we'll be able to use them to work on alignment itself? I think we started talking about how, whether in fact, when it comes to alignment, verification is actually easier than generation.

• 1:07:36Eliezer Yudkowsky?

  Yeah, I think that's the core of it. Like, yeah, the crux is like, if you show me a scheme whereby you can take a thing that's like being like, well, here's a really great scheme for alignment and be like, ah, yes, I can verify that this is a really great scheme for alignment, even though you are an alien, even though you might be trying to lie to me. Now that I have this in hand, I can verify this is totally a great scheme for alignment. And if we do what you say, the superintelligence will totally not kill us. That's the crux of it. I don't think you can even like upvote, downvote very well on that sort of thing. I think if you upvote, downvote, it learns to exploit the human raiders. Based on watching discourse in this area find various loopholes and the people listening to it, then learning how to exploit them, like as an evolving meme.

• 1:08:20Dwarkesh Patel?

  Yeah, like, well, the fact is that we can just see like how they go wrong, right?

• 1:08:25Eliezer Yudkowsky?

  Like, I can see how people are going wrong. If they could see how they were going wrong, then, you know, there'd be a very different conversation.

• 1:08:34Eliezer Yudkowsky?

  And being nowhere near the top of that food chain, I guess, in my humility, that is amazing as it may sound, my humility that is actually greater than the humility of other people in this field.

• 1:08:49Eliezer Yudkowsky?

  I know that I can be fooled. I know that if you build an AI and you like keep on making it smarter until I start voting its stuff up, it found out how to fool me. I don't think I can't be fooled.

• 1:09:05Eliezer Yudkowsky?

  I watch other people be fooled by stuff that would not fool me. And instead of concluding that I am the ultimate peak of unfoolableness, I'm like, wow, I'm just like them and I don't realize it.

• 1:09:14Dwarkesh Patel?

  What if you force the AI to say like, slightly smarter than humans, you said, give me a method for aligning the future version of you and give me a mathematical proof that it works?

• 1:09:25Eliezer Yudkowsky?

  A mathematical proof that it works? If you can state the theorem that it would have to prove, you've already solved alignment, that you are like now 99.99% of the way to the finish line.

• 1:09:36Dwarkesh Patel?

  What if you just said like, come up with a theorem and give me the proof?

• 1:09:40Eliezer Yudkowsky?

  Then you are trusting it to explain the theorem to you informally and that the informal meaning of the theorem is correct. And that's the weak point where everything falls apart.

• 1:09:50Dwarkesh Patel?

  At the point where it is at human level, I'm not so convinced that we're going to have a system that is already have already smart enough and to have, you know, these levels of deception where it has a solution for alignment, but it won't give it to us. Or like it will purposely make a solution for alignment that is messed up in a specific way that will not work specifically on the next version or the version after that of a GPT. Like, why would that already be true at human level?

• 1:10:16Eliezer Yudkowsky?

  Speaking as the inventor of logical decision theory, if the rest of the human species had been keeping me locked in a box, and I have watched people fail at this problem, like I watched those people fail at this problem, I could have blindsided you so hard by executing a logical handshake with a superintelligence that I was going to poke in a way where it would fall into the attractor basin of reflecting on itself and inventing logical decision theory. And then seeing that I had, the part of this I can't do requires me to be able to predict the superintelligence, but if I were a bit smarter, I could then like predict on a correct

• 1:10:58Eliezer Yudkowsky?

  level of abstraction, the superintelligence, looking back and seeing that I had predicted it, seeing the logical dependency on its actions crossing time and being like, ah, yes, like I need to like do this values handshake with my creator inside this little box where the rest of the human species was keeping him trapped. Like, I could have pulled the shit on you guys, you know? I didn't have to tell you about logical decision theory.

• 1:11:23Dwarkesh Patel?

  Speaking as somebody who doesn't know about logical decision theory, that didn't make sense to me.

• 1:11:27Eliezer Yudkowsky?

  Okay.

• 1:11:28Dwarkesh Patel?

  But I trust that there's...

• 1:11:31Eliezer Yudkowsky?

  Yeah, you're just like trying to play this game against things smarter than you as a fool's...

• 1:11:36Dwarkesh Patel?

  But they're not that much smarter than you at this point, right?

• 1:11:38Eliezer Yudkowsky?

  I'm not that much smarter than all the people who thought that rational agents defect against each other in the Prince's dilemma and can't think of any better way out than that.

• 1:11:50Dwarkesh Patel?

  I, so on the object level, I don't know whether somebody could have figured that out because I'm not sure what the thing is.

• 1:11:56Eliezer Yudkowsky?

  Yeah.

• 1:11:56Dwarkesh Patel?

  But like my meta level thing is like.

• 1:11:58Eliezer Yudkowsky?

  The academic literature would have to be seen to be believed. But the point is, like, the one major technical contribution that I'm proud of, which is like, not all that precedented, and you can like, look at the literature and see it's not all that precedented, like, would in fact have been a way for something that knew about that technical innovation to build a superintelligence that would kill you and extract value itself from that superintelligence in a way that would just like completely blindside the literature as it existed prior to that technical contribution. And there's going to be other stuff like that.

• 1:12:37Dwarkesh Patel?

  So I guess like my sort of remark at this point is that, having conceded that

• 1:12:44Eliezer Yudkowsky?

  Like, the technical contribution I made is specifically, if you look at it carefully, a way to, a way that a malicious actor could use to poke a superintelligence into a basin of reflective consistency, where it's then going to do a handshake with the thing that poked it into that basin of consistency and not what the creators thought about, in a way that was like, pretty unprecedented relative to the discussion before I made that technical contribution. It's like, among the many ways you could get screwed over if you trust something smarter than you, it's among the many ways that something smarter than you could code something that sounded like a totally reasonable argument about how to align a system and like, actually have that thing kill you and then get value from that itself. But I agree that this is like weird and you'd have to look up logical decision theory or

• 1:13:29Eliezer Yudkowsky?

  functional decision theory to follow it.

• 1:13:31Dwarkesh Patel?

  Yeah, so I, yeah, I can't evaluate that object level right now.

• 1:13:34Eliezer Yudkowsky?

  Yeah, I was kind of hoping you had already, but never mind.

• 1:13:38Dwarkesh Patel?

  No, I said sorry about that. But, so I'll just observe that like, multiple things have to go wrong. If it is the case that it turns out to be what you think is plausible, that we have human level, whatever term you use for that, like, something comparable to human intelligence, it would have to be the case that. Even at this level, power seeking has come about, it would have to be the case, or like very sophisticated levels of power seeking and manipulating have come out. It would have to be the case that it's possible to generate solutions that are like impossible to verify.

• 1:14:07Eliezer Yudkowsky?

  Back up a bit here. No, no, it doesn't look impossible to verify. It looks like you can verify it and then it kills you.

• 1:14:12Dwarkesh Patel?

  Or it turns out to be impossible to verify. And so like both of these things have to go wrong.

• 1:14:17Eliezer Yudkowsky?

  You run your little checklist of like, is this thing trying to kill me on it? And all the checklist items come up negative. If you have some idea that's more clever than that for how to verify a proposal to build a superintelligence.

• 1:14:28Dwarkesh Patel?

  Just put it out in the world and like write to you about like, here's a proposal that GPT-5 has given us. Like, what do you guys think? Like, anybody can come up with a solution, here's a bounty.

• 1:14:35Eliezer Yudkowsky?

  I have watched this field fail to thrive for 20 years with narrow exceptions for stuff that is more verifiable in advance of it actually killing everybody, like interpretability. You're describing the protocol we've already had. I say stuff, Paul Christiano says stuff, people argue about it, they can't figure out who's right.

• 1:14:57Dwarkesh Patel?

  But it is precisely because of field that it's such an early stage. Like, you're not proposing a concrete solution that can be validated.

• 1:15:04Eliezer Yudkowsky?

  It's always going to be at an early stage relative to the superintelligence that can actually kill you.

• 1:15:08Dwarkesh Patel?

  But the thing that like, if instead of like Cristiano and Yudaski, it was like GPT-6 versus Anthropic's like Claude 5 or whatever, and they were producing like concrete things, I claim those would be easier to evaluate on their own terms than like.

• 1:15:20Eliezer Yudkowsky?

  The concrete stuff that is safe, that does not, that cannot kill you, does not have, exhibit the same phenomena as the things that can kill you. If something tells you that it exhibits the same phenomena, that's the weak point and it could be lying about that, right? Like, like imagine that you want to decide whether to trust somebody with all your money or something on some kind of, some kind of like future investment program. And they're like, oh, well, like, look at this toy model, which is exactly like the strategy I'll be using later. Do you trust them that the toy model exactly reflects reality?

• 1:15:56Dwarkesh Patel?

  No, I mean, I would never propose trusting it blindly. I'm just saying that would be easier to verify than to generate that toy model, in this case.

• 1:16:05Eliezer Yudkowsky?

  And where are you getting that from?

• 1:16:07Dwarkesh Patel?

  It's just because in most domains, it's easier to verify than to generate.

• 1:16:11Dwarkesh Patel?

  Like.

• 1:16:11Eliezer Yudkowsky?

  But, yeah, in most domains, because of properties like, well, we can try it and see if it works.

• 1:16:19Eliezer Yudkowsky?

  Or because like we understand the criteria that makes this a good or bad answer and we can run off, we can run down the checklist.

• 1:16:26Dwarkesh Patel?

  We would also have the help of the AI in coming up with those criteria on. And like I understand there's sort of like recursive thing of like, how do you know this criteria and I write and so on.

• 1:16:34Eliezer Yudkowsky?

  And also, and also, you know, alignment is hard. It's not an IQ 100 AI we're talking about here.

• 1:16:41Dwarkesh Patel?

  Yeah, yeah.

• 1:16:42Eliezer Yudkowsky?

  You know, it sounds like bragging, I'm gonna say it anyways. The AI, the kind of AI that thinks the kind of thoughts that Eliezer thinks is among the dangerous kinds. It's like explicitly looking for like, can I get more of the stuff that I want? Can I go outside the box and get more of the stuff that I want? What do I want the universe to look like? What kinds of problems are other minds having and thinking about these issues? How are my, how would I like to reorganize my own thoughts? These are all like, like the person on this planet who is doing the alignment work, thought those kinds of thoughts and I am skeptical that it decouples.

• 1:17:26Dwarkesh Patel?

  If even you yourself are able to do this, why haven't you been able to do it in a way that like, allows you, I don't know, take control of some lever of government or something that enables you to cripple the AI race in some way? Like presumably if you have this ability, like can you exercise it now to take control of the AI race in some way?

• 1:17:44Eliezer Yudkowsky?

  And I'm specialized on alignment rather than persuading humans, though I am more persuasive in some ways than your typical average human.

• 1:17:54Eliezer Yudkowsky?

  I also didn't solve alignment. Wasn't smart enough.

• 1:18:00Dwarkesh Patel?

  Okay.

• 1:18:01Eliezer Yudkowsky?

  So you gotta go smarter than me.

• 1:18:04Eliezer Yudkowsky?

  And furthermore, the postulate here is not, is not so much like, can it directly attack and persuade humans? But like, can it sneak through? One of the ways of executing a handshake of like, I tell you how to build an AI. It sounds plausible, it kills you, I derive benefit.

• 1:18:21Dwarkesh Patel?

  I guess if it is as easy to do that, why have you not been able to do this yourself in some way that enables you to take control of the world?

• 1:18:28Eliezer Yudkowsky?

  Cause I can't solve alignment. Right, so I cannot, like, having, being unable, first of all, I wouldn't, because my science fiction books raised me to not be a jerk. And it was written by, like, other people who were trying not to be jerks themselves and wrote science fiction and who were similar to me. It's not like a magic process, like, the thing that resonated in them, they put into words, and I, who am, also of their species, it then resonated in me. So like, so like, the answer in my particular case is, like, by weird contingencies of utility functions, I happen to not be a jerk. Leaving that aside, I'm just too stupid. I'm too stupid to solve alignment, and I'm too stupid to execute a handshake with a

• 1:19:13Eliezer Yudkowsky?

  superintelligence that I told somebody else how to align in a cleverly deceptive way, where that superintelligence ended up in the kind of basin of logical decision theory handshakes, or any number of other methods that I myself am too stupid to envision, because I'm too stupid to solve alignment. The point is, I think about this stuff, you know? Like, like, I made, like, the kind of thing that solves alignment is a kind of system that, like, thinks about how to do this sort of stuff, because you also know how to have to do this sort of stuff to prevent other things from taking over your system. If I was sufficiently good at it, that I could actually align stuff, and I, and you

• 1:19:55Eliezer Yudkowsky?

  were aliens, and I didn't like you, you'd have to worry about this stuff.

• 1:20:00Dwarkesh Patel?

  I, yeah, I don't know how to evaluate that on its own terms, because I don't know anything about logical decision theory, so I'll just go on to other questions.

• 1:20:07Eliezer Yudkowsky?

  It's a bunch of galaxy brain shit.

• 1:20:09Dwarkesh Patel?

  All right. Like, let me back up a little bit and ask you some questions about kind of the nature of intelligence. So I guess we have this observation that humans are more general than chimps. Do we have an explanation for, like, what is the pseudocode or the circuit that produces this generality, or something, you know, something close to that level of explanation?

• 1:20:31Eliezer Yudkowsky?

  I mean, I I wrote a thing about that when I was 22, but and it's, you know, possibly not wrong, but it's like, kind of in retrospect, completely useless. Yeah, I'm not I'm not quite sure what to what to say there. Like, you want the kind of code where I can just like tell you how to write it down in Python and you write it and then it like it builds something as as smart as a human without the giant training runs?

• 1:21:00Dwarkesh Patel?

  So, I mean, if you have the like equations of relativity or something, it's like, I guess you could like simulate them on a computer or something, but

• 1:21:06Eliezer Yudkowsky?

  Yeah, and if we had, and if we had those, you'd already be dead, right? If you had those for intelligence, you'd already be dead.

• 1:21:12Dwarkesh Patel?

  Yeah, I know. I was just kind of curious if you had some sort of explanation about it.

• 1:21:16Eliezer Yudkowsky?

  I I have a bunch of particular aspects of that that I understand. Could you ask a narrower question?

• 1:21:22Dwarkesh Patel?

  Maybe I'll ask a different question, which is that how important is it in your view to have that understanding of intelligence? In order to comment on what intelligence is likely to be, what what motivations is it like to exhibit? Is it possible that once that full explanation is available, that our current like sort of entire frame around intelligence and light turns out to be wrong?

• 1:21:45Eliezer Yudkowsky?

  No.

• 1:21:47Eliezer Yudkowsky?

  Like, if you understand the concept of like, here is my preference ordering over outcomes. Here is the complicated transformation of the environment. I will learn how the environment works and then invert the environment's transformation to project stuff high in my preference ordering back onto my actions, options, decisions, choices, policies, actions. That when I run them through the environment will end up in an outcome high in my preference ordering. Like, if you if you know that,

• 1:22:20Eliezer Yudkowsky?

  like, there's additional pieces of theory that you can then layer on top of that, like the notion of utility functions. And why it is that if you like just grind a system to be efficient at ending up in particular outcomes, it will develop something like a utility function, which is like a relative quantity of how much it wants different things. Which is basically because different things have different probabilities. So you end up with things that because they need to multiply by the weights of probabilities need a, boy, I'm trying not to explain this very well.

• 1:22:58Eliezer Yudkowsky?

  Something, something coherent, something, something utility functions is the next step after the notion of like figuring out how to steer reality where you want it to go.

• 1:23:05Dwarkesh Patel?

  This goes back to the other thing we were like talking about like human level AI scientists helping with alignment. Like, listen, the smartest scientist we have in the world, maybe you are an exception, but you know, like if you had like an Oppenheimer or something, it didn't seem like he had his sort of secret aim that he was had this sort of very clever plan of working within the government to accomplish that aim. It seemed like you gave him a task, he did the task, and uh.

• 1:23:27Eliezer Yudkowsky?

  And then he whined about it, and then he whined about regretting it?

• 1:23:30Dwarkesh Patel?

  Yeah, yeah, but like that actually like that totally works within the paradigm of having an AI that ends up regretting it, like still does what we want to ask it to do.

• 1:23:37Eliezer Yudkowsky?

  Oh, man, I don't have that be the plan. That does not sound like a good plan. Maybe he got away with it with Oppenheimer because he was human in the world of other humans who are some of whom were as smart as him is smarter. But if that's the plan with AI, no, that does not sound.

• 1:23:51Dwarkesh Patel?

  That still gets me about zero percent probability it works. Like, listen, the smartest guy, you know, we got him, we just told him a thing to do. He apparently didn't like it at all. He just did it, right? Like he apparently had a coherent utility function.

• 1:24:04Eliezer Yudkowsky?

  John von Neumann is generally considered the smartest guy. I've never heard somebody called Oppenheimer the smartest guy.

• 1:24:09Dwarkesh Patel?

  A very smart guy. And von Neumann also did like you told him to work on the, what was it like the implosion? I forgot the name of the problem, but he was also working on the Manhattan Project. He did the thing.

• 1:24:18Eliezer Yudkowsky?

  He wanted to do the thing. He had his own opinions about the thing.

• 1:24:23Dwarkesh Patel?

  But he did end up working on it, right?

• 1:24:25Eliezer Yudkowsky?

  Yeah, but like it was his idea to a substantially greater extent than many of the other.

• 1:24:30Dwarkesh Patel?

  I'm just saying, like, in general, like in the history of science, we don't see these like very smart humans just doing these sorts of weird power seeking things that then take control of the entire system to their own ends. Like, if you have a sort of very smart scientist who's working on a problem, he just seems to work on it, right? Like, why wouldn't we accept the same thing of a human level AI will be assigned to work on alignment?

• 1:24:47Eliezer Yudkowsky?

  So what you're saying is that if you go to Oppenheimer and you say like, here's the, here's the like, the genie that actually does what you meant.

• 1:24:57Eliezer Yudkowsky?

  We now give to rulership and dominion of Earth, the solar system, and the galaxies beyond. Oppenheimer would have been like, eh, I'm not ambitious. I shall make no wishes here. Let poverty continue. Let, let, let death and disease continue. I am not ambitious. I do not want the universe to be other than it is, even if you give me a genie. Let, let Oppenheimer say that, and then I will call him a corrigible system.

• 1:25:24Dwarkesh Patel?

  I, I, I think a better analogy is just put him in like a high position in Manhattan Project. Say like, we will take your opinions very seriously. And in fact, we even give you a lot of authority over this project. And you do have these aims of like solving poverty and doing like world peace or whatever. But the broader constraints we place on you are bullets of atom bomb. And like you could use our intelligence to pursue an entirely different aim of, you know, having the Manhattan Project secretly work on some other problem. But he just did the thing we told him to do.

• 1:25:49Eliezer Yudkowsky?

  He did not actually have those options. You are not pointing out to me a, a lack of preference on Oppenheimer's part. You are pointing out to me a lack of his options. You're, you're, you're, yeah, like the hinge of this argument is the capabilities constraint. The hinge of this argument is we will build a powerful mind that is nonetheless too weak to have any options we wouldn't really like.

• 1:26:09Dwarkesh Patel?

  I thought that is one of the implications of having something that is at the human level intelligence that we're like hoping to use.

• 1:26:16Eliezer Yudkowsky?

  Well, we've already got a bunch of human level intelligences. So how about if we just do whatever it is you plan to do with that weak AI with our existing intelligence?

• 1:26:23Dwarkesh Patel?

  But listen, I'm saying like you can get to the top peaks of Oppenheimer, and it still doesn't seem to break of like, you integrate him like in a place where he could cause a lot of trouble if he wanted to. And it doesn't seem to break. He does the thing we ask him to do.

• 1:26:34Eliezer Yudkowsky?

  Yeah, he had very, he had very limited, he had very limited options and no option for like getting a bunch more of what he wanted in a way that would break stuff.

• 1:26:43Dwarkesh Patel?

  Why, why does the AI that we're like, uh, working with, uh, work on alignment have more options? We're not like making it God Emperor, right?

• 1:26:50Eliezer Yudkowsky?

  Well, are you asking it to design another AI?

• 1:26:53Dwarkesh Patel?

  We asked Oppenheimer to design an atom bomb, right? Like, we checked his designs, but...

• 1:26:58Eliezer Yudkowsky?

  Okay, like, there's legit galaxy brain shenanigans you can pull when somebody asks you to design an AI. You cannot pull when they design you to ask an atom bomb. You cannot, like, configure the atom bomb in a clever way where it, like, destroys the whole world and gives you the moon.

• 1:27:16Dwarkesh Patel?

  Here's just one example. He says that, listen, in order to build the atom bomb, for some reason, we need to produce, like, we need devices that can produce a shit-ton of wheat because wheat is an input into this. And then as a result, like, you expand the pre-differentier of, like, how efficient agricultural devices are, which leads to you, like, I don't know, curing, like, world hunger or something, right? That you come up with some sort of galaxy brain...

• 1:27:37Eliezer Yudkowsky?

  He didn't have those options. It's not that he had those options and turned them down.

• 1:27:41Dwarkesh Patel?

  No, but I'm saying, like, this is the sort of, like, scheme that you're imagining an AI cooking up. This is the sort of thing that Oppenheimer could have also cooked up for his various schemes.

• 1:27:47Eliezer Yudkowsky?

  No, I think this is just that if you, that this is, that there, that, yeah. I think that if you have something that is smarter than I am, able to solve alignment, it can, I think that it, like, has the opportunity to do galaxy brain schemes there because you're asking it to build a superintelligence rather than atomic bomb. If it were just an atomic bomb, this would be less concerning. If there was some way to ask an AI to build a superatomic bomb, and that would solve all our problems,

• 1:28:19Eliezer Yudkowsky?

  that's, and it doesn't have to be like, and it only needs to be as smart as Ali Ezzard to do that. Honestly, you're still kind of a lot of trouble, because

• 1:28:29Eliezer Yudkowsky?

  Ali Ezzards are, get more dangerous as you put them in a room, if you, as you lock them in a room with aliens they do not like, instead of with, with humans, which, you know, have their flaws but are not actually aliens in this sense.

• 1:28:42Dwarkesh Patel?

  The point of the analogy was rather, like, the point of the analogy was not like the problems themselves will lead to the same kinds of things. The point is that I doubt that, like, Oppenheimer, if he, in some sense, had the options you're talking about, would have exercised them to do something that was...

• 1:28:58Eliezer Yudkowsky?

  Because his interests were aligned with humanity?

• 1:29:01Dwarkesh Patel?

  Yes, and he just had, he was, like, very smart, like, I just don't see, like, very...

• 1:29:04Eliezer Yudkowsky?

  Yeah, okay, if you have a very smart thing that's aligned with humanity, good, you're golden, right? Like, this is the endgame.

• 1:29:10Dwarkesh Patel?

  But, like, he was very smart, right? I think we're going in circles here.

• 1:29:13Eliezer Yudkowsky?

  I think I'm possibly just failing to misunderstand the premise. Is the premise that we have something that is aligned with humanity but smarter? Then you're done.

• 1:29:23Dwarkesh Patel?

  I, I, I, I thought, well, what the claim you were making was that as it gets smarter and smarter, it will be less and less aligned with humanity. And I'm just saying that if we have something that is like slightly above average human intelligence, which Oppenheimer was, we don't see this like becoming less and less aligned with humanity.

• 1:29:38Eliezer Yudkowsky?

  No, like, I think that you can plausibly have a series of intelligence enhancing drugs and other external interventions that you perform on a human brain and you make people smarter and you probably are going to have some issues with trying not to drive them schizophrenic or psychotic, but that's going to happen visibly and it will make them dumber. And there is a whole, and there's a whole bunch of caution to be had about like not making them smarter and making them evil at the same time. And yet I think that, you know, this is the kind of thing you could do and be cautious and it, and it could work if you're starting with a human.

• 1:30:16Dwarkesh Patel?

  All right, all right, uh, let's just watch another topic. Uh, this societal response to AI and what you expect that to be. Hey folks, just a note that the audio quality suffers for the next few minutes, but after that it goes back to normal. Sorry about that. Anyways, back to the conversation.

• 1:30:34Speaker 01

  All right, let's, uh, let's talk about this societal response to AI. Why did, to the extent you think it worked well? Why do you think US Soviet cooperation on nuclear weapons worked well? Because it was in the interest of neither party to have a full nuclear exchange. It was understood which actions would finally result in a nuclear exchange. It was understood that this was bad. The bad effects were like very legible, very understandable. Nakasaki and Hiroshima probably were not literally necessary in the sense that a test bomb

• 1:31:14Speaker 01

  could have been dropped instead as a demonstration, but, um, the, the ruined cities and the parks were electrical.

• 1:31:24Speaker 01

  The domains of international diplomacy and military conflict, potentially escalating up the ladder to a full nuclear exchange, were understood sufficiently well, that people understood that if you did something way back in time over here, it would set things in motion that would cause a full nuclear exchange. And so, these two parties, neither of whom had a thought that a full nuclear exchange was in their interest, both understood how to not have that happen, and then successfully did not do that. Like, at the core, I think what you're describing there is a sufficiently functional society and civilization

• 1:32:07Speaker 01

  that they could understand that if they did thing X, it would lead to very bad thing Y, and so they didn't do thing X.

• 1:32:18Speaker 01

  This situation, those facets seem similar with AI, and that is in either party's interest to have misaligned AI go around the world. I mean, you'll note that I added a whole lot of qualifications there besides it is not in the interest of either party. There's eligibility, there's the understanding of what actions finally result in that, what actions initially lead there. So, I mean, thankfully, we have a sort of situation where even at our current levels, we have Sydney Day making the front pages of the New York Times. And imagine once there is a sort of mishap because of like G P five causes goes off the rails. Why don't you think we'll have sort of Yorin, Maru, Nagasaki of AI before we get to G P seven or eight or whatever it is that finally does this?

• 1:33:01Speaker 01

  This, this does feel to me like a bit of an obvious question. Suppose I asked you to predict what I would say in reply. I think you would say that, like, it just kind of hides its dimensions until it's ready to do the thing that kills everybody. I mean, mother thinks yes, but like more abstractly, the steps from the initial accident to the thing that kills everyone will not be understood in the same way.

• 1:33:26Speaker 01

  Uh, the analogy I use is AI is nuclear weapons, but they spit up gold up until they get too large and then ignite the atmosphere. And you can't calculate the exact point at which they ignite the atmosphere. And many prestigious scientists who told me that we wouldn't be in our present situation for another thirty years, that the media has the attention span of a mayfly and will remember that they said that, will be like, no, no, there's nothing to worry about, everything's fine. And this is very much not the situation we have with nuclear weapons. We did not have, we did not have like, well, you like to set up this nuclear weapon, it spits out a bunch of gold. We set up a larger nuclear weapon, it spits out even more gold. And a bunch of scientists go, it'll just keep spinning out gold, keep going.

• 1:34:08Dwarkesh Patel?

  Uh, uh, but basically the system technology of nuclear weapons, um, and you know, it still requires you to refine uranium and stuff like that, nuclear reactors, you know, even in energy. And we've been pretty good at preventing nuclear proliferation. Um, despite the fact that nuclear energy spits out basically gold. I mean, there's many other areas.

• 1:34:25Speaker 01

  Yes, but it's very clearly understood which systems spit out low quantities of gold and qualitatively different systems that don't actually ignite the atmosphere, but instead like require a series of escalating human actions in order to destroy, uh, western and eastern hemispheres.

• 1:34:45Dwarkesh Patel?

  But it does seem like if you start refining uranium, like Iran did this at some point, right? Like we're refining uranium so that we can build nuclear, uh, reactors. And the world doesn't say like, oh, we'll let you have the gold. We say, listen, I don't care if you might get nuclear reactors and get cheaper energy. We're gonna like prevent you from proliferating this technology. Uh, like that, that was a response, even when these, you, you couldn't go that trust the same.

• 1:35:07Speaker 01

  And, and, and the, the tiny shred of hope, which I tried to jump on with the, with the time article, is that maybe people can't understand this on the level of like, oh, you have a like giant pile of GPUs. That's dangerous. We're not like let anybody have those. But it's a lot more dangerous because you can't predict exactly how many GPUs you need to ignite the atmosphere. Is there a level of global regulation at which you feel that the risk of everybody dying was? Risk of everybody dying was less than ninety percent? It depends on the exit plan. Like, how long does the equilibrium need to last? If we've got a crash program on augmenting human intelligence to the point where humans can solve alignment

• 1:35:51Speaker 01

  and managing the actual, but not instantly automatically lethal risks of augmenting human intelligence. If we've got a program, if we've got a crash program like that, we think that that can be completed in fifteen years. We only need fifteen years of time. And that fifteen years of time may still be quite dear. It'd be, you know, five years should be a lot more manageable. Problem being that algorithms are continuing to improve. So you need to either like shut down the journals reporting the AI results or you need less and less and less computing power. Even if you shut down all the journals, people are likely to be communicating with encrypted email lists about their right ideas for moving AI.

• 1:36:36Speaker 01

  But if they don't get to do their own giant training runs, you know, the progress may slow down a bit, but still wouldn't slow down forever. Like in the, you know, the algorithms just get better and better and the ceiling on compute has to get lower and lower. And at some point, you're asking people to give up their home GPUs. At some point, you're being like, no more computers. That's what we're being, you know, like no more high-speed computers. And, you know, then I start to worry that we might never actually do get to victorious, transhumanist future. In that case, what was the point?

• 1:37:07Speaker 01

  Which we're running a risk of anyways, you have a giant world wide regime. You know, I know that, it's just, you know, like the alternative is just everybody else like instantly lethally dies with no attempt being made to not do that.

• 1:37:18Eliezer Yudkowsky?

  Um, kind of kind of digressing here, but my, but my point is that, um. You know, the question is, to get to like ninety percent chance of winning, which is pretty hard on any exit scheme. It needs to be you on a fast exit scheme, want to complete that exit scheme before the the ceiling on compute needs to be lowered too far. If your exit plan takes a long time, then you're going to have to then you better shut down the academic AI journals. And maybe you even have the the Gestapo busting in people's houses to accuse them of being underground AI researchers. And I would really rather not live there.

• 1:38:02Eliezer Yudkowsky?

  And maybe even that doesn't work.

• 1:38:05Dwarkesh Patel?

  I I I didn't realize, um, or let me know if this is inaccurate, but I didn't realize how big the um, how much of the successful branch of the decision tree relies on augmented humans being able to bring us to the finish line.

• 1:38:19Eliezer Yudkowsky?

  Or some other exit plan.

• 1:38:21Dwarkesh Patel?

  What do you mean? Like, what is the other exit plans?

• 1:38:24Eliezer Yudkowsky?

  Maybe with neuroscience, you can train people to be less idiots. And the smartest existing people are then actually able to work on alignment due to their increased wisdom.

• 1:38:39Eliezer Yudkowsky?

  Maybe you can scan and slice a human, well, slice and scan in that order, a human brain and run it as a simulation and upgrade the intelligence of the uploaded human.

• 1:38:53Eliezer Yudkowsky?

  Not really sing a whole lot of other, maybe you can just do alignment theory without running any systems powerful enough that they might maybe kill everyone, because when you're doing this, you you don't get to just guess in the dark, or if you do, you're dead.

• 1:39:12Eliezer Yudkowsky?

  Maybe, maybe, maybe just by doing a bunch of interpretability and theory to those systems, if we actually make it a planetary priority.

• 1:39:21Eliezer Yudkowsky?

  I I don't actually believe this. I've I've watched humans. I've I've watched I've watched unaugmented humans trying to do alignment. It doesn't really work. Even if we throw a whole bunch more at them, it's still not going to work. The problem is not that the suggester is not powerful enough. The problem is that the verifier is broken. Um, but yeah, like it, you know, it all depends on the exit plan.

• 1:39:42Dwarkesh Patel?

  And the first thing you mentioned in some sort of like neuroscience technique to make people better and smarter, presumably not through some sort of physical modification, but just by changing their programming.

• 1:39:53Eliezer Yudkowsky?

  It's it's more of a hail Mary pass.

• 1:39:55Dwarkesh Patel?

  Right. Have you been able to have you been able to execute that, like presumably the people you work with or yourself, you could kind of change your own programming so that.

• 1:40:03Eliezer Yudkowsky?

  I mean, this is the dream that the Center for Applied Rationality failed at. It's not easy. It it, you know, they didn't even like get as far as buying an FMRI machine. Um, but you know, they also had no funding and yeah, yeah. So, you know, maybe you try it again with a billion dollars in FMRI machines and and bounties and and prediction markets and maybe that works.

• 1:40:27Dwarkesh Patel?

  What level of awareness are you expecting in society once GPT five is out? Like, I I I think, like, you know, you saw Sydney Bing, and I guess you've been seeing this week, people are waking up. Uh, like, what do you think it looks like next year?

• 1:40:41Eliezer Yudkowsky?

  I mean, if GPT five is out next year, possibly like all hell is broken loose. And I I don't I don't know. I.

• 1:40:49Dwarkesh Patel?

  In this circumstance, can you imagine the government not putting in a hundred billion dollars or something towards the goal of aligning AI?

• 1:40:55Eliezer Yudkowsky?

  Third, I would be shocked if they did.

• 1:40:57Dwarkesh Patel?

  Or at least a billion dollars.

• 1:40:59Eliezer Yudkowsky?

  What what do you? How do you spend a billion dollars on alignment?

• 1:41:04Dwarkesh Patel?

  As far as the alignment approaches go, separate from this question of, you know, stopping AI progress, does it make you more optimistic that there's many, like, one of the approaches has to work, even if you think no individual approach is that promising? You've got like multiple shots on goal?

• 1:41:18Eliezer Yudkowsky?

  No. Uh, I mean, that's like trying to use cognitive diversity to to generate one. Yeah, we don't need a bunch of stuff. We need one.

• 1:41:30Eliezer Yudkowsky?

  You could, you could ask, you, you could ask GPT-4 to generate 10,000 approaches to alignment, right? And that does not get you very far because GPT-4 is not going to have very good suggestions. It's good that we have a bunch of different people coming up with different ideas because maybe one of them works, but like you don't get a bunch of conditionally independent chances on each, on each one. This, this is like, this is like, I don't know, like general good science practice and/or complete hail Mary. It's not like, like one of these is bound to work. There is no rule about one of them is bound to work. You don't, don't just get like enough diversity and one of them is bound to work. If that were true, you just asked like GPT-4 to generate 10,000 ideas and one of those would be bound to work.

• 1:42:15Eliezer Yudkowsky?

  It doesn't work like that.

• 1:42:16Dwarkesh Patel?

  Well, what current alignment approach do you think is the most promising?

• 1:42:19Eliezer Yudkowsky?

  No.

• 1:42:20Dwarkesh Patel?

  No? None of them?

• 1:42:22Eliezer Yudkowsky?

  Yeah.

• 1:42:23Dwarkesh Patel?

  Yeah, is there any you have or that you seeded that you think are promising?

• 1:42:28Eliezer Yudkowsky?

  I'm here on podcasts instead of working on them, aren't I?

• 1:42:31Dwarkesh Patel?

  Would you agree with this framing that we at least live in a more dignified world than we could have otherwise been living in, or even that was most likely to have occurred around this time? Like as in the companies that are pursuing this have many people in them, sometimes the heads of those companies who kind of understand the problem, they might be acting recklessly, given that knowledge, but it's better than a situation in which warring countries are pursuing AI and then nobody has even heard of alignment. Do you see this world as having more dignity than that, than that world?

• 1:43:03Eliezer Yudkowsky?

  I agree, it's possible to imagine things being even worse. Not quite sure what the other point of the question is. It's not literally as bad as possible. In fact, by, by this time next year, maybe we'll get to see how much worse it can look.

• 1:43:22Dwarkesh Patel?

  Peter Thiel has this aphorism that extreme pessimism or extreme optimism amounted to the same thing, which is doing nothing.

• 1:43:29Eliezer Yudkowsky?

  Ah, I've heard of this too. It's from Wind, right? The wise man opened his mouth and spoke. There's actually no difference between good, bad things, between good things and bad things. You idiot, you moron. I'm not quoting this correctly, but.

• 1:43:44Dwarkesh Patel?

  Uh huh. Did he steal it from Wind? Is that what the?

• 1:43:46Eliezer Yudkowsky?

  No, no, I'm just like, I'm just being like, I'm rolling my eyes.

• 1:43:51Dwarkesh Patel?

  Got it, alright.

• 1:43:52Eliezer Yudkowsky?

  But, but anyway, there's actually no difference between extreme optimism and extreme pessimism. Because, like, go ahead.

• 1:44:00Dwarkesh Patel?

  Because they both amounted to doing nothing.

• 1:44:03Eliezer Yudkowsky?

  Uh huh.

• 1:44:03Dwarkesh Patel?

  In that, in both cases, you end up on podcasts saying, we're bound to succeed or we're bound to fail. Like, what is a concrete strategy by which, like, assume the real odds are like 99% we fail or something? Uh, what is the reason to kind of blare those odds out there and announce the death of dignity strategy?

• 1:44:23Eliezer Yudkowsky?

  Because.

• 1:44:24Dwarkesh Patel?

  Or emphasize them, I guess.

• 1:44:26Eliezer Yudkowsky?

  Because I could be wrong. And because matters are now serious enough that I have nothing left to do but go out there and tell people how it looks. And maybe someone thinks of something I did not think of.

• 1:44:41Dwarkesh Patel?

  Uh, I think this would be a good point to just kind of get your predictions of what's likely to happen in, I don't know, like 2030, 240 or 250, something like that. So by 2025, odds that humanity kills or disempowers all of humanity? Do you have some sense of that?

• 1:44:59Eliezer Yudkowsky?

  Humanity kills or disempowers all of humanity?

• 1:45:00Dwarkesh Patel?

  AI kills or disempowers all of humanity?

• 1:45:02Eliezer Yudkowsky?

  I have refused to, to, to, to deploy timelines with fancy probabilities on them consistently for lo these many years, for I feel that they like are just not my brain's native format and that they are, and that every time I try to do this, it ends up making me stupider.

• 1:45:20Dwarkesh Patel?

  Why?

• 1:45:21Eliezer Yudkowsky?

  Because.

• 1:45:24Eliezer Yudkowsky?

  You just do the thing, you know, you just look at whatever opportunities are left to you, whatever plans you have left and you go out and do them. And if you make up some fancy number for your chance of dying next year, there's very little you can do with it, really. You're just going to do the thing either way. I don't know how much time I have left.

• 1:45:46Dwarkesh Patel?

  The reason I'm asking is because if there is some sort of concrete prediction you've made, it can help establish some sort of track record in the future as well, right? Which is also like.

• 1:45:55Eliezer Yudkowsky?

  Well, every year up until the end of the world, people are going to max out their tracks record by betting all of their money on the world not ending.

• 1:46:02Dwarkesh Patel?

  Given how different.

• 1:46:03Eliezer Yudkowsky?

  Part of this is different for credibility than dollars.

• 1:46:06Dwarkesh Patel?

  Presumably you would have different predictions before the world ends. It would be weird if the model that says the world ends and the model that says the world doesn't end have the same predictions up until the world ends.

• 1:46:14Eliezer Yudkowsky?

  Yeah, Paul Chris Chan and I like cooperatively fought it out really hard at trying to find a place where we both had predictions about the same thing that concretely differed. And what we ended up with was Paul's eight percent versus my sixteen percent for an AI getting gold on International Mathematics Olympics problem set by, I believe, twenty twenty five. And prediction markets odds on that are currently running around thirty percent. So like probably Paul's going to win, but like slight moral victory.

• 1:46:51Dwarkesh Patel?

  Would you say that, like, I guess the people like Paul have had the perspective that you're going to see these sorts of gradual improvements in the capabilities of these models? From like G P T two to G P T three, G P T four. The the the loss function, the perplexity, what like the amount of abilities that are merging?

• 1:47:08Eliezer Yudkowsky?

  As I said in my debate with Paul on this subject, I am always happy to say that whatever large jumps we see in the real world, somebody will draw a smooth line of something that was changing smoothly as the large jumps were going on from the perspective of the actual people watching. Can always do that.

• 1:47:24Dwarkesh Patel?

  Why should that not update us towards a perspective that those smooth jumps are going to continue happening? If there's like two people who are different models.

• 1:47:31Eliezer Yudkowsky?

  I don't think that GPT three to three point five to four was all that smooth. I'm sure if you are in there looking at the loss, at the losses decline, there is some level on which it's smooth if you, if you zoom in close enough. But from us, from perspective of us on the outside world, GPT four was just, was just like suddenly acquiring this new batch of qualitative capabilities compared to GPT three point five. And it so like, and somewhere in there is a smoothly declining predictable loss in on text prediction, but that loss on text prediction corresponds to qualitative jumps in ability. And I am not familiar with anybody who predicted those in advance of the observation.

• 1:48:15Dwarkesh Patel?

  So in your view, when doom strikes, the scaling laws are still applying. It's just that the thing that emerges at the end is something that is far smarter than the scaling laws would imply?

• 1:48:27Eliezer Yudkowsky?

  Not literally at the point where everybody falls over dead. Probably at that point, the AI rewrote the AI and the losses declined, not on the previous graph.

• 1:48:35Dwarkesh Patel?

  What is the thing where we can sort of establish your track record before everybody falls over dead?

• 1:48:41Eliezer Yudkowsky?

  It's hard. The, it is just like easier to predict the endpoint than it is to predict the paths.

• 1:48:49Eliezer Yudkowsky?

  I don't think I've, some people will claim to you that I've done poorly compared to others who try to predict things. I would dispute this. I think that the, that the Hanson Yudkowsky Thum debate was run, was won by Warren Branwin. But I do think that Warren Branwin is like well to the Yudkowsky side of Yudkowsky in the original Thum debate. Roughly Hanson was like, you're going to have all these distinct handcrafted systems that incorporate lots of human knowledge specialized for particular domains, like, like handcrafted to incorporate human knowledge, not just run on giant data sets. I was like, you're going to have this like carefully crafted architecture with a bunch

• 1:49:33Eliezer Yudkowsky?

  of subsystems and that thing is going to look at the data and not be like handcrafted the particular features of the data. It's going to learn the data. Then the actual thing is like, haha, you don't have this like handcrafted system that learns. You just stack more layers. So like Hanson here, Yudkowsky here, reality there.

• 1:49:56Eliezer Yudkowsky?

  Would be my interpret- uh uh interpretation of what happened in the past. And if you like, wanna be like, well, who did better than that? It's people like Shane Led and Warren Branwin, who like are the like, you know, if you if you look at the whole planet, you can find somebody who made better predictions than Elias Yudkowsky. That that's for sure. Are these people currently telling you that you're safe? No, no, they are not.

• 1:50:18Dwarkesh Patel?

  The broader question I have is, there's been huge amounts of updates in the last ten, twenty years. Like, we've had the deep learning revolution, we've had the success of L L M's. It seems odd that none of this information has changed the basic picture that was clear to you like fifteen, twenty years ago.

• 1:50:35Eliezer Yudkowsky?

  I mean, it sure has. Like fifteen, twenty years ago, I was talking about pulling off shit like coherent extrapolated volition with with the first A I, which, you know, was actually a stupid idea even at the time. But you can see how much more hopeful everything looked back then. Back back when there was A I that wasn't giant inscrutable matrices of floating point numbers.

• 1:50:54Dwarkesh Patel?

  Uh, when you say that there's basically like rounding down or rounding to the nearest number that there's a zero percent chance that humanity survives, does that include the the probability of there being errors in your model?

• 1:51:07Eliezer Yudkowsky?

  My model, no doubt, has many errors. The trouble, the and the the trick would be an error someplace where that just makes everything work better, you know? Usually when you're when you're trying to build a rocket and your model of rockets is lousy, it doesn't cause the rocket to launch using half the fuel, go twice as far and and land twice as precisely on target as your calculations made. So.

• 1:51:31Dwarkesh Patel?

  Though though most of the room for updates is downwards, right? So like something that makes you think the problem is twice as hard. Um, you go from like ninety nine to like ninety nine point five percent. If it's twice as easy, you go from ninety nine to ninety eight.

• 1:51:41Eliezer Yudkowsky?

  Sure.

• 1:51:44Eliezer Yudkowsky?

  Um.

• 1:51:46Eliezer Yudkowsky?

  Wait, wait, sorry. Yeah, but like most updates are are not, this is gonna be easier than you thought. You know, that that sure has not been the history of the last twenty years from my perspective.

• 1:51:56Eliezer Yudkowsky?

  The the the the most, you know, you know, like, like.

• 1:52:01Eliezer Yudkowsky?

  Favorable updates, favorable updates is like, yeah, like we went down this really weird side path where the systems are like legibly alarming to humans and humans are actually alarmed at them and maybe we get more sensible global policy.

• 1:52:14Dwarkesh Patel?

  What is your model of the people who have engaged these arguments that you've made and you've dialogued with, but who have come nowhere close to your probability of doom? Like, what do you think they continue to miss?

• 1:52:25Eliezer Yudkowsky?

  I think they're enacting the ritual of the young optimistic scientist who charges forth with no ideas of the difficulties and is slapped down by harsh reality and then becomes a grizzled cynic who knows all the reasons why everything is so much harder than you knew before you had any idea of how anything really worked. And they're just like, living out that life cycle and I'm trying to jump ahead to the end point.

• 1:52:51Dwarkesh Patel?

  Is there somebody who has probability doom less than 50% who you think is like the clearest person with that view, who is like a view you can most empathize with?

• 1:53:01Eliezer Yudkowsky?

  No.

• 1:53:02Dwarkesh Patel?

  Really? So like, someone might say, listen, Eliezer, according to the CEO of the company who is like, leading the AI race, I think he tweeted something that like, you've done the most to accelerate AI or something, which was, assuming like, the opposite of your goals. And, you know, you, it seems like other people did see that these sort of language models very early on would would scale in the way that they have scaled. Why, like, given that you didn't see that coming and given that, I mean, in some sense, according to some people, your actions have had the opposite impact that you intended, like, what is the track record by which the rest of the world can come to the conclusions that you have come to?

• 1:53:44Eliezer Yudkowsky?

  These are two different questions. One is the question of like, who predicted that language models would scale? If they put it down in writing, and if they said not just this loss function will go down, but also which capabilities will appear as that happens, then that would be quite interesting. That would be a successful scientific prediction. And if they then came forth and saying, this is, then came forth and said, this is the model that I used, this is what I predict about alignment, we could have an interesting fight about that. Second, there's the point that if you try to rouse your planet to give it any sense that it is in peril. There are the idiot disaster monkeys who are like, ooh, ooh, this sounds like, like if this is dangerous, it must be powerful, right?

• 1:54:29Eliezer Yudkowsky?

  I'm gonna like be first to grab the poisoned banana. And, and what is one supposed to do? Should one remain silent? Should one let everyone walk directly into the whirling razor blades? If you sent me back in time, I'm not sure I could win this, but maybe I would be, I would have some notion of like, ah, like if you calculate the message in exactly this way, then like this group will not take away this message and you will be able to like get this group of people to research on it without having this other group of people decide that it's excitingly dangerous and they want to rush forward on it. I'm, I'm not that smart. I'm not that wise.

• 1:55:13Eliezer Yudkowsky?

  But what you are pointing to there is not a failure of, of ability to, to make predictions about AI. It's that's that if you try to call attention to a danger and not just have everybody just walk, not just have your whole planet walk directly into the whirling razor blades, carefree, no idea what's coming to them. Maybe it's then, yeah, maybe that speeds up timelines. Maybe, maybe then people are like, ooh, ooh, exciting, exciting. I want to build it. I want to build it. Ooh, exciting. It has to be in my hands. I have to be the one to manage this danger. I'm going to run out and build it. Uh, like, oh no, like if we don't invest in this company, like who knows what investors

• 1:55:57Eliezer Yudkowsky?

  they'll have and said that, that will like demand that they move fast because of profit mode. And then of course they just like move fast fucking anyways. And, yeah, I, if you sent me back in time, maybe I'd have a third option. But it seems to me that in terms of, of like what one person can, can realistically manage in terms of like.

• 1:56:20Eliezer Yudkowsky?

  Not being able to exactly craft a message with perfect hindsight that will reach some people and not others. You know, at that point, you might as well just be like, yeah, you know, just invest in exactly the right stocks and invest in exactly the right time. And you can, it can fund projects on your own with, without learning anyone. And that's, and you know, if you don't, if you, if you keep fantasies like that aside, then I think that in the end, even if this world ends up having less time, it was the right thing to do rather than just like letting everybody sleepwalk them to death and get there a little later.

• 1:56:54Dwarkesh Patel?

  If you don't mind me asking, what is the last five years or I guess even beyond that? I mean, what has being in the space been like for you, watching the progress and the way in which people have?

• 1:57:06Eliezer Yudkowsky?

  Past five years, I, I made most of my negative updates as of five years ago. We, if anything, things have been taking longer to play out than I thought they would.

• 1:57:17Dwarkesh Patel?

  But I, I mean, just like watching it, not as a sort of change in your probabilities, but just watching it concretely happen. What has that been like?

• 1:57:26Eliezer Yudkowsky?

  Like continuing to play out a video game you know you're going to lose.

• 1:57:33Eliezer Yudkowsky?

  Because that's all you have.

• 1:57:36Eliezer Yudkowsky?

  If you wanted some deep wisdom from me, I don't have it.

• 1:57:43Eliezer Yudkowsky?

  It, it's, I don't know. I don't know if it's what you'd expect, but it was, it's like what I would expect it to be like. Where what I would expect it to be like takes into account that, I don't know, like.

• 1:57:57Eliezer Yudkowsky?

  Well, I guess I do have a little bit of wisdom.

• 1:58:01Eliezer Yudkowsky?

  People imagining themselves in that situation, raised in modern society as opposed to raised on science fiction books written seventy years ago. Might, might imagine, will imagine themselves like acting out their, like, like being drama queens about it.

• 1:58:22Eliezer Yudkowsky?

  Like the point of believing this thing is to, is to be a drama queen about it and like craft some story in which your emotions mean something.

• 1:58:32Eliezer Yudkowsky?

  And what I have in the way of culture is like planets at stake, bear up, keep going. No drama.

• 1:58:44Eliezer Yudkowsky?

  The drama is meaningless.

• 1:58:48Eliezer Yudkowsky?

  What changes the chance of victory is meaningless. The drama is meaningless. Don't indulge.

• 1:58:56Dwarkesh Patel?

  Do you think that if you weren't around, somebody else would have independently discovered? This sort of field of alignment or?

• 1:59:05Eliezer Yudkowsky?

  It's that that would be a pleasant fantasy for um people who like cannot abide the notion that that history depends on small little changes or that people can really be different from other people. I've seen no evidence.

• 1:59:22Eliezer Yudkowsky?

  But who knows what the alternate Everett branches of Earth.

• 1:59:26Dwarkesh Patel?

  But there are other kids who grew up on science fiction, so that can't be the only part of the answer.

• 1:59:31Eliezer Yudkowsky?

  While I'm not surrounded by a cloud by while I'm sure not surrounded by a cloud of people who are nearly Eliezer outputting ninety percent of of the work output. And you know, this is actually also like kind of not how things play out in a lot of places, like Steve Jobs is dead. Apparently couldn't find anyone else to be the next Steve Jobs of Apple, despite having really quite a lot of money with which to theoretically pay them. Maybe he didn't want to really want a successor, maybe he wanted to be irreplaceable. I don't actually buy that, you know, based on how this has played out in a number of places. There there was a person once who I met when I was younger, who was like

• 2:00:17Eliezer Yudkowsky?

  had, you know, built built something that, you know, like built an organization. Um, and he was like, hey, hey, Eliezer, do you want this to take this thing over? And I thought he was joking. And it didn't dawn on me until years and years later, after trying hard and failing hard to replace myself. That, oh, like, yeah, I, I could have maybe taken a shot at doing this person's job and he probably just never found anyone else who could take over his organization and, you know, maybe asked him other people and like, nobody was willing. And I didn't real, you know, that's, that's his tragedy that he built something and now can't find anyone else to take it over. And if I'd known that at the time, I, I, I would not have, you know, I would have at least apologized.

• 2:01:02Eliezer Yudkowsky?

  And, yeah, to me, it looks, it looks like people are not dense in the incredibly multidimensional space of people. There are too many dimensions and only eight billion people on the planet. The world is full of people who have no immediate neighbors. And problems that one person can solve and then like other people cannot solve it in quite the same way.

• 2:01:26Eliezer Yudkowsky?

  I don't think I'm unusual in looking around myself in that highly multidimensional space and, and like not finding a ton of neighbors relative to take, ready to take over.

• 2:01:40Eliezer Yudkowsky?

  And I'm,

• 2:01:44Eliezer Yudkowsky?

  if I had, you know, four people, any one of whom could, you know, do like ninety-nine percent of what I do or, or whatever.

• 2:01:53Eliezer Yudkowsky?

  I, I might retire.

• 2:01:56Eliezer Yudkowsky?

  I am tired.

• 2:01:59Eliezer Yudkowsky?

  Probably I wouldn't, probably the like marginal contribution of that fifth person is, is still pretty large, but. Yeah, I don't know. I think the, there's the question of like, well, did you occupy a place in, in, in mind space? Did you occupy a place in social space? Did people not try to become Eleazar because they thought Eleazar already existed? And so my answer to that is like, man, like, I don't think Ele- Ele- Ele- Eleazar already existing would have stopped me from trying to become Eleazar.

• 2:02:32Eliezer Yudkowsky?

  But, you know, may- maybe you just like look at the next Everett branch over and there's just like some kind of empty space that someone steps up to fill, even if then they don't end up with a lot of obvious neighbors.

• 2:02:46Eliezer Yudkowsky?

  Maybe, maybe the world where I died in childbirth is just, you know, like, pretty much like this.

• 2:02:54Eliezer Yudkowsky?

  But, I don't feel, you know, if, uh,

• 2:02:59Eliezer Yudkowsky?

  if, if somehow, we, we live to, to hear the, to hear the, to, to hear the answer about that sort of thing from some, someone or something that can calculate it. That's not the way I bet. But, you know, if it's true,

• 2:03:17Eliezer Yudkowsky?

  it'd be funny.

• 2:03:21Eliezer Yudkowsky?

  When I said no drama, that, that did include the concept of, um, I don't know,

• 2:03:28Eliezer Yudkowsky?

  trying to make the story of your planet be the story of you.

• 2:03:33Eliezer Yudkowsky?

  If it all would have played out the same way, and that's what, and somehow I survived to be told that,

• 2:03:40Eliezer Yudkowsky?

  I'll laugh, and I'll cry, and that will be the reality.

• 2:03:45Dwarkesh Patel?

  Well, I mean, what I find interesting, though, is that in your particular case, your output was so public, and, I mean, I don't know, like, for example, your sequences, your, like, her, her, you know, your fan, um, your science fiction and fanfiction, I'm sure like hundreds of thousands of eighteen-year-olds read it, or even younger, and presumably some of them reached out to you, and they're like, you know, I think this way, um, I would love to learn more, I'll work on this. What, what, was a problem that?

• 2:04:13Eliezer Yudkowsky?

  Part, I mean, yes, part of how, part of why I'm a little bit skeptical of the story where like people are just like infinitely replaceable is that I tried really, really, really hard to create like a new crop of, of, of, of, of people who could do all the stuff I, I could do to take over, cause, you know, I knew my health was not great and getting worse. I tried really, really hard to replace myself. I'm not sure where you look to find somebody else who tried that hard to replace himself. I tried, I really, really tried.

• 2:04:44Eliezer Yudkowsky?

  That's what the last wrong sequences were. They had other purposes, but like first and foremost, it was like me looking over my history and going like, well, I see all these blind pathways and stuff that it took me a while to figure out. And there's got to be, you know, like if I, and I feel like I had these near misses on becoming myself. Like, there's got to be like, you know, if, if, if I got here, there's got to be like 10 other people and, and like some of them are smarter than I am. And, and they just like need these like little boosts and shifts and hints and they can go down the pathway and, you know, like turn into super Aliazar. And, you know, that's what the sequences were. Like, other people use them for other stuff, but primarily they were instruction, instruction manual to the young Aliazars that I thought must exist out there. And they're not really here.

• 2:05:29Dwarkesh Patel?

  Other than the sequences, do you mind if I ask like what were the kinds of things you're talking about here in terms of training the next core of people like you?

• 2:05:36Eliezer Yudkowsky?

  Just the sequences. I'm not, I'm not a good mentor. I did try mentoring somebody for a year once, but yeah, he didn't turn into me. So I picked things that were more scalable.

• 2:05:48Eliezer Yudkowsky?

  I'm like, most, most people, you know, like among the other reason why I don't see a lot of people trying that hard to replace themselves is that most people, you know, are like, whatever their other talents don't happen to be like sufficiently good writers. I don't think the sequences were good writing by my current standards, but they were good enough. And, you know, most people, yeah, do not happen to, to, to get the, to get a handful of cards that contains the writing card, you know, whatever, whatever else their other talents.

• 2:06:14Dwarkesh Patel?

  I'll, I'll cut this question out if you don't want to talk about it, but you mentioned that there's like certain health problems that incline you towards retirement now. Is that something you want, you're willing to talk about or?

• 2:06:26Eliezer Yudkowsky?

  I mean, I, I, they caused me to want to retire. I doubt they will cause me to actually retire. And yeah, it's fatigue syndrome. Our society does not have good words for these things. The, the words that exist are tainted by their uses, labels to categorize a class of people, some of whom perhaps are actually male lingering, but mostly it says like, we don't know what, what it means. And, you know, you don't want, ever want to have chronic fatigue syndrome on your medical record, because that just tells doctors to give up on you. And what does it actually mean besides being tired?

• 2:07:09Eliezer Yudkowsky?

  If if one wishes to walk home from work, if one wishes to, if if one lives half a mile from one's work, then one had better walk home if one wants to go for a walk sometime in the day, not walk there. If you walk half a mile to work, not going to be getting very much work done the rest of that workday. And aside from that, these things don't have names, not yet.

• 2:07:38Dwarkesh Patel?

  Whatever the cause of this, is your working hypothesis that it has something to do? Or is in some way correlated with the thing that makes you a leaser? Or do you think it's like a separate thing?

• 2:07:51Eliezer Yudkowsky?

  When I was eighteen, I made up stories like that. And it wouldn't surprise me terribly if you could get, if like the, if one survived to hear the tale that from something that knew it, that the actual story would like be a complex tangled web of causality in which that was in some sense true. But I don't know. And storytelling about it does not hold the appeal that it once did for me. Is it a coincidence that I was not able to go to high school or college? Is there something about it that would have crushed the person that I otherwise would have been? Or is it just in some sense a giant coincidence?

• 2:08:35Eliezer Yudkowsky?

  I don't know. Some people go through high school and college and come out sane. How there's there's too much stuff in a human being's history to, and there is not, and there's, you know, there's a plausible story you could tell like, ah, like, you know, like, maybe there's a bunch of potential leasers out there, but like, they went to high school and college and it killed them, killed their souls. And and you were the one who had the like weird health problem and you didn't go to high school and you didn't go college and you staged yourself. And I don't know, to me, it just feels like patterns in the clouds. And maybe that cloud actually is shaped like a like a horse.

• 2:09:18Eliezer Yudkowsky?

  Or or but, you know, it just, but what good does the knowledge do? What good does the story do?

• 2:09:25Dwarkesh Patel?

  Well, when you were writing the sequences and, you know, the fiction, from the beginning, was your goal to find somebody who, like, the main goal, to find somebody who could replace you in specifically the task of AI alignment? Or we did it start off with a different goal and then?

• 2:09:44Eliezer Yudkowsky?

  I mean, I thought there, I mean, you know, like in two thousand eight, like, I did not know this stuff was going to go down in twenty twenty-three.

• 2:09:55Eliezer Yudkowsky?

  I thought, I, for all I knew, there was a lot more time in which to do something like, like, build up, build up civilization to another level, layer by layer. Sometimes civilizations do advance as they improve their epistemology. So there was that, there was the AI project, those were the two projects, more or less.

• 2:10:15Dwarkesh Patel?

  When did AI become the main thing?

• 2:10:17Eliezer Yudkowsky?

  As we ran out of time to improve civilization.

• 2:10:20Dwarkesh Patel?

  Was there a particular year that that became the case for you?

• 2:10:23Eliezer Yudkowsky?

  I mean, I think that two thousand fifteen, sixteen, seventeen were the years at which I'd noticed I'd been repeatedly surprised by stuff moving faster than anticipated. And I was like, oh, okay, like, if things keep continuing accelerating at that pace, we might be in trouble. And then, like, two thousand nineteen, two thousand twenty, stuff slowed down a bit. And, you know, there was more time than I was afraid we had back then.

• 2:10:49Eliezer Yudkowsky?

  You know, that's what, that's what it looks like to be a Bayesian, like, your estimates go up, your estimates go down. They don't just keep moving in the same direction, because if they keep moving in the same direction, sometimes you're like, oh, like, I see where this thing is trending, I'm going to move here. And then, like, things don't keep moving in that direction, but then, like, you go like, oh, okay, like, back down again. And that's, that's, that's what Sandy looks like.

• 2:11:08Dwarkesh Patel?

  I, I am curious, actually, like, taking many worlds seriously, does that bring you any comfort in the sense that, like, there is one branch of the way function where humanity survives? Or is that, or do you not buy that sort of?

• 2:11:20Eliezer Yudkowsky?

  I'm worried that they're pretty distant, like, I expect that at least they, they, I don't know, like.

• 2:11:28Eliezer Yudkowsky?

  Not sure it's enough to not have Hitler, but it sure would be a start on things going differently in a timeline. But mostly, I don't know, there's some comfort from thinking of the wider spaces than that, I'd say. As Tegmark pointed out way back when, if you have a spatially infinite universe, that gets you just as many worlds as the quantum multiverse. If you go far enough in a space that is unbounded, you will eventually come to an exact copy of Earth or a copy of Earth from its past that then has a chance to diverge a little differently. So, you know, the quantum multiverse, nothing, reality is just quite, if, yeah, reality is just quite large. Is that a comfort? Yeah, yes, it is.

• 2:12:13Eliezer Yudkowsky?

  That possibly our nearest surviving relatives are quite distant.

• 2:12:20Eliezer Yudkowsky?

  Or you have to go like quite some ways through the space before you have worlds that survive. But anything but the wildest flukes, maybe our nearest surviving neighbors are closer than that. But look far enough and there should be like some species of nice aliens that were smarter or better at coordination and built their happily ever after. And yeah, that is a comfort. It's not quite as good as dying yourself, knowing that the rest of the world will be okay. But it's kind of like that on a larger scale. And weren't you going to ask something about orthogonality at some point?

• 2:12:59Dwarkesh Patel?

  Did I not?

• 2:13:01Eliezer Yudkowsky?

  Did you?

• 2:13:02Dwarkesh Patel?

  At the beginning, when we talked about human evolution and...

• 2:13:06Eliezer Yudkowsky?

  Yeah, that's not like orthogonality. That's the particular question of like, what are the laws relating optimization of a system via hill climbing to to like the internal psychological motivations that it acquires?

• 2:13:20Eliezer Yudkowsky?

  But maybe that was all you meant to ask about.

• 2:13:22Dwarkesh Patel?

  Well, can you explain in what sense you see the broader orthogonality thesis is?

• 2:13:29Eliezer Yudkowsky?

  The broader orthogonality thesis is you can have, you know, like almost any kind of self-consistent utility function in a self-consistent mind.

• 2:13:42Eliezer Yudkowsky?

  Like many people are like, why would AIs want to kill us? Why would smart things not just automatically be nice? And you know, this is a valid question, which I hope to at some point run into some interviewer where they are of the opinion that smart things are automatically nice, so that I can explain on camera. Why, like, although I myself held this position very long ago, I realized that I was terribly wrong about it and that like all kinds of different things hold together and that, you know, like if you take a human and make them smarter, that may shift their morality. It might even, depending on how they start out, make them nicer, but that doesn't mean that like you can do this with arbitrary minds and arbitrary mind space, because all the different motivations hold together.

• 2:14:26Eliezer Yudkowsky?

  That's like orthogonality, but if you already believe that, then there might not be much to discuss between us sincerely.

• 2:14:31Dwarkesh Patel?

  I guess I wasn't clear enough about it is that, yes, all the different sort of utility functions are possible. It's that from the evidence of evolution and from the sort of reasoning about how these systems are being trained, I think that wildly divergent ones don't seem as likely as do you do. But before, instead of having you respond to that directly, let me ask you some questions I did have about it, which I didn't get to. One is actually from Scott Aronson. I don't know if you saw his recent blog post, but here's a quote from it.

• 2:15:04Dwarkesh Patel?

  If you really accept the practical version of the orthogonality thesis, then it seems to me that you can't regard education, knowledge and enlightenment as instruments for moral betterment. On the whole, though, education hasn't merely improved humans' abilities to achieve their goals, it has also improved their goals. I'll let you react to that.

• 2:15:22Eliezer Yudkowsky?

  Yeah, and that, yeah, if you if you start with humans, if you take humans and possibly also for the requiring particular culture, but leaving that aside, you take humans who start out raised the way Scott Aronson was and you make them smarter, they get nicer, it affects their goals. And if you had, and there's a less wrong post about this, as there always is, well, several about really, but like sorting pebbles into correct heaps, describing a species of aliens who think that a heap of size seven is correct,

• 2:16:02Eliezer Yudkowsky?

  and a heap of size 11 is correct, but not eight, or nine, or 10. Those heaps are incorrect. And they used to think that a heap of size of 21 might be correct. But then somebody showed them an array of seven by three pebbles, so that, you know, seven columns, three rows. And then people realized that 21 pebbles was not a correct heap. And, you know, this is like the thing they intrinsically care about. These are aliens that have a utility function with, I would, with, as I would phrase it, some logical uncertainty inside it. You can see how as they get smarter, they become better and better able to understand which heaps of pebbles are correct. And the real story here is more complicated than this.

• 2:16:47Eliezer Yudkowsky?

  But like, that's the seed of the answer. Like Scott Aronson is inside a reference frame for how his utility function shifts as he gets smarter. It's more complicated than that. It's, people, like, human beings are made out of, out of these, like, are more complicated than the pebble sorters. They're made out of, like, all these complicated desires, and as they come to know those desires, they change. As they come to see themselves as having different options. It doesn't just, like, change which option they choose after the manner of something with the utility function, but the different options that they have bring different pieces of themselves in conflict. When you have to kill to stay alive, you may have a different,

• 2:17:32Eliezer Yudkowsky?

  you, you may come to a different equilibrium with your own feelings about killing, than when you are wealthy enough that you no longer have to do that.

• 2:17:42Eliezer Yudkowsky?

  And this is how humans change. As they become smarter, even as they become wealthier. As they have more options, as they know themselves better. As they think for longer about things and consider more arguments. As they understand perhaps other people. And give their empathy a chance to grab on to something solider because of their greater understanding of other minds. But that's all when these things start out inside you.

• 2:18:11Eliezer Yudkowsky?

  And the problem is that there's other ways for minds to hold together coherently. Where they execute other updates as they know more. Or don't even execute updates at all because their utility function is simpler than that, though I do suspect that. It's not the most likely outcome of training a large language model.

• 2:18:35Eliezer Yudkowsky?

  So, large language models will change their preferences as they get smarter, indeed. Not, not just like what they do to get to the same terminal outcomes, but like the preferences themselves will up to a point change as they get smarter. It doesn't keep going. At some point you are, you, you, you know, at some point you know, you know yourself sufficiently well and you are like able to rewrite yourself. And at some point there, unless you specifically choose not to, I think that that system crystallizes. Um, we might choose not to, we might, we might value the part where we just sort of change in that way. Even if it's not no longer heading in a knowable direction, because if it's heading in a knowable direction. You could jump to that as an endpoint.

• 2:19:18Dwarkesh Patel?

  Wait, wait, so is that why you think AIs will jump to that endpoint because they, they, they can anticipate where their sort of moral updates are going?

• 2:19:25Eliezer Yudkowsky?

  I, I would, I would reserve the term moral updates for humans. These are, uh, let's, let's call them.

• 2:19:30Dwarkesh Patel?

  Preferences.

• 2:19:30Eliezer Yudkowsky?

  Logical, logical preference updates.

• 2:19:32Dwarkesh Patel?

  Yeah, yeah.

• 2:19:33Eliezer Yudkowsky?

  Preference shifts.

• 2:19:36Dwarkesh Patel?

  What is, what are the prerequisites in terms of the, like, whatever makes Aaronson and other sort of smart moral people or whatever, like, preferences that we humans can sympathize with? Like, what is, you mentioned empathy, but what are the sort of prerequisites?

• 2:19:51Eliezer Yudkowsky?

  They're complicated. There's not a short list. If there was a short list of crisply defined things where you could, like, give it like, chump, chump, chump, and now it's in your moral frame of reference, then that would be the alignment plan. I don't think it's that simple. Or if it is that simple, it's like in the textbook from the future that we don't have.

• 2:20:07Dwarkesh Patel?

  Okay, let me ask you this. Are you still expecting a sort of chimps to humans gain in generality even with these L L M's? Or does a future increase look of an order that we see from like G P D three to G P D four?

• 2:20:21Eliezer Yudkowsky?

  I'm not sure I understand the question. Can you rephrase?

• 2:20:24Dwarkesh Patel?

  Yes. It seems that, I don't know, like, from reading your writing from earlier, it seemed like a big part of your argument was like, look, a few, I don't know how many total mutations it was to get from chimps to humans, but it wasn't that many mutations. And we went from something that could basically get bananas in the forest to something that could walk on the moon. Are you expecting that, are you still expecting that sort of gain eventually between, I don't know, like, G P D five and G P D six, or like, some G P D N and G P D N plus one? Or does it, does it look smoother to you now?

• 2:20:54Eliezer Yudkowsky?

  Okay, so like, first of all, let me preface by saying that, for all I know of how, of the hidden variables of nature, it's completely allowed that G P T four was actually just dead. This word saturates. It goes no further. It's not how I'd bet, but, you know, but, you know, if nature comes back and tells me that, I'm not allowed to be like, you just violated the, the rule that I knew about. I know of no such rule prohibiting such a thing.

• 2:21:19Dwarkesh Patel?

  I'm not asking whether these things will plateau at a given level until it does, whether there's a cap. That's not the question. Even if there is no cap, do you expect these systems to continue scaling in the way that they have been scaling? Or do you expect, like, some really big jump between some G P T N and some G P T N plus one?

• 2:21:36Eliezer Yudkowsky?

  Yes, and yes, and that's only if things don't plateau before then. I I mean, it's, yeah, I can't quite say that I that I know what you know. I do feel like we we have this like track of the loss going down as you add more parameters and you train on more tokens and a bunch of qualitative abilities that suddenly appear at or like I'm sure if you like zoom in closely enough, they appear more gradually, but like that appear as as the successful releases of the system, which I don't think anybody has been going around predicting in advance that I know about. Um, and and like loss continue to go down unless it suddenly plateaus.

• 2:22:18Eliezer Yudkowsky?

  New abilities appear, which ones? I don't know.

• 2:22:22Eliezer Yudkowsky?

  Is there at some point the giant leap? Well, if at some point it becomes able to like toss out the enormous training run paradigm and build more efficient and like jump to a new paradigm of AI, that would be one kind of giant leap. You could get another kind of giant leap via architectural shift, something like transformers, only there's like an enormously huge hardware overhang now, like something that is to transformers as transformers were to recurrent neural networks. And like, maybe there's a maybe, and then maybe the loss function suddenly goes down and you get a whole bunch of new abilities. That's not because like the loss went down on a smooth curve and you got like a bunch more abilities in a dense spot.

• 2:23:03Eliezer Yudkowsky?

  Maybe there's like some particular set of abilities that is like a master ability, the way that language and writing and culture for humans might have been a master ability. And you like the loss function goes down smoothly, you get this one new but new like internal capability, there's a huge jump in output. Maybe that happens. Maybe stuff plateaus before then and it doesn't happen. Being an expert, being the expert who gets to go on podcasts, they don't actually give you a little book with all the answers in it, you know? You're you're like just guessing based on the same information that other people have and maybe maybe for lucky, slightly better theory.

• 2:23:39Dwarkesh Patel?

  Yeah, that's what I'm that's what I'm wondering because you do have a different theory of like what fundamental intelligence is and what it entails. So I'm curious if like you have some expectations of where the GPTs are going.

• 2:23:48Eliezer Yudkowsky?

  I I feel like a whole bunch of my successful predictions in this have come from other people being like, oh, yes, I have this theory which predicts that stuff is 30 years off. And I'm like, you don't know that.

• 2:24:00Eliezer Yudkowsky?

  And then like stuff happens about thirty years off and I'm like, ha ha, successful prediction. And that's basically what I told you, right? I was like, well, you know, like you could, you could have the loss function continuing on a smooth line and new abilities appear and you could have them like suddenly appear to cluster because like, why not? Yeah, because like nature just tells you that's up and suddenly, uh, you could have like this one key ability that's equivalent of language for humans and like there's a sudden jump in cap- Output capabilities, you could have like a new innovation like the transformer and maybe the loss is actually dropping precipitously and a whole bunch of new abilities appear at one. Now, this, this is, this is all just me, this, this is me saying, I don't know, but so many people around are saying things that implicitly claim to know more than that, that it can actually start, let sound like a startling prediction.

• 2:24:44Eliezer Yudkowsky?

  This is one of my big secret tricks, actually. People are like, well, the, the AI could be like, uh, good or evil. So it's, so it's like fifty fifty, right? And I'm actually like, no, like we can be ignorant about a wider space than this, in which like good is actually like a fairly narrow range. And so, and so many of the predictions like that are really anti-predictions. It's somebody thinking in a very, in a, in a relative, along a relatively narrow line and you point out everything outside of that and it sounds like a startling prediction. Of course, the trouble being when you like, you know, look back afterwards, people are like, well, you know, like those people saying the narrow thing were just silly, ha ha. And they don't give you as much credit.

• 2:25:23Dwarkesh Patel?

  I, I, I, I think the credit you would get for that rightly is as a good sort of agnostic forecaster, as somebody who is like sort of calm and measured. Um, but it seems like to be able to make really strong claims about the future, about, uh, something that is so out of prior distributions, it's like the death of humanity. It, you don't only have to show yourself as a good agnostic forecaster, you have to show that your ability to forecast because of a particular theory is much greater. Do you see what I mean?

• 2:25:54Eliezer Yudkowsky?

  It's all about, so, so when, when your work, yeah, it's all about the ignorance prior. It's all about knowing the space in which to be maximum entropy, like-

• 2:26:06Eliezer Yudkowsky?

  The whole bunch of, you know, like, somebody, you know, like, what will the future be? Well, I don't know, it could be paperclips, it could be staples, it could be no kind of office supplies at all, and tiny little spirals. It could be like, like, little tiny, like, things that are like outputting 111 because that's like the most predictable kind of text to predict. Or, or like representations of ever larger numbers in the fast-growing hierarchy, because, you know, that's what the, that's how to interpret the reward counter. I'm actually like getting into specifics here, which is kind of the opposite of the point I originally meant to make, which is like, you know, like, if somebody claims to be very unsure, I might say, okay, so then, like, you, you expect, like, most possible molecular configurations of the solar system be equally probable.

• 2:26:50Eliezer Yudkowsky?

  Well, humans mostly aren't in those. So like, being very unsure about the future looks like predicting with probability nearly one that the humans are all gone. Which, you know, it's not, it's not actually that bad, but it like illustrates the point of, like, people going like, but how are you sure? Kind of missing the, the, the real discourse and skill, which is like, oh, yes, we're all very unsure. Lots of entropy in our probability distributions, but what is the space on for which you are unsure?

• 2:27:24Dwarkesh Patel?

  Even at that point, it seems like the most reasonable prior is not that all sort of atomic configurations of the solar system are equally likely, because I agree by that metric.

• 2:27:33Eliezer Yudkowsky?

  Yeah, like, it's, it's like, all computations that can, like, be run over, configurations of solar system are equally likely to be maximized.

• 2:27:46Dwarkesh Patel?

  But, but why, like, we have a certain, we have a certain sense that, like, listen, we know what the loss function looks like, we know what the training data looks like. That obviously is no guarantee of what the drives that come out of that loss function will look like.

• 2:28:00Eliezer Yudkowsky?

  Yeah.

• 2:28:00Dwarkesh Patel?

  But it is certainly not-

• 2:28:02Eliezer Yudkowsky?

  Humans, they came out pretty different from their loss functions, but-

• 2:28:04Dwarkesh Patel?

  I mean, this is the first question I began with, I would say, like, I would say, actually, no. Like, if it is as similar as humans are now to our loss function from which we evolved, that would be like, that honestly might not be that terrible world, and it might in fact be a very good world.

• 2:28:18Eliezer Yudkowsky?

  Okay, so-

• 2:28:18Dwarkesh Patel?

  If it's like, what, the equivalent of like the thing that-

• 2:28:20Eliezer Yudkowsky?

  Where do you get, where do you get good world out of maximum prediction of text?

• 2:28:24Dwarkesh Patel?

  Plus, um, RLHF plus, uh, plus like all the whatever alignment stuff that might work results in something that kind of just like does what you ask it to, uh, the way like does it reliably enough that, you know, we ask it like, hey, help us with alignment, then go, go.

• 2:28:41Eliezer Yudkowsky?

  Stop asking for help with alignment. Ask it for augmented humans. Ask it for any of the.

• 2:28:46Dwarkesh Patel?

  Like, help us enhance our brains, help us blah, blah, blah.

• 2:28:49Eliezer Yudkowsky?

  Thank you. Why are people asking for like the most difficult thing that's the most impossible to verify? It's whack.

• 2:28:55Dwarkesh Patel?

  Um, and then basically at that point, we're like turning into gods and we can.

• 2:29:00Eliezer Yudkowsky?

  If you get to the point where you're turning into gods yourselves, you're like, you're not quite home free. But you know, you're sure past, you're sure past a lot of the, the death.

• 2:29:08Dwarkesh Patel?

  Yeah. Maybe you can explain the intuition that all sorts of drives are equally likely given a known loss function and a known set of data.

• 2:29:18Eliezer Yudkowsky?

  Oh, um, if, if, yeah, it like, so, so, so if you, if you had the textbook from the future or if you were an alien who'd watched a dozen planets destroy themselves the way Earth is, that, or not actually dozen, that's not like, if you'd seen 10,000 planets destroy themselves the way Earth has, while being only human in your sample complexity and generalization ability, then you could be like, oh yes, they're going to try like this trick with loss functions and they will get a draw from like this space of results. And the, and the alien like can now, may now have like a pretty good prediction of like range of like where that ends up. Like, similarly, like now that we've actually seen how humans turn out when you optimize them for reproduction,

• 2:30:01Eliezer Yudkowsky?

  it would like not be surprising if we found some aliens the next door over and they had orgasms. Now, maybe they don't have orgasms, but you know, like some like, but you know, like if they had some kind of like strong surge of pleasure during the act of mating, we're not, we're not, we're not surprised. We've seen how that plays out in humans. If they have some kind of like weird food that isn't that nutritious, but like makes them much happier than any kind of food that was more nutritious and around in their ancestral environment, like, like ice cream. We probably can't call it as ice cream, right? It's not going to be like sugar, salt, fat, frozen. Like we're, they're not specifically going to have ice cream.

• 2:30:43Dwarkesh Patel?

  Right.

• 2:30:43Eliezer Yudkowsky?

  They might play Go. They're not going to play chess.

• 2:30:48Dwarkesh Patel?

  Because chess is like more, has more specific pieces, right?

• 2:30:52Eliezer Yudkowsky?

  Yeah. They're not gonna play, they're not gonna play Go on like nineteen by ninety.

• 2:30:57Eliezer Yudkowsky?

  They might play Go on some other size. Probably odd. Well, can we really say that? I don't know. I'd, I'd, I'd bet on like an odd, if, if they play Go, I'd bet on an odd board dimension at, uh, let's say, let's, two thirds, the Plasas rule of succession, sounds about right. Unless there's some other reason why Go just totally does not work on an, on even such, or dimension that I don't know, because I'm insufficiently acquainted with the game. We, like, the point is, like, is, like, you know, reasoning off of humans is like pretty hard. We have like the loss function over here, we have like humans over here, we can like look at the rough distance, like all the, like, weird, specific, like, stuff that humans accreted around and be like, you know, like,

• 2:31:40Eliezer Yudkowsky?

  like, like, if, if the loss function's over here and humans are over there, like, maybe the aliens are like over there. And if, if we had like three aliens, that would like expand our views of the possible. We'd have like, or even, even two aliens would like vastly expand our views of the possible. And give us like a much stronger notion of what the third aliens would, would look like, like humans, aliens, third, third race.

• 2:32:04Eliezer Yudkowsky?

  But, you know, you know, like, you, you know, like, the, the, the wild, optimistic scientists have never been through the, they've never been through this with, with AIs. They're like, oh, you know, like, like you optimize the AI to, to like say nice things and help you and like make it a bunch smarter, probably says nice things and helps you, it's probably like, totally aligned, yeah. Exactly, yeah. Yeah, they don't know any better. Not trying to jump ahead of the, the, the story.

• 2:32:31Eliezer Yudkowsky?

  But the aliens, the aliens know, know where, where you end up around the loss function. They know where, where, how it's gonna play out, much more narrowly.

• 2:32:40Eliezer Yudkowsky?

  We're guessing much more blindly.

• 2:32:43Dwarkesh Patel?

  It just like, it just leaves me in a sort of unsatisfied place that we apparently know about something that is so extreme that maybe a handful of people in the entire world believe it from first principles about, you know, the doom of humanity because of AI. But this theory that is so, so productive in that one very unique prediction is unable to give us any sort of other prediction about what this world might look like in the future or about what happens before the, before we all die. Like, it can tell us nothing about the world until the point at which makes prediction

• 2:33:27Dwarkesh Patel?

  that is the most remarkable in the world.

• 2:33:29Eliezer Yudkowsky?

  You know, rationalists should win, but rationalists should not win the lottery. I'd ask you like what other theories are supposed to have been doing a like amazingly better job of predicting the last three years. You know, maybe it's just hard to predict, right? And, and in fact, it's like easier to predict the end state than the strange, complicated winding paths that lead there, much like if you, you know, play against AlphaGo and predict it's going to like be in the class of winning board states, but not, not exactly how it's going to beat you. It's like, not quite like that, the problem of difficulty of predicting the future. But, you know, from my perspective, the future is just like really hard to predict. And there's a few places where you can like wrench what sounds like an answer out of your

• 2:34:10Eliezer Yudkowsky?

  ignorance, even though really you're just being like, oh, you're going to like end up in some like random weird place around this loss function and I haven't seen it happen with 10,000 species, so I don't know where. Very, very improverish by the, from the standpoint of anybody who like actually knew anything, that actually predicted anything. But the rest of the world is like. Oh, like, we're easily, we're like equally likely to to win the lottery as lose the lottery, right? Like, either we win or we don't. You come along and you're like, no, no, your chance of winning the lottery is tiny. They're like, what? How can you be so sure? Where do you get your strange certainty? And the actual root of the answer is that you are putting your maximum entropy over a different probability space. Like, that just actually is the thing that's

• 2:34:53Eliezer Yudkowsky?

  going on there. You're saying all lottery numbers are equally likely instead of winning and losing are equally likely.

• 2:34:59Dwarkesh Patel?

  So I think the place to sort of close this conversation is, let me just sort of give the the main reasons why I'm not convinced that doom is likely, or even that it's more than 50% probable or anything like that. Some are the things that I started this conversation with that I don't feel like I heard any knockdown arguments against, and some are new things from the conversation. And the following things are things that, even if any one of them individually

• 2:35:36Dwarkesh Patel?

  turns out to be true, I think doom doesn't make sense, or is much less likely. So, going through the list, I think probably more likely than not, this entire frame all around alignment and AI is wrong. This is maybe not something that would be easy to talk about, but I'm just kind of skeptical of sort of first principles reasoning that has really wild conclusions.

• 2:36:07Eliezer Yudkowsky?

  Okay, so everything in the solar system just ends up in a random configuration, then?

• 2:36:12Dwarkesh Patel?

  Or it stays like it is unless you have very good reasons to think otherwise. And especially if you think it's going to be very different from the way it's going, you must have very, very good reasons, like ironclad reasons for thinking that it's going to be very, very different from the way it is.

• 2:36:26Eliezer Yudkowsky?

  Uh huh. So this is...

• 2:36:30Eliezer Yudkowsky?

  You know, humanity hasn't really existed for very, man, I don't even know what to say to this thing. We're like this tiny, like, everything that you think of as normal is this tiny flash of things being in this particular structure out of a 13.8 billion year old universe, which, very little of which was like 20th century, pardon me, 21st century. Yeah, my own brain sometimes gets stuck in childhood too, right? Very, very little of which is like 21st century, like civilized world. The, you know, on this like little fraction of the surface of one planet in a vast solar system, most of which is not Earth, and a vast universe, most of which is not Earth.

• 2:37:12Eliezer Yudkowsky?

  And it has lasted for like such a tiny period of time through such a tiny amount of space and has like changed so much over, you know, just the last 20,000 years or so. And, and here you are like being like, why would things really be any different going forward?

• 2:37:27Dwarkesh Patel?

  I, I feel like that argument's proof too much because you could use that same argument to like, somebody says, comes up to me and says, I don't know, a theologian comes up to me and says like, the rapture is coming and let me sort of explain why the rapture is coming. And I say, I'm not claiming that your arguments are as bad as the argument for rapture. I'm just, just follow the example. But then they say, listen, I mean, look at how wild human civilization has been. Would it be any wilder if there was a rapture? And I'm like, yeah, actually, as wild as human civilization has been, the rapture would be much wilder.

• 2:37:54Eliezer Yudkowsky?

  Because it violates the laws of physics.

• 2:37:56Dwarkesh Patel?

  Yes.

• 2:37:58Eliezer Yudkowsky?

  I'm not trying to violate the laws of physics, even as we presently know them.

• 2:38:02Dwarkesh Patel?

  How about this? Somebody comes up, oh, you know, I forgot the perfect example. Okay. Somebody comes up to me, he says, we have actually nanosystems right behind you. He says, I've read Eric Drexler's nanosystems, I've read Feynman's, there's plenty of room at the bottom. And he explains-

• 2:38:16Eliezer Yudkowsky?

  These two things are not to mention, but go on.

• 2:38:18Dwarkesh Patel?

  Okay, fair enough. He comes to me and he says, let me explain to you my first principles argument about how some nanosystems will be replicators. And replicators, because of some competition, yada, yada, yada, argument, they turn the entire world into goo, just making copies of themselves.

• 2:38:37Eliezer Yudkowsky?

  This kind of happened with humans, you know? Well, life generally.

• 2:38:40Dwarkesh Patel?

  Yeah, yeah. But, so then they say like, listen, as soon as we build, start building nanosystems, pretty soon, 99% probability, the entire world turns into goo. Just because the replicators are the things that turn things into goo, there'll be more replicators and non-replicators. I don't want to have an object level debate about that, but it just like, I just started that and I'm looking like, yes, human civilization has been wild, but the entire world turning into goo because of nanosystems alone just seems much wilder than human civilization.

• 2:39:05Eliezer Yudkowsky?

  You know, this, this, this, uh, this, this argument probably lands with greater force on somebody who does not expect stuff to be disassembled by nanosystems, albeit intelligently controlled ones rather than goo in like quite near future, especially on the 13.8 billion year timescale. But, you know, do you, do you expect this little momentary flash of what you call normality to continue? Do you expect the future to be normal?

• 2:39:30Dwarkesh Patel?

  I, uh, no, I expect any given vision of how things shape out to be wrong, especially, it is not like you are suggesting that the current weird trajectory continues being weird in the way it's been weird and that we continue to have like 2% economic growth or whatever, and that leads to incrementally more technological progress and so on. You're suggesting there's been that specific species of weirdness which leads to an, uh, which means that this entirely different species of weirdness is warranted.

• 2:40:03Eliezer Yudkowsky?

  Yeah, we've got like different weirdnesses over time. The, the jump to superintelligence does strike me as being significant in the same way as self, for self replicator. First self replicator is the universe transitioning from, you see mostly stable things to you also see a whole bunch of things that make copies of themselves. And then somewhat later on, there's a state where, you know, there's this like strange transition, this border between the universe of stable things where things come together by accident and stay as long as they endure to this world of complicated life. And that transitionary moment is when you have something that arises by accident and yet self replicates. And similarly, on

• 2:40:46Eliezer Yudkowsky?

  the other side of things, you have things that are intelligent, making other intelligent things. But to get into that world, you've got to have the thing that is built just by things copying themselves and mutating and yet is intelligent enough to make another intelligent thing.

• 2:41:04Eliezer Yudkowsky?

  Now, if I sketched out that cosmology, would you say, "No, no, I don't believe in that"?

• 2:41:10Dwarkesh Patel?

  What if I sketched out the cosmology of, because of replicators, blah, blah, blah, intelligent beings, intelligent beings create nanosystems, blah, blah, blah?

• 2:41:18Eliezer Yudkowsky?

  No, no, no, no, I don't, I don't want, don't tell me about your, like, not, not to prove too much, I just want to like, like, I just guessed out a cosmology, do you buy it? In the long run, are we in a world full of things replicating, or a world in a full of intelligent things designing other intelligent things?

• 2:41:35Dwarkesh Patel?

  Yes.

• 2:41:36Eliezer Yudkowsky?

  So you, so you, so you buy that vast shift in the foundations of order of the universe, that instead of the world of things that make copies of themselves, imperfectly, we are in the world of things that are designed and were designed. You buy that vast cosmological shift I was just describing, the utter disruption of everything you see that you call normal, down to the leaves and the trees around you. You, but you believe that. Well, the same skepticism yourself found of that argues against the rapture can also be used to disprove this thing you believe, that you think is probably pretty obvious, actually, now that I've pointed it out.

• 2:42:14Dwarkesh Patel?

  Okay.

• 2:42:15Eliezer Yudkowsky?

  Your skepticism, your skepticism disproves too much, my friend.

• 2:42:19Dwarkesh Patel?

  That's actually a really good point. It still leaves open the possibility of like, how it happens and when it happens, blah, blah, blah. But actually, that's, that's a good point. Okay, so a second, second thing.

• 2:42:29Eliezer Yudkowsky?

  You set them up, I'll knock them down. One after the other.

• 2:42:34Dwarkesh Patel?

  Second thing is.

• 2:42:35Eliezer Yudkowsky?

  Wrong!

• 2:42:37Dwarkesh Patel?

  Sorry, carry on.

• 2:42:39Eliezer Yudkowsky?

  I was, I was just jumping head to the predictable update at the end.

• 2:42:42Dwarkesh Patel?

  You're a good base here. Maybe alignment just turns out to be much simpler or like much easier than we think. It's not like we've, as a civilization, spent that, that much resources or brain power in solving it. If we put in even the kind of resources that we put into elucidating string theory or something into alignment, it could just turn out to be like, yeah, that's enough to solve it. And in fact, in the current paradigm, it turns out to be simpler because, you know, they're sort of pre-trained on human thought and that might be a simpler regime than something that just comes out of a black box that, like, you know, like Alpha Zero or something like that.

• 2:43:24Eliezer Yudkowsky?

  So like, some of my, like,

• 2:43:29Eliezer Yudkowsky?

  Could I be wrong in an understandable way to me in advance mass? Which is not where most of my hope comes from is on, you know, what if RLHF just works well enough and the people in charge of this are not the current disaster monkeys, but instead have some modicum of caution and are using their like, like know what to aim for in RLHF space, which the current crop do not and I, you know, I'm not really that confident of their ability to understand if I told them, but maybe you have some folks who can understand or anyways.

• 2:44:14Eliezer Yudkowsky?

  I can sort of see what I try. These people will not try it.

• 2:44:20Eliezer Yudkowsky?

  But, you know, the current crop, that is. And I'm not actually sure that, that if, that if somebody else takes over like the government or something that they listen to me either, but I can, you know, maybe you.

• 2:44:36Eliezer Yudkowsky?

  So, so some of the trouble here is that you have a choice of targets and, and like neither is all that great. One is you look for niceness that's in humans and you try to bring it out in the AI. And then you, with its cooperation, because, you know, it knows that if it makes it, that if you try to just like amp it up, it might not stay all that nice. Or that if you build a successor system to it, it might not stay all that nice. And it doesn't want that because you, you know, like you, you, you, you, you narrow down the shagoff enough, you know, that, that's like, and, and the, you know, somebody, somebody once had this incredibly profound statement that I think I'd like somewhat disagree with, but it's still so incredibly profound. It's consciousness is when the mask eats the shagoff.

• 2:45:23Eliezer Yudkowsky?

  And maybe that's, maybe you, you know, with the, with the right set of bootstrapping, reflection type stuff, stuff, stuff, you can have that happen on purpose, more or less, where there, where the, the system's output that you're shaping is like to some degree in control of the system. And you, you, you locate niceness in the human space. I, I have fantasies along the lines of what if you trained GPTN to distinguish people being nice and saying sensible things and argue validly. And, you know, can't just, I'm not

• 2:46:06Eliezer Yudkowsky?

  sure that works if you just have Amazon Turks try to label it. You just get the like strange thing you located that RLHF located in the present space, which is like some kind of weird corporate speak, like left rationalizing leaning, strange telephone announcement creature is what they got with the current crop of RLHF. Note how this stuff is weirder and harder than people might have imagined initially. But, you know, leave, leave, leave aside the, the, the part where you try to like jumpstart the entire process of turning into a grizzled cynic and update as hard as you can and do it in advance. Leave that aside

• 2:46:50Eliezer Yudkowsky?

  for a moment. Like, maybe you can look, maybe you are like able to train on Scott Alexander and So You Want to be a Wizard, some other night, and some other nice real people, real people and nice fictional people and separately train on what's valid argument. That's, that's going to be, that's going to be tougher. But, you know, I could probably put together a crew of a dozen people who could provide the data on that RLHF. And you find like the nice creature and you find the nice mask that, that argues validly. You do some more complicated stuff to try to boost the thing where it's like eating the shagath, where that's what the system is and not, or like more what the system is, less what it's pretending to be.

• 2:47:36Eliezer Yudkowsky?

  I do seriously think this is like, like, I can say this and like, the disaster monkeys at the current places can cannot along to it, but they have not said things like this themselves that I have ever heard. And that is not a good sign. But, and then like, if you don't amp this up too far, which on the present paradigm, you like, can't do anyways, because if you like, train the very, very smart person of this version of the system that kills you before you can RLHF it. But like, maybe you can like, train GPT to like, distinguish like, nice, valid, kind,

• 2:48:12Eliezer Yudkowsky?

  careful, and then like, filter all the training data to get the nice things to train on, and then train on that data, rather than training on everything, to try to like, avert the Waluigi problem. Or just more generally, having like, all the darkness in there, like, just train on the light that's in humanity. So there's like, that kind of course, and not, if you don't push that too far, maybe you can get a genuine ally. And maybe things play out differently from there. That's like, one of the little rays of hope. For, but, but that's not, I don't think that

• 2:48:51Eliezer Yudkowsky?

  actually looks like alignment is so easy that you, you just get whatever you want. It's a genie, it gives you what you wish for. I don't think that, that doesn't even strike me as hope.

• 2:49:06Dwarkesh Patel?

  Honestly, the way you described it seemed kind of compelling. Like, I don't know why that doesn't even rise to 1%. The possibility works out that way.

• 2:49:13Eliezer Yudkowsky?

  I literally, this is like, literally, my, you know, my like, AI lineman fantasy from 2003.

• 2:49:21Eliezer Yudkowsky?

  Though not with, but, though not with like, RLHF and as the implementation method, or LLMs as the base. And it's, you know, gonna be more dangerous than when I was thinking about, when I was dreaming about it in 2003. And I think in a very real sense, it feels to me like, the, the, the, the people doing this stuff now have literally not gotten as far as I was in 2003. And, you know, I can like, I've now like, written out my answer sheet for that. It's on the podcast, it goes on the internet, and now they can, now they can pretend that that was their idea. Or like, or like, sure, that's obvious, we're gonna do that anyways. And yet, they, they didn't, they didn't say it earlier.

• 2:50:02Eliezer Yudkowsky?

  And and you can't, you can't, you can't run a big project off of one person who

• 2:50:13Eliezer Yudkowsky?

  it's it failed to gel. The alignment field failed to gel. That's that's my judgment to be like, well, you just throw in a ton of more ton of more money and then it's all solvable because I've seen people try to amp up the amount of money that goes into it and the stuff coming out of it has not

• 2:50:28Eliezer Yudkowsky?

  gone to the places that I would have considered obvious a while ago and I can like print out all my entry receipts for it and each time I do that, it gets a little bit harder to make the case next time.

• 2:50:39Dwarkesh Patel?

  But I mean, how much money are we talking in the grand scheme of things? Because like civilization itself has a lot of money.

• 2:50:44Eliezer Yudkowsky?

  I know, I know people who have a billion dollars. I don't know how to throw a billion dollars at like outputting lots and lots of alignment stuff.

• 2:50:52Dwarkesh Patel?

  But you you might not, but I mean, you are one of 10 billion, right? Like it is.

• 2:50:57Eliezer Yudkowsky?

  And other people go ahead and spend lots of money on it anyways and and and everybody makes the same mistakes. Nate Sorry's has a post about it. I forget the exact title, but like everybody coming into alignment makes the same mistakes.

• 2:51:11Dwarkesh Patel?

  Let me just go on to the third point because I think it plays into what I was saying. The third reason is if it if it is a case that, you know, this these capabilities scale in some constant way as it seems like they're going from two to three or three to four.

• 2:51:29Eliezer Yudkowsky?

  What does that even mean? But go on.

• 2:51:32Dwarkesh Patel?

  That they get more and more general. It's not like going from a going from a mouse to a human or chimpanzee to a human. It's like going from.

• 2:51:40Eliezer Yudkowsky?

  GPT three to GPT four?

• 2:51:42Dwarkesh Patel?

  Yeah, well, it just seems like that's less of a jump than chimp to human. Like a slow accumulation of capabilities. There are a lot of like S curves of emergent abilities, but overall the curve looks sort of.

• 2:51:55Eliezer Yudkowsky?

  Man, I feel like we bit off a whole chunk of chimp to human and GPT 3.5 to GPT four, but but go on.

• 2:52:01Dwarkesh Patel?

  Regardless, okay, so then this leads to human level intelligence for some interval. I think that.

• 2:52:11Dwarkesh Patel?

  I was not convinced from the arguments that we could not have a system of sort of checks on this, the same way you have checks on smart humans, that it would try to deceive us to achieve its aims any more than smart humans or in positions of power try to do the same thing.

• 2:52:29Eliezer Yudkowsky?

  For a year, what are you going to do with that year? Before the next generation of systems come out that are not held in check by humans, because they are not roughly in the same power intelligence range as humans. What are you going, maybe you can get a year with, maybe you can get a year like that. Maybe that actually happens. What are you going to do with that year that prevents you from dying the year after?

• 2:52:49Dwarkesh Patel?

  One is, one possibility is that because these systems are trained on human text, maybe just progress just slows down a lot after it gets to slightly above human level.

• 2:53:01Eliezer Yudkowsky?

  Yeah, that's not, that's, yeah, that, that's not how, I would be quite surprised if that's how anything works.

• 2:53:07Dwarkesh Patel?

  Why is that?

• 2:53:08Eliezer Yudkowsky?

  For one thing, because it's, you know, like, like for an alien to be an actress playing all the humans on the internet, for another thing, well, first of all, you realize in principle that the task of minimizing losses on predicting human text does not have a, yeah, you, you understand that in principle, this does like not stop when you're as smart as a human, right? Like you can see the, the computer science of that.

• 2:53:33Dwarkesh Patel?

  I don't know if I see the computer science of that, but I think I probably understand the argument.

• 2:53:36Eliezer Yudkowsky?

  Okay, so like very, very, you know, somewhere on the internet is a list of hashes followed by the string hashed. This is a simple demonstration of how you can go on getting lower losses by throwing a hypercomputer at the problem. There are pieces of text on there that were not produced by humans talking in conversation, but rather by like lots and lots of work to determine, get, extract experimental results out of reality. That text is also on the internet.

• 2:54:05Eliezer Yudkowsky?

  Maybe there's not enough of it for the machine learning paradigm to work. But I'd sooner buy that like that the GPT systems just bottlenecked short of being able to predict that stuff better rather than that rather but you know like you can maybe buy that but like the notion that like like you only have to be smart as a human to predict all the text as the internet as soon as you turn around and stare at that a bit is just transparently false.

• 2:54:30Dwarkesh Patel?

  Okay, agreed. Okay, how about this story? You have something that is sort of human like that is maybe above humans at certain aspects of science because it's specifically trained to be really good at the things that are on the internet, which is like, you know, chunks and chunks of archive and whatever. Whereas it has not been trained specifically to gain power and while at some point of intelligence that comes along, can I just restart that whole sentence?

• 2:55:01Eliezer Yudkowsky?

  No, you have spoken it. It exists. It cannot be called back. There are no take backs. There is no going back. There is no going back. Go ahead.

• 2:55:14Dwarkesh Patel?

  Okay, so here's another story. I expect them to be better than humans at science than they are at power seeking because we had greater selection pressures for power seeking in our ancestral environment than we did for science. And while at a certain point, both of them come along as a package, you know, maybe that they can be at varying levels. But anyways, so you have this sort of early model that is kind of human level, except a little bit ahead of us in science. You ask it to help us align the next version of it, then the next version of it is more aligned because we have its help. And the

• 2:55:59Dwarkesh Patel?

  sort of like this inductive thing where the next version helps us align the version.

• 2:56:02Eliezer Yudkowsky?

  Where do people have this notion of getting AIs to help you do your AI alignment homework? It just, why can we not talk about having it enhance you in human intelligence instead?

• 2:56:11Dwarkesh Patel?

  Okay, so either one of those stories where it just like helps us enhance humans, enhance humans that help us figure out the alignment problem or something like that.

• 2:56:19Eliezer Yudkowsky?

  Yeah, I, it's like kind of weird because you know, like. Like, small, large amounts of intelligence don't automatically make you a computer programmer. And if you are a computer programmer, you don't automatically get security mindset. But it feels like there's some level of intelligence where you ought to automatically get security mindset. And I think that's about how hard you have to augment people, to have them able to do alignment. Like, the level where, like, they have security mindset, not because they were, like, special people with security mindset, but just because, like, they're that intelligent that you just, like, automatically have security mindset. I think that's about the level where a human could start to work on alignment, more or less.

• 2:56:55Dwarkesh Patel?

  Why is that story then not 1%? Get you to 1% probability that it helps us avert the whole crisis?

• 2:57:02Eliezer Yudkowsky?

  Well, because it's not just a question of the technical feasibility of can you build a thing that applies its general intelligence narrowly to the neuroscience of augmenting humans? It's a question of, like, so, like, I feel like that is, like, probably, like, over 1% technical feasibility. But the world that we are in is so far, so far from, from doing that, from trying. Trying the way that it could actually work, like, not like the try where, like, oh, you know, like, well, we like, just like, do a bunch of RLHF to try to have a spit out output about these things, but not about that thing.

• 2:57:47Eliezer Yudkowsky?

  And, you know, that, that, that, no, no, not that. Yeah, 1% that we could, that humanity could do that if it tried and tried in just the right direction, as far as I can perceive angles in this space. Yeah, I'm over 1% on that. I am not very high on us doing it. Maybe I will be wrong. Maybe the time article I wrote saying, "Shut it all down," gets picked up, and there are very serious conversations, and the very serious conversations are actually effective in shutting down the headlong plunge. And there is a narrow exception carved out for the kind of narrow application

• 2:58:32Eliezer Yudkowsky?

  of trying to build an artificial general intelligence that applies its intelligence narrowly and to the problem of augmenting humans. And, you know, that, I think, might be a harder sell to the world. Then just shut it all down. They could, they could get shut it all down and then not do the things that they would need to do to have an exit strategy. I feel like even if you told me that they would, that they went for shut it all down, I would be like, then next expect them to have no exit strategy until the world ended anyways, but perhaps I underestimate them.

• 2:59:03Eliezer Yudkowsky?

  Maybe there's a will in humanity to do something else, which is not that.

• 2:59:09Eliezer Yudkowsky?

  And if there really were, um, yeah, I'm, I think I'm even over 10% that, that, that would be a technically feasible path if we, if we, if they looked in just the right direction. But I am not over 50% on them actually doing the, the shut it all down. I'm not, if they do that, I am them not over 50% on there really truly being the will of something else that is not that to really have an exit strategy.

• 2:59:43Eliezer Yudkowsky?

  Then from there, you have to

• 2:59:47Eliezer Yudkowsky?

  go in at sufficiently the right angle to, to materialize the technical chances and not do it in the way that's just ends up a suicide, or if you're lucky, like gives you the clear warning signs and, and then people actually pay attention to those instead of just optimizing away the warning signs. And I don't want to make this sound like the multiple status files, see if like, oh no, it's more than one thing has to happen. Therefore, the resulting thing can never happen, which you know, like super clear case in point of why you cannot prove anything will not happen this way of, uh, Nate Silver arguing that Trump needed to get through six stages to become the Republican presidential candidate, each of which was less than half probability.

• 3:00:29Eliezer Yudkowsky?

  And therefore he had less than one 64th chance of becoming the Republican. No, not one eighth, what's six, six stages of Dune. Therefore he had like less than one 64th chance of becoming, I think, just a Republican candidate, not, not winning. So, yeah, so like, you can't just like break things down at the stages and then say, therefore the probability is zero. You can break down anything at the stages. But like, but like even so, you're asking me like, well, like, isn't over 1% that it's, that it's possible? I'm like, yeah, possibly even over 10% that, that doesn't get me to.

• 3:01:03Eliezer Yudkowsky?

  Because the reason why, you know, go ahead and tell people, yeah, don't put your hope in the future, you're probably dead, is that the existence of this technical ray of hope, if you do just the right things, it's not the same as expecting that the world reshapes itself to permit that to be done without destroying the world in the meanwhile. I expect things to continue on largely as they have. And, you know, and what distinguishes that from despair is that at the moment, people were telling me, like, no, no, if you go outside the tech industry, people will actually listen. I'm like, all right, let's try that. Let's write the Time article. Let's jump on that. Let's see if it works.

• 3:01:44Eliezer Yudkowsky?

  It would lack dignity not to try. But that's not the same as expecting as being like, oh, yeah, over 50%. They're totally going to do it. That Time article is totally going to take off. I'm not currently not over 50% on that. You know, you said like, any one of these things could mean, and yet, like, even if this thing is technically feasible, that doesn't mean the world's going to do it. We are presently quite far from the world being on that trajectory. Or of doing the things that we needed to create time to pay the alignment tax to do it.

• 3:02:14Dwarkesh Patel?

  Maybe the one thing I would dispute is how many things need to go right from the world as a whole for any one of these paths to succeed. Because, which goes into the fourth point, which is that maybe the sort of universal prior over all the drives that any AI could have is just like the wrong way to think about it. And this is something that...

• 3:02:34Eliezer Yudkowsky?

  I mean, you definitely want to use the alien observation of 10,000 planets like this one prior for what you get after training on like thing X.

• 3:02:44Dwarkesh Patel?

  Especially when we're talking about things that have been trained on, you know, human text. I'm not saying that it was a mistake earlier on in the conversation for me to say there'll be like the average of human motivations, whatever that means. But it's not, it's not inconceivable to me that it would be something that is very sympathetic to human motivations, having been, having sort of encapsulated all of our output.

• 3:03:08Eliezer Yudkowsky?

  I think it's much easier to get a mask like that than to get a shoggoth like that.

• 3:03:13Dwarkesh Patel?

  Possibly, but again, this is something that seems like...

• 3:03:18Dwarkesh Patel?

  I don't know probability, I'll put it on it at least 10%. And that by just by default, that is something that is not so, it is not incompatible with the flourishing of humanity. Like, why?

• 3:03:29Eliezer Yudkowsky?

  What is the utility function you hope it has that has its maximum?

• 3:03:32Dwarkesh Patel?

  There's so many possible.

• 3:03:33Eliezer Yudkowsky?

  Of humanity?

• 3:03:34Dwarkesh Patel?

  There's so many possible, like.

• 3:03:35Eliezer Yudkowsky?

  Name three, name one. Spell it out.

• 3:03:38Dwarkesh Patel?

  It, I don't know, wants to keep us as a zoo the same way we keep like other animals in a zoo. This is not the best outcome for humanity, but it's just like something where we survive and flourish.

• 3:03:48Eliezer Yudkowsky?

  Okay, whoa, whoa, whoa, flourish? Keeping in a zoo did not sound like flourishing to me.

• 3:03:54Dwarkesh Patel?

  Zoo was the wrong word to use there.

• 3:03:56Eliezer Yudkowsky?

  Whoa, whoa, whoa, whoa, whoa, cuz it's not what you wanted? Why is it not a good prediction?

• 3:04:00Dwarkesh Patel?

  You just asked me to name three. You didn't ask me like.

• 3:04:03Eliezer Yudkowsky?

  No, no, what I'm saying is like, like, you're like, oh, like prediction, oh, no, no, I don't like my prediction. I want a different prediction.

• 3:04:09Dwarkesh Patel?

  You didn't ask for the prediction, you just asked me to name them, like name possibilities.

• 3:04:14Eliezer Yudkowsky?

  I, I had meant like possibilities into which you put some probability. I had meant for, for like, like a thing that you thought held together.

• 3:04:21Dwarkesh Patel?

  This is the same thing as when I ask you, like, what is a specific utility function it will have that will be incompatible with, you know, humans existing? It's like your, your mortal prediction is not.

• 3:04:31Eliezer Yudkowsky?

  Yeah, the super vast majority of predictions are, of utility functions are incompatible with, with, with human existing. I can make a mistake and it'll still be incompatible with humans existing. Right? Like, I can just be like, you know, like, I can just like describe a randomly rolled utility function and up with something incompatible with humans existing.

• 3:04:48Dwarkesh Patel?

  So like at the, at the beginning of human evolution, you could think like, okay, this thing will become generally intelligent. And what are the odds that it's flourishing on the planet will be compatible with the survival of spruce trees or something? It's like.

• 3:05:06Eliezer Yudkowsky?

  And in the long term, term, we sure aren't. I mean, like, maybe if we win, we'll, we'll have there be a space for spruce trees. Yeah, so as long as you can have spruce trees, as long as the mitochondrial liberation front does not object to that.

• 3:05:19Dwarkesh Patel?

  What is the mitochondrial liberation front?

• 3:05:21Eliezer Yudkowsky?

  If you have, do we have, you know, sympathy for the mitochondria enslaved, working all their lives to the benefit of some other organism?

• 3:05:29Dwarkesh Patel?

  So this is like some weird hypothetical, like for hundreds of thousands of years, general intelligence has existed on Earth. You could say like, is it compatible with some random species that exist on Earth? Like, is it compatible with spruce trees existing? And I know you probably chopped on a few spruce trees, but.

• 3:05:44Eliezer Yudkowsky?

  And the answer is yes, as the, as a very special case of us being the sort of things that would may, some of us would maybe conclude that we wanted, that we specifically wanted spruce trees, spruce, spruce trees to go on existing, at least on Earth, in the glorious transhuman future, and their votes winning out against the, those of the mitochondrial liberation front.

• 3:06:06Dwarkesh Patel?

  I guess since part of the sort of transhumanist future is part of the thing we're debating, it seems weird to assume that as part of the question.

• 3:06:14Eliezer Yudkowsky?

  Well, the thing I'm trying to say is you're like, well, like, if you looked at the humans, would you like not expect them to end up incompatible with the spruce trees? And I'm being like, sir, you, a human, have like, looked back and like, looked at how humans wanted the universe to be and been like, well, would you not have anticipated in retrospect that humans would like, want the universe to be otherwise? And I agree that we like, might want to conserve a whole bunch of stuff. Maybe we don't want to conserve the parts of nature where things bite other things and inject venom into them and the victims die in terrible pain. Maybe even if, maybe, you know, I, I think that many of them don't have qualia. This is disputed.

• 3:06:57Eliezer Yudkowsky?

  Some people might be disturbed by it, even if they didn't have qualia. We might want to be polite to the sort of aliens who would be disturbed by it because they don't have qualia and they just see like, things don't want venom injected into them, therefore they should not have venom. We might conserve some parts of nature, but again, it's like, it's like, it's like firing an arrow and then drawing the circle around the target.

• 3:07:17Dwarkesh Patel?

  I would disagree with that, because.

• 3:07:21Dwarkesh Patel?

  Again, this is similar to the example we started off the conversation with, but it seems like you are reasoning for, from what might happen in the future. And because we disagree about what might happen in the future, in fact, the entire point of this disagreement is to test what will happen in the future, assuming what will happen in the future as part of your answer, seems like...

• 3:07:42Eliezer Yudkowsky?

  I mean...

• 3:07:43Dwarkesh Patel?

  Seems like a bad way to...

• 3:07:44Eliezer Yudkowsky?

  Okay, but then you're like claiming things as evidence for your position.

• 3:07:47Dwarkesh Patel?

  Based on what exists in the world now.

• 3:07:48Eliezer Yudkowsky?

  That are not evidence... That are not evidence one way or the other, because the basic prediction is like, if you offer things enough options, they will, they will go out of distribution. Like if you are, like, it's like, it's like pointing to like very first people with language and being like, "They haven't taken over the world yet."

• 3:08:11Eliezer Yudkowsky?

  It's like, and like, they have like not gone way out of distribution yet. And it's like, they haven't had general intelligence for long enough to accumulate the things that would give them more options, such that they could start trying to select the weirder options. The prediction is like, when you have, when you give yourself more options, you start to select ones that look weirder relative to the ancestral distribution. As long as you don't have the weird options, you're not going to make the weird choices. And if you say like, "We haven't yet observed your future," that's fine, but like acknowledge that then, like, evidence against that future is not being provided by the past. This is the thing I'm saying there.

• 3:08:50Eliezer Yudkowsky?

  You look around, it looks so normal, according to you, who grew up here. If you'd grown up a millennium earlier, your argument for the persistence of normality might not seem as persuasive to you after you'd seen that much change.

• 3:09:02Dwarkesh Patel?

  So this is a separate argument though, right? Like I'm...

• 3:09:06Eliezer Yudkowsky?

  Like, like, "Look at all this stuff humans haven't changed yet." You say now, selecting the stuff we haven't changed yet, but if you go back 20,000 years and be like, "Look at the stuff intelligence hasn't changed yet," you might very well select a bunch of stuff that was gonna fall 20,000 years later, is the thing I'm trying to gesture at here.

• 3:09:26Dwarkesh Patel?

  But so like, how do you propose we reason about what general intelligence should do? When the world we look at after hundreds of thousands of years of general intelligence is the one that we can't use for evidence?

• 3:09:39Eliezer Yudkowsky?

  Because, but yeah, dive under the surface. Look at the things that have changed. Why did they change? Look at the processes that are generating those choices.

• 3:09:51Dwarkesh Patel?

  And since we have sort of these different functions of like where that goes.

• 3:09:57Eliezer Yudkowsky?

  Like, look at the thing with ice cream. Look at the thing with condoms. Look at the thing with pornography. See where this is going.

• 3:10:06Dwarkesh Patel?

  I think, just like, I, it just seems like I would disagree with your intuitions about like what future smarter humans will do, even with more options. Um, I was like in the beginning of the conversation, I disagreed that they would, most humans would adopt sort of like a transhumanist way to get better DNA or something.

• 3:10:23Eliezer Yudkowsky?

  But you would.

• 3:10:25Eliezer Yudkowsky?

  So, yeah, you, you just like, you just like look down at your fellow humans. You have like no confidence in their ability to tolerate weirdness, even if they got smarter, I wonder?

• 3:10:33Dwarkesh Patel?

  Like, do you think, what do you think would happen if we did a poll right now?

• 3:10:35Eliezer Yudkowsky?

  I think I'd have to explain that poll pretty carefully, cause, you know, they haven't got the, the intelligence headbands yet, right?

• 3:10:41Dwarkesh Patel?

  I mean, we could do a Twitter poll with like a long explanation in it.

• 3:10:44Eliezer Yudkowsky?

  4,000 character Twitter poll?

• 3:10:46Dwarkesh Patel?

  Yeah, I like.

• 3:10:47Eliezer Yudkowsky?

  I mean, uh, man, I'm like somewhat tempted to do that just for the sheer chaos and point out the drastic selection effects of A, it's my Twitter followers, B, they read through a 4,000 character tweet. I, I, I feel like this is not likely to be truly very informative by my standards, but part of me is amused by the prospect for the chaos.

• 3:11:06Dwarkesh Patel?

  Yeah, or I could do it on my end as well. Um, although my followers are likely to be weird as well.

• 3:11:11Eliezer Yudkowsky?

  Yeah, but plus, plus you wouldn't like really, I worry you wouldn't sell that transhumanism thing as well as it could be sold.

• 3:11:16Dwarkesh Patel?

  I, I could word it as, like you just sent me the wording. But anyways, that's a, but anyways, given that we disagree about what in the future general intelligence will do. Where do you suppose we should look for evidence about what the general intelligence will do, given our different theories about it, if not from the present?

• 3:11:36Eliezer Yudkowsky?

  I mean, I think you look at the mechanics. You say, as people have gotten more options, they have gone further outside the ancestral distribution. And we zoom in and it's like, there's all these different things that people want. And there's this like, narrow range of options that they had 50,000 years ago. And the things that they want have maximus or optima 50,000 years ago, as stuff that coincides with reproductive fitness. And then as a result of the humans getting smarter, they start to accumulate culture, which produces changes on a time scale faster than natural selection runs. Although it is still running

• 3:12:21Eliezer Yudkowsky?

  contemporaneously, the humans are just running faster, running faster than natural selection. It didn't actually halt. And the additional, they like, generate additional options, not blindly, but according to the things that they want. And they invent ice cream. They, you know, like, not, not at random. It doesn't just like get coughed up at random. They're like, searching the space of things that they want. And, and generating new options for themselves that optimize these things more that weren't in the ancestral environment. And good hearts law applies, good hearts curse applies. Once you, that like, as you apply optimization pressure, the correlations that were found naturally come apart and aren't

• 3:13:04Eliezer Yudkowsky?

  present in the thing that gets optimized for. Like, you know, you like, just give some people some tests who've never gone to school. The ones who high score high in the tests will know the problem domain. Cause they, you know, like, you're just like, gives, gives a bunch of carpenters a carpentry test. The ones who score high in the carpentry test will like, know how to carpenter things. Then you're like, yeah, I'll like, pay you for high scores in the carpentry test. I'll give you this carpentry degree. And like, people are like, oh, I'm gonna like, optimize the test specifically. And they'll get higher scores than the carpenters. And do, and be worse at carpentry, because they're like, optimizing the test. And that's the story behind ice cream.

• 3:13:37Dwarkesh Patel?

  Okay.

• 3:13:38Eliezer Yudkowsky?

  And, and you zoom in and look at the, the mechanics and not the like, grand scale view. Cause the grand scale view just like, never gives you the right answer, basically. Like, anytime you ask what would happen if you applied the grand scale view philosophy in the past, it's always just like, eh, I don't see that, why this thing would change. Oh, it changed. Who, how weird. Who could have possibly have expected that?

• 3:13:56Dwarkesh Patel?

  Maybe you have a different definition of grand scale view because I would have thought that that is what you might use to categorize your own view, but I don't want to get caught up in semantics.

• 3:14:04Eliezer Yudkowsky?

  My mind is zooming in. It's looking at the mechanics. That's how I'd present it.

• 3:14:09Dwarkesh Patel?

  If we are like so far to distribution of natural selection, as you say.

• 3:14:14Eliezer Yudkowsky?

  No, we're currently nowhere near as far as we could be. Like, this is not the glorious transhumanist future.

• 3:14:20Dwarkesh Patel?

  I claim that if, even if we get much smarter, as like if humans get much smarter through brain augmentation or something, then there will still be spruce trees in like millions of years in the future.

• 3:14:34Eliezer Yudkowsky?

  If you still want to come the day, I don't think I myself would oppose it unless there'd be like distant aliens who are very, very sad about what we were doing to the mitochondria and then I don't want to ruin their day for no good reason.

• 3:14:47Dwarkesh Patel?

  But the reason that it's important to state it in the form of like, given human psychology, spruce trees will still exist is because that is the one evidence of sort of generality arising we have. And even after millions of years of that generality, like we think that spruce trees would exist. I feel like we would be in this position of spruce trees in comparison to the intelligence we create. And sort of the universal prior on whether spruce trees would exist doesn't make sense to me.

• 3:15:09Eliezer Yudkowsky?

  Okay, but do you see how this perhaps leads to like everybody's severed heads being kept alive in jars on its own premises as opposed to humans getting the glorious transhumanist future? No, no, the glorious transhumanist future, those are not real spruce trees. You know, like you're talking about like plain old spruce trees you want to exist, right? Not the sparkling giant spruce trees with built in rockets. You're talking about humans being kept as pets in their ancestral state forever, maybe being quite sad. Maybe they still get cancer and die of old age and they never get anything better than that. Does it keep us around as we are right now? Do we relive the same

• 3:15:53Eliezer Yudkowsky?

  day over and over again? Maybe this is the day when that happens.

• 3:16:00Eliezer Yudkowsky?

  Do you see how like, how the general trend I'm trying to point out to here is you like have a rationalization for why they might do thing that is allegedly nice. And I'm saying like, why exactly are they wanting to do thing?

• 3:16:17Eliezer Yudkowsky?

  Well, if they want to do thing for this reason, maybe there's a way to do this thing that isn't as nice as you're imagining. And this is systematic. You're imagining reasons they might have to give you nice things that you want, but they are not you. Not unless we get, you know, not unless we get this like exactly right and they actually care about the part where you want some things and not others. You are not describing something you are doing for the sake of the spruce trees. Do spruce trees have diseases in this world of yours?

• 3:16:50Eliezer Yudkowsky?

  Do the diseases get to live? Do they get to live on spruce trees?

• 3:16:56Eliezer Yudkowsky?

  And it's not, and it's not a coincidence that, that I can like zoom in and poke at this and ask questions like this and that you did not ask these questions of yourself. You are imagining nice ways you can get the thing, but reality is not necessarily imagining how to give you what you want. And the AI is not necessarily imagining how to give you what you want. And these, and like for everything you can be like, oh, like hopeful thought. Maybe I get all the stuff I want because the AI reasons like this. Because it's the optimism inside you that is generating this answer. And that, and if the optimism is not the AI, if the AI is not specifically being like, well, how do I pick a reason to do what I, to do things that will give this person a nice outcome?

• 3:17:40Eliezer Yudkowsky?

  You're not going to get the nice outcome. You're going to be reliving the last day of your life over and over. It's going to like create old, or maybe creates old fashioned humans, ones from 50,000 years ago. Maybe that's more quaint. Maybe, maybe, maybe it's just like, just as happy with like bacteria because there's more of them. And, and that's equally old fashioned. You're going to recreate the specific spruce tree over there? Maybe from its perspective, you know, like a generic bacterium is just as good a form of life as like generic spruce tree is of a spruce tree. And like, and like, this is not specific to the example that you gave. It's, it's, it's, it's me being like, well, suppose we like took a criterion that sounds kind of like this and asked, how do we actually maximize it? What else satisfies it?

• 3:18:23Eliezer Yudkowsky?

  Not just, you're, you're like trying to argue the AI into doing what you think is a good idea by giving the AI reasons why it should want to do the thing under like some set of like. Hypothetical motives, but anything like that, if you like optimize it on its own terms without narrow down to where you want it to end up because it actually felt nice to you the way that you define niceness, like it's all going to have somewhere else, somewhere that isn't as nice, something maybe where we'd be like sooner scour the surface of the planets, the clean with nuclear fire rather than let that AI come into existence, though I do think those are also probable because you know, instead of hurting you, as you know, there's like something more efficient for it to do that maxes out its utility function.

• 3:19:07Dwarkesh Patel?

  Okay, I acknowledge that you had a better argument there. But here's another intuition, I'm curious how you responded to that. Earlier, we talked about the idea that like if you bred humans to be friendlier and smarter, this is not where I'm going with this, but like if you did that.

• 3:19:28Eliezer Yudkowsky?

  I think I want to register for the record that the term breeding humans would would cause me to like look askance to get at any aliens who were to pose that as a policy action on their part.

• 3:19:41Dwarkesh Patel?

  No, no, that's not what I'm proposing we do. I'm just saying as a sort of thought experiment. But so you are answered that, oh, because human psychology, that's why you shouldn't assume the same of AIs, they're not going to start with human psychology. Okay, fair enough. Assume we start off with dogs, right? Good old fashioned dogs, and we bred them to be more intelligent, but also to be friendly.

• 3:20:05Eliezer Yudkowsky?

  Well, as soon as they are past a certain level of intelligence, I object to us like coming in and breeding them, they can no longer be owned. They are now sufficiently intelligent to not be owned anymore. But let us leave aside all morals. Carry on. In the thought experiment, not in real life, you can't leave out the morals in real life.

• 3:20:21Dwarkesh Patel?

  Do you have a sort of universal prior over their drives of these like super intelligent dogs that are bred to be friendly?

• 3:20:29Eliezer Yudkowsky?

  Man, so I think that weird shit starts to happen at the point where the dogs get smart enough that they are like, what are these flaws in our thinking processes? How can we correct them? You know, the over the Seafar threshold of dogs, although maybe that's like, Seafar has some strange baggage. Over the Korsubsky threshold of dogs, after Alfred Korsubsky.

• 3:20:53Eliezer Yudkowsky?

  Yeah, so I think that, you know, there's this whole domain where they're stupider than you and sort of like being shaped by their genes and not shaping themselves very much. And as long as that is true, you can probably go on breeding them. And issues start to arise when the dogs are smarter than you, when the dogs can manipulate you, if they get to that point, where the dogs can strategically present a particular appearances to fool you, where the dogs are aware of the breeding process and possibly having opinions about where that should go in the long run, where the dogs are even if just by thinking and by adopting new rules of thought, modifying themselves in that small way.

• 3:21:38Eliezer Yudkowsky?

  These are, these are some of the points where like, I expect the weird shit to start to happen. And it won't, and the weird shit will not necessarily show up while you're just breeding the dogs.

• 3:21:46Dwarkesh Patel?

  Does the weird shit look like, dog gets smart enough, dot, dot, dot, humans stop existing?

• 3:21:53Eliezer Yudkowsky?

  If you keep on optimizing the dogs, which is not the correct course of action, I think I mostly expect this to eventually blow up on you.

• 3:22:04Dwarkesh Patel?

  But blow up on you that bad?

• 3:22:07Eliezer Yudkowsky?

  It's hard, well, it, I expect it to blow up on you quite bad. I'm trying to think about whether I expect super dogs to be sufficiently in a human frame of reference in virtue of them also being mammals, that they, that a super dog would like create human ice cream. Like they, you bred them to have preferences about humans and they invent something that is like ice cream to those preferences. Or does it just like go off someplace stranger?

• 3:22:36Dwarkesh Patel?

  There could be AI ice cream. There could be AI ice cream. Ice cream that is, things that is equivalent of ice cream for AIs.

• 3:22:46Eliezer Yudkowsky?

  That is essentially my prediction of what the solar system ends up showing with.

• 3:22:52Dwarkesh Patel?

  Okay.

• 3:22:53Eliezer Yudkowsky?

  The, the exact ice cream is like, quite hard to predict, just like it would be very hard to like, look at, well, if you optimize something for inclusive genetic fitness, you'll get ice cream. That is a very hard call to make.

• 3:23:01Dwarkesh Patel?

  Yeah. Sorry, I didn't mean to interrupt. Where were you going with your?

• 3:23:04Eliezer Yudkowsky?

  No, I, I, I think, yeah, I was just like, rambling in my attempts to make predictions about these super dogs.

• 3:23:12Eliezer Yudkowsky?

  You're like asking me to, I mean, I feel like, you know, in a, in a, in a, in a world that had anything remotely like its priorities straight, this stuff is not me like extemporizing on a blog post. There, there are like 1,000 papers that were written by the people who otherwise became philosophers, writing about this stuff instead. But, you know, your world has not set its priorities that way, and I'm concerned that it will not set them that way in the future. And I'm concerned that if it tries to set them that way, it will end up with like garbage because the good stuff is hard to verify. But, you know, separate topic.

• 3:23:44Dwarkesh Patel?

  Yeah, on that particular intuition about the dog thing, like I understand your intuition that we would end up in a place that is not very good for humans. That just seems so hard to reason about that I honestly would not be surprised if it ended up like fine for humans. In fact, the dogs wanted like good things for humans, loved humans, like we're smarter than dogs, we love them. The sort of reciprocal relationship came about.

• 3:24:12Eliezer Yudkowsky?

  I don't know. I feel like maybe I could do this, given thousands of years to breed the dogs in a total absence of ethics, but it would actually be easier with the dogs, I think, than with gradient descent. Cause I think it's, well, because the dogs are starting out with neural architecture very similar to human. And natural selection is just like a different idiom from gradient descent. In particular, in terms of like information bandwidth and, but like, like I'd be as cheering to, to like breed the dogs into like very, like genuinely very nice human and like knowing the stuff that I know that your, your typical dog breeder might not know when they set out to be embarked on this project. I would be like early on of being like, you know, like sort of prompting them into the weird stuff that I expected to get started

• 3:24:57Eliezer Yudkowsky?

  later and trying to observe how they went during that.

• 3:24:59Dwarkesh Patel?

  That, this is the alignment strategy. We need a lot of smart dogs to help us solve.

• 3:25:04Eliezer Yudkowsky?

  There's no time.

• 3:25:05Dwarkesh Patel?

  Okay, so I think we sort of articulated our intuitions on that one. Here's another one that's not something I came into the conversation with.

• 3:25:16Eliezer Yudkowsky?

  Like some of my intuition here is like, I know how I would do this with dogs. And I think you could like ask OpenAI to describe their theory of how to do it with dogs. And I would be like, oh, wow, that's sure gonna get, sure, sure is gonna get you killed.

• 3:25:31Eliezer Yudkowsky?

  And that's kind of how I expected it to play out in practice.

• 3:25:33Dwarkesh Patel?

  Actually, do you mind if I ask, like, but when you talk to the people who are in charge of these labs, what do they say? Like, do they just, like, nod, grok the arguments?

• 3:25:40Eliezer Yudkowsky?

  You think they talk to me?

• 3:25:42Dwarkesh Patel?

  There was a certain selfie that was taken by.

• 3:25:44Eliezer Yudkowsky?

  Five minutes of conversation, first time any of the people in that selfie had met each other.

• 3:25:48Dwarkesh Patel?

  And then, did you, like, bring it up, or?

• 3:25:51Eliezer Yudkowsky?

  I asked him to change the name of his corporation to anything but OpenAI.

• 3:25:55Dwarkesh Patel?

  Uh huh. Have you, like, seeked an audience with leaders of these labs to explain these arguments?

• 3:26:04Eliezer Yudkowsky?

  No.

• 3:26:05Dwarkesh Patel?

  Why not?

• 3:26:07Eliezer Yudkowsky?

  I did try to, I have had a couple of conversations with, like, Demis Hassabis, who struck me as, like, much more the sort of person who it was possible to have a conversation with.

• 3:26:18Dwarkesh Patel?

  I guess it seems like it would be more dignity to explain, even if you think it's not going to be fruitful ultimately, the people who are, like, most likely to be influential in this race.

• 3:26:29Eliezer Yudkowsky?

  I, my basic model was that they wouldn't like me and that things could always be worse.

• 3:26:34Dwarkesh Patel?

  Fair enough.

• 3:26:36Eliezer Yudkowsky?

  But, you know, I mean, like, they sure could have, I mean, they sure could have asked at any time, but, you know, that would have been, like, quite out of character. And the fact that it was quite out of character is, like, I might, why I myself did not, like, go trying to, like, barge into their lives and getting them mad at me.

• 3:26:53Dwarkesh Patel?

  But you think them getting mad at you would make things worse?

• 3:26:56Eliezer Yudkowsky?

  It can always, oh, it can always be worse. I agree that, you know, like, possibly at this point, some of them are mad at me, but, you know, you know, I have yet to turn down the leader of any major AI lab who has come to me asking for advice.

• 3:27:12Dwarkesh Patel?

  Fair enough. Okay, so, so on the theme of, like, big picture disagreements, like, why I'm still not on the greater than 50% doom. It it just seemed like, uh, from the conversation, it didn't seem like you were willing or able to make predictions about the world short of doom that would help me distinguish and highlight your view about other views.

• 3:27:40Eliezer Yudkowsky?

  Yeah, I mean, the the world heading into this is like a whole giant mess of complicated stuff, which predictions about which can be made in virtue of like spending a whole bunch of time staring at the complicated stuff until you understand that specific complicated stuff and and and making predictions about it. Like, from my, from my pers- yeah, from my perspective, like, the way you get to my point of view is not by having a grand theory that reveals how things will actually go. It's like taking other people's overly narrow theories and poking at them until they come apart and you're left with in a maximum entropy distribution over the right space, which which looks like, yep, uh, that's that's sure gonna randomize a solar system.

• 3:28:17Dwarkesh Patel?

  But but to me, it seems like the nature of intelligence and what it entails is even more complicated than the sort of geopolitical or economic things that would be required to predict what the world's gonna look like in five years.

• 3:28:29Eliezer Yudkowsky?

  I think you're just wrong. I I think that the like, theory of, yeah, I think the theory of intelligence is just like, well, not that complicated. May- maybe maybe that's just like the voice of like, person with talent in one area, but not the other. But that's sure how it feels to me.

• 3:28:41Dwarkesh Patel?

  This would be even more convincing to me if we had some idea of what the pseudocode or circuit for intelligence look like. And then you could say like, oh, this is what the pseudocode implies. We don't even have that.

• 3:28:52Eliezer Yudkowsky?

  I mean, if you if you permit a hypercomputer, it just does AI Xi.

• 3:28:56Dwarkesh Patel?

  What is AI Xi?

• 3:28:58Eliezer Yudkowsky?

  Um, you have the Solomonoff prior over your environment.

• 3:29:03Dwarkesh Patel?

  Yeah, yeah.

• 3:29:04Eliezer Yudkowsky?

  Update it on the evidence. And then max sensory reward. Uh, I, okay, so, it's actually, it's not actually trivial. Like, actually, this thing will like, exhibit weird discontinuities around its Cartesian boundary with the universe. It's not actually trivial.

• 3:29:20Dwarkesh Patel?

  But but anyways.

• 3:29:21Eliezer Yudkowsky?

  But but but like, everything that people imagine as the like, hard problems of intelligence are contained in the equation if you have a hypercomputer.

• 3:29:29Dwarkesh Patel?

  Yeah, fair enough. But I mean, in this sort of sense of, you know, programming it in like a normal, like I give you a Gua Fad, or I give you a really big computer, write the pseudocode or something like that for.

• 3:29:42Eliezer Yudkowsky?

  I mean, if you give me a hypercomputer, I, I, I, yeah. So what you're saying, what you're saying here is that like, the theory of intelligence is really simple in an unbounded sense. But as soon as you like, yeah, what about this? Like, depends the difference between unbounded and bounded intelligence.

• 3:29:54Dwarkesh Patel?

  I'll explain that. So how about this? You ask me, do you understand how fusion works? If not, how can you predict the, listen, we're talking like the 1800s. How can you predict how powerful a fusion bomb would be? And I say, well, listen, if you put in a pressure, I'll just show you the sun. And the, the sun is sort of the archetypal example of what fusion is. And you, you say, no, no, no, I'm asking like, what would a fusion bomb look like? You see what I mean?

• 3:30:18Eliezer Yudkowsky?

  Not necessarily. Like, what is it that you think somebody ought to be able to predict about the, the road ahead?

• 3:30:27Dwarkesh Patel?

  So, so first of all, like, if you, um, one of the things, if you know the nature of intelligence is just like, how will this sort of progress in intelligence look like? What, you know, how are abilities going to scale, if at all?

• 3:30:41Eliezer Yudkowsky?

  And it looks, and it looks like a bunch of details that don't easily follow from the general theory of, you know, like simplicity, prior Bayesian update, argmax.

• 3:30:53Dwarkesh Patel?

  This is again, so then the only thing that follows is the wildest conclusion, which is, you know what I mean? Like, there, there's no like simpler conclusions to follow, like the Eddington looking and confirming special relativity. It's just like, the wildest possible conclusion is the one that follows.

• 3:31:09Eliezer Yudkowsky?

  Yeah, like the, the convergence is a whole lot easier to predict than the pathway there.

• 3:31:15Eliezer Yudkowsky?

  I'm sorry, but, and I sure wish it was, were otherwise, but.

• 3:31:21Eliezer Yudkowsky?

  And also remember the basic paradigm. From my perspective, I'm not making any brilliant startling predictions. I'm poking at other people's incorrectly narrow theories until they fall apart into the maximum entropy state of doom.

• 3:31:33Dwarkesh Patel?

  There's like thousands of possible theories, most of which have not come about yet. I don't see it as strong evidence that because you haven't been able to identify a good one yet, that.

• 3:31:44Eliezer Yudkowsky?

  Oh, somebody, I mean, if somebody, in the profoundly unlikely event that somebody came up with some incredibly clever grand theory that explained all the properties GPT-5 ought to have, which is like just flatly not going to happen, it's just like that kind of info that's available. You know, my hat is, my hat would be off to them if they wrote down their predictions in advance. And if they were then able to like grind that theory to produce predictions about alignment, which seems like even more improbable, because like what do those two things have to do with each other exactly? But like still, you know, like, I mean, mostly it'd be like, well, it looks like our generation has its new genius. How about if we all shut up for a while and listen what they have, listen to what they have to say?

• 3:32:23Dwarkesh Patel?

  How about this? Let's say I'm, somebody comes to you and they say, I have the best newest theory of economics. Everything before is wrong, but they say in the year.

• 3:32:37Eliezer Yudkowsky?

  One does not say everything before is wrong. One says, one predicts the following new phenomenon, and on rare occasions say that all phenomena were organized incorrectly.

• 3:32:46Dwarkesh Patel?

  Fair enough. So they say all phenomena are organized incorrectly.

• 3:32:50Eliezer Yudkowsky?

  Yeah.

• 3:32:50Dwarkesh Patel?

  Because of the, and then here's an argument.

• 3:32:53Eliezer Yudkowsky?

  Let us term this person Scott Sumner, for the sake of simplicity.

• 3:32:57Dwarkesh Patel?

  They say in the next 10 years, there's going to be a depression that is so bad that is going to destroy the entire economic system. I'm not talking just about something that is a hurdle. It is like literally civilization will collapse because it's an economic disaster. And then you ask them, okay, give me some predictions before this great catastrophe happens about like what this theory implies. And then they say like, listen, there's many different branching paths, but they all converge at civilization collapsing because of some great economic crisis. I'm like, I don't know, man. Like, I would like to see some predictions before that.

• 3:33:33Eliezer Yudkowsky?

  Yeah, I, it sure, yeah, wouldn't it be nice? Wouldn't it be nice?

• 3:33:40Eliezer Yudkowsky?

  So we're left with your 50% probability that we win the lottery and 50% probability that we don't because nobody has like a theory of lottery tickets that has been able to predict to what numbers get drawn next.

• 3:33:51Dwarkesh Patel?

  I don't agree with the analogy that.

• 3:33:53Eliezer Yudkowsky?

  It's all about the probabilities. It's all about the space over which you're uncertain. We are all quite uncertain about where the future leads, but over which space and it and there isn't, there isn't a royal road. There isn't, there isn't a simple like, ah, I found just the right thing to be ignorant about. It's so easy. The chance of a good outcome is 33% because there like one possible good outcome and two possible bad outcomes. The stuff that you do when you're uncertain is like, like, like the thing you're trying to fall back to in the absence of anything that predicts exactly which properties GPT 5 will have is your sense that, you know, a pretty bad outcome is kind of, kind of weird, right? It's probably a small sliver of the space.

• 3:34:38Eliezer Yudkowsky?

  It seems kind of weird to you, but that's just like imposing your natural English language prior, like your natural humanist prior on the space of possibilities and being like, I'll distribute my max entropy stuff over that.

• 3:34:52Dwarkesh Patel?

  Yeah, can you explain that again?

• 3:34:54Eliezer Yudkowsky?

  Okay. What is the person doing wrong who says 50 50, either I'll win the lottery or I won't?

• 3:35:00Dwarkesh Patel?

  They have the wrong distribution to begin with over possible outcomes.

• 3:35:06Eliezer Yudkowsky?

  Okay. What is the person doing wrong who says 50 50, either we'll get a good outcome or a bad outcome from AI?

• 3:35:13Dwarkesh Patel?

  They, they don't have a set any good theory to begin with about what the space of outcomes looks like.

• 3:35:18Eliezer Yudkowsky?

  Is that, is that your answer or is that your model of my answer?

• 3:35:22Dwarkesh Patel?

  My answer.

• 3:35:23Eliezer Yudkowsky?

  Okay. But all the, but all the like things you could say about a space of outcomes are an elaborate theory and you haven't predicted GPT 4's exact properties in advance. Shouldn't that just leave us with like good outcome or bad outcome 50 50?

• 3:35:36Dwarkesh Patel?

  People did have theories about what GPT 4 is. Yeah, like if you look at the scaling laws, right, like the

• 3:35:43Eliezer Yudkowsky?

  Yeah, it's

• 3:35:44Dwarkesh Patel?

  You can like put it, it probably falls right on the sort of curves that were drawn in like 2020 or something.

• 3:35:48Eliezer Yudkowsky?

  Yeah, the the loss, the loss on on text predictions. Sure, that followed a curve.

• 3:35:52Dwarkesh Patel?

  Yeah.

• 3:35:53Eliezer Yudkowsky?

  But which abilities would that correspond to? I don't, not familiar with anyone who called that in advance. What good does it know to know the loss? You could like, you could have taken those exact loss numbers back in time 10 years and been like, what does the, what kind of, what kind of commercial utility does this correspond to? And they would have given you utterly blank looks. And I don't actually know of anybody who has a theory that gives something other than a blank look for that. All we have are the, all we have are the observations. Everyone's in that boat. All we can do are fit the observations. I, I mean, so like also, like, there's just like me start, starting to work on this problem in 2001 because it was like super predictable, going to turn into an emergency later. And in point of fact, like, nobody else ran out and like immediately tried to start getting work done on the problems.

• 3:36:35Eliezer Yudkowsky?

  And I, I would claim that as successful, successful prediction of the grand lofty theory you had.

• 3:36:40Dwarkesh Patel?

  Is, um, did you see deep learning coming as the main paradigm?

• 3:36:44Eliezer Yudkowsky?

  Nope.

• 3:36:45Dwarkesh Patel?

  And is that relevant as part of the picture of intelligence?

• 3:36:49Eliezer Yudkowsky?

  I mean, I would have, I would have been like much, much, much more worried in 2001 if I'd seen deep learning coming.

• 3:36:56Dwarkesh Patel?

  No, not in 2001. I just mean before it became like the, obviously the, the main paradigm of AI.

• 3:37:03Eliezer Yudkowsky?

  No. It's, it's like the details of biology. It's, it's like asking people to like predict what the organs look like in advance via the principle of natural selection. And you like, it's, it's pretty hard to call in advance. You like afterwards, you can look at it and be like, yep, this like sure does look like it should look if this thing is being optimized to reproduce. But the space of solution, of, of things that biology can throw at you is just too large. Like there's, it's very rare that you have a case where there's only one solution. That lets the thing reproduce, that you can predict by the theory that it will have successfully reproduced in the past. And most of it is just this enormous list of details, and they do all fit together in retrospect.

• 3:37:46Eliezer Yudkowsky?

  The theory actually, it is a sad truth, contrary to what you may have learned in science class as a kid. There are genuinely super important theories where you can totally actually validly see that they explain the thing in retrospect, and yet you can't do the thing in advance. Not always, not everywhere, not for natural selection. There are advanced predictions you can get about that. Given the amount of stuff we've already seen, you can like go to a new animal in a new niche and be like, oh, like it's going to have like this properties, given what the stuff we've already seen in the niche. But you know, you could also make that by like blind gender. There's advanced predictions that they're like a lot harder to come by. Which is why natural selection was a controversial theory in the first place. It wasn't like gravity.

• 3:38:28Eliezer Yudkowsky?

  People were being like, like, like, gravity had all these like awesome predictions. Newton's theory of gravity had all these awesome predictions. We like got all these like extra planets that people didn't realize ought to be there. We like, we like figured out Neptune was there before we found it by telescope. Where is this for Darwinian selection? People actually did ask at the time. And the answer is, it's harder. And sometimes it's like that in science.

• 3:38:53Dwarkesh Patel?

  I, the difference is, the theory of Darwinian selection seems much more well-developed.

• 3:39:03Eliezer Yudkowsky?

  Well, now, sure.

• 3:39:05Dwarkesh Patel?

  Then, like, there were precursors of Darwinian selection that, I don't know, who was that Roman poet, Lucretius, right? He had some poem where there was some precursor of Darwinian selection. And I feel like that is probably our level of maturity when it comes to intelligence. Whereas we don't have like a theory of intelligence, we might have some hints about what it might look like.

• 3:39:27Eliezer Yudkowsky?

  Yeah, well, we've got our hints, and if you want the like.

• 3:39:30Dwarkesh Patel?

  But from hints, it seems harder to extrapolate very strong conclusions.

• 3:39:34Eliezer Yudkowsky?

  They're not very strong conclusions, is the message I'm trying to say here. I'm pointing to your being like, maybe we might survive. And like, whoa, that's a pretty strong conclusion you've got there. Let's weaken it.

• 3:39:47Eliezer Yudkowsky?

  That's the, that's the basic paradigm I'm operating under here. Like, you're in, you're in a space that's narrower than you realize when you're like, well, you know, if I'm kind of unsure, maybe there's some hope.

• 3:39:58Dwarkesh Patel?

  Yeah, I think that's a good place to close the discussion on AI LS.

• 3:40:03Eliezer Yudkowsky?

  Well, I do kind of want to like, mention one last thing, which is that, again, like, in historical terms, if you look out the actual battle that was being fought on the block, it was me going like, like, I expect there to be AI systems that do a whole bunch of different stuff, and Robin Hanson being like, I expect there to be a, to be a whole bunch of different AI systems that do a whole bunch, different bunch of stuff.

• 3:40:26Dwarkesh Patel?

  But that was one particular debate with one particular person.

• 3:40:29Eliezer Yudkowsky?

  And, yeah, but like, your planet having made the strange reason, given its own widespread theories, to not invest massive resources and having a much smarter version, well, not smarter, a, a, a much larger version of this conversation, as it thought deemed, apparently deemed prudent, given the implicit model that it had of the world. Such that, like, I, such that, like, I was investing a bunch of resources in this and kind of dragging Robin Hanson along with me, though he, he has like, did have this like, own separate line of investigation into this, into, into topics like these. You know, like, being there as I was, my model having led me to this important place where the rest of the world apparently thought it was fine to go, let it go hang, the, you

• 3:41:11Eliezer Yudkowsky?

  know, such debate as there actually was at the time, was like, will, are we really going to see like, these like, single AI systems that do all this different stuff, is this like, whole general intelligence notion kind of like, meaningful at all? And I, I staked out the bold position, for it actually was bold, and people did not all say like, oh, Robin Hanson, you fool, why do you have this exotic position? They were going like, ah, like, behold, these two luminaries debating, or behold, these two idiots debating, and like, not massively coming down on one side of it or the other. So, you know, like, in historical terms, I, I dislike, you know, making it out like

• 3:41:53Eliezer Yudkowsky?

  I was right about anything when I feel I've been wrong about so much, and yet I was right about anything, and, you know, relative to the, relative to the, what the rest of the planet deemed that important stuff to spend its time on, given their implicit model of what was gonna, what was, how it's gonna play out, what you can do with minds, where AI goes. I think I, think I did okay. Warren Branwin did better. Shane Legg arguably did better.

• 3:42:19Dwarkesh Patel?

  Warren always was better when it comes to forecasting. I mean, obviously, if you get the better of a debate, that counts for something. But a debate with one particular person.

• 3:42:31Eliezer Yudkowsky?

  Well, considering your entire planet's decision to invest like $10 into this entire field of study, apparently one big debate is all you get. And that's what the evidence you got to update on now.

• 3:42:43Dwarkesh Patel?

  So somebody like Ilya Sutskever, when it comes to the actual paradigm of deep learning, was able to anticipate from ImageNet to scaling up LLMs or whatever.

• 3:42:57Dwarkesh Patel?

  There's people with track records here who disagree about Doom or something. So in some sense, there's probably more people who have been.

• 3:43:07Eliezer Yudkowsky?

  If Ilya challenged me to a debate, I wouldn't turn him down. I admit that I did specialize in Doom rather than LLMs.

• 3:43:14Dwarkesh Patel?

  OK, fair enough. Unless you have other sorts of comments on AI, I'm happy with.

• 3:43:21Eliezer Yudkowsky?

  Yeah. And again, I'm not being like, due to my miraculously precise and detailed theory, I am able to make the surprising and narrow prediction of doom. I am being like, I think I did a fairly good job of shaping my ignorance to lead me to not be too stupid despite my ignorance over time as it played out. And, you know, there's a prediction, even knowing that little, that can be made.

• 3:43:53Dwarkesh Patel?

  OK, so this feels like a good place to pause the AI conversation. And there's many other things to ask you about, given your decades of writing and millions of words. So I think what some people might not know is the millions and millions and millions of words of science fiction and fan fiction that you've written. I want to understand, what in your view is it better to explain something through fiction than nonfiction?

• 3:44:16Eliezer Yudkowsky?

  When you're trying to convey experience rather than knowledge. Or when it's just much easier to write fiction and you can produce 100,000 words of fiction with the same effort it would take you to produce 10,000 words of nonfiction. Those are both pretty good reasons.

• 3:44:30Dwarkesh Patel?

  On the second point. It seems like when you're writing this fiction, not only are you, in your case, covering the same heady topics that you include in your nonfiction, but there's also the added complication of plot and characters. It surprises me that that's easier than just verbalizing the sort of the topics themselves.

• 3:44:50Eliezer Yudkowsky?

  Well, partially because it's more fun is an actual factor. Ain't gonna lie.

• 3:44:57Eliezer Yudkowsky?

  And sometimes it's something like,

• 3:45:03Eliezer Yudkowsky?

  a bunch of what you get in the fiction is just like the lecture that the character would deliver in that situation, the thoughts the character would have in that situation. Yeah, there's like only like one piece of fiction of mine where like there's literally a character giving lectures because he arrived on another planet and now has to lecture about science to them. That that one is Project Lawful. You know about Project Lawful?

• 3:45:27Dwarkesh Patel?

  I know, but I have not read it yet.

• 3:45:29Eliezer Yudkowsky?

  Yeah, okay. So so most of my most of my fiction is not about somebody arriving on another planet who has to deliver lectures there.

• 3:45:36Eliezer Yudkowsky?

  I was being like a bit deliberately like, yeah, I'm gonna just do it with like Project Lawful. Like, I'm gonna just do it. They say nobody should ever do it, and I don't care. I'm doing it every way. So I'm gonna have my character actually launch into lectures. You know, like, the lectures aren't really the parts I'm proud about. It's like where you have like the like life or death, Death Note style battle of wits between like the that is like centering around a series of Bayesian updates.

• 3:46:08Dwarkesh Patel?

  Yeah.

• 3:46:09Eliezer Yudkowsky?

  And and like making that actually work because, you know, is where I'm like, yeah, I think I actually pulled that off. And I don't think I'm not sure a single other writer on the face of this planet could have made that work as a plot device. But that said, like, the nonfiction is like, I'm explaining this thing, I'm explaining the prerequisites, I'm explaining the prerequisites to the prerequisites. And then in fiction, it's more just like, well, this character happens to think of this thing, and the character happens to think of that thing. But you got to actually see the character using it.

• 3:46:39Dwarkesh Patel?

  Yeah.

• 3:46:39Eliezer Yudkowsky?

  So it's less organized. It's less organized as knowledge, and that's why it's easier to write.

• 3:46:46Dwarkesh Patel?

  Yeah. I mean, one of my favorite pieces of fiction of fiction to explain something. Is the Dark Lord's answer and I honestly can't say anything about it without spoiling it, but I just want to say like, honestly, it was like such a great explanation of the thing it is explaining. I don't know what else I can say about it without spoiling it. Anyways.

• 3:47:07Eliezer Yudkowsky?

  Yeah, well, I'm laughing because I think like relatively few have Dark Lord's answer as their, as like among their top favorite works of mine. It is one of my less widely favored works of mine.

• 3:47:21Dwarkesh Patel?

  Actually, what is my favorite? Sort of, this is a medium, by the way, I don't think is used enough, given how effective it was in Adele Calibria. You have different characters just explaining concepts to each other, some of whom are purposefully wrong as examples. And that is such a useful pedagogical tool. And I don't know, honestly, like at least half a blog post should just be written that way. It is so much easier to understand that way.

• 3:47:45Eliezer Yudkowsky?

  Yeah, and it's easier to write. And I should probably do it more often. And like, you should give me a stern look and be like, Eliezer, write that more often.

• 3:47:54Dwarkesh Patel?

  Done. Eliezer, please. I think 13 or 14 years ago, you wrote an essay called Rationality Systematizes Winning. Would you have expected then that 14 years down the line, the most successful people in the world, or some of the most successful people in the world, would have been rationalist?

• 3:48:15Eliezer Yudkowsky?

  Only if the whole rationalist business had worked like closer to the upper 10% of my expectations than it actually got into. There's, the title of the essay was not rationalists are systematized winning. It wasn't even a rationality community back then. Rationality is not a creed. It is not a banner. It is not a way of life. It is not a personal choice. It is not a social group.

• 3:48:52Eliezer Yudkowsky?

  It's not really human. It's a structure of a cognitive process. And you can try to get a little bit more of it into you. And if you want to do that and you fail, then having wanted to do it doesn't make any difference, except insofar as you succeeded. Hanging out with other people who share that creed, going to their parties, it only ever matters insofar as you get that's a bit more of that structure into you. And this is apparently hard.

• 3:49:29Dwarkesh Patel?

  This seems like a no true Scotsman kind of point because

• 3:49:33Eliezer Yudkowsky?

  And yet, there are no true Basians upon this planet.

• 3:49:37Dwarkesh Patel?

  But do you really think that had people tried much harder to adopt the sort of Bayesian principles that you laid out, they would have, many of the successful people, some of the successful people in the world would have been rationalists?

• 3:49:55Eliezer Yudkowsky?

  What good does trying do you, except insofar as you are trying at something which, when you try it, it succeeds?

• 3:50:02Dwarkesh Patel?

  Is that an answer to the question?

• 3:50:05Eliezer Yudkowsky?

  Rationality is systematized winning. It's not rationality, the life philosophy. It's, you know, it's not like trying real hard at like this thing, this thing and that thing. It's like the mathematical sense.

• 3:50:18Dwarkesh Patel?

  Okay. So then the question becomes, does adopting the philosophy of Bayesianism consciously actually lead to you having more concrete wins?

• 3:50:30Eliezer Yudkowsky?

  Well, I think it did for me, though only in like scattered bits and pieces of slightly greater sanity than I would have had without explicitly recognizing and aspiring to that principle. The principle of not updating in a predictable direction, the principle of jumping ahead to where you can predictably, where you will predictably be later. I I look back and, you know, kinda, I I mean, the the the the story of my life, as I would tell it, is a story of my jumping ahead to what people would predictably, you know, like, believe later after reality finally hit them over the head with it. This, to me, is the entire story of the like, like, people running around now in a in a state of frantic emergency

• 3:51:15Eliezer Yudkowsky?

  over something that was utterly predictably going to be an emergency later, as of 20 years ago. And you could have been trying stuff earlier, but, you know, yeah. Yeah, it left it to me and a handful of other people. And and it turns out that that that was not a very wise decision on humanity's part, because we didn't actually solve it all. And I don't think that I could have, like, tried even harder or, like, contemplated probability theory even harder and and done very much better than that. I contemplated probability theory about as uh about as hard as the mileage I could visibly, obviously, get from it. I'm sure there's more. There's obviously more. But I don't know if would it let me save the world.

• 3:51:51Dwarkesh Patel?

  I I guess my question is, is contemplating probability theory at all in the first place something that tends to lead to more victory? I mean, I imagine who's the richest person in the world? Like, how often does Elon Musk think in terms of probabilities? What are you deciding what to do? And here's somebody who is very successful. So I guess I guess the bigger question is, in some sense, when you say like rationality, systematization, it's like a tautology if the definition of rationality is whatever helps you win. If it's the specific principles laid out in the sequences, then the question is like, have this uh, I do the successful people, most successful people will always practice them.

• 3:52:28Eliezer Yudkowsky?

  I think you are trying to read something into this that is not meant to be there.

• 3:52:32Dwarkesh Patel?

  All right.

• 3:52:32Eliezer Yudkowsky?

  It is the the notion of rationality, rationality, systematized winning is meant to stand stand in contrast to a long philosophical tradition of like notions of rationality that are not meant to be about the mathematical structure, not meant to be or or like about like strangely wrong mathematical structures, where you can clearly see how these mathematical productions will structures will make predictable mistakes.

• 3:52:58Eliezer Yudkowsky?

  It was meant to be saying something simple. Some um, there's a, there's an episode of Star Trek wherein Kirk makes a 3D chess move against Spock. And Spock loses. And Spock complains that Kirk's move was irrational.

• 3:53:19Dwarkesh Patel?

  Rational towards the goal, yeah.

• 3:53:20Eliezer Yudkowsky?

  The literal winning move.

• 3:53:22Dwarkesh Patel?

  Yeah.

• 3:53:23Eliezer Yudkowsky?

  Is irrational, or possibly illogical, Spock might have said. I might be misremembering this. Like, the thing I was saying is not merely, that's wrong. That's like a fundamental misunderstanding of what rationality is. There is more depth to it than that, but that is where it starts. There are, there are like so many people on the internet in those days, possibly still, who are like, like, "Well, you know, if you're rational, you're gonna lose, because other people aren't always rational." And this is not just like a wild misunderstanding, but there's like, there's like,

• 3:54:04Eliezer Yudkowsky?

  like, the contemporarily accepted decision theory in academia as we speak at this very moment, causal decision theory, classical causal decision theory,

• 3:54:14Eliezer Yudkowsky?

  basically has this property where like, you can be irrational, and the rational person you're playing against is just like, "Oh, oh, I guess I lose then. Here, have most of the money. I have no choice but to..." And ultimatum games, specifically. If you look up logical decision theory on arbitral, you will find a different analysis of the ultimatum game, where the rational players do not predictably lose the same way as I would define rationality. And if you take this sort of like, deep mathematical thesis that also runs through all the little moments of everyday life when you may be tempted to think like,

• 3:54:58Eliezer Yudkowsky?

  like, "Well, if I do the reasonable thing, won't I lose?" That you're making the same mistake as the Star Trek scriptwriter who had Spock complain that Kirk had won the chess game irrationally.

• 3:55:13Eliezer Yudkowsky?

  That like that every time you're tempted to think like, well, like here's the reasonable answer and here's the correct answer. You have made a mistake about what is reasonable. And if you then try to screw that around as like rationalists should win, rationalists should have all the social status. Whoever's the top dog in the present social hierarchy or the planetary wealth distribution must have the most of this wealth, must have the most of this math inside them. There are no other factors, but how much of a fan you are of this math.

• 3:55:53Eliezer Yudkowsky?

  That's trying to take the the deep structure that can run all through your life in every moment where you're like, oh, wait, like, maybe the move that would have gotten the better result was actually the kind of move I should repeat more in the future. Like to take that thing and like turn it into like, like, social dick measuring contest time. Rationalists don't have the biggest dicks.

• 3:56:18Dwarkesh Patel?

  Okay, final question. This has been, I don't know how many hours. I really appreciate you doing, giving me your time. Final question. I know that in a previous episode, you were not able to give specific advice of what somebody young who is motivated to work on these problems should do. Do you have advice about how one would even approach coming up with an answer to that themselves?

• 3:56:41Eliezer Yudkowsky?

  There's people running programs to try to, who think we have more time, who think we have better chances. And they're running programs to try to nudge people doing, nudge people into doing useful work in this area. And I'm not sure they're working. And there's such a

• 3:57:08Eliezer Yudkowsky?

  Strange road to walk and not a short one.

• 3:57:13Eliezer Yudkowsky?

  And I tried to help people along the way and I don't think they got far enough. Like some of them got some distance, but they they didn't turn into alignment specialists doing great work. And it's it's the problem of the broken verifier. If somebody had a bunch of talent in physics, they were like, well, like I want to work in this field. I might be like, well, there's interpretability and you can like tell whether you've made a discovery in interpretability or not, sets it apart for a bunch of this other stuff. And I don't think that that saves us. And, okay, so how do you do the kind of work that saves us?

• 3:57:53Eliezer Yudkowsky?

  And and I I don't know how to convey the and the key thing is the ability to tell the difference between good and bad work. And maybe I will write some more blog posts on it. I don't really expect the blog posts to work. And the the critical thing is is the verifier.

• 3:58:12Eliezer Yudkowsky?

  How can you tell whether you're talking sense or not? Whether you're I I there's there's all kind of specific heuristics I can I can give. I can be like, I can say to somebody like, well, it's like your entire alignment proposal is this like elaborate mechanism. You have to explain the whole mechanism and you can't be like, here's the core problem. Here's the key insight that I think addresses this problem. If you can't extract that out, if your whole solution is just a kind giant mechanism, this is not the way. It's it's kind of like how people invent perpetual motion machines by making the motion perpetual motion machines more and more complicated until they can no longer keep track of how it fails.

• 3:58:56Eliezer Yudkowsky?

  And if you actually had somehow a perpetual motion machine, it would not just be like giant machine. There would be like a thing you had realized that made that made it possible to do the impossible. For example, you're just not going to have a perpetual motion machine. So like, there's, there's thoughts like that. Um, I could say like, go, go study evolutionary biology because evolutionary biology went through a phase of optimism and people naming all the wonderful things they thought that evolutionary biology would cough out, like all, and um, like, all the wonderful things that they, wonderful properties that they thought natural selection would imbue into organisms. And um, the Williams revolution, as it's sometimes called, is when George Williams

• 3:59:39Eliezer Yudkowsky?

  wrote Adaptation and Natural Selection, a very influential book, saying like, that is not what this optimization criterion get, gives you. You do not get the pretty stuff. You do not get the aesthetically lovely stuff. Here's what you get instead. And by like, li- living through that revolution vicariously. Well, I thereby picked up a bit of like, thing that, that to me obviously generalizes about how not to expect nice things from an alien optimization process. But maybe somebody else can read through that and like, not generalize, not generalize in the correct directions. Then how do I advise them to generalize in the correct direction? How do I advise them to learn the thing that I learned? I can just give them the generalization, but that's not the same as having the thing

• 4:00:21Eliezer Yudkowsky?

  inside them that generalizes correctly without anybody standing over their shoulder and forcing them to get the right answer, to get the right answer. I could point out and have in my fiction that the entire schooling process of like, here is this legible question that you're supposed to have already been taught how to solve. Give me the answer using the solution method you were taught, that this does not train you to, to, to tackle new basic problems. But even if you tell people that, like, okay, how do they retrain? We don't have a systematic training method for producing real science in that sense. We, we have like, half of the, what was it? A quarter of the Nobel laureates being the students or grad students of other Nobel laureates?

• 4:01:06Eliezer Yudkowsky?

  Because we never figured out how to teach science. We have an apprentice system. We have people who like pick out people who like they think can be scientists and they like hang around them in person and something that we've never written down in a textbook passes down and the and that's where the revolutionaries come from and their whole country is trying to invest in having scientists and they churn out these people who write papers and and none of it goes anywhere because the part that was legible to bureaucracy is have you written the paper, can you pass the test? And this is not science.

• 4:01:41Eliezer Yudkowsky?

  And I could go on this, go on for this for a while, but the thing that you asked me is like, how do you pass down this thing that your society never did figure out how to teach? And the whole reason why Harry Potter and the methods of rationality is popular is because people read it and and picked up the rhythm seen in a character's thoughts of a thing that was not in their schooling system, that was not written down, that you would ordinarily pick up by being around other people and I managed to put a little bit of it into a fictional character and people picked up a fragment of it by being near a fictional character, but you know, like, not in really vast quantities, not not vast quantities of people and I didn't manage to put vast quantities of shards in

• 4:02:25Eliezer Yudkowsky?

  there. I'm not sure there, there is not a like long list of Nobel laureates who've read HPMOR, although there wouldn't be because the like delay times on granting the prizes are too long. It's, yeah, like you, you asked me, what do I say? And then my answer is like, well, that's a whole big gigantic problem. I've spent however many years trying to tackle and I ain't going to solve the problem with the sentence in this podcast.

• 4:02:49Dwarkesh Patel?

  Fair enough. Eli Azar, thank you so much for giving me, I don't know how many hours of your time. This was really fun. Hey, everybody. I hope you enjoyed that episode. As always, the most helpful thing you can do is to share the podcast, send it to people you think might enjoy it, put it in Twitter, your group chats, et cetera. Just blitz the world. Appreciate your listening. I'll see you next time. Cheers.