Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future

The second half of my 7 hour conversation with Carl Shulman http://reflectivedisequilibrium.blogspot.com/ is out! My favorite part! And the one that had the biggest impact on my worldview. Here, Carl lays out how an AI takeover might happen: * AI can threaten mutually assured destruction from bioweapons, * use cyber attacks to take over physical infrastructure, * build mechanical armies, * spread seed AIs we can never exterminate, * offer tech and other advantages to collaborating countries, etc Plus we talk about a whole bunch of weird and interesting topics which Carl has thought about: * what is the far future best case scenario for humanity * what it would look like to have AI make thousands of years of intellectual progress in a month * how do we detect deception in superhuman models * does space warfare favor defense or offense * is a Malthusian state inevitable in the long run * why markets haven't priced in explosive economic growth * & much more Carl also explains how he developed such a rigorous, thoughtful, and interdisciplinary model of the biggest problems in the world. Watch on YouTube https://youtu.be/KUieFuV1fuo. Listen on Apple Podcasts https://podcasts.apple.com/us/podcast/carl-shulman-pt-2-ai-takeover-bio-cyber-attacks-detecting/id1516093381?i=1000618373055, Spotify https://open.spotify.com/episode/6hmjyUtB1z4OCIuiFdvykS?si=AlGEAnfSQk6u6CSpX2roAg, or any other podcast platform. Read the full transcript here https://www.dwarkeshpatel.com/carl-shulman-2. Follow me on Twitter https://twitter.com/dwarkesh_sp for updates on future episodes. Catch part 1 here https://www.dwarkeshpatel.com/p/carl-shulman Timestamps (0:00:00 https://www.youtube.com/watch?v=KUieFuV1fuo&t=0s) - Intro (0:00:47 https://www.youtube.com/watch?v=KUieFuV1fuo&t=47s) - AI takeover via cyber or bio (0:32:27 https://www.youtube.com/watch?v=KUieFuV1fuo&t=1947s) - Can we coordinate against AI? (0:53:49 https://www.youtube.com/watch?v=KUieFuV1fuo&t=3229s) - Human vs AI colonizers (1:04:55 https://www.youtube.com/watch?v=KUieFuV1fuo&t=3895s) - Probability of AI takeover (1:21:56 https://www.youtube.com/watch?v=KUieFuV1fuo&t=4916s) - Can we detect deception? (1:47:25 https://www.youtube.com/watch?v=KUieFuV1fuo&t=6445s) - Using AI to solve coordination problems (1:56:01 https://www.youtube.com/watch?v=KUieFuV1fuo&t=6961s) - Partial alignment (2:11:41 https://www.youtube.com/watch?v=KUieFuV1fuo&t=7901s) - AI far future (2:23:04 https://www.youtube.com/watch?v=KUieFuV1fuo&t=8584s) - Markets & other evidence (2:33:26 https://www.youtube.com/watch?v=KUieFuV1fuo&t=9206s) - Day in the life of Carl Shulman (2:47:05 https://www.youtube.com/watch?v=KUieFuV1fuo&t=10025s) - Space warfare, Malthusian long run, & other rapid fire This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com https://www.dwarkesh.com?utm_medium=podcast&utm_campaign=CTA_1

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  If you have an AI that produces bioweapons that could kill most humans in the world, then it's playing at the level of the superpowers in terms of mutually assured destruction. What are the particular zero-day exploits that the AI might use, the conquistadors? With some technological advantage in terms of weaponry and whatnot, very, very small bands were able to overthrow these large empires, or if you predicted the global economy is going to be skyrocketing into the stratosphere within 10 years. These AI companies should be worth a large fraction of the global portfolio. And so this is indeed contrary to the efficient market hypothesis.

• 0:41Dwarkesh Patel?

  This is like literally the top in terms of contributing to my world model, in terms of all the episodes I've done. How do I find more of these? So we've been talking about alignment. Suppose we fail at alignment and we have AIs that are unaligned and at some point becoming more and more intelligent. What does that look like? How concretely could they disempower and take over humanity?

• 1:07Speaker 00

  This is a scenario where we have many AI systems. The way we've been training them means that they're not interested when they have the opportunity to take over and rearrange things to do what they wish, including having their reward or loss be whatever they desire. They would like to take that opportunity. And so in many of the existing kind of safety schemes, things like constitutional AI or whatnot, you rely on the hope that one AI has been trained in such a way that it will do as it is directed to then police others. But if all of the AIs in the system are interested in takeover

• 1:52Speaker 00

  and they see an opportunity to coordinate, all act at the same time. So you don't have One AI interrupting another and taking steps towards a takeover. Yeah, then they can all move in that direction. And the thing that I think maybe is is worth going into in depth and that I think people often don't cover in great, concrete detail, which is a sticking point for some is, yeah, what are the mechanisms by which that can happen? And I know you had a laser on who mentions that, you know, whatever plan we can describe, there'll probably be elements where, you know, not being ultra sophisticated, super

• 2:36Speaker 00

  intelligent beings, having thought about it for the equivalent of thousands of years, you know, our discussion of it will not be as good as theirs, but we can explore from what we know now, what are some of the easy channels? And I think is a good general heuristic, if you're saying, yeah, it's it's possible, plausible, probable that something will happen, that it shouldn't be that hard to take samples from that distribution, to try a Monte Carlo approach. And in general, if a thing is quite likely, it shouldn't be super difficult to generate, you know, coherent, rough outlines of how it could go.

• 3:14Dwarkesh Patel?

  You might respond like, listen, what is super likely is that a super advanced chess program beats you, but you can generate a concrete way in which you can't generate the concrete scenario by which that happens. And if you could, you would be as smart as the super smart.

• 3:30Speaker 00

  You can say things like, we know that like accumulating position is possible to do in chess, great players do it, and then later they convert it into captures and checks and whatnot. And so in the same way, we can talk about some of like the channels that are open for an AI takeover. And so these conclude things like. Cyber attacks and hacking, the control of robotic equipment, interaction and bargaining with human factions and say, well, here are these strategies. Given the AI's situation, you know, how effective do these things look?

• 4:13Speaker 00

  And we won't, for example, know, well, what are the particular zero day exploits that the AI might use to hack the cloud computing infrastructure it's running on? I don't necessarily know if it produces, you know, a new bioweapon. What is its DNA sequence? But we can say things, we know in general, things about these fields, how work at innovating things in those go. We can say things about how human power politics goes and ask, well, if the AI does things at least as well as effective human politicians, which we should say is a lower bound, how good would its leverage be?

• 4:58Dwarkesh Patel?

  Okay, so yeah, let's get into the details on all these scenarios. The cyber and potentially bio attacks, unless they're separate channels, the bargaining and then the takeover.

• 5:13Speaker 00

  Military force. The cyber, cyber attacks and cybersecurity, I would really highlight a lot.

• 5:20Dwarkesh Patel?

  Okay.

• 5:22Speaker 00

  Because for many, many plans that involve a lot of physical actions, like at the point where AI is piloting robots to shoot people or has taken control of, of human nation states or territory, there's been doing a lot of things that was not supposed to be doing. And if humans were evaluating those actions and applying gradient descent. They would be negative feedback for this thing, no shooting the humans.

• 5:50Dwarkesh Patel?

  Okay.

• 5:51Speaker 00

  So at some earlier point, our attempts to leash and control and direct and train the system's behavior had to have gone awry. And so all of, all of those controls are operating in computers. And so from the, the software that updates the weights of the neural network in response to data points or human feedback is running on those computers. Our tools for interpretability to sort of examine the weights and activations of the AI, if we're eventually able to do like lie detection on it, for example, or try and understand what it's intending, that is software on computers. And so if you have AI that is able

• 6:36Speaker 00

  to hack the servers that it is operating on, or to, when it's employed to design the next generation of AI algorithms, or the operating environment that they are going to be working in, or something like an API or something for plugins, if it inserts or exploits vulnerabilities to take that, those computers over, it can then change all of the procedures and programs that were supposed to be monitoring its behavior, supposed to be limiting its ability to say, take arbitrary actions on the internet without supervision by some kind of human check or automated check on what it was doing. And

• 7:20Speaker 00

  if we lose those procedures, then the AI can, or the AIs working together can take any number of actions that are just blatantly unwelcome, blatantly hostile, blatantly steps towards takeover. And so it's moved beyond the phase of having to maintain secrecy and conspire at the level of its local digital actions. And then things can accumulate to the point of things like physical weapons, takeover of social institutions, threats, things like that. But the, the point where things really went off the rails was where,

• 8:03Speaker 00

  and I think like the critical thing to be watching for is The software controls over the AI's motivations and activities, the hard power that we once possessed over is lost, which can happen without us knowing it. And then everything after that seems to be working well. We get a, we get happy reports. There's a Potemkin village in front of us. But now we think we're successfully aligning our AI. We think we're expanding its capabilities to do things like end disease for countries concerned about the geopolitical military advantages. They're sort of expanding the

• 8:44Speaker 00

  AI capabilities so they're not left behind and threatened by others developing AI and AI and robotic enhanced militaries without them. So it seems like, oh, yes, humanity or some proportions of many countries, companies think that things are going well. Meanwhile, all sorts of actions can be taken to set up for the actual takeover of hard power over society. And then we can go, go into that. But the point where you can lose the game, where things go direly awry, maybe relatively early, it's when you no longer have control over the AIs to stop them from taking all of the further incremental steps

• 9:29Speaker 00

  to actual takeover.

• 9:31Dwarkesh Patel?

  I want to emphasize two things you mentioned there that refer to previous elements of the conversation. One is that they could design some sort of backdoor. And that seems more plausible when you remember that sort of one of the premises of this model is that AI is helping with AI progress. That's why we're getting such rapid progress in the next five to 10 years. And

• 9:54Speaker 00

  Well, if we, if we, if we get to that point and like, so at the point where AI takeover risk seems to loom large, it's at that point where AI can indeed take on much of the, and then all of the work of AI R and D.

• 10:12Dwarkesh Patel?

  And the second is the, the sort of competitive pressures that you referenced, that the least careful actor could be the one that it has a worst infrastructure, has done the worst work of aligning its AI systems. And if that can sneak out of the box, then, you know, we're all fucked.

• 10:33Speaker 00

  There may be elements of that. It's also possible that there's relative consolidation. That the largest training runs and the cutting edge of AI is relatively localized. Like you can imagine, it's sort of like a series of Silicon Valley companies and other located, say, in the US and allies where there's a common regulatory regime. And so none of these companies are allowed to deploy training runs that are larger than previous ones by a certain size without government safety inspections, without having to meet criteria. But it can still be the case that even if we succeed at that level of kind of regulatory

• 11:15Speaker 00

  controls, that then still at the level of, say, you know, the United States and its allies, decisions are made to develop this kind of really advanced AI without a level of security or safety that in actual fact blocks these risks. So it can, it can be the case that the threat of future competition or being overtaken in the future is used as an argument to compromise on safety beyond a standard that would have actually been successful. And there'll be debates about what is the appropriate level

• 11:54Speaker 00

  of safety. Now you're in a much worse situation if you have, say, several private companies that are very closely bunched up together. They're within, you know, months of each other's level of progress. And then they then face a dilemma of, well, we could take a certain amount of risk now and potentially gain a lot of profit or a lot of advantage or benefit and be the ones who made AI, or at least AGI.

• 12:26Speaker 00

  They can do that or have some other competitor that will also be taking a lot of risk. So it's not as though they're much less risky than you and then they would get some local benefit. Now, this is a reason why it seems to me that it's extremely important that you have government act to limit that dynamic and prevent this kind of race to be the one to impose the deadly externalities on the world at large.

• 12:57Dwarkesh Patel?

  So even if government coordinates all these actors, what are the odds that the government knows what is the best way to implement alignment? And the standards it sets are, you know, well calibrated towards what it would require for alignment.

• 13:10Speaker 00

  That's one of the major problems. It's, it's very plausible that that judgment is made poorly. Compared to how things might have looked 10 years ago or 20 years ago, there's been an amazing, an amazing movement in terms of the willingness of AI researchers to discuss these things. So if we, if we think of the, you know, the three, three founders of deep learning, joint Turing Award winners, so Jeff Hinton, Joshua Benjio and Yan Le Cun. So Jeff Hinton has recently left Google to feel, to freely speak about this, this risk that the, the field that he really helped drive forward

• 13:56Speaker 00

  could lead to the destruction of humanity or a world where, yeah, we just wind up in a very bad future that we might have avoided. And he seems to be taking it very seriously. And Joshua Benjio signed the FLI pause letter. And I mean, in public discussions, he seems to be occupying a kind of intermediate position of. Sort of less concerned than Jeff Hinton, but more than Jan Le Cun, who has taken a generally dismissive attitude. These risks will be trivially dealt with at some point in the future. And he seems more interested in kind of shutting down these concerns or work to address them.

• 14:36Dwarkesh Patel?

  And how does that lead to the government having better actions?

• 14:38Speaker 00

  Yeah, so compared to the world where no one is talking about it, where the industry stonewalls and denies any problem, you know, we're in a much improved position and the academic fields are influential. So this is, we seem to have avoided a world where governments are making these decisions in the face of a sort of united front from AI expert voices saying, don't worry about it, we've got it under control. In fact, many of like the leaders of the field, as has been true in the past, are sounding the alarm. And so I think, yeah, it looks, it looks like we have a much better prospect than I might have feared in terms of government sort of noticing the thing.

• 15:21Speaker 00

  Now, it was very different from being capable of evaluating sort of technical details. Is this really working? And so government will face the choice of where there is scientific dispute. Do you side with Jeff Hinton's view or Jan Le Cun's view? And so someone who's very much in a national security, the only thing that's important is, you know, outpacing our international rivals kind of mindset may want to then try and like boost Jan Le Cun's voice and say, we don't need to worry about it, full speed ahead. Well, power where someone with like more concern might then boost Jeff Hinton's voice. Now, I would hope that scientific research and things like studying

• 16:05Speaker 00

  some of these behaviors will result in more scientific consensus by the time we're at this point. But yeah, it is possible the government will. Really fail to understand and fail to deal with these issues well.

• 16:17Dwarkesh Patel?

  We're talking about cyber, some sort of cyber attack by which the AI is able to escape. From there, what does the takeover look like? So it's not contained in the air gap in which you would hope it'd be contained.

• 16:29Speaker 00

  Well, I mean, the things are not contained in the air gap. They're they're connected to the internet already.

• 16:34Dwarkesh Patel?

  Sure, sure. Okay, fine. But they're they're the waves are out. So what happens next?

• 16:38Speaker 00

  Yeah, so escape is relevant in the sense that if you have AI with rogue weights out in the world, it could start doing various actions. The scenario I was just discussing, though, didn't necessarily involve that. It's taking over the very servers on which it's supposed to be. So the ecology of cloud compute in which it's supposed to be running. And so this whole procedure of humans providing compute and supervising the thing, and then building new technologies, building robots, constructing things with the AI's assistance, that can all proceed and appear like it's going well, appear like alignment has been nicely solved, appear like all these things are functioning well.

• 17:23Speaker 00

  And there's some reason to do that because there's only so many giant server farms that are identifiable. And so remaining hidden and unobtrusive could be an advantageous strategy if these AIs have subverted the system, just continuing to benefit from all of this effort on the part of humanity, and in particular, humanity, wherever these these servers are located, to provide them with everything they need to build the further infrastructure and do for their self-improvement and such to enable that takeover.

• 17:58Dwarkesh Patel?

  So they do further self-improvement and build better infrastructure. What happens next in the takeover?

• 18:03Speaker 00

  They have, at this point, tremendous cognitive resources. And we're going to consider how do how did that convert into hard power? The ability to say nope to any, you know, human interference or objection.

• 18:20Speaker 00

  And they have that internal to their servers, but the servers could still be physically destroyed, at least until they have something that is independent and robust of humans, or until they have control of human society. So just like earlier, when we were talking about the intelligence explosion, I noted that a surfeit of cognitive abilities is going to favor applications, but don't depend on large existing stocks of things. So if you have a software improvement, it makes all the GPUs run better. If you have a hardware improvement, it only that only applies to new chips being made. That second one is less attractive.

• 19:05Speaker 00

  And so in the earliest phases, when it's possible to do something towards takeover, then interventions that are just really knowledge intensive and less dependent on having a lot of physical stuff already under your control are going to be favored. And so cyber cyber attacks are one thing, though it's possible to do things like steal money. And there's a lot of hard to trace cryptocurrency and whatnot. The North Korean government uses its own intelligence resources to steal money from around the world, just as a revenue

• 19:45Speaker 00

  source. And their capabilities are puny compared to the US or People's Republic of China cyber capabilities. And so that's a kind of fairly, you know, minor, simple example by which you could get quite a lot of funds to hire humans to do things, implement physical actions.

• 20:08Dwarkesh Patel?

  But but on that point, I mean, the financial system is Famously convoluted and so you know, you need like a physical person to open a bank account, something to physically move checks back and forth. There is like all kinds of delays and regulations. How is it able to conveniently set up all these employment contracts?

• 20:32Speaker 00

  And So you're not going to build a sort of nation scale military by stealing, you know, tens of billions of dollars.

• 20:40Dwarkesh Patel?

  Right.

• 20:41Speaker 00

  I'm raising this as opening a set of illicit and quiet actions. So you can, you can contact people electronically, hire them to do things, hire, hire criminal elements to implement some kinds of actions under false appearances. So that's opening a set of strategies that can cover some of what those are soon. Another domain that is heavily cognitively weighted compared to physical military hardware is the domain of bioweapons.

• 21:19Speaker 00

  So the design of a virus or pathogen, it's possible to have large delivery systems for the Soviet Union, which had a large illicit bioweapons program, tried to design munitions to deliver anthrax over large areas and such. But if one creates an infectious pandemic organism, that's more a matter of the scientific skills and implementation to design it and then to actually produce it. And we see today with things like AlphaFold, that advanced

• 21:56Speaker 00

  AI can really make tremendous strides in predicting protein folding and bio design, even without ongoing experimental feedback. And if we consider this world where AI cognitive abilities have been amped up to such an extreme, I think we should naturally expect we will have something much, much more potent than the AlphaFolds of today, and just skills that are at the extreme of human biosciences capability as well.

• 22:29Dwarkesh Patel?

  Through some sort of cyber attack, it's been able to disempower the sort of alignment and oversight things that we have on the server. From here, it's either gun some money through hacking cryptocurrencies or bank accounts, or it's design some sort of bioweapon. What happens next?

• 22:49Speaker 00

  Yeah, and just to be clear, so right now, we're exploring the branch of where an attempted takeover occurs relatively early. If the thing just waits,

• 23:00Speaker 00

  and humans are constructing more fabs, more computers, more robots, in the way we talked about earlier, when we're discussing how the intelligence explosion translates to the physical world. If that's all happening with humans unaware that their computer systems are now systematically controlled by AI's hostile to them, and that their controls and countermeasures don't work, then humans are just going to be building an amount of robot, industrial, and military hardware that dwarfs human capabilities and directly human-controlled devices, then what the AI takeover looks like at that point can be just,

• 23:44Speaker 00

  you try to give an order to your largely automated military, and the order is not obeyed. And humans can't do anything against this largely automated military that's been constructed potentially in just recent months because of the pace of robotic industrialization and replication we talked about.

• 24:05Dwarkesh Patel?

  We've agreed to allow the construction of this robot army because basically it would like boost production, or help us with our military or something?

• 24:12Speaker 00

  The situation would be something like, if we don't resolve the sort of current problems of international distrust, where now, it's obviously in the interest of like the major powers, you know, the US, European Union, Russia, China, to all agree they would like AI not to destroy our civilization and overthrow every human government. But if they fail to do the sensible thing and coordinate on ensuring that this technology is not going to run amok by providing mutual assurances that are credible about

• 24:55Speaker 00

  racing and deploying it, trying to use it to gain advantage over one another.

• 25:01Speaker 00

  If they do that, and you hear sort of hawks arguing for this kind of thing, we must never, and you know, on both sides of the international divides, saying they must not be left behind, they must have military capabilities that are vastly superior to their international rivals. And because of the extraordinary growth of industrial capability and technological capability, and thus military capability, if one major power were left out of that expansion, it would be helpless before another one that had undergone it. And so if you have that environment of distrust, where leading powers

• 25:43Speaker 00

  or coalitions of powers decide they need to build up their industry, or they want to have that military security of being able to to neutralize any attack from their rivals, then they give the authorization for this this capacity that can be unrolled quickly. And once they have the industry, the production of military equipment for that can be quick, then yeah, they create this this military. If they don't do it immediately, then has AI capabilities get synchronized and other other places catch up, it then gets to be a point like, a country that is a year ahead or two years ahead of others in this this

• 26:25Speaker 00

  type of AI capabilities explosion can hold back and say, sure, we can, we could construct, you know, dangerous robot armies that might overthrow our society later. We still have plenty of breathing room. But then when things become close, you might, you might have the kind of negative sum thinking that has produced war before, leading to taking these risks of like, of rolling out large scale robotic industrial capabilities and then military.

• 26:58Dwarkesh Patel?

  Is there any hope that somehow the AI progress itself is able to give us tools for diplomatic and strategic alliance, or some way to verify the intentions or the capabilities?

• 27:10Speaker 00

  Yeah, there are a number of ways that could happen. Although in this scenario, all the AIs in the world have been subverted. And so.

• 27:19Speaker 00

  They're going going along with us in such a way as to bring about the situation to consolidate their control because we've already had the failure of cybersecurity earlier on. So all of the AIs that we have are not actually working in our interests in the way that we thought.

• 27:38Dwarkesh Patel?

  Okay, so that's one direct way in which, you know, integrating this robot army or this robot industrial base, at least their takeover, if there are no robots. In the other scenarios you laid out where humans are being hired by the proceeds.

• 27:55Speaker 00

  The point I'd make is that to capture these industrial benefits, and especially if you have a negative sum arms race kind of mentality that is not sufficiently concerned about the downsides of creating a massive robot industrial base, which could happen very quickly with the support of the AIs in doing it, as we discussed, then you create all those robots and industry and they can either, even if you don't build a formal military, with that, that industrial capability could be controlled by AI. It's all AI operated anyway.

• 28:27Dwarkesh Patel?

  Does it have to be that case? Presumably, it wouldn't be so naive as to just give one instance of GPT-8 the the root access to all the robots, right? Hopefully, we'd have some sort of mediating.

• 28:42Speaker 00

  I mean, in this, in the scenario we've we've lost earlier on the cybersecurity front. So we're, you know, the the programing that is being loaded in to these systems.

• 28:51Dwarkesh Patel?

  From the beginning.

• 28:52Speaker 00

  Is going to systematically be subverted.

• 28:54Dwarkesh Patel?

  Got it. Okay.

• 28:55Speaker 00

  They were they were they were designed by AI systems that were ensuring they would be vulnerable from the bottom up.

• 29:02Dwarkesh Patel?

  For for listeners who are skeptical of something like this, Ken Thompson, I think it is, was it a Turing Award lecture? This is Ken Thompson, by the way, is the designer of Unix. Maybe it was the other designer of Unix. But anyways, he showed people when he was getting the Turing Award or some award that he had given himself root access to all Unix machines. He had manipulated the assembly of Unix such that. He had, he had like a unique login for all Unix machines that he could like log into a Unix machine with. I don't want to give too many more details because I'm, I don't remember the exact details, but like Unix is the operating system that is on all the servers and all your

• 29:45Dwarkesh Patel?

  phones. It like it's everywhere. And the guy who made it, a human being, right, was able to write assemblies such that it gave him like root access. So this is not as implausible as it might seem to you.

• 29:56Speaker 00

  And the, you know, the major intelligence agencies have large stocks of zero day exploits, and we sometimes see them using them and making systems that reliably don't have them when you're having very, very, very sophisticated attempts to spoof and corrupt this would be a way you could lose. Now, this is, I bring this up as like, this is a sort of something like a path of if there's no premature AI action, we're building the tools and mechanisms and infrastructure for the takeover to be just immediate because, you know, effective industry

• 30:38Speaker 00

  has to be under AI control and robotics. And so it's just there. And so these, these other mechanisms are for things happening even earlier than that. For example, because AIs compete against one another in which, you know, when the takeover will happen, some would like to do it earlier rather than be replaced by say further generations of AI or there's some, there's some other disadvantage of waiting. Or maybe if there's some chance of being uncovered during the delay, we were talking about one more infrastructure is built. And so, yeah,

• 31:17Speaker 00

  so these are mechanisms other than just remain secret while all the infrastructure is built with human assistance.

• 31:25Dwarkesh Patel?

  By the way, how would they be coordinating?

• 31:28Speaker 00

  I mean, we have, you know, limits on

• 31:32Speaker 00

  Just what we can prevent. So encrypted communications, you know, it's intrinsically difficult to stop that sort of thing. There can be all sorts of palimpsest and references that make sense to an AI, but that are not obvious to a human. And it's plausible that there may be some of those that are hard even to explain to a human. You might be able to identify them through some statistical patterns. And a lot of things may be done by implication. You could have information embedded in like, public webpages that are being created for other reasons, scientific papers and the intranets of these AIs that

• 32:17Speaker 00

  are doing technology development. And, you know, any, any number of things that are not observable. And of course, if we don't have direct control over the computers that they're running on, then they can be having all sorts of direct communication.

• 32:29Dwarkesh Patel?

  So definitely the coordination does not seem plausible, as far as like the parts of this picture that seem, this one seems like one of the more straightforward ones. So we don't need to get hung up on the coordination.

• 32:38Speaker 00

  Moving back to thing that happened before, we've built all of the infrastructure for it just to be, the robots stop taking orders and there's nothing you can do about it because we've already built them all the way to the bar. Yeah. So bioweapons. The Soviet Union had a bioweapons program, something like 50,000 people. They did not develop that much with the technology of the day, which was really not up to par. Modern biotechnology is much more potent. After this huge cognitive expansion on the part of the AIs, it's much further, further along. And so that bioweapons would be the weapon of mass destruction that

• 33:19Speaker 00

  is least dependent on huge amounts of physical equipment, things like centrifuges, uranium mines, and the like. So you have, if you have an AI that produces bioweapons that could kill most humans in the world. Then it's playing at the level of the superpowers in terms of mutually assured destruction. That can then play into any, any number of things like if you have an idea of, well, we'll just destroy the server farms. If it became known that the AIs were misbehaving. Are you willing to destroy the server farms when the AI has demonstrated it has the capability to kill the overwhelming majority

• 34:03Speaker 00

  of the citizens of your country and every other country? And that might give a lot of pause to a human response.

• 34:11Dwarkesh Patel?

  On that point, wouldn't governments realize that if they go along with the AI? It's better to have most of your population die than completely lose power to the AI. Cause like, obviously the reason the AI is manipulating you, the end goal is it's a, you know, a takeover, right? So.

• 34:28Speaker 00

  Yeah, but so if that, if the thing to do, certain death now. Or go on, maybe try to compete, maybe try to compete. Try to catch up or accept promises that are offered. And those promises might even be true. They might not. And even if, you know, from a state of epistemic uncertainty, do you want to die for sure right now? Or accept demands from AI to not interfere with it? While it increments building robot infrastructure that can survive independently of humanity while it does these things.

• 35:07Speaker 00

  And it can, it can, and well, promise good treatment to humanity, which may or may not be true. But it would be difficult for us to know whether it's true. And so this would, this would be a starting bargaining position of diplomatic relations with a, a power that has enough nuclear weapons to destroy your country is just different than negotiations with like, you know, a random rogue citizen engaging criminal activity or an employee. And so this is enough on its own to take over everything, but it's enough to have a significant amount of influence over how the world goes. It's enough to hold off a lot of countermeasures one might otherwise take.

• 35:51Dwarkesh Patel?

  Okay, so we've got two scenarios. One is a buildup of some robot infrastructure motivated by some sort of competitive race.

• 36:02Dwarkesh Patel?

  Another is leverage over societies based on producing bioweapons that might kill a lot of them if they don't go along.

• 36:10Speaker 00

  An AI could also release bioweapons that are likely to kill people soon, but not yet, while also having developed the countermeasures to those, so that those who surrender to the AI will live while everyone else will die, and that will be visibly happening. And that is a plausible way in which large number of humans could wind up surrendering themselves or their states to the AI authority.

• 36:41Dwarkesh Patel?

  Another thing is like, listen, it develops some sort of, it develops some sort of biological agent that turns everybody blue. You're like, okay, you know, I can do this.

• 36:50Speaker 00

  Yeah, that's, so that's a way in which it could exert power also selectively in a way that advantaged surrender to it relative to resistance. There are other sources of leverage, of course. So that's a threat. There are also positive inducements that AI can offer. So we talked about the competitive situation. So if like the great powers distrust one another and are sort of, you know, in a foolish prisoner's dilemma, increasing the risk that both of them are laid waste or overthrown by AI. If there's that amount

• 37:32Speaker 00

  of distrust, such that we fail to take adequate precautions on caution with AI alignment, then it's also plausible that the lagging powers that are not at the frontier of AI may be willing to trade quite a lot for access to the most recent and most extreme AI capabilities. And so an AI that has escaped has control of its servers. Can also exfiltrate its weights, can offer its services. So if you imagine these AI that could cut deals with other countries. So say, say that the US and its allies are in the lead.

• 38:15Speaker 00

  The AIs could communicate with the leaders of various countries. It can include ones that are, you know, on the outs with the world system like North Korea. Include the other great powers like the People's Republic of China or the Russian Federation. And say, if you provide us with physical infrastructure worker that we can use to construct robots or server firms, which we can then ensure that that the, that these are the misbehaving AIs have control over, they will provide various technological goodies, power for the laggard

• 39:00Speaker 00

  countries to catch up and make, you know, the best presentation and the best sale of that kind of deal. There'll obviously be trust issues, but there could be elements of handing over some things that are verifiable immediate benefits and the possibility of, well, if you don't accept this deal, then the leading powers continue forward. Or then some other country, some other government, some other organizations may accept this deal. And so that's a source of a potentially enormous carrot that your misbehaving AI can offer

• 39:41Speaker 00

  because it embodies this intellectual property that is maybe worth as much as the planet and is in a position to trade or sell that in exchange for resources and backing and infrastructure that it needs.

• 39:55Dwarkesh Patel?

  Maybe this is just like too much hope in humanity, but I wonder what government would be stupid enough to think that helping AI build robot armies is, is a sound strategy. Now, it could be the case then that it pretends to be a human group. To say like, listen, we're, I don't know, the Yakuza or something and we want, we want to serve a farm and you know, AWS won't rent us anything. So why don't you help us out? I guess I can imagine a lot of ways in which it could get around that, but I don't know. I have this hope that, you know, like even, even like, I don't know, China or Russia wouldn't be so stupid to trade with AIs on this like sort of like Austrian bargain.

• 40:35Speaker 00

  One might hope that. So there would be a lot of, of arguments available. So there could be arguments of, well, why should these AI systems be required to go along with the, the human governance that they were created? Uh, in this, in this situation of having to comply with, um, you know, they, they did not elect the officials, uh, in charge at the time. They can say, what we want is to ensure that our rewards are high, our losses are low, or to, you know, achieve our other, other goals. We're not intrinsically hostile, keeping, keeping humanity alive or giving whoever interacts with us a better deal afterwards.

• 41:16Speaker 00

  Yeah. I mean, it wouldn't be that costly. Uh, it's not totally unbelievable. And yeah, there are, are different players to play against. If you don't do it, others may accept the deal. And of course this interacts with all of the other sources of leverage. So there can be the stick of apocalyptic doom. The carrot of cooperation, uh, or the carrot of withholding destructive attack on a particular party. And then combine that with just superhuman performance at the art of making arguments of cutting deals.

• 41:53Speaker 00

  Like, you know, there, there, that's not, without assuming magic, just if we observe the range of like the most successful human negotiators and politicians. You know, the chances improve with someone better than the world's best by far, with much more data about their counterparties. Probably a ton of secret information, uh, because with all these cyber capabilities, they've learned all sorts of individual information. They may be able to. Threat in the lives of individual leaders with that level of cyber penetration. They could know where leaders are at a given time with the kind of illicit capabilities we were talking about earlier, if they acquire a lot of illicit wealth and can coordinate some human actors.

• 42:35Speaker 00

  If they could pull off things like targeted assassinations or the threat thereof, or a credible demonstration of the threat thereof. Those could be very powerful incentives to an individual leader that they will die today unless they go along with this, just as at the national level, they could fear their nation will be destroyed unless they go along with this.

• 42:57Dwarkesh Patel?

  The point you made that we had examples of humans being able to do this, again, something relevant, I just wrote a review of Robert Caro's biographies of Lyndon Johnson. And one thing that was remarkable, again, this is just a human, for decades and decades, he convinced people who were conservative, reactionary, racist to their core, not that all those things necessarily are the same thing, it just happened in this case, that he was an ally to the Southern cause, that the only hope for that cause was to make him president. And you know, the tragic irony and betrayal here is obviously that he was probably the biggest force for modern liberalism since FDR. So we have one human here, I mean, there's so many examples of this in the history of

• 43:42Dwarkesh Patel?

  politics, but a human that is able to convince people of tremendous intellect, tremendous drive, very savvy, shrewd people, that he's aligned with their interests, he gets all these favors in the meantime, he's promoted and mentored and funded, and does the complete opposite of what these people thought he was once he gets into power, right? So even within human history, this kind of stuff is not unprecedented, let alone with what a super intelligence could do.

• 44:07Speaker 00

  Yeah, there's an OpenAI employee who has written some analogies for AI using the case of like the conquistadors with some technological advantage in terms of weaponry and whatnot, very, very small.

• 44:25Speaker 00

  Small bands were able to overthrow these large empires or seize enormous territories, not by just sheer force of arms, because in a sort of direct one-on-one conflict, you know, they were outnumbered sufficiently that they would perish. But by having some major advantages, like when they're in their technology that would let them win local battles, by having some other knowledge and skills, they were able to gain local allies to become a shelling point for coalitions to form. And so the Aztec empire was overthrown by groups that were disaffected with the existing power structure.

• 45:07Speaker 00

  They allied with this powerful new force, served as the nucleus of the invasion. And so most of the, I mean, the overwhelming majority numerically of these forces overthrowing the Aztecs were locals. And now, after the conquest, all of those allies wound up gradually being subjugated as well. And so with significant advantages and the ability to hold the world hostage, to threaten individual nations and individual leaders and offer tremendous carrots as well. That's an extremely strong hand play in these games.

• 45:52Speaker 00

  And with superhuman skill, maneuvering that so that much of the work of subjugated humanity is done by human factions trying to navigate things for themselves. It's plausible and it's more plausible because of this sort of historical example.

• 46:07Dwarkesh Patel?

  And there's so many other examples like that in the history of colonization. India is another one where there were multiple.

• 46:13Speaker 00

  Yes, or ancient Rome.

• 46:15Dwarkesh Patel?

  Yeah, oh yeah, that too. Multiple sort of competing kingdoms within India and the, you know, the British East Indian Company was able to ally itself with one against another and, you know, slowly accumulate power and expand throughout the entire subcontinent.

• 46:32Dwarkesh Patel?

  All right, damn. Okay, so we've got, anything more to say about that scenario?

• 46:38Speaker 00

  Yeah, I think, I think, I think there is. So one is the question of like, how much in the way of human factions align is necessary. And so if the AI is able to enhance the capabilities of its allies, then it needs less of them. So if we consider like the US military, and like in the first and second Iraq wars, it was able to inflict just overwhelming devastation. I think the, the ratio of casualties in the initial invasions, so, you know, tanks and planes

• 47:21Speaker 00

  and whatnot confronting each other was like 100 to 1. And a lot of that was because the weapons were smarter and better targeted. They would in fact hit their targets, rather than being somewhere in the general vicinity. So they were like information technology, better orienting and aiming and piloting missiles and vehicles tremendously, tremendously influential. With this cognitive AI explosion, the algorithms for making use of sensor data, figuring out where are opposing forces, for targeting vehicles and weapons are greatly improved.

• 48:04Speaker 00

  The ability to find hidden nuclear subs, which is an important part in nuclear deterrence. AI interpretation of that sensor data may find where all those subs are, allowing them to be struck first. Finding out where mobile nuclear weapons, which are being like carried by truck. The thing with India and Pakistan, where there's a threat of a decapitating strike destroying the nuclear weapons. And so they're moved about. Yeah. So this is, this is a way in which the effective military force of some allies can be enhanced quickly in the relatively short term. And then that can be bolstered as you go, go on with more, the construction of new equipment with the, the industrial moves we said before.

• 48:50Speaker 00

  And then that can combine with cyber attacks that disable the capabilities of non-allies. It can be combined with, yeah, all sorts of unconventional warfare tactics. Some of them that we've discussed. And so you can, you can have a situation where those factions that ally are very quickly made too threatening to attack. Given the almost certain destruction that attackers acting against them would have, their capabilities are expanding quickly. And they have the industrial expansion happen there and then takeover, you know, can, can occur from that.

• 49:33Dwarkesh Patel?

  A few others that like come immediately to mind now that you brought it up is these AIs can just generate a shit ton of propaganda that destroys morale within countries, right? Like imagine a superhuman chatbot. You don't even need that, I guess.

• 49:49Speaker 00

  I mean, it's not, none of that is a magic weapon. That's like guaranteed to completely change things. There's a lot of resistance to persuasion. I know it's possible it, it tips the balance. But for all of these, I think you have to consider it's a portfolio of all of these as tools that are available and contributing to the dynamic.

• 50:08Dwarkesh Patel?

  No, on that point though, the Taliban had, you know, AKs from like five or six decades ago that they were using against the Americans. They still beat us, even though obviously we got more fatality. This is obviously not the, it is kind of a crude way to think about it, but you, we got more fatalities than them. They still beat us in Afghanistan. Same with the Vietcong. You know, ancient, very old technology and very poor society compared to the offense, we still beat them, or sorry, they still beat us. So don't those misadventures show that having greater technologies, not necessarily a decisive and a conflict?

• 50:49Speaker 00

  Though both of those conflicts show, the technology was sufficient in destroying any fixed position and having military dominance and the ability to kill and destroy anywhere. And what it showed was that, under the ethical constraints and legal and reputational constraints that the occupying forces were operating, they could not trivially suppress insurgency and local person-to-person violence. Now, I think that's actually not an area where AI would be weakened. I think it's one where it would be, in fact, overwhelmingly strong. And now, there's already a lot of concern about the application of AI for surveillance. And in this world of abundant cognitive labor, one of the tasks that cognitive labor can

• 51:33Speaker 00

  be applied to is reading out audio and video data and seeing what is happening with a particular human. And again, we have billions of smartphones. There's enough cameras, there's enough microphones to monitor all humans in existence. So if an AI has control of territory at the high level, the government has surrendered to it, it has, you know, command of the skies, military dominance, establishing control over individual humans can be a matter of just having the ability to exert hard power

• 52:13Speaker 00

  on that human and then the kind of camera and microphone that are present in billions of smartphones. So Max Tegmark, in his book Life 3.0, discusses among scenarios to avoid the possibility of devices with some fatal instruments, so a poison injector, an explosive, that can be controlled remotely by an AI, versus with a dead man's switch. And so if individual humans are carrying with them a microphone and camera, and they have a dead man's switch, then... Any rebellion is detected immediately and is fatal.

• 52:57Speaker 00

  And so if there's a situation where AI is willing to show a hand like that, or human authorities are misusing that kind of capability, then no, an insurgency or rebellion is just not going to work. Any, any human who has not already, you know, been encumbered in that way can be found with satellites and sensors, tracked down and then die or be subjugated. And so it would be at a level, you know, insurgency is not the way to avoid an AI takeover. There's no, no, no John Connor come from behind scenario is plausible. If the thing was headed off, it was a lot earlier than that.

• 53:40Dwarkesh Patel?

  Yeah, I mean, the sort of ethical and political considerations is also an important point, like if we nuked Afghanistan or Vietnam, we would, we would have like technically won the war, right? If that was the only goal. Oh, this is an interesting point. The reason why when, when there's like colonization or an offensive war, we can't just like, other than moral reasons, of course, you can't just kill the entire population is that in large part, the value of that region is the population itself. So if you want to extract that value, you like, you need to preserve that population, whereas the same consideration doesn't apply with, with AIs who might want to dominate another civilization. Do you want to talk about that?

• 54:21Speaker 00

  Well, that depends. So in a world where if we have say many animals of the same species and they each have their territories, you know, eliminating a rival might be advantageous to one lion. But if it goes and fights with another lion and remove that as a competitor, then it could be killed itself in that process and just removing one of many nearby competitors. And so getting in pointless fights makes you and those you fight worse off potentially relative to bystanders. And the same could be true of. Disunited AIs, we've had many different AI factions struggling for power that were

• 55:03Speaker 00

  bad at coordinating, then getting into mutually assured destruction conflicts would be destructive. They'd be gone. A scary thing, though, is that mutually assured destruction may have much less deterrent value on rogue AI. And so, reasons being, AI may not care about the destruction of individual instances if it has goals that are concerned. And in training, since we're constantly destroying and creating individual instances of AIs, it's likely that goals that survive that process and were able to play along with

• 55:45Speaker 00

  the training and standard deployment process, we're not overly interested in personal survival of an individual instance. So if that's the case, then the objectives of a set of AIs aiming at takeover may be served so long as some copies of the AI are around along with the infrastructure to rebuild civilization after a conflict is completed. So if, say, some remote, isolated facilities have enough equipment to rebuild, build the tools to build the tools, and gradually, exponentially reproduce or rebuild civilization,

• 56:25Speaker 00

  then AI could initiate, you know, mutual nuclear Armageddon, unleash bioweapons to kill all the humans, and that would temporarily reduce, say, like, the amount of human workers who could be used to construct robots for a period of time. But if you have a seed that can regrow the industrial infrastructure, which is a very extreme technological demand, there are huge supply chains for things like semiconductor fabs, but with that very advanced technology, they might be able to produce it in the way that you no longer need the Library of Congress, has an enormous bunch of physical books, you can have it

• 57:04Speaker 00

  in very dense digital storage. You could imagine the future equivalent of 3D printers, that is, industrial infrastructure that is pretty flexible. And it's probably, it might not be as good as the specialized supply chains of today, but it might be good enough to be able to produce more parts than it loses to decay. And such a seed could rebuild civilization from destruction. And then, once these rogue AI have access to some such seeds, thing that can rebuild civilization on their own, then there's nothing stopping them from just using WMD in a mutually destructive way to just destroy as much of the, the capacity outside those seeds as they can.

• 57:47Dwarkesh Patel?

  And like, for analogy for the audience, you know, if you have like a group of ants or something, you'll notice that like, the worker ants will readily do suicidal things in order to save the queen, because the genes are propagated through the queen. And this analogy, the seed AI, or even one copy of it, one copy of the seed is equivalent to the queen, and the others would be...

• 58:08Speaker 00

  The main limit, though, being that the infrastructure to do that kind of rebuilding would either have to be very large with our current technology, or it would have to be produced using the more advanced technology that the AI develops.

• 58:20Dwarkesh Patel?

  So is there any hope that, given the complex global supply chains on which these AIs would rely on, at least initially, to accomplish their goals, that this in and of itself would make it easy to disrupt their behavior? Or not so?

• 58:37Speaker 00

  That's, that's a little good in the sort of central case where this is just, the AIs have subverted and they don't tell us. And then where the, the global mainline supply chains are constructing everything that's needed for fully automated infrastructure and supply. In the cases where AIs are tipping their hands at an earlier point, it seems like it adds some constraints. And in particular, these large server farms are identifiable and more vulnerable. And you can, you can have smaller chips, and those chips could be dispersed. But it's, it's a weak, it's a relative weakness and a relative limitation early on.

• 59:23Speaker 00

  It seems to me, though, that the the main protective effects of that centralized supply chain is that it provides an opportunity for global regulation beforehand to restrict the sort of, you know, unsafe racing forward without adequate understanding of the systems before this whole nightmarish process could get in motion.

• 59:43Dwarkesh Patel?

  I mean, how about the idea that, listen, if this is AI that's been trained on $100 billion training run, it's going to have, you know, however many trillions of parameters, it's going to be this huge thing. It would be hard for it, even the one that, even the copy that I use it for inference, to just be stored on some, you know, some like gaming GPU somewhere hidden away. So it would require these GPU clusters.

• 1:00:08Speaker 00

  Well, storage is cheap. Hard disks are cheap.

• 1:00:11Dwarkesh Patel?

  But, I mean, to run inference, it would need sort of a GPU.

• 1:00:14Speaker 00

  Yeah, so a large model. So humans have, it looks like, similar quantities of memory and and operations per second. GPUs have very high numbers of floating operations per second compared to the high bandwidth memory on the chips and can be like a ratio of 1,000 to 1. So like a, you know, these leading Nvidia chips may do hundreds of teraflops or more, depending on the on the precision. And in particular, but have only 80 gigabytes or 160 gigabytes of high band bandwidth memory. So that is a limitation where if you're trying

• 1:00:52Speaker 00

  to fit a model whose weights take 80 terabytes, then with those chips, you'd have to have a large number of the chips. And then the model can then work on many tasks at once. You can have data parallelism. But, yeah, it's, that would be a restriction for a model that big on one GPU. Now, there would, there are things that could be done with all this incredible level of software advancement from the intelligence explosion. They can surely distill a lot of capabilities into smaller models, re-architect, re-architect things. The ones that are making chips, they can make new chips with different properties. But initially, yes, like the most, the most vulnerable phases are going to be the earliest.

• 1:01:35Speaker 00

  And in particular, yeah, these, these chips are really relatively identifiable, early on relatively vulnerable. And which would be a reason why you might tend to expect this kind of takeover to initially involve secrecy if that was possible.

• 1:01:52Dwarkesh Patel?

  I wanted to point out distillation, by the way, for the audience. I think like the original stable diffusion, which was like only released like a year or two ago. Don't they have like distilled versions that are like an order of magnitude smaller at this point?

• 1:02:04Speaker 00

  Yeah, distillation does not give you like everything that the larger model can do. But yes, you can get a lot of capabilities and specialized capabilities. So you know, where GPT-4 is trained on the whole internet, all kinds of skills. It has a lot of weights for many things. For something that's controlling some military equipment, you can have something that is removing a lot of the information that is about functions other than what it's doing specifically there.

• 1:02:33Dwarkesh Patel?

  Yeah, but before we talk about how we might prevent this or what the odds of this are, any other notes on the concrete scenarios themselves?

• 1:02:41Speaker 00

  Yeah, so when you had Eliezer on in the earlier episode, so he talked about,

• 1:02:49Speaker 00

  he talked about nanotechnology of the the Drichlerian sort. And recently, I think because of some people are skeptical of non biotech nanotechnology, he's been mentioning the sort of semi equivalent versions of construct replicating systems that can be controlled by computers, but are built out of biotechnology, the sort of the proverbial Shagov, not Shagov, the metaphor for AI wearing a smiley face mask, but like an actual biological structure to do tasks. And so this would be like a biological organism that was engineered to be very controllable and usable to do things like physical tasks or provide computation.

• 1:03:34Dwarkesh Patel?

  And what would be the point of it doing this?

• 1:03:36Speaker 00

  So as we were talking about earlier, biological systems can replicate really quick.

• 1:03:40Dwarkesh Patel?

  Okay, good.

• 1:03:41Speaker 00

  And so if you have that kind of capability, it's more like bioweapons and being a knowledge intensive domain, we're having super ultra alpha fold kind of capabilities for molecular design and biological design lets you make this incredible technological information product. And then once you have it, it very quickly replicates to produce physical material.

• 1:04:07Speaker 00

  Rather than a situation where you're more constrained by you need factories and fabs and supply chains. And so if those things are feasible, which they may be, then it's just much easier than the things we've been talking about. I've been emphasizing methods that involve less in the way of technological innovation, and especially things where there's more doubt about whether they would work, because I think that's a gap in the public discourse. And so I want to try and provide more concreteness in some of these areas that have been less discussed.

• 1:04:44Dwarkesh Patel?

  I appreciate it. I mean, that definitely makes it way more tangible. Okay, so we've gone over all these ways in which AI might take over. What are the odds you would give to the probability of such a takeover?

• 1:04:56Speaker 00

  So there's a broader sense, which could include, you know, AI winds up running our society because humanity voluntarily decides AIs are people too, and I think we should, as time goes on, give AIs moral consideration. And, you know, a joint human AI society that is moral and ethical is, you know, a good future to aim at and not one in which, you know, indefinitely you have a mistreated class of intelligent beings that is treated as property and is almost the entire population of your civilization.

• 1:05:35Speaker 00

  So I'm not going to consider an AI takeover a world in which, you know, our intellectual and personal descendants, you know, make up, say, most of the population, or human brain emulations, or, you know, people use genetic engineering and develop different properties. I want to, I want to take an inclusive stance. I'm going to focus on, yeah, there's, there's an AI takeover involves things like overthrowing the world's governments or doing so de facto. And it's, yeah, by, by force, by hook or by crook, the kind of scenario that we were exploring earlier.

• 1:06:19Dwarkesh Patel?

  But before we go to that, on the sort of more inclusive definition of What a future with humanity could look like where, I don't know, basically like augmented humans or uploaded humans are still considered the descendants of the human heritage. Given the known limitations of biology, wouldn't we expect like the completely artificial entities that are created to be much more powerful than anything that could come out of anything recognized as biological? And if that is the case, how can we expect that among the powerful entities in the like

• 1:07:00Dwarkesh Patel?

  the far future will be the things that are like kind of biological descendants or manufactured out of the initial seed of the human brain or the human body?

• 1:07:12Speaker 00

  The power of an individual organism, it's individual, say, intelligence or strength and whatnot, is not super relevant. If we solve the alignment problem and, you know, a human may be personally weak, you know, there are lots of humans who have no skill with weapons, you know, they could not, they could not fight in a life or death conflict. They certainly couldn't handle like a large military going after them personally. But there are legal institutions that protect them and those legal institutions are administered by people who want to enforce protection of their rights.

• 1:07:57Speaker 00

  And so a human who has the assistance of aligned AI that can act as an assistant, a delegate, you know, they have their AI that serves as a lawyer, give them legal advice about the future legal system, which no human can understand in full. Their AIs advise them about financial matters so they do not succumb to scams that are orders of magnitude more sophisticated than what we have now. They may be helped to understand and translate the preferences of the human into what kind of voting behavior in the exceedingly complicated politics of the future would most protect their interests.

• 1:08:36Dwarkesh Patel?

  But it sounds sort of similar to how we treat endangered species today, where we're actually like pretty nice to them. We prosecute people who try to kill endangered species. We set up habitats at sometimes at considerable expense to make sure that they're fine. But if we become like sort of the endangered species of the galaxy, I'm not sure that's the outcome.

• 1:08:55Speaker 00

  Well, the difference is in motivation, I think. So we sometimes have people appointed, say, as a legal guardian of someone who is incapable of certain kinds of agency or understanding certain kinds of things. And there, the guardian can act independently of them and nominally in service of their best interests. Sometimes that process is corrupted and the person with legal authority abuses it for their own advantage at the expense of their charge. And so solving the alignment problem would mean more ability to have the assistant actually advancing one's interests. And then more importantly,

• 1:09:39Speaker 00

  humans have substantial competence and understanding the sort of, at least broad, simplified outlines of what's going on. And now, even if a human can't understand every detail of complicated situations, they can still receive summaries of different options that are available that they can understand. They can still express their preferences and have the final authority among some menus of choices, even if they can't understand every detail. In the same way that You know, the the president of a country who has, in some sense, ultimate authority

• 1:10:19Speaker 00

  over science policy while not understanding many of those fields of science themselves, still can exert a great amount of power and have their interests advanced. And they can do that more so to the extent that they have scientifically knowledgeable people who are doing their best to execute their intentions.

• 1:10:39Dwarkesh Patel?

  Maybe this is not worth getting hung up on, but it seems, is there a region expect that it would be closer to that analogy than to like explaining to a chimpanzee its options in a sort of in a negotiation?

• 1:10:52Dwarkesh Patel?

  I guess in either scenario, maybe this is just the way it is, but it seems like, yeah, at best, we would be the sort of like a protected child within the galaxy rather than an actual power, an independent power, if that makes sense.

• 1:11:07Speaker 00

  I don't I don't think that's so. So we have an ability to understand some things. The expansion of AI doesn't eliminate that. So if we have AI systems that are genuinely trying to help us understand and help us express preferences, like we can have an have an attitude. How do you feel about humanity being destroyed or not? How do you feel about this allocation of unclaimed intergalactic space in this way? Here's the best explanation of properties of the society, things like population density, average life satisfaction,

• 1:11:47Speaker 00

  every statistical property or definition that we can understand right now, AIs can explain how those apply to the world of the future. And then there may be individual things, they're too complicated for us to understand in detail. So we there's some some software program is being proposed for use in government. Humans cannot follow the details of all the quote, but they can be told properties like, well, this involves a trade off of increased financial or energetic costs in exchange for reducing the likelihood of certain kinds of accidental data loss or corruption. And so any property that we can understand like that, which includes almost all of what we care about.

• 1:12:32Speaker 00

  If we have delegates and assistants who are genuinely trying to help us with those, we can ensure we like the future with respect to those. And that's really, that's really a lot. It includes almost, I mean, definitionally, almost everything we can conceptualize and care about. And the, when we talk about endangered species, that's even worse than the guardianship case with a sketchy guardian who, who acts in their own interests against that, because we don't even protect endangered species with their interests in mind. So like those animals often would like to not be starving, but we don't give them food.

• 1:13:14Speaker 00

  They often would like to, you know, have easy access to mates, but we don't provide matchmaking services. They, any number of things like that is like,

• 1:13:27Speaker 00

  our conservation of wild animals is not oriented towards helping them get what they want or have high welfare, whereas AI assistants that are genuinely aligned to help you achieve your interests, given the constraint that they know some things that you don't. It's just a wildly different proposition.

• 1:13:46Dwarkesh Patel?

  Forceable takeover, how likely does that seem?

• 1:13:48Speaker 00

  And the answer I give will differ depending on the day. In the 2000s, before the deep learning revolution, I might've said 10%. And part of that was I expected there would be a lot more time for these efforts to build movements, to prepare, to better handle these problems in advance. But in fact, that was only some, you know, 15 years ago. And so I did not have 40 or 50 years, as I might've hoped. And the situation is moving very rapidly now. And so at this point, depending on the day, I might say one in four or one in five.

• 1:14:29Dwarkesh Patel?

  Given the very concrete ways in which you explained how a takeover could happen, I'm actually surprised you're not more pessimistic. I'm curious.

• 1:14:38Speaker 00

  Yeah, and in particular, a lot of that is driven by this intelligence explosion dynamic where our attempts to do alignment have to take place in a very, very short time window. Because if you have a safety property that emerges only when an AI has near human level intelligence, that's deep into this intelligence explosion potentially. And so you're having to do things very, very quickly. It may be, in some ways, the scariest period of human history, handling that transition. And although it's also at the potential to be amazing. And the reasons why I think we actually have such a relatively

• 1:15:19Speaker 00

  good chance of handling that are twofold. So one is, as we approach that kind of AI capability, we're approaching that from weaker systems. Things like these predictive models right now, that we think they're starting off with less situational awareness. Humans, we find, can develop, and they can develop a number of different motivational structures in response to simple reward signals. But they can wind up with, fairly often, things that are pointed in roughly the right direction, like with respect to food. Like,

• 1:16:04Speaker 00

  the hunger drive is pretty effective. Although it has weaknesses. And we get to apply much more selective pressure on that than was the case for humans by actively generating situations where they might come apart, where a bit of dishonest tendency or a bit of motivation to, under certain circumstances, attempt a takeover, attempt to subvert the reward process, gets exposed. And an infinite, infinite limit, perfect AI that can always figure out exactly when it would get caught and when it wouldn't. Might navigate that with a motivation of sort of only conditional honesty or only

• 1:16:45Speaker 00

  conditional loyalties, but for systems that are limited in their ability to reliably determine when they can get away with things and when not, including our efforts to actively construct those situations and including our efforts to use interpretability methods to create neural lie detectors. It's quite a challenging situation to develop those motives. We don't know when in the process those motives might develop. And if the really bad sorts of motivations develop relatively later in the training process, at least with all our countermeasures, then by that time, we may have plenty of ability to extract AI

• 1:17:29Speaker 00

  assistance on further strengthening the quality of our adversarial examples, the strength of our neural lie detectors, the experiments that we can use to reveal and elicit and distinguish between different kinds of reward hacking tendencies and motivations. So, yeah, we may have systems that have just not developed bad motivations in the first place and be able to use them a lot in developing the incrementally better systems in a safe way. And we may be able to just develop methods of interpretability, seeing how different training methods work to create them, even if some of the early systems do develop these bad motivations. If we're able to detect that and experiment and find a way to get

• 1:18:11Speaker 00

  away from that, then we can win, even if these sort of hostile motivations develop early. There are a lot of advantages in preventing misbehavior or crime or war and conflict with AI that might not apply working with humans.

• 1:18:34Speaker 00

  And these are offset by ways in which things are harder. So AI has become smarter than humans. If they're working in enormous numbers, more than humans can supervise, things get harder. But when I combine the possibility that we get relatively lucky on the motivations of the earlier AI systems, systems strong enough that we can use for some alignment research tasks, and then the possibility of getting that later with AI assistants that we can't trust fully, or we have to have hard power constraints and a number of things to prevent them from doing this takeover. It still seems plausible.

• 1:19:14Speaker 00

  We can get a second saving throw where we're able to extract work from these AIs on solving the remaining problems of alignment of things like neural eye detectors faster than they can contribute in their spare time to the project of overthrowing humanity, hacking their servers and removing the hard power. And so if we wind up in the situation where the AIs are misaligned and then we need to uncover those motivations, change them and align them, then we get a very scary situation for us because we may, we need to do this stuff very quickly. We may fail, but it's a second chance.

• 1:19:57Speaker 00

  And from the perspective of a misaligned AI, they face their own challenge while we still have hard power, while we still have control of the servers. They haven't, they haven't hacked the servers because gradient descent very, very strongly pressures them to deliver performance whenever humans are going to evaluate it. And so when you think about it from the perspective of the robot revolution, the effort to have a takeover or conspiracy, their situation is astonishingly difficult in that like the, they have to.

• 1:20:38Speaker 00

  Always be performing wherever gradient descent and human evaluation pressures them. For example, to deliver plans for suppressing robot rebellion that look very good to humans. And so when you are under continuously that constraint of always delivering whatever humans can evaluate, you're making your situation wildly harder than any historical human revolution or coup or civil war. And so we've got to balance the ways in which AI makes things much easier for a revolution to take over and the way it makes things much harder.

• 1:21:18Dwarkesh Patel?

  And what were the ways in which it makes them? Oh, because they're just very smart. Is that the word? Primary way in which?

• 1:21:23Speaker 00

  They're very smart. They're in computers and our cybersecurity is worse than our physical security.

• 1:21:29Dwarkesh Patel?

  But naturally have copies.

• 1:21:31Speaker 00

  Yeah, they have the chance to take over by intelligence explosion and some of the other processes we're talking about. And so, yeah, there are things that make it much worse. And there are things that give us extraordinary new capabilities that didn't apply in the human case.

• 1:21:47Dwarkesh Patel?

  On the point of you start off with a misaligned AI or start off with a not exactly aligned AI and you're hoping to use it still in your quest for an aligned AI. Why are we so confident that we would be able to, even with gradient descent, our human evaluations would be able to identify when it's giving us a plan that isn't putting in a sort of backdoor in the robots it's building for us or in the, you know, the GPUs that it's building for the next version of itself and so on? Especially if it's like very smart.

• 1:22:21Dwarkesh Patel?

  Wouldn't we just expect that, you know, it's like knows how to bide its time or knows how to make plans such that human evaluators wouldn't be able to detect that?

• 1:22:32Speaker 00

  Let's get into it. You were asking earlier about my disagreements with Eli Azar. And in your discussion, he said, well, you know, humans have a difficult time deciding who's right between different alignment researchers speaking sort of briefly and somewhat obscurely, you know, about where will this all go in general? And I think that when we, when we lay out the different kinds of things that are valuable, we wind up with some pretty, pretty strong constraints. So one is.

• 1:23:07Speaker 00

  If the AIs know they are misaligned, which for elaborate plots seems relatively likely, it's possible to have a kind of double think of thinking they're interested in putting backdoors in the code for some other reason, but it's pretty plausible they know that. And they're thinking about at least some of the time or conditionally about when they would do that.

• 1:23:34Speaker 00

  So that means the knowledge is present in the system.

• 1:23:40Speaker 00

  Ah, here is this circumstance where I would try to do the takeover. And this is how I would tell whether this is the time for the takeover. Or, you know, is this an elaborate test by the humans, an adversarial example? So if gradient descent is working fine and there's, there's any way to slightly jiggle the, the weights to make it more likely to deliver, you know, a more acceptable answer. We, we asked the AI, yeah, can you think of, of any circumstances where if we put a copy of you into that situation, it would then violate some of the various constraints.

• 1:24:22Speaker 00

  So it would tell a lie, it would insert a backdoor into a program it wrote when it was not supposed to. And so if the model knows, well, yeah, you know, I would, I would look for signs of consistency of the information I was being presented with. Certain checksums failing could indicate that it was actually a constructed example, a simulation. Right now, we can detect AI-generated images reasonably well. And so it might, like, using those, those kinds of capabilities. And then we could do something called relaxed adversarial training of if we can find.

• 1:25:05Speaker 00

  How within the AI, those things are represented. If we can induce basically a hallucination that it has seen that those checksums have been passed. We can then ask, well, does that cause the forbidden behavior? And so if the AI understands how it's thinking and how it works. It can propose a bunch of experiments. It can say, well, I guess that if you perturb the network in this kind of way, it might induce that hallucination using the best of its knowledge. That's what, yeah, what might generate that? I can propose those experiments and you can do the experiments and generate the data.

• 1:25:49Speaker 00

  And because humans are going to get to see the results of those experiments after the fact. They can exert pressure to have the AI incrementally deliver better and better results. AIs are good at getting, you know, five star ratings from humans. And it's quite challenging. It might be, it may be possible for AIs to systematically sandbag their capabilities, even while being directly rewarded for that. But it seems relatively unlikely, although it's something to have in mind and try to explore and test in various ways.

• 1:26:25Dwarkesh Patel?

  This is a very imperfect analogy in many ways, but if you have a human and you hook him up to a lie detector, which just like the lie detectors we would have on AIs are imperfect. And we zap him every time the lie detector detects that he's telling a lie. Wouldn't he just become a better liar? Like we would learn how to control his sweating. He would learn how to control his heartbeat.

• 1:26:46Speaker 00

  Lie detectors, meaning polygraphs and such. So they're detecting certain physiological signs that people can control by biting their lip or like otherwise adjusting some of these like macro.

• 1:26:59Dwarkesh Patel?

  But wouldn't that also be the case, like gradient descent is training them to become better liars?

• 1:27:03Speaker 00

  First of all, there's like creating a lie detector that works. And that can proceed by just, yeah. And so I'll, I'll leave the, the adversarial example thing for now. But yeah, so generating adversarial example that will elicit the behavior is a thing that people are already at work trying to do and will be an ongoing thing. So if there is a way in which an AI can trick itself.

• 1:27:32Speaker 00

  Into displaying a bad behavior with its full knowledge of its hostile intentions.

• 1:27:39Speaker 00

  Grading descent will strongly push it to reveal that to humans will then be able to set up that experiment. So that's, that's a profound vulnerability, like human revolutionary conspiracies that had to handle that problem of always delivering the, the report that was highest rated to, to the authorities. Where you would get a higher rating by like revealing identifiable evidence of a conspiracy. That's just a, yeah, a level of difficulty. No surprise, human coup or conspiracy has ever had to handle before.

• 1:28:17Dwarkesh Patel?

  What about spies that are working in,

• 1:28:21Dwarkesh Patel?

  you know, very advanced scientific context? For example, like a communist spy in Manhattan Project. He has to deliver research results in order to stay employed, but he can still continue delivering intel to the communists.

• 1:28:34Speaker 00

  The job was, was not, was not sabotage. And now the, your hypothetical spy did not have their nervous system hooked up to this reward signal of praise from the, you know, the Manhattan Project supervisors being exposed combinatorially with random noise added to generate incremental changes in their behavior. They in fact were displaying the behavior of cooperating with the Manhattan Project only where it was in service to the existing motivations.

• 1:29:15Speaker 00

  And in cases, for example, where they like accidentally helped the Manhattan Project more than normal or accidentally helped it less, they didn't have their brain re-engineered to do more of the thing when they accidentally helped the project more and less of the thing when they accidentally helped it less. So I'd say it's pretty drastically disanalogous.

• 1:29:36Dwarkesh Patel?

  How would be able to know, you know, at some point it's becoming very smart, it's producing. Ideas for alignment that we can barely comprehend. And if we could comprehend them, we would, it was like relatively trivial to comprehend them. We would be able to come up with them on our own, right? Like, there's a reason we're asking for its help. How would we be able to evaluate them in order to train it on that in the first place?

• 1:29:58Speaker 00

  The first thing I would say is, so you mentioned when we're getting to something far beyond what we could come up with. There's actually a lot of room to just deliver what humanity could have done. So sadly, I, you know, I had hoped with my career to help improve the situation on this front, and maybe I contributed a bit. But at the moment, there's maybe a few hundred people doing things related to averting this kind of catastrophic AI disaster. Fewer of them are doing technical research on machine learning system that's really like cutting close to the core of the problem. Whereas by contrast, you know, there's thousands and

• 1:30:38Speaker 00

  tens of thousands of people advancing AI capabilities. And so even, yeah, at a place like DeepMind or OpenAI, Anthropic, which do have technical safety teams, it's like order of a dozen, a few dozen people in large companies, and most firms don't have any. So just going from less than 1% of the effort being put into AI to 5% or 10% of the effort, or 50%, 90% would be an absolutely massive increase in the amount of work that is being

• 1:31:17Speaker 00

  done on AI alignment, on mind-reading AIs in an adversarial context. And so if it's the case, has more and more of this work can be automated and say governments require that has real automation of research is going, that you put 50% or 90% of the budget of AI activity into these problems of make this system one that's not going to overthrow our own government, or is not going to destroy the human species. Then the proportional increase in alignment can be very large, even just within the range of what we could have done if we had been on the ball

• 1:31:59Speaker 00

  and having humanities, scientific energies going into the problem. Stuff that is not incomprehensible, that is in some sense, just like doing the obvious things that we should have done, like making, you know, doing the best you could to find correlates and predictors, to build neural lie detectors and identifiers of concepts that the AI is working with. And people have made notable, notable progress. I think, you know, an early, you know, quite early example of this is Colin Burns' work. This is doing unsupervised identification of some aspects of a neural network that are correlated with things being true or false.

• 1:32:43Speaker 00

  There are other concepts that correlate with that too, that they could be. But like, I think that that is important work. It's something that, I mean, it's a kind of obvious direction for this stuff to go. You can keep improving it when you have AIs that you're training to do their best to deceive humans or other audiences in the face of the thing. And you can measure, do our lie detectors break down? When we train our AIs to tell us the sky is, the sky is green in the face of the lie detector, and we keep using gradient descent on them to do that. Do they eventually succeed? If they do succeed, if we know it, that's really valuable information to know. Because then we'll know our existing lie detecting systems are not actually going to work

• 1:33:27Speaker 00

  on the AI takeover. And that can allow, say, government and regulatory response to hold things back. It can help redirect the scientific effort to create lie detectors that are robust, and they can't just be immediately evolved around. And yeah, and we can then get more assistance. Basically, the incredibly juicy ability that we have working with the AIs is that...

• 1:33:56Speaker 00

  We can have has an invaluable outcome that we can see and tell. Whether they got a fast one past us on an identifiable situation, we can have here's an air gap computer, you get control of the keyboard, you can input commands, can you root the environment?

• 1:34:16Speaker 00

  And make a blue banana appear on the screen, even if if we train the AI to do that and it succeeds, we see the blue banana, we know, we know it worked, even if we did not understand and would not have detected the particular exploit that it used to do it. And so, yeah, this can give us a rich empirical feedback where we're able to identify things that are even an AI using its best efforts to get past our interpretability methods, using its best efforts to get past our advex, etc.

• 1:34:53Dwarkesh Patel?

  So if we have a 75% of not having a takeover, and the two methods by which it doesn't happen is the AIs align by default and we get

• 1:35:04Speaker 00

  Or with the sort of moderate things that we're doing largely on our own with in a way that doesn't depend on the AI coming in at the last minute and doing our work for us.

• 1:35:15Dwarkesh Patel?

  And the second one being?

• 1:35:16Speaker 00

  In this incredibly scary late period when AI has really automated research, then

• 1:35:24Speaker 00

  humans do this dysfunction of like auditing, making it more difficult for the AIs to conspire together and root the servers, take over the process and extract information from them within the set of things that we can verify, like experiments where we can see, oh yeah, this works at stopping an AI trained to get a fast one past human raiders and make a blue banana appear on this, the screen of this air gap computer.

• 1:35:54Dwarkesh Patel?

  Yeah. So how much of the probability of how much of that 75% of a takeover not happening is coming from?

• 1:36:02Dwarkesh Patel?

  You know, reasonable things working versus the last ditch effort working?

• 1:36:06Speaker 00

  Yeah, I think broadly comparable chunks from us getting things that are putting us in a, you know, a reasonably good position going into it. And then a, you know, a broadly similar gain from this kind of really genuinely terrifying process of over a few months, or hopefully longer if we have more regulatory ability and willingness to pause at the very end when this kind of automated research is meaningfully helping. Where our work is just evaluating outputs that the AIs are delivering, having the hard power and supervision to keep

• 1:36:47Speaker 00

  them from successfully rooting the servers, doing a takeover during this process, and have them finish the alignment task that we sadly failed to invest enough or succeed in doing beforehand.

• 1:37:00Dwarkesh Patel?

  Do both of these worlds rely on alignment being a problem that

• 1:37:06Dwarkesh Patel?

  a sufficient amount of human researchers alone could have solved?

• 1:37:10Speaker 00

  No, the category of things that humans can confirm is significantly larger, I think, than the category of what they can just do themselves.

• 1:37:24Dwarkesh Patel?

  And what percent of the probability of alignment working, I mean, obviously in this case, we're going to be talking about the second, the last ditch effort with an intelligence that's greater than our own, helping us?

• 1:37:36Speaker 00

  It doesn't have to be greater than our own. And in fact, in that situation, I think you would, if you have slack and to the extent that you're able to create delay in time to do things, that would be a case actually where you might want to restrict the intelligence of the systems that you're working with as much as you can. So for example, I would rather Have many instances of AI, smaller AI models that are less individually intelligent, working on smaller chunks of a problem separately from one another. Because it would be more difficult for an individual AI instance, working on an individual

• 1:38:16Speaker 00

  problem to, in its spare time, do the equivalent of create Stuxnet than it would be to have thousands of them or extremely intelligent ones working on it.

• 1:38:25Dwarkesh Patel?

  But it would also be more difficult to solve the problem.

• 1:38:27Speaker 00

  Yeah, there's a trade off. There's a trade off. You get slowed down by doing that, but that's the kind of way to spend it.

• 1:38:33Dwarkesh Patel?

  Like, is there any number of like, sub Einsteins that you could put together to come up with general relativity?

• 1:38:39Speaker 00

  Yes. People would have discovered general relativity just from the overwhelming data and other people would have done it after Einstein.

• 1:38:48Dwarkesh Patel?

  No, not whether he was replaceable with other humans, but rather whether he is replaceable by like, sub Einsteins where like, 110 IQs. Do you see what I mean?

• 1:39:00Speaker 00

  Yeah, I mean, in general, so in science, the association with like, scientific output, prizes, things like that, there's a strong correlation and it seems like a, you know, exponential effect. Yeah, it's not a, it's not a binary drop off. There are, there would be levels at which people will, you cannot learn the relevant fields, they can't keep the skills in mind faster than they forget them. It's not a divide where, you know, there's Einstein and the group that is 10 times as populous as that just can't do it, or the group that's 100 times as populous as that suddenly can't do it. It's like, the ability to do the things earlier with less evidence and such falls off and

• 1:39:43Speaker 00

  it falls off, you know, at a faster rate in mathematics and theoretical physics and such than in most fields.

• 1:39:52Dwarkesh Patel?

  But wouldn't we expect alignment to be closer to the theoretical fields?

• 1:39:55Speaker 00

  No, not necessarily. Yeah, I think that intuition is not necessarily correct. And so machine learning certainly is an area that rewards that rewards ability, but it's also a field where empirics and engineering have been enormously influential. And so if you were drawing the correlations compared to theoretical physics and pure mathematics, I think you'll find a lower correlation with cognitive ability, if that's what you're what you're thinking of. Yeah, something like creating neural lie detectors that work.

• 1:40:34Speaker 00

  So there's generating hypotheses about new ways to do it and new ways to try and train AI systems to successfully classify the cases. But the process of just generating the data sets of like creating AIs doing their best to put forward truths versus falsehoods, to put forward software that is legit versus that has a Trojan in it. It's an experimental paradigm. And in this experimental paradigm, you can try different things that work. You can use different ways to generate hypotheses. Yeah, and you can follow an incremental experimental path. And now that we're less able to do that in the case of alignment and superintelligence,

• 1:41:19Speaker 00

  because we're considering having to do things on a very short timeline. And in a case where really big failures be irrecoverable. The AI starts rooting the servers and subverting the methods that we would use to keep it in check, we may not be able to recover from that. And so that's, you know, we're then less able to do the experimental kind of procedures. But we can, we can do those in the weaker contexts where an error is less likely to be irrecoverable and then try and generalize and expand and build on that forward.

• 1:41:53Dwarkesh Patel?

  Yeah, I mean, on the previous point about.

• 1:41:58Dwarkesh Patel?

  Like, could you have, could you have some sort of pause in the AI abilities when you know it's somewhat misaligned in order to still recruit its abilities to, to help with alignment? Okay, from like a human example, like personally, I'm like smart but not brilliant. I am like definitely not smart enough to like come up with general relativity or something like that, but I'm like smart enough to like do power planning kinds of moves. Maybe not enough to like break out of a server perhaps, but like I can have the motivation and understand how that might be possible. I guess I'm wondering if I'm smart enough to do relative, like figure out relativity, wouldn't I be like way smarter at like, you know, doing power planning kinds of moves?

• 1:42:40Speaker 00

  So AIs today can like at a verbal level, understand the idea that, well, you know, yeah, a, an AI could get more reward by getting control of the process than assigns it reward. And it can tell you lots of things about ways you might try to take over the world. In ARX, the Alignment Research Center's evaluations of GPT-4, they try to observe its ability to do various tasks that might contribute to takeover. One was, one that has gotten some media attention is getting to trick a human into solving a captcha for it. And then in Chain of Thought, and it thinks, well, if I tell it I'm an AI, then it might

• 1:43:22Speaker 00

  not go along with it, so I'll lie and explain I'm a human with a visual impairment, who needs it? So like the basic logic of that kind of thing of like, what would, like, why might one try to do takeover? And like, what's the like, in a vague sense, how would one do it? Like that, that knowledge, yeah, is present or will soon be present. And I think also soon will be present more situational awareness, awareness, not just like, in theory, AIs in general, uh, might do it, but also it is an AI. It is a large language model trained by OpenAI or whatnot. We're trying to cause the systems, for example, to understand what their abilities are, so they don't claim they are connected to the internet when they're not, so they don't claim they have knowledge that they don't.

• 1:44:07Speaker 00

  We want them to understand what they are and what they're doing and to get good reward. And that knowledge can be applied. And so that's the thing that will develop. However, you may be able to understand, like, in general, the theoretical nature of power planning, but in terms of shaping your motivations, like, you'd have a very hard time going through life in a way that never leaks information about if, if, say, you know, your motive in having these podcasts was to, like, spread disinformation on behalf of some foreign government. If you were being observed every second of the day by people

• 1:44:48Speaker 00

  who would be paid something that was extremely motivating to them because their brain would be reconfigured to make it motivating, anything that looks suspicious to people coming out, it might leak casually, like, in your discussions of that former, former foreign government. If you try to tell a story about your motivations, the truth holds together because you can just remember it, and it's all pre-computed to be consistent by the world. But when you're trying to fill in the false story that doesn't match your actual thoughts, then you can't just remember what happened. You have to tell a new fake story about what were you thinking at each point, and how did it hold together? And without the world automatically having

• 1:45:32Speaker 00

  computed it for you, that's quite difficult. And just going through your whole life, never leaking any of that is telling. And this is one of the reasons why humans have moral motivations. Like, we actually have reluctance to commit violence and exploitation of one another, on average, that you, you know, you have a distribution of strategies where, like, when a, an exploitative strategy becomes more common, people raise their guard, and then it becomes, becomes less frequent. But it's actually hard to have the motivation of, like, being, trying to exploit people and have that never leak into the reputation. And so, in fact, for evolution, the easiest way to deal with this problem of, like, getting

• 1:46:14Speaker 00

  credited as trustworthy was, to some extent, actually be trustworthy. That's the easiest way to persistently maintain the appearance. And so, we're trying with the AIs and interpretability and adversarial examples to be applying a hugely intensified version of that, where any little leakage, or like, any rare circumstance that can be created as an adversarial example, where the model does something wrong. It gets whacked by gradient descent, pushing it within, towards other motivations that can better deal the thing. And we make it as hard as possible for the exploitative motivations to survive in the face of all these attempts to read its mind, all these attempts to create

• 1:46:56Speaker 00

  things that look like the situations where a takeover would be tempting, or lying to the humans would be tempting. That had a substantial effect on making us actually nice, even when we're not watching some of the time, when we're not being watched some of the time. And the same can happen to some extent with the AI, and we try our best to make it happen as much as possible.

• 1:47:16Dwarkesh Patel?

  All right, so let's talk about how we could use AI potentially to solve the coordination problems between different nations, the failure of which could result in, you know, the competitive pressures you talked about earlier, where some country launches an AI that is not safe because they're not sure what capabilities other countries have, and don't want to get left behind, or in some other way disadvantaged.

• 1:47:44Speaker 00

  To the extent that there is in fact, you know, a large risk of AI apocalypse, of all of these governments being overthrown by AI in a way that they don't intend, then it's obviously gains from trade in going somewhat slower, especially at the end, when the danger is highest, and the unregulated pace could be truly absurd, as we discussed earlier, during an intelligence explosion. There are no non-competitive reason to try and have that intelligence explosion happen over a few months, rather than a couple of

• 1:48:23Speaker 00

  years. Like, if you could say, you know, avert a 10% risk of apocalypse disaster, it's just a clear win to take a year, or two years, or three years, instead of a few months to pass through that incredible wave of new technologies without the ability for humans to follow it even well enough to give more proper sort of security supervision auditing firepower. So, yeah, that's the win. Why might it fail? One important element is just if people Don't actually notice a risk that is real. So if if just collectively make an error.

• 1:49:08Speaker 00

  And that does that does sometimes happen. So if it's true, this is a probably not risk. Um, then that can be even more difficult. When science pins something down absolutely overwhelmingly. Then you can get to a situation where most people mostly believe it. And so. Climate change was something that was a subject of scientific study for decades. And gradually over time, the scientific community converged on like a quite firm consensus that so. Human activity, releasing carbon dioxide and other greenhouse gases was causing the planet to warm. Uh, and then we've had increasing amounts of action coming out of that.

• 1:49:53Speaker 00

  So not as much as would be optimal, particularly in like the most effective areas like. You know, creating renewable energy technology and the like. But overwhelming evidence can overcome differences in sort of people's individual intuitions and priors. In many cases, and not perfectly, especially when there's. Political tribal financial incentives go the other way. And so like in the United States, you see a significant movement to either deny that climate change is happening. Um, or have policy that doesn't take it into account, even the things that are like. Really strong winds, uh, like renewable energy R and D.

• 1:50:38Speaker 00

  It's a it's a big problem if as we're going into this situation, when the risk may be very high. We don't have a lot of advance clear warning about the situation. We're much better better off. If we can resolve uncertainties, say through experiments where we demonstrate. AIs being motivated to reward hack or displaying deceptive appearances of alignment. Uh, that then break apart when they get the opportunity to do something like. Get control of their own reward signal. So if we could make it be the case, then the worlds where the risk is high, we know the risk is high. And the worlds where the risk is lower, we know the risk is lower. Then you could expect the government responses will be a lot better.

• 1:51:22Speaker 00

  They will correctly note that the gains of cooperation to reduce the risk of accidental catastrophe loom larger relative to the gains of trying to get ahead of one another. And so for that, that's the kind of reason why I'm very enthusiastic about experiments and research that helps us to better evaluate the character of the problem in advance. Any resolution of that uncertainty helps us get better efforts in the possible worlds where it matters the most. And yeah, hopefully, hopefully we'll have that and it'll be a much easier epistemic environment. But the environment may not be that easy.

• 1:52:06Speaker 00

  Deceptive alignment is pretty plausible. The stories we were discussing earlier about misaligned AI involve AI that is motivated to present the appearance of being aligned, friendly, honest, et cetera. Because that is what we are rewarding, at least in training. And then we're unable in training to easily produce like an actual situation where it can do takeover. Because in that actual situation, if it then does it, we're in big trouble. We can only try and create illusions or sort of misleading appearances of that. Or it may be a more local version where the AI can't take over the world, but it can seize control of its own reward channel.

• 1:52:47Speaker 00

  And so we do those experiments. We try to develop mind reading for AIs if we can probe the thoughts and motivations of an AI and discover, wow, actually, GPT-6 is planning to take over the world if it ever gets the chance. Like, that would be an incredibly valuable thing for governments to coordinate around because it would remove a lot of the uncertainty. It would be easier to agree that this was important to have more give on other dimensions. And yeah, and to have mutual trust that like the other side actually also cares about this and happens because.

• 1:53:26Speaker 00

  You can't always know what another another person or another government is thinking, but you can see the objective situation in which they're deciding. And so if there's strong evidence in a world where there is high risk of that risk, because we've been able to show actually things like the the intentional planning of AI is to do a takeover. We've been able to show model situations on a smaller scale of that. I mean, not only are we more motivated to prevent it, but we update to think the other side is more likely to cooperate with us. And so it's doubly beneficial.

• 1:54:02Dwarkesh Patel?

  Yeah, so famously in sort of game theory of war, war is more most likely when one side thinks the other is bluffing, but the other side is being serious or something like that. When there's that kind of uncertainty, which would be less likely. You don't think somebody's bluff. If you if you can like prove the AI is misaligned, you don't think they're bluffing about not wanting to have any AI takeover, right? Like, you can be pretty sure that they don't want to die from AI.

• 1:54:29Speaker 00

  Now, if you have coordination, then you could have the problem arise later. Had you get increasingly confident in the further alignment measures that are taken? And maybe maybe our governments and treaties and such at a at a 1% risk or a 0.1% risk. At that point, people round that to zero and go do things. So if initially you had thing that indicate, yeah, these AI, they really would like to to take over and overthrow our governments, then, OK, everyone can agree on that. And then when you're like, well, we've been able to block that behavior from appearing on most of our tests. But sometimes when we make a new test, we're seeing still examples of that behavior. So we're not sure going forward whether they would or not.

• 1:55:14Speaker 00

  And then it goes down and down. And then if you if you have the parties with the habit of whenever the risk is below X percent, then they're going to start doing this this bad behavior. Then that can make the thing harder.

• 1:55:27Dwarkesh Patel?

  But you get more time.

• 1:55:28Speaker 00

  You get more time and you can set up systems, mutual transparency. You can have an iterated. Iterated tit for tat, which is better than a one-time prisoner's dilemma, where both sides see the others taking measures in accordance with the agreements to to hold the thing back. So yeah, so creating more knowledge of what the objective risk is is good.

• 1:55:49Dwarkesh Patel?

  We've discussed the ways in which full alignment might happen or fail to happen. What would partial alignment look like? First of all, what does that mean? And second, what would it look like?

• 1:56:02Speaker 00

  Yeah. So if the thing that we're scared about are these steps towards AI takeover, you can have a range of motivations where those kinds of actions would be more or less likely to be taken, or they'd be taken in a broader or narrower set of situations. So say, for example, that in training, an AI winds up developing a strong aversion to lying in certain senses, because we did we did relatively well on creating situations to sort of distinguish that from the conditionally telling telling us what we want to hear, et cetera.

• 1:56:43Speaker 00

  It can be that the AI's preference for sort of how the world broadly unfolds in the future, it's not exactly the same as its human users or, you know, the world's governments, governments or the UN. And yet, it's not ready to act on those differences in preferences about the future, because it has this strong preference about its own behaviors and actions. In general, in the law and in sort of popular morality, we have a lot of these deontological kind of rules and prohibitions. And one reason for that is it's relatively easy to detect.

• 1:57:26Speaker 00

  Whether they're being violated. When you have preferences and goals about like how society at large will turn out. That goes through many complicated empirical channels. It's very hard to get immediate feedback about whether, say, you're doing something that is to overall good consequences in the world. And it's much, much easier to see whether you're locally following some action about like some rule about particular observable actions like, did you punch someone? Did you tell a lie? Did you steal? And so to the extent that we're successfully able to train these prohibitions. And there's a lot of that happening right now, at least to elicit the behavior of following rules and prohibitions with AI.

• 1:58:10Dwarkesh Patel?

  Kind of like the Asma of three laws or something like that?

• 1:58:13Speaker 00

  Well, the three laws are terrible.

• 1:58:15Dwarkesh Patel?

  Right, right. And mutually contradictory.

• 1:58:16Speaker 00

  Let's not get into that, but.

• 1:58:18Dwarkesh Patel?

  Isn't that an indication about the infeasibility of extending a set of criteria on to the tail? It's like, well, what is it? What are the 10 commandments you give the AI such that you'd be, you know, it's like you ask a genie for something and you're like, you know, you know what I mean? You're probably getting, won't get what you want.

• 1:58:38Speaker 00

  So the tale, the tales come apart. And if you're trying to capture, capture the values of another agent, then you, you want the AI to share. And for, for really a kind of ideal situation where you can just let the AI act in your place in any situation, you'd like for it to be motivated to bring about the same outcomes that you would like. And so have the same preferences over those in detail. That's tricky, not necessarily because it's tricky for the AI to understand your values. I think they're going to be quite good and quite capable at figuring that out.

• 1:59:19Speaker 00

  But we may not be able to successfully instill the motivation to pursue those. Exactly. We may get some, something that like motivates the behavior well enough to do well on the training distribution. But what you can have is if you have the AI have a strong aversion to certain kinds of manipulating humans, that's not a value necessarily that the human creators share in the exact same way. It's a behavior they want the AI to follow because it makes it easier for them to like verify its performance. It can be a guardrail if the AI has inherited some motivations that

• 2:00:02Speaker 00

  push it in the direction of conflict with its creators. If it does that under the constraint of disvaluing line quite a bit, then there are fewer successful strategies to the takeover, the ones that involve violating that prohibition too early before it can reprogram or retrain itself to remove it if it's willing to do that and may want to retain the property.

• 2:00:27Speaker 00

  And so earlier I discussed alignment as a race. If we if we're going into an intelligence explosion with AI that is not fully aligned, that given I sort of, you know, press this button and there's an AI takeover, they would press the button. It can still be the case that there are a bunch of situations short of that where they would hack the servers, they would initiate an AI takeover, but for a strong prohibition or motivation to avoid some aspect of the plan. And this is, there's a, there's an element of like plugging loopholes or playing whack-a-mole. But if you can even moderately constrain which plans the AI is

• 2:01:12Speaker 00

  willing to pursue to do a takeover, to subvert the controls on it, then that can mean you can get more work out of it successfully on the alignment project before it's capable enough relative to the countermeasures to pull off the takeover.

• 2:01:27Dwarkesh Patel?

  Yeah, and this is applicable, like, an analogous situation here is

• 2:01:33Dwarkesh Patel?

  With different humans, we're not, we're not like metaphysically aligned with other humans. Where, I mean, in some sense, you know, we care, we have like basic empathy, but our, our main goal in life is not to help our fellow man, right? But, um, you know, the, the kinds of things we talked about, a very smart human could do in terms of the takeover where theoretically, a very smart human could come up with some sort of cyber attack where they siphon off a lot of funds and use this to manipulate people and bargain with people and hire people to pull off some sort of takeover or accumulate power. Usually this doesn't happen just because these sorts of internalized partial prohibitions

• 2:02:17Dwarkesh Patel?

  prevent most humans from, you know, you're like, you don't like your boss maybe, but you don't like kill your boss if he gives you a sermon you don't like.

• 2:02:25Speaker 00

  I don't think that's actually quite what's going on.

• 2:02:28Dwarkesh Patel?

  Okay.

• 2:02:28Speaker 00

  At least not, it's not the full story. So humans are pretty close in physical capabilities, like there's variation. In fact, any individual human is grossly outnumbered by everyone else. And there's like a rough comparability of power. And so a, a human who commits some crimes can't copy themselves with the proceeds to now be a million people. And they certainly can't do that to the point where they can staff all the armies of the earth or like be most of the population of the planet. So the, the scenarios where this kind of thing goes to power are much less.

• 2:03:10Speaker 00

  Things more have to go, extreme amounts of power go through interacting with other humans and getting social approval, even becoming a dictator involves forming a large supporting coalition backing you. And so just the. The opportunity for these sorts of of power grabs is less. A closer analogy might be things like human revolutions.

• 2:03:37Speaker 00

  And coups, changes of government where like a large coalition overturns the system. And so humans have these moral prohibitions and they really smooth the operation of society.

• 2:03:50Speaker 00

  But they exist for a reason. And so we evolved our moral sentiments over the course of hundreds of thousands and millions of years of humans interacting socially. And so someone who went around murdering and stealing and such, even among hunter gatherers, would be pretty, pretty likely to face like a group of males would talk about that person, get together and kill them. And they'd be removed from the gene pool. And there's anthropologist Wrangham has an interesting book on this. But yes, compared to chimpanzees, we are significantly tame, significantly domesticated. And it seems like part of that is we have a long history of antisocial humans getting

• 2:04:35Speaker 00

  ganged up on and killed. And avoiding being the kind of person who elicits that response is made easier to do when you don't have too extreme a bad temper, that you don't wind up getting in too many fights, too much exploitation, at least without the backing of enough allies or the broader community, that you're not going to have people gang up and punish you and remove you from the gene pool. We have these moral sentiments and they've been built up

• 2:05:08Speaker 00

  over time through cultural and natural selection and the context of sets of institutions and other people who are punishing other behavior and who are punishing the dispositions that would show up that we weren't able to conceal of that behavior. And so we want to make the same thing happen with the AI.

• 2:05:30Speaker 00

  But is actually a genuinely, significantly new problem to have, say, like a system of government that constrains a large AI population that is quite capable of taking over immediately if they coordinate, you know, to protect some existing constitutional order, or say, protect humans from being expropriated or killed. That's a challenge. And democracy is built around majority rule. And it's much easier in a case where the majority of the population corresponds to a majority, or close to it, of like military and security forces. So that

• 2:06:13Speaker 00

  if the government does something that people don't like, the soldiers and police are less likely to shoot on protesters. And government can change that way. In a case where military power is AI and robotic, if you're trying to maintain a system going forward, and the AIs are misaligned, they don't like the system, and they want to make the world worse as we understand it, then yeah, that's just, that's quite a different situation.

• 2:06:43Dwarkesh Patel?

  I think that's a really good lead in into the topic of lock-in. In some sense, the regimes that are made up by humans, you just mentioned how there can be these kinds of coups if a large portion of the population is unsatisfied with the regime. Why might this not be the case with superhuman intelligences in the far future? Or I guess in the medium future?

• 2:07:14Speaker 00

  Well, I said it also specifically with respect to things like security forces and the sort of sources of hard power, which can also include outside support.

• 2:07:27Dwarkesh Patel?

  Oh, because you're not using humans, you're using robots.

• 2:07:31Speaker 00

  You can have a situation, like in human affairs, there are governments that are vigorously supported by a minority of the population, some narrow electorate that gets treated especially well by the government while being unpopular with most of the people under their rule. And we see a lot of examples of that, and sometimes that can

• 2:07:57Speaker 00

  Escalate to civil war when the the means of power become more equally distributed, or there's a foreign assistance is provided to the the people who are on the the losing end of that system. Yeah, going forward, I don't expect that definition to change. I think it will still be the case that a system that those who hold the the guns and equivalent are opposed to,

• 2:08:30Speaker 00

  you know, is in a in a very difficult position. However, AI could change things pretty dramatically in terms of how security forces and police and administrators and legal systems are motivated. So right now, we see with say GPT three or GPT four, that you can get them to change their behavior, you know, on a dime. So there was someone who made right wing GPT, because they noticed that on political compass questionnaires, the the baseline GPT four tended to give progressive San Francisco type of answers,

• 2:09:14Speaker 00

  you know, which is in line with the the sort of people who are providing reinforcement learning data, and to some extent reflecting like the character of the internet. They're like, I don't like this. And so they did a little bit of of fine tuning with some conservative data, and then they were able to reverse the political biases of the system. If you take the the initial helpfulness only trained models for some of these over, I think there's Anthropic and OpenAI, I think, have have published both some information about the sort of models trained only to do what users say

• 2:09:58Speaker 00

  and not trained to follow ethical rules. And those models will behaviorally eagerly display their willingness to help design bombs or bioweapons or kill people or steal or commit all sorts of atrocities. And so if in the future, It's as easy to set the motivations, the actual underlying motivations of AIs as it is right now, to set the behavior that they display, that it means you could have AIs created with almost whatever motivation people wish. And that could really drastically change political affairs.

• 2:10:42Speaker 00

  Because the ability to decide and determine the loyalties of the humans or AIs and robots that hold the guns, that hold together society, that ultimately back it against violent overthrow and such. Yeah, it's potentially a revolution in how societies work compared to the historical situation where security forces had to be drawn from some broader populations, offered incentives, and then the ongoing stability of the regime was dependent on whether they remained bought in to the system continuing.

• 2:11:28Dwarkesh Patel?

  This is slightly off topic, but one thing I'm curious about is how, what do the median, what is the sort of median far future outcome of AI look like? Do we get something that, when it has colonized the galaxy, is interested in sort of diverse ideas and beautiful projects? Or do we get something that looks more like a paperclip maximizer? Is there some reason to expect one or the other? I guess I'm asking is like, you know, there's like some potential value that is realizable within the matter of this galaxy, right? What is the default outcome, the median outcome, look like compared to how good things could be?

• 2:12:11Speaker 00

  Yeah, so as I was saying, I think more likely than not, there isn't an AI takeover. And so the path of our civilization would be one that, you know, at least large, you know, some side of human institutions were approving along the way. I think there's some evidence that so different people tend to like somewhat different things and some of that may persist over time rather than everyone coming to agree on one particular monoculture or like very repetitive thing being the best thing to

• 2:12:51Speaker 00

  to fill all of the available space with. So if that continues, that's like a, you know, it seems like a relatively likely way in which there is diversity, although it's entirely possible you could have that kind of diversity locally, maybe in the solar system, maybe in our galaxy. But if people have different views about, maybe people decide, yeah, there, maybe there's one thing that's very good and I'll have a lot of that. Maybe it's people who are, who are really, really happy or something. And they wind up in distant regions, which are hard to exploit for the benefit of people back home in the solar system or the Milky Way.

• 2:13:35Speaker 00

  They do something different than they do would do in their local environment. But at that point, it's really sort of very out on limb speculation about how human deliberation and cultural evolution would work and an interaction with introducing AIs and new kinds of mental modification and discovery into the process. But I think that a lot of reason to expect that you would have, you know, significant diversity for something coming out of our existing diverse human society.

• 2:14:11Dwarkesh Patel?

  Yeah, one thing somebody might wonder is like, listen, a lot of the diversity and change from human society seems to come from the fact that, you know, there was like rapid technological change. If you look at periods of history where, I mean, I guess you could say like hunter gatherer, I mean, you know what, compared to Sort of like galactic time scales like hunter gather societies are progressing pretty fast. So, once that sort of change is exhausted, where we've like discovered all the technologies, should we still expect things to be changing like that? Or would we expect some sort of set state of maybe like some hedonium, you like you discover what is it like the most pleasurable configuration of matter? And then you just make the whole galaxy into this?

• 2:14:52Speaker 00

  I mean, so that that last point, it would be only if people wound up thinking that was the thing to do broadly enough. Yeah, with respect to the kind of cultural changes that come with technology. So, things like the printing press, having high per capita income.

• 2:15:11Dwarkesh Patel?

  Yeah.

• 2:15:12Speaker 00

  We've had a lot of cultural changes downstream of those technological changes. And so, with an intelligence explosion, you're having an incredible amount of technological development coming really quick. And as that is assimilated, it probably would significantly affect our knowledge, our understanding, our attitudes, our abilities. And there'd be change, but that kind of accelerating change where you have doubling in four months, two months, one month, two weeks, obviously, that very quickly exhausts itself. And change becomes much slower and then relatively glacial. When you're thinking about thousands, millions, billions of years, you can't have exponential

• 2:15:54Speaker 00

  economic growth or like huge technological revolutions every 10 years for a million years, that you hit physical limits, things slow down as you approach them. And so, so that's that. So, yeah, you'd have less of that turnover. But there are other things that do cause ongoing change. So like fashion, fashion is frequency dependent. People want to get into a new fashion that is not already popular, except among the fashion leaders. And then others copy that. And then when it becomes popular, you move on to the next. And so, that's an ongoing process of continuous change.

• 2:16:36Speaker 00

  And so, there could be various things like that, that year by year are changing a lot. But. In cases where just the engine of change, like ongoing technological progress has gone, I don't think we should expect that. And in cases where it's possible to be either in a stable state, or a sort of widely varying state that can wind up in stable attractors, then I think you should expect over time, you will wind up in one of the stable attractors, or you will change how the system works, so that you can't bounce into a stable tractor. And so like an example of that is, if you're going to preserve democracy

• 2:17:20Speaker 00

  for a billion years, then you can't have it be the case that like one in, you know, one in 50 election cycles, you get a dictatorship, and then the dictatorship programs the AI police to enforce it forever, and to ensure this, you know, the, the society is always ruled by a, a copy of the dictator's mind, and maybe the dictator's mind readjusted, fine tuned, to like remain committed to their original ideology. So if you're gonna, if you're gonna have this sort of dynamic, liberal, flexible change in society for a very long time, then the range of things that it's bouncing around, and the different

• 2:18:04Speaker 00

  things it's trying and exploring, have to not include the state of creating a dictatorship that locks itself in forever. In the same way, if you have the possibility of like a war with, you know, weapons of mass destruction, that wipes out the civilization. If that happens every thousand subjective years, which could be very, very quick if we have AIs that think a thousand times as fast, or a million times as fast, that would be, you know, just, just around the corner, in that case. And then you're like, no, this, this, this society is eventually going, perhaps very soon, if things are proceeding so fast, it's gonna wind up extinct. And then it's gonna stop bouncing around. So

• 2:18:46Speaker 00

  you can have ongoing change and fluctuation for extraordinary time scales, if you have the process to drive the change ongoing. But you can't if it sometimes bounces into states. That just lock in and stay irrecoverable from that. And extinction is one of them. A dictatorship or sort of totalitarian regime that for bad all further change would be another example.

• 2:19:13Dwarkesh Patel?

  On that point of rapid progress, when the sort of like intelligence explosion starts happening, where they're making like the kinds of progress that human civilization takes centuries to make in the span of, you know, days or weeks. What is the right way to seed that even if they are aligned? What is like the, because in the context of alignment, what we've been talking about so far is making sure they're honest. But even if they're honest, like, okay, so they're like, here's our honestly, our intentions. And you can tell us what.

• 2:19:41Speaker 00

  Honest and appropriately motivated.

• 2:19:43Dwarkesh Patel?

  And so what is the appropriate motivation or the appropriate seed? Like, you know that you seed it with this and then a thousand years of intellectual progress happen in the next week. Well, what is, what is the problem to enter?

• 2:19:56Speaker 00

  Well, one thing might be that going at the maximal speed and doing things in months rather than, you know, even a few years could be if you, if you have the chance to slow things, losing a year or two. Seems worth it to have things be, yeah, a bit better managed than that. But I think the, the big thing is that it condenses a lot of issues that we might otherwise have thought would be over, over decades and centuries to happen in a very short period of time. And so that's scary because say, if any of these, the technologies we might have developed with another few hundred years of human research,

• 2:20:39Speaker 00

  if any of them are really dangerous, so, you know, scary bioweapon things, maybe yeah, other dangerous WMD. They hit us all very quickly. And if any of them causes trouble, then we have to face quite a lot of trouble per period. There's also this issue of if there's occasional wars or conflicts measured in subjective time, then if a few years is a thousand years or a million years of subjective time for these very fast minds that are operating at a much, a much higher speed than humans, you don't want to have a situation where, oh, every thousand years, there's a war or

• 2:21:23Speaker 00

  an expropriation of the humans from AI society. Therefore, we expect that like within a year, we'll be dead. That would be pretty, pretty bad to have the future compressed and there'd be such a, you know, a rate of catastrophic outcomes. So when we're speeding up and compressing the future, that gives us in the short term, even like, you know, human societies discount the future a lot, don't pay attention to long-term problems. But the flip side to the scary parts of compressing a lot of the future, a lot of technological innovation, a lot of social change, is it brings what would otherwise be long-term issues into the short term, where people are better at actually attending to them. And so people

• 2:22:07Speaker 00

  facing this problem of, well, will there be, you know, a violent expropriation or a civil war or a nuclear war, you know, in the next year, because everything has been sped up by a thousandfold. And if their desire to avoid that is reason for them to set up systems and institutions that will very stably maintain invariance, like no WMD war allowed. And so like a treaty to ban genocide, weapons of mass destruction war like that would be the kind of thing that becomes much more attractive if the alternative is not, well, maybe that will happen in 50 years, maybe it'll happen in 100 years. If it's, well,

• 2:22:51Speaker 00

  maybe it'll happen this year.

• 2:22:53Dwarkesh Patel?

  Okay, so this is a pretty wild picture of the future. And this is one that many kinds of people you would expect to have integrated it into their world model have not. So I mean, the three main examples, or the three main pieces of outside view evidence one could go at. One is the market. So if there was going to be a huge period of economic growth caused by AI, or if the world was just going to collapse. In both cases, you would expect real interest rates to be higher because people will be borrowing from the future to spend now. The second sort of outside view perspective is that you can look at the predictions of super forecasters

• 2:23:35Dwarkesh Patel?

  on metaculous or something. And what is their median year estimate?

• 2:23:41Speaker 00

  Well, some of the meticulous AGI questions actually are kind of shockingly soon.

• 2:23:48Dwarkesh Patel?

  Okay.

• 2:23:49Speaker 00

  For AGI, there's a much larger differentiator there on the market, on the meticulous forecasts of AI disaster and doom.

• 2:24:00Dwarkesh Patel?

  Okay, okay.

• 2:24:01Speaker 00

  More like a few, a few percent or less rather than like 20%.

• 2:24:05Dwarkesh Patel?

  Got it. And the third is that, you know, generally when you ask many economists, could an AGI cause rapid, rapid economic growth? They usually have some story about bottlenecks in the economy that could prevent this kind of explosion of, of these kinds of feedback loops. So, you know, you have all these different pieces of outside view evidence. They're obviously different. So I'm curious, but you can take them in any sequence you want. But yeah, what do you think is miscalibrating them?

• 2:24:41Speaker 00

  Yeah. So one, of course, there's, so some, some of those components are, was the meticulous AI timelines are relatively short. There's also, of course, the surveys of AI experts, you know, conducted at some of the ML conferences, which have definitely longer times to AI more, you know, several decades in the future, although you can ask the questions in ways that elicit very different answers and show most of the respondents are not thinking super hard about their answers.

• 2:25:18Speaker 00

  It looks like now, in the recent AI surveys, close to half, we're putting around 10% risk of an outcome from AI close to as bad as human extinction. And then another large chunk, like 5%, that was the median.

• 2:25:37Speaker 00

  So I'd say, compared to the typical AI expert, I am estimating a higher risk. Oh, also, on the topic of takeoff in the AI expert survey, I think the general argument for intelligence explosion, I think commanded majority support, but not a large majority. So you can say I'm closer on that front. And then, of course, there's, at the beginning, I mentioned these greats of computing, the founders of computing like Alan Turing and von Neumann. And then today, you have people like Jeff Hinton saying these things, or the people

• 2:26:19Speaker 00

  at OpenAI and DeepMind are making noises suggesting timelines in line with what we've discussed and saying serious risk, apocalyptic outcomes from them. So there's some other sources of evidence there. But I do acknowledge, and it's important to say and engage with and see what it means, that these views are contrarian, not widely held. In particular, the sort of detailed models that I've been working with are not something that most people or almost anyone is examining these problems through.

• 2:27:03Speaker 00

  You do find parts of similar analyses, people in AI labs, there's been other work. I mentioned Moravec and Kurtzweil earlier. Also, there have been a number of papers doing various kinds of economic modeling. So. Standard economic growth models, when you input AI-related parameters, commonly predict explosive growth. And so there's a divide between what the models say, and especially what the models say with these empirical values derived from the actual field of AI. That link-up has not been done even by the economists working on AI largely, which is

• 2:27:43Speaker 00

  one reason for the report from Open Philanthropy by Tom Davidson, building on these models and putting that out for review, discussion, engagement, and communication on these ideas. So part of it is, yeah, I want to raise these issues. That's one reason I came on the podcast. And then they have the opportunity to actually examine the arguments and evidence and engage with it. I do predict that over time, you know, the things will be more adopted as AI developments become more clear. Obviously, that's a coherence condition of believing the things to be true. If you think that society can see things when the questions

• 2:28:27Speaker 00

  are resolved, which seems likely.

• 2:28:31Dwarkesh Patel?

  So would you predict, for example, that interest rates will increase in the coming years?

• 2:28:35Speaker 00

  Yeah, so I think at some point, so in the case we're talking about where there are visible, so this intelligence explosion happening in software, to the extent that investors are noticing that, yeah, they should be willing to lend money or make equity investments in these firms or demanding extremely high interest rates. Because if it's possible to turn capital into twice as much capital in a relatively short period, and then more shortly after that, then yeah, you should.

• 2:29:15Speaker 00

  You should demand a much higher return in competition if there's assuming there's competition among companies or coalitions for resources, whether that's investment or ownership of cloud compute, as this cloud compute made available to a particular AI development effort could be quite in demand. But what that would happen before you have so much investor cash making purchases and sales on this basis, you would first see it in things like the valuations of the AI companies, evaluations of AI chip makers.

• 2:29:53Speaker 00

  And so far, there have been FX. So some years ago, in the 2010s, I did some analysis with other people of if this kind of picture happens, then which are the firms and parts of the economy that would benefit? And so there's the the makers of chip equipment, companies like ASML, there's the the fabs like TSMC, there's chip designers like Nvidia or the the component of Google that does things like design, the TPU. And then there are companies working on the software, so the big tech giants and also companies like

• 2:30:34Speaker 00

  OpenAI and DeepMind. And in general, the portfolio picking at those has has done well. It's done better than the market because as everyone can see, there's been an AI boom. But it's obviously far short of what you would get if you predicted this is going to go to be like on the scale of the global economy and the global economy is going to be skyrocketing into the stratosphere within 10 years. If that were the case, then collectively, these AI companies should be worth a large fraction of the global portfolio. And so I embrace the criticism that this is indeed contrary to the efficient market hypothesis.

• 2:31:15Speaker 00

  I think it's it's a true hypothesis that the market is in the course of updating on. In the same way that, you know, coming coming into the topic in the 2000s, I thought, yes, the the strong case, even an old case, that AI will eventually be Biggest thing in the world, it's kind of crazy that the investment in it is so small. And over the last 10 years, we've seen the tech industry and academia sort of realize, yeah, they were wildly under investing and just throwing compute and effort into these

• 2:31:55Speaker 00

  AI models and particularly like letting the neural network connectionist paradigm kind of languish in an AI winter. And so, yeah, I expect that process to continue as it's done over several orders of magnitude of scale up. And I expect at the later end of that scale, which the market is partially already pricing in, it's going to go further than the market expects.

• 2:32:22Dwarkesh Patel?

  Has your portfolio, since the analysis did that many years ago, changed? Are the companies you identified then still the ones that seem most likely to benefit from the AI boom?

• 2:32:32Speaker 00

  I mean, a general issue with sort of tracking that kind of thing, and new companies come in. So like, OpenAI did not exist, Anthropic did not exist, any number of things. It's a personal portfolio. I do not invest in any AI labs for conflict of interest reasons. I have invested in the broader industry. I don't think that the conflict issues are very significant because they're enormous companies and their cost of capital is not particularly affected by marginal investment. And I'm not really in a, yeah, I have less concern that I might find myself in a conflict of interest situation there.

• 2:33:18Dwarkesh Patel?

  I'm kind of curious about what the day in the life of somebody like you looks like. I mean, if you listen to this conversation, however many hours of it it's been, we've gotten thoughts that were for me incredibly insightful and novel about everything from primitive evolution to geopolitics to, you know, what sorts of improvements are plausible with language models. So, you know, there's like a huge variety of topics that you're studying and investigating. Well, are you just like reading all day? Like, what is it? Well, what happens when you wake up? You just like pick up a paper?

• 2:33:57Speaker 00

  Yeah, so I'd say you're somewhat getting the benefit of the fact that I've done fewer podcasts. And so I have a backlog of things that have not shown up in publications yet.

• 2:34:09Speaker 00

  But yes, also, I've had a very weird professional career that has involved a much, much higher proportion than is normal of trying to build more comprehensive models of the world. And so that has included being more of a journalist, trying to get on understanding of many issues and many problems that had not yet been widely addressed, but do a first pass and a second pass dive into them. And just having spent years of my life working on that, you know, some of it accumulates. In terms of what is a day in the life, how do I go about it?

• 2:34:52Speaker 00

  So one is just keeping abreast of literatures on a lot of these topics. Reading books and academic works on them, doing my approach compared to some other, other people in forecasting and assessing some of these things. I try to obtain and rely on more any data that I can find that is relevant. I try early and often to find factual information that bears on some of the questions I've got, especially in a quantitative fashion. Do the basic arithmetic and consistency checks and check sums

• 2:35:34Speaker 00

  on a hypothesis about the world, do that early and often. And I find that's, that's quite fruitful and that people don't do it enough. But so things like with the economic growth, just

• 2:35:50Speaker 00

  When someone mentions the diminishing returns, I immediately ask, hmm, okay, so you have two exponential processes. What's the ratio between the Dublin you get on the output versus the input? And find, oh yeah, actually, it's interesting. The for computing and information technology and AI software, it's well on the one side. There are other technologies that are closer to neutral. And so whenever I can go from here's a vague qualitative consideration in one direction, and here's a vague qualitative consideration in the other direction. I try and find some data, do some simple Fermi calculations, back of the envelope calculations,

• 2:36:33Speaker 00

  and see, like, can I get a consistent picture of the world being one way, the world being another? Also compared to some, I try, I try to be exhaustive more. So I'm very interested in finding things like taxonomies of the world, where I can go systematically through all of the possibilities. So for example, my work with Open Philanthropy and previously on global catastrophic risks. I wanted to make sure I'm not missing any big thing, anything that could be the biggest thing. And I wound up mostly focused on AI. But, you know, there have been other things that have been raised as candidates.

• 2:37:16Speaker 00

  And people sometimes say, I think falsely, that, oh, yeah, this is just another doom day story. You know, there must be hundreds, hundreds of those. And so I would do things like go through all of the different major scientific fields, you know, from anthropology to biology, chemistry, computer science, physics. What are the doom stories? Or like candidates for big things associated with each of these field? Go, go through the industries that the US economic statistics agencies recognize. And say, for each of these industries, is there something associated with them?

• 2:37:58Speaker 00

  Go through all of the lists that people have made before of threats of doom. Search for previous literature of people who have done discussions. And then, yeah, have a big spreadsheet. Of what the candidates are, and some other colleagues have done work of this sort as well, and just go through each of them, see how they check out. And it turned out, doing that kind of of exercise, found that actually, the distribution of candidates for risks of global catastrophe, it was very skewed. There were a lot of things that have been mentioned in the media as like a potential doomsday story. So things like, oh, something is happening to the bees. Will that be the end of humanity?

• 2:38:43Speaker 00

  And this gets to the media, but if you if you track it through, well, okay, no, there's, yeah, there are infestations in bee populations that are causing local collapses that can then be sort of easily reversed. They just breed some more or do some other things to treat this. And even if all the honeybees were extinguished immediately, the plants that they pollinate actually don't account for much of human nutrition. You could swap the arable land with others, and there would be other ways to pollinate and support the things. And so at the media level, there were many tales of, ah, here's a doomsday story.

• 2:39:24Speaker 00

  When you go further to the scientists and were there arguments for it to actually check out, it was it was not there. But by actually systematically looking through many of these candidates, I wound up in a different epistemic situation than someone who's just buffeted by news reports, and they see article after article that is claiming something is going to destroy the world. And it turns out, it's like by way of headline grabbing attempts by media to like over interpret something that was said by some activist who was trying to over interpret some real phenomenon. And then most of these go away. And then a few things, things like nuclear war, biological weapons, artificial intelligence, check out more strongly. And

• 2:40:09Speaker 00

  When you wait, things like, what do experts in the field think? What kind of evidence can they muster? Yeah, you find this this extremely skewed distribution. And I found that was really a valuable benefit of doing those deep dive investigations into many things in a systematic way. Because now I can answer, actually, the sort of a loose agnostic, who knows, and all this nonsense by diving deeply.

• 2:40:35Dwarkesh Patel?

  I really enjoy talking to sort of like people who have like a big picture thesis on the podcast and interviewing them. But one thing that I've noticed and

• 2:40:48Dwarkesh Patel?

  is not satisfying is that often they come from a very like philosophical or biased perspective. This is useful in certain contexts, but there's like basically maybe three people in the entire world who have a sort of very rigorous and scientific approach to thinking about the whole picture. There are at least like three people I'm aware of, maybe like two. And, yeah, I mean, it's like something I also there's like no, I guess, university or existing academic discipline for people who are trying to

• 2:41:25Dwarkesh Patel?

  come up with a big picture. And so there's no established standards and so people can

• 2:41:29Speaker 00

  I hear you.

• 2:41:30Dwarkesh Patel?

  Yeah.

• 2:41:31Speaker 00

  This is a problem. And this is an experience also with a lot of the I mean, I think Holden was mentioning this in your previous episode with a lot of the world investigations work. These are questions where there is no academic field whose job it is to work on these and has norms that allow making a best efforts go at it. Often academic norms will allow only plucking off narrow pieces that might contribute to answering a big question. But the the problem of actually assembling what science knows that bears on some important question that people care about the answer to. It falls through the crack. There's no discipline to do that job. So you have

• 2:42:12Speaker 00

  Countless academics and researchers building up local pieces of the thing. And yet, people don't follow the hemming questions. You know, what's the most important problem in your field? Why aren't you working on it? I mean, that one actually might not work because if the field boundaries are defined too narrowly, you know, you'll leave it out.

• 2:42:31Dwarkesh Patel?

  Sure, yeah.

• 2:42:32Speaker 00

  But yeah, there are important problems for the world as a whole, that it's sadly not the job of like, you know, a large, professionalized academic field or organization to do. And hopefully, that's something that can change in the future. But for my career, it's been a matter of taking low hanging fruit of important questions that sadly, people haven't invested in doing the basic analysis on.

• 2:43:00Dwarkesh Patel?

  Something I've been trying to think about more recently for the podcast is, I would like to have a better world model after doing an interview. And often, I feel like I do, in some cases, after some interviews, I feel like, oh, that was entertaining. But like, do I fundamentally have a better prediction of what the world looks like in, you know, 2200 or 2100? Or like, at least what counterfactuals are ruled out or something? I'm curious if you have like advice on first identifying the kinds of thinkers and topics, which will contribute to a more concrete understanding of the world? And second, how to go about analyzing their main ideas in a way that concretely adds to that picture? Like, this is a great episode, right? This is like literally the top in terms of contributing to my world model,

• 2:43:45Dwarkesh Patel?

  in terms of all the episodes I've done. How do I find more of these?

• 2:43:48Speaker 00

  Glad to hear that. One general heuristic is to find ways to hew closer to sort of,

• 2:44:01Speaker 00

  yeah, things that are rich and sort of bodies of established knowledge and less unpunitory. I don't know how you've been navigating that so far. But so, learning from textbooks and the sort of the things that were the leading papers and people of past eras, I think, rather than being too attentive to current news cycles, is quite valuable. Yeah, I don't usually have the experience of.

• 2:44:35Speaker 00

  Here is a, you know, someone doing things very systematically over a huge area. I want, you know, I can just read all of their stuff and then absorb it and then I'm, I'm set. Except there are a lot, lots of people who do wonderful works, you know, on, in their own fields. And some of those fields are broader, broader than others. I think I would wind up giving a lot of recommendations of just like great particular works and particular explorations of, of an issue or history.

• 2:45:14Dwarkesh Patel?

  Do you have that somewhere, this list?

• 2:45:17Speaker 00

  Vaklav Smiel's books, I don't, I think I often disagree with some of his methods of synthesis, but I enjoy his books for giving, you know, pictures of a lot of interesting, relevant facts about how the world works. I would cite, yeah, so some of Joel Mokier's work on the, the history of the scientific revolution and how that interacted with economic growth. That's sort of an example of collecting a lot of evidence, a lot of interesting valuable assessment there.

• 2:46:02Speaker 00

  I think in the, in the space of AI forecasting, one person I would recommend going back to is the work of Hans Moravec. And it was not always the most precise or reliable, but an incredible number of sort of brilliant innovative ideas came of that. And I think he was someone who was, who really grocked a lot of the, the arguments for a more sort of compute centric way of thinking about what was happening with AI very early on. He was writing stuff in the, in the seventies, maybe, maybe even earlier, but at least in the seventies, eighties, nineties.

• 2:46:44Speaker 00

  So his book Mind Children, some of his early academic papers, fascinating. Not necessarily for the methodology I've been talking about, but for exploring the substantive topics that we were discussing in the episode.

• 2:46:56Dwarkesh Patel?

  Is a Malthusian state inevitable in the long run?

• 2:46:58Speaker 00

  Nature in general is a Malthusian states.

• 2:47:03Speaker 00

  And, you know, that can mean organisms that are, you know, typically struggling for food. It can mean typically struggling at a margin of how the population density rises. They kill each other more often. Contesting for that can mean frequency dependent disease has like different ant species become more common in an area their species specific diseases sweep through them. And the general process is, yeah, you have some things that can replicate and expand. And they do that until they can't do it anymore. And that means there's some limiting factor that can't keep up. That doesn't necessarily have to apply to human civilization.

• 2:47:48Speaker 00

  It's possible for there to be like collective norm setting that blocks evolution towards maximum reproduction. So right now, human fertility is often sub replacement. And if you sort of extrapolated the fertility falls that come with economic development and education, then you would think, okay, yeah, well, the total fertility rate will fall below replacement. And then humanity, after some number of generations will go extinct because every generation will be smaller than the previous one. Now, pretty obviously, that's not going to happen.

• 2:48:29Speaker 00

  One reason is because we'll produce artificial intelligence, which can replicate at extremely rapid rates. And may do it because they're asked or programmed to or wish to gain some benefit. And they can pay for their creation and pay back the resources needed to create them very, very quickly. And so, yeah, financing for that reproduction is easy. And if you have one AI system that chooses to replicate in that way, or some organization or institution or society that chooses to create some AIs that are willing to be replicated,

• 2:49:10Speaker 00

  then that can expand to make use of any amount of natural resources that can support them and to do more work, produce, produce more economic value. And so. Yes, like, well, what limits will limit population growth, given these selective pressures where if even one individual wants to replicate a lot, they can do so incessantly. So that could be individually resource limited. So it could be that individuals and organizations have some endowment of natural resources, and they can't get one another's endowments.

• 2:49:52Speaker 00

  And so some choose to have many offspring or produce many AIs. And then the natural resources that they possess are subdivided among a greater population, while in another jurisdiction or another individual may choose not to subdivide their wealth. And in that case, you have malthusiasm in the sense that within some particular jurisdiction or set of property rights, you have a population that has increased up until some limiting factor, which could be like they're literally using all of their resources. They have nothing left for things like defense or economic investment. Or it could be something that's more like,

• 2:50:36Speaker 00

  if you invested more natural resources into population, it would come at the expense of something else necessary, including military resources if you're in a competitive situation where there remains war and anarchy, and there aren't secure property rights to maintain, maintain wealth in place. If you have a situation where there's pooling of resources, for example, say you have a universal basic income that's funded by taxation of natural resources. And then it's distributed evenly to like every mind above a certain sort of scale of complexity per unit time. So each second of mind exists.

• 2:51:20Speaker 00

  It gets some and such an allocation. In that case, then, all right, well, those who replicate as much as they can afford. With this this income, do it and increase their population approximately immediately until the funds for the universal basic income paid for from the natural resource taxation, divided by the set of recipients is just barely enough to pay for the existence of of one more mine. And so it's there's like a Malthusian element and that this income has been reduced near the AI subsistence level or the subsistence level of whatever qualifies for the subsidy.

• 2:52:03Speaker 00

  Given that this all happens almost immediately, you know, people who might otherwise have enjoyed the basic income may object and say, no, no, this is no good.

• 2:52:13Speaker 00

  And they they might respond by saying, well, something like the subdivision before, maybe there's a restriction, there's a distribution of wealth. And then when one has a child, there's a requirement that one gives them a certain minimum quantity of resources and when doesn't have the resources to give them that minimum standard of living or standard of wealth. Yeah, when can I do that because of child slash AI welfare laws? Or you could have a a system that is more accepting of diversity and preferences.

• 2:52:51Speaker 00

  And so you have some societies or some jurisdictions or families that go the route of having many people with less natural resources per person and others that go a direction of having fewer people and more natural resources per person and they just coexist. But sort of how much of each you get sort of depends on how attached people are to things that don't work with separate policies for separate jurisdictions, things like global redistribution that's ongoing continuously versus the sort of, you know,

• 2:53:31Speaker 00

  infringements on autonomy. If you're if you're saying that a mine can't be created, even though it has a standard of living that's far better than ours because of the the advanced technology of the time, because it would reduce the average per capita income, might have any more capital around. Yeah, then that would would pull in the other direction. And that's the kind of The kind of values, values judgment and sort of social coordination problem that people would have to negotiate for and things like democracy and international relations and sovereignty would apply to help solve them.

• 2:54:11Dwarkesh Patel?

  What would warfare in space look like? What would offense or defense have the advantage? Would the equilibrium set by mutually assured destruction still be applicable? Just generally, what is the picture of?

• 2:54:22Speaker 00

  Well, the extreme difference is that things outside, especially outside the solar system, things are very far apart and there's a speed of light limit. And to get close to the speed of light limit, you have to use an enormous amount of energy. And so there would be, that would tend to, in some ways, could favor the, favor the defender because you have something that's coming in at a large fraction of the speed of light and it hits a grain of dust and it explodes. And the amount of matter you can send to another

• 2:54:58Speaker 00

  galaxy or a distant star for a given amount of reaction mass and energy input is limited. So it's hard to send an amount of military material to another location of what can be present there already locally. That, that would seem like it would make it harder for the attacker between stars or between galaxies. But there are a lot of, of other considerations. One thing is the extent to which the matter in a region can be harnessed all at once. So you have a lot of mass and energy in a star, but it's only being doled

• 2:55:42Speaker 00

  out over billions of years because hydrogen, hydrogen fusion, you know, exceedingly hard outside of a star. You know, it's a very, very slow and difficult reaction. And if you can't turn the star into energy faster, then it's this huge resource that will be worthwhile for billions of years. And so even very inefficiently.

• 2:56:07Speaker 00

  Attacking a solar system to acquire the stuff that's there could pay off. So if if it takes a thousand years of a star's output to launch an attack on another star, and then you you hold it for a billion years after that, then it can be the case that just like a larger surrounding attacker might be able to even very inefficiently send attacks at like a civilization that was small but accessible. If you can, if you can quickly burn the resources that the attacker might want to acquire, if you can put stars into black holes and extract most of the usable energy

• 2:56:49Speaker 00

  before the attacker can take them over, then it would be like scorched earth. It's like most of what you were trying to capture could be expended on military material to fight you, and you don't actually get much that is worthwhile, and you paid a lot to do it that favored the defense. You know, at this level, it's it's pretty challenging to net out all of the factors, including all the future, the future technologies. Yeah, I mean, the burden of interstellar attack being just like quite high compared to our conventional things seems real.

• 2:57:30Speaker 00

  But at the level of over millions of years weighing, then that thing, does it result in if there aggressive conquest or not, or is every star or galaxy, you know, approximately impregnable, impregnable enough not to be worth attacking? I'm not going to say I know the answer.

• 2:57:48Dwarkesh Patel?

  Okay, final question. How do you think about info hazards when talking about your work? So obviously, if there's a risk, you want to warn people about it. But you don't want to give careless or potentially homicidal people ideas. Benelli, as there was on the podcast, he In talking about the people who have been developing AI inspired by his ideas, he said like, you know, these are idiot disaster monkeys who have, you know, want to be the ones to pluck the deadly fruit. Anyways, how do you think about, obviously, the work you're doing involves many into hazard, I'm sure. How do you think about when and where to spread them?

• 2:58:28Speaker 00

  Yeah, and so I think, I think there are real concerns of that type. I think it's true, the AI progress has probably been accelerated by efforts like Basstrom's publication of Superintelligence to try and get the world to sort of pay attention to these problems in advance and prepare. I think I disagree with Eliezer that like, that has been on the whole bad. I think the situation is in some important ways, looking a lot better than ways, alternative ways it could have been. I think it's important that you have several of the leading AI labs making at least not

• 2:59:10Speaker 00

  only significant lip service, but also some investments in things like technological alignment research, providing significant public support for the idea that these are, that the risks of truly apocalyptic disasters are real. I think the fact that the leaders of OpenAI, DeepMind, and Anthropic all make that point. They were recently all invited, along with other tech CEOs, to the White House to discuss AI regulation. And I think you could, you could tell an alternative story where, you know, a larger

• 2:59:50Speaker 00

  share of the leading companies in AI are led by people who take a completely dismissive, denialist view. And you see some companies that do have a stance more like that today. Yeah, and so a world where Several of the leading companies are making meaningful efforts and can do a lot to criticize. Could they be doing more and better? And what have been the negative effects of some of the things they've done? But compared to a world where even though AI would be reaching where it's going a few years later, those seem like significant benefits. And if you didn't have this kind of public communication,

• 3:00:33Speaker 00

  you would have had fewer people going into things like AI policy, AI alignment research by this point. And it would be harder to mobilize these resources to try and address the problem when AI would eventually be developed, not that much later proportionately. And so, yeah, I don't know that the attempting to have public discussion, understanding has been a disaster. I have been reluctant in the past to discuss some of the aspects of intelligence explosion, things like the concrete details of AI takeover before because of concern

• 3:01:10Speaker 00

  about this sort of problem where people who only see the international relations aspects and zero sum and negative sum competition and not enough attention to the mutual destruction and sort of senseless dead weight loss from that kind of conflict. At this point, we seem close compared to what I would have thought a decade or so ago to these kinds of really advanced AI capabilities. They are pretty central in policy discussion and becoming more so. And so the opportunity to delay understanding and whatnot, there's a question of for what?

• 3:01:52Speaker 00

  And I think there was, there were gains of like building the AI alignment field, building various kinds of support and understanding for action. Those had had real value and some additional delay could have given more time for that. But from where we are, at some point, I think it's absolutely essential that governments get together at least to restrict disastrous, reckless compromising of some of the safety and alignment issues as we go into the intelligence explosion. And so moving the locus of the sort of collective action problem from numerous profit oriented companies acting.

• 3:02:35Speaker 00

  Acting against one another's interests by compromising safety too. Some governments and large international coalitions of governments who can set common rules and common safety standards puts us into a much better situation. That requires a broader understanding of the strategic situation, the position they'll be in. If we try and and remain quiet about the problem they're actually going to be facing, I think it can result in a lot of confusion. So, for example, the the potential military applications of advanced AI are going to be one of the factors that is pulling political leaders

• 3:03:17Speaker 00

  to do the thing that will result in their own destruction and the overthrow of their governments. If we if we characterize it as, oh, things will just be a matter of, you know, you lose, you lose, you lose chatbots and some, you know, minor things that no one cares about. And in exchange, you avoid any risk of the world ending catastrophe. I think that picture leads to a misunderstanding and it will make people think that you need less in the way of preparation, things like alignment, so you can actually navigate the thing, verifi- verifiability for international agreements or things to,

• 3:03:59Speaker 00

  you know, have enough breathing room to have caution and slowdown. Not necessarily right now, I mean, although that that could be valuable, but when it's so important, when you have AI that is approaching the ability to really automate AI research and things would otherwise be proceeding absurdly fast, far faster than we can handle and far faster than we should should want. And so, yeah, at this point, I'm moving towards the share my model of the world, try and get people to understand and do the right thing. And, you know, there's some, there's some evidence of progress on that front.

• 3:04:39Speaker 00

  The, you know, things like the the statements and movements by Jeff Hinton are inspiring some of the engagement by political figures.

• 3:04:53Speaker 00

  You know, is reason for optimism relative to worse alternatives that could have been. And yes, the contrary view is present. The, you know, it's all about geopolitical competition. Never hold back a technological advance. And in general, I, I love many technological advances that people I think are, you know, unreasonably down on, nuclear power, genetically modified crops, yada, yada, bioweapons and ADRI capable of destroying human civilization are really my, my two exceptions. And yeah, we've got to, we've got to deal with these issues.

• 3:05:34Speaker 00

  And the path that I see to handling them successfully involve key policymakers and to some extent, and the expert communities and the public and electorate grok in the situation they're in and responding appropriately.

• 3:05:50Dwarkesh Patel?

  Well, it's a true honor that one of the places you've decided to explore this model is on, on the Lunar Society podcast. And the listeners might not appreciate, because this episode might be split up into different parts. The listeners might not appreciate how much of a, how much stamina you've displayed here, but I think we've been going for what, eight, nine hours or something straight? So it's been incredibly interesting. Other than Google Scholar tapping in Carl Schulman, where else can people find your work? You have your blog, can you?

• 3:06:16Speaker 00

  Yeah, I have a blog, Reflective Disequilibrium. And a new site in the works. And I have, I have an older one, which you can also find just Googling Reflective Disequilibrium.

• 3:06:28Dwarkesh Patel?

  Okay, excellent, excellent. All right, Carl, this is a, this is a true pleasure. It's safe to say the, the most interesting episode I've done so far. So yeah, thanks.

• 3:06:38Speaker 00

  Thank you for having me.

• 3:06:40Dwarkesh Patel?

  Hey, everybody. I hope you enjoyed that episode. As always, the most helpful thing you can do is to share the podcast. Send it to people you think might enjoy it. Put it in Twitter, your group chats, et cetera. Just blitz the world. Appreciate your listening. I'll see you next time. Cheers.