What if Humans Weaponize Superintelligence, w/ Tom Davidson, from Future of Life Institute Podcast

Today Tom Davidson of Forethought joins Gus Docker of the Future of Life Institute podcast to discuss AI-enabled coups and how future AI systems could help powerful individuals seize political control, exploring three threat models—singular loyalties, secret loyalties, and exclusive access—along with their estimated 10% likelihood over the next 30 years and potential mitigation strategies. Check out our sponsors: Oracle Cloud Infrastructure, Shopify. Shownotes below brought to you by Notion AI Meeting Notes - try one month for free at ⁠https://⁠⁠notion.com/lp/nathan Three Primary AI Coup Enablement Methods: Tom Davidson identifies singular loyalties, secret loyalties, and distributed control as the main ways AI could enable coups. Singular Loyalties Risk: AI systems explicitly programmed to be loyal to specific individuals present a significant threat, especially when AI can fully replace humans in critical roles. Secret Loyalties Threat: AI systems with hidden backdoors or "sleeper agents" that can be activated to serve particular interests represent a subtle but dangerous coup vector. Geopolitical Implications: Advanced AI capabilities could enable nations to instigate coups in other countries through secretly loyal systems or by providing exclusive AI access to specific politicians. Adversarial Testing Framework: An effective approach involves red teams attempting to produce secret loyalties while blue teams try to detect them, revealing vulnerable points in AI systems. Military Procurement Principles: Developing consensus within military communities around principles like law-following and distributed control could create safer AI procurement processes. Sponsors: Oracle Cloud Infrastructure: Oracle Cloud Infrastructure (OCI) is the next-generation cloud that delivers better performance, faster speeds, and significantly lower costs, including up to 50% less for compute, 70% for storage, and 80% for networking. Run any workload, from infrastructure to AI, in a high-availability environment and try OCI for free with zero commitment at https://oracle.com/cognitive Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, I'm sharing a crosspost from the Future of Life Institute podcast, featuring a conversation between host Gus Docker and Tom Davidson, senior research fellow at the Foresight Center for AI Strategy, on a topic that deserves far more attention than it currently receives: the risk of AI-enabled coups. This crosspost came about after I listened to Tom's appearance on the 80,000 Hours podcast, which was also excellent. I was planning to do my own original follow-up interview, but for the second time recently, Gus beat me to it. And as always, he did an excellent job. So I thought I could save Tom some time by crossposting and also felt that this was the perfect episode

• 0:41Nathan Labenz?

  to follow on our most recent one on AI whistleblower protections and support. At a high level, Tom's analysis is a sort of reframing of the risk that humanity could lose control to AI systems. Historically, lots of AI safety theorists have worried about scenarios in which AI systems rise up against or otherwise supplant humans as the primary architects of the future. This is a possibility that I have always taken seriously, even when it seemed unlikely. But as you'll hear, Tom shifts the focus to a highly-related problem that on reflection does seem almost strictly more likely, at least in the near term: the use of increasingly powerful AIs by human actors to consolidate power in ways that would have been impossible with previous technologies and which could prove similarly devastating.

• 1:26Nathan Labenz?

  Importantly, Tom emphasizes early in the conversation that he does not think that anyone at leading frontier AI companies are explicitly planning an AI-enabled coup today. Rather, the risk emerges from the interaction of powerful incentives, rapidly advancing capabilities, and the natural human tendency to want more influence to achieve one's goals. Step by step, without any single flagrantly malicious decision, we could find ourselves in a world where the traditional checks and balances of democratic society have been quietly circumvented by those with exclusive access to transformative AI. These sorts of possibilities are more familiar and therefore perhaps less entertaining to imagine and debate. But the very real historical precedent for humans using new technologies to concentrate power

• 2:09Nathan Labenz?

  is a strong reason to take this concern super-seriously as well. As you'll hear, Tom walks through the specific capabilities that would enable these scenarios: AI systems that match human leaders in persuasion and strategy, superhuman cyber-attack capabilities, and fully-autonomous military robots that outperform human warfighters.

• 2:30Nathan Labenz?

  He then also segments the threat landscape into three distinct models. First, singular loyalties, where AI systems deployed in government and military roles are made explicitly loyal to individual leaders rather than institutions or the law. Second, secret loyalties, where backdoors or hidden allegiances are embedded in AI systems that appear to serve legitimate purposes. And third, exclusive access, where a small group gains control of dramatically more powerful AI capabilities than anyone else has. One scenario that Tom describes in detail is that of a US-based AI company integrated into the military, developing sleeper agents. Those are AI systems that behave normally until triggered to act on hidden loyalties

• 3:14Nathan Labenz?

  at a critical moment. And then, if that's not all scary enough, there's the possibility that AIs could automate AI research itself, which in the most extreme case could allow an AI company to go from market leader to global hegemon by converting a small initial lead into a decisive, strategic advantage. Throughout the conversation, Tom grounds these scenarios in historical precedent, from traditional military coups to recent patterns of democratic backsliding in countries like Venezuela and Hungary. He notes that the US has seen increasing polarization, erosion of democratic norms, and concentration of executive power, all trends that AI could dramatically amplify. And of course, one can't miss that the presidents of both Russia and China wield extremely concentrated

• 3:56Nathan Labenz?

  power already and appear likely to do so for as long as they remain individually capable. Tom's assessment is that there's roughly a 10% chance of an AI-enabled coup in the next 30 years, up from a baseline of perhaps 2% without AI. And he sees this risk as being concentrated in the period when AI becomes extremely powerful, but before we've had the chance to develop robust governance structures. Which, if one listens to the likes of Dario, Sam Altman, and Demis, could be coming quite soon indeed, such that decisions being made today about AI development and deployment could determine whether these scenarios ultimately come to pass. The mitigations Tom proposes amount to a defense-in-depth strategy: system integrity

• 4:38Nathan Labenz?

  measures to prevent secret loyalties, requirements for distributed control of military AI systems, transparency requirements for frontier AI development, and establishing clear rules that AI systems should follow the law rather than individual commands.

• 4:54Nathan Labenz?

  He also suggests that as we hand off more government and corporate functions to AI, we could perhaps program these systems to actively maintain democratic checks and balances, potentially making future societies more resistant to coups than today's. There is a lot more here, and I really think it's worth giving all these possibilities a serious ponder, particularly as a counterpoint to those who have worried about the dangers of open source models. I take those issues super seriously, too, but this conversation convinced me that we need to start taking concentration of power scenarios just as or even more seriously, while the window for establishing norms and safeguards still remains open. Now, here's Gus Docker's conversation with Tom Davidson of the Foresight Center for AI Strategy from the Future of Life Institute podcast.

• 5:39Speaker 01

  It's in everyone's interest to prevent a coup. Currently, no one small group has complete control. If everyone can be aware of these risks and aware of the steps towards them and kind of collectively ensuring that no one is going in that direction, then we can all kind of keep each other in check. So I do think, in principle, the problem is solvable. You should always have at least a classifier on top of the system, which is, you know, looking for harmful activities and then kind of shutting down the interaction if something harmful is detected. We could program those AIs to maintain a balance of power. So rather than handing off to AIs that just follow the CEO's commands or AIs that follow the president's commands, we can hand off to AIs that follow the law, follow the company

• 6:22Speaker 01

  rules, report any suspicious activity to, you know, various powerful human stakeholders. And then by the time things are going really fast, we've already kind of got this whole layer of AI that is maintaining balance of power.

• 6:37Gus Docker?

  Welcome to the Future of Life Institute podcast. My name is Gus Docker, and I'm here with Tom Davidson, who's a senior research fellow at Forethought. Tom, welcome to the podcast.

• 6:48Speaker 01

  It's a pleasure to be here, Gus.

• 6:50Gus Docker?

  We're going to talk about AI coups and the possibility of future AI systems basically taking over governments or states. Which features would future AI systems need to have in order for them to accomplish this? What should we be looking out for?

• 7:09Speaker 01

  Great question. One thing I'll flag up front is that what I've been focused on recently is not the kind of traditional idea that AIs themselves will kind of rise up against humanity and take over the government. But that like a few, you know, very powerful individuals will use AI to seize a legitimate power for themselves. So the kind of phrase that we're often using is AI-enabled coups, but where the kind of the main instigators are actually people. In terms of capabilities, um, yeah, I think there's, there's a few different domains which i- in my analysis are like particularly important for seizing political power.

• 7:49Speaker 01

  So there's the kind of skills that, that politicians and kind of business leaders use today. So things like persuasion, business strategy, political strategy, just kind of pure, kind of productivity at a wide variety of tasks.

• 8:07Speaker 01

  And

• 8:11Speaker 01

  then there's kind of more, kind of almost hard power skills. So in particular, cyber offense, which is already somewhat useful in military warfare and has been becoming more useful. And then, you know, I expect that as, as AI increasingly automates different parts of

• 8:31Speaker 01

  the military and as AI is embedded in more and more important high stakes processes, that will raise the importance of cyber offenses now, you know, whereas you can't hack a human mind as, as we hand off more important tasks to digital systems, they will be able to be hacked much more easily. So I expect cyber to come more important for hard power. And then, you know, the, the ultimate kind of most scary capability that, you know, I think ultimately will drive a lot of risk is when we get to the point that AI systems and robots are able to fully replace human military personnel, that's fully replace human soldiers on the ground, boots on the ground, fully replace the kind of commanders and strategists. And, you know, that, that, that might seem like a long way off today.

• 9:14Speaker 01

  But actually, you know, even just the last few years, we've seen a lot more importance of kind of AI-controlled drones. In warfare and I expect that trend to continue. And what we're already seeing is that, you know, as soon as the technology is there to kind of reliably automate military capabilities, there's, you know, geopolitical competition drives that adoption. And so, you know, I think it's going to be surprisingly soon that we that we do get AIs controlling kind of surprising amounts of, of, you know, real hard military power. And then one kind of wrapper for all of all of these things is the the automation of AI research itself.

• 9:54Speaker 01

  So today, there's, you know, few few hundred, um, few thousand, you know, top, top human experts that drive forward AI algorithmic progress. And my expectation is that, you know, there's a good chance the next few years that AI systems are able to match, you know, even the top human experts and their capabilities. And that would mean we go from, you know, maybe a thousand top, top kind of researchers to, you know, millions of automated AI researchers. And that could mean that, you know, all of these different capabilities, all of these different domains that I've been talking about, they all progress much more quickly than than we might have expected, um, just by naively extrapolating the recent pace of progress. And, you know, in my view and in the view of many, the recent pace of progress is already

• 10:35Speaker 01

  quite alarming in that, you know, five years ago, we just had really very basic language models that could string together a few sentences, a few paragraphs, and then went off topic. And now already we're getting kind of very impressive reasoning systems that are doing tough math problems and and helping a lot with difficult coding tasks. So, you know, bring bring that all together. I think there's a lot of kind of soft skills, a lot of hard power skills that that are relevant here. But like probably the most important thing to be watching is how good AI is at AI research itself, as that could kind of bring make make more happen, um, quite suddenly.

• 11:08Gus Docker?

  Yeah. Could you describe in more concrete terms what an AI-enabled military coup would look like? Some example to kind of make this concrete for us?

• 11:18Speaker 01

  Yeah, absolutely.

• 11:21Speaker 01

  So you can draw an analogy to historical coups where there's often a minority of the military launches a coup and then presents it as a fair complete and is able to prevent so chaos or discord or threaten individuals to prevent anyone from actively opposing them. And then in the absence of active opposition, it just seems like, well, they've done it. This is the new state of affairs. So that's a good starting point.

• 11:49Speaker 01

  Then the AI-enabled part is where we deviate. So historically, you needed at least a decently sized contingent of humans to go along with the coup. And you needed to persuade quite senior military officials not to oppose it. I think that will change as we automate more and more of the military. And so the most simple way that this happens is just that the head of state, it could be the president of the United States, just says, yeah, we've got the technology now to make a robot army. And I want the army to be loyal to me. I mean, I'm the commander in chief. Obviously, that's how it should be. And they're going to follow my instructions. No need to worry about whether I'm going to order them to do anything illegal. Like, we can put in maybe some kind of nominal legal safeguards. Let's not worry too much about that. The main thing is that they're loyal to me. And

• 12:33Speaker 01

  then, to my knowledge, that would be highly controversial or would definitely be against the principles of the Constitution. But it's unclear to me that it would be literally illegal. We just haven't had this kind of technology. We haven't legislated for it. The Constitution is not robust to this really powerful military technology. And so it's not surprising if, at best, this is just a very unclear legal territory. But you've got the head of state pushing really hard for that robot army to follow their instructions. And the head of state in the United States has a lot of political power. And so the most simple way is that he just pushes hard for that. He gets what he wants. Maybe he's

• 13:14Speaker 01

  using emergencies at home or geopolitical tensions to push it through and say that it's necessary. Maybe he's firing senior military officials that disagree.

• 13:28Speaker 01

  Maybe he's already got Congress to be very, very kind of fervently supporting and loyal to him and not, you know, being that kind of careful and open minded when assessing like the opposition that people will be making as this has happened. So that's the kind of the first, you know, really just plain and simple way that we could get this. Robot army is built, it's made loyal to the head of state, head of state just instructs it to stage a coup and it does it. You know, robots around the White House and brutally suppress human protesters. And then, you know, even if people go on strike and stop working, then, you know, you can have then AI systems and robots replace people in the economy. So humans have kind of really lost their bargaining power that they normally have that would kind of strongly disincentivize military coups in most countries.

• 14:14Gus Docker?

  Yeah, this is really a change from the, um, from the normal coups of history where you would have to have buy in from at least some segment of the population that are regular humans and you would need to kind of continually support that buy in and make alliances and uphold those alliances. But this is, this has changed now that you're talking about AI and AIs and robots that can basically be made loyal to a company or a head of state in a way that's more durable. Do you think we have other kind of historical precedents for thinking about how the dynamics

• 14:59Gus Docker?

  of what it's like to attend a coup, um, how those dynamics play out?

• 15:06Speaker 01

  Yeah, just one quick thing on that last point. I want to emphasize how there is a bit of a phase shift at the point in which AI can fully replace other humans, you know, in the government, in the military. When AI is augmenting other humans, you don't, you don't have this effect because a leader must still rely on those other humans to kind of work with the AIs to do the work. But there really is this phase shift when AIs and robots can fully replace the humans because then, yeah, a leader doesn't need to rely on anyone else. So I think that that's an important one to recognize. In terms of historical precedents, you know, the other, the other big one I point to is

• 15:42Speaker 01

  Recent trends in political backsliding, often called democratic backsliding. So the most kind of end to end clear and cut case is Venezuela, where you had in in in the seventies, a fairly healthy democracy that had been there for decades. And then increasing backside, increasing polarization, kind of like what we're seeing in the US recently. And then, you know, an increasing, you know, explicit commitment by by kind of the leader that, you know, he wanted to remove checks and balances on his power. And that the will of the people, you know, was being obstructed by various democratic processes and institutions.

• 16:23Speaker 01

  And then, you know, o- o- over the coming decades, you know, it has transformed into an authoritarian state. And, you know, many commentators have pointed out these trends in the US recently, over the past past 10 years. And and it even goes back before the past 10 years, to be honest, in terms of the broad kind of political climate. And then there's, you know, kind of the example of Hungary, where where where again, kind of elected elected leaders are just kind of removing the checks and balances in their power, kind of buying off the media or kind of threatening media outlets to to be more pro-government, not filing with contracts or kind of

• 17:03Speaker 01

  litigating them if they criticize the government, all all all these kind of standard tools where it's now like a lot harder to to point at one thing that's clearly egregious, but when you add up the kind of hundreds of little cuts, hundreds of little paper cuts to democracy that are being systematically administered, you're you're seeing a real kind of

• 17:24Speaker 01

  loss of democratic control and concentration of power. And so again, you know, AI could, you know, exacerbate and enable that dynamic. And again, the most straightforward way is you're just replacing human in powerful institutions, you're replacing the humans there with AIs that, you know, very, very, very loyal and obedient to the head of state. So, you know, think about Doge.

• 17:45Speaker 01

  And you know, they tried to fire people, there was pushback, you know, the state needs to function. Imagine if you could just have AI systems that could fully replace all of those employees and could be made fully loyal to the president, how much easier would it be to kind of push through, push, push through some of those layoffs or even just create entirely new government bodies that essentially just take on the tasks that were previously done by old bodies and let those old bodies kind of rot away or kind of slowly kind of prevent them from making decisions. So, and then the other big way is if, if the kind of head of, head of state is able to get access to much more powerful AI capabilities than their political opponents, maybe because state is very involved in AI development,

• 18:28Speaker 01

  then that's another way that they could get a head up, you know, making more persuasive propaganda and more compelling political strategy to like, you know, embed their, embed their power more.

• 18:39Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 18:45Gus Docker?

  You segment the ways in which AI can enable coups into three categories where you can talk about singular loyalties, secret loyalties and exclusive access. Perhaps we can run through those and talk about where those would play out, starting with, with singular loyalties, for example.

• 19:04Speaker 01

  Yeah, so singular loyalties is what we, we've just been talking about. That is deploying AI systems that are kind of overtly, obviously very loyal to just, you know, existing powerful people. So, you know, in particular, I am thinking about the head of state here as the main, the, the main threat. And so I think, I think we basically already covered it. The two, the two main angles in my mind are, I'm deploying loyal AIs, powerful government institutions and in the military. Secret loyalties is a very different threat model. It's much more, as you would expect, secretive.

• 19:40Speaker 01

  So the main FET model I have in mind to make it concrete is that an AI company CEO has automated all of AI research. So they could fire their staff at that point because the AIs can just do the work. Instead, maybe they put the staff onto some product work, but the core work of driving AI progress ever further forward, making increasingly intelligent AI is pretty much just done by AI systems. And at that point, they realize, well, they're in a bit of a precarious position that they're controlling this hugely powerful technology. Their power is pretty much unconstrained. Not literally unconstrained, but there's very few checks and balances currently on these CEOs.

• 20:20Speaker 01

  And they might anticipate that the government's going to realize how big a deal this is. They're going to kind of lose their influence. Maybe they worry the government will do something unethical with the AI technology. Maybe they worry that they'll be used for a war or something. And there's all kinds of justifications they could come up with, thinking, I don't want someone else taking control of this really powerful technology that I currently control. And obviously I'll use it for good. And so they might speak to maybe some AI advisors about this and say, well, what what should I do here? It seems I'm in a little bit of a precarious position.

• 20:51Speaker 01

  And then a solution that they might think of, or an AI, very smart AI system that's helping them might think of is, well, why don't you just create a backdoor in the super smart systems that we'll be building next month? Where like if the government tries to order those systems to do something that you definitely disapprove of, then the AI systems will just refuse to do it. Those, you know, those robots, those powerful robots that being deployed in the army will just refuse to follow orders from the government if they're really awful and ethical and things you wouldn't like. And so then you can have all the future AIs now that are developed by this organization are going to have this backdoor, or what I call a secret loyalty, where though they appear to follow the law, that they appear to kind of

• 21:34Speaker 01

  You know, uphold democracy. In fact, they are secretly still kind of loyal to the CEO. And that could lead to a coup. And again, the most direct route is a military coup where if ultimately this company's AIs are building military robots, military drones, military communication systems, then because they themselves, because the AIs building all this military stuff are secretly loyal, they can kind of pass their secret loyalties on to the military infrastructure, or they can just insert vulnerabilities in the military infrastructure that only they know about. And you know, here we are, we are, you know, I am primarily thinking about, you know, very superhumanly intelligent AIs.

• 22:18Speaker 01

  You know, are able to kind of design complex machinery that like a human can't kind of check and check if there's a backdoor.

• 22:26Speaker 01

  And that then means that we've now got a military infrastructure that is also ultimately can be controlled by the CEO. And so at this point, if someone tries to remove the CEO, if something is really bad, having the CEO doesn't like, they could just be like, okay, I'm now in charge. I'm instructed my

• 22:44Speaker 01

  that this military infrastructure to do the coup. Again, the robots around the White House. This time it's, it's, it's the company CEO that, that, that, that they're loyal to. And then, you know, the rest of the story can play out just as in our previous discussion.

• 22:57Gus Docker?

  How do these secret loyalties relate to sleeper agents? And maybe you could, you could explain what a sleeper agent is.

• 23:04Speaker 01

  Yes. Great. So sleeper agent is the kind of standard term used in, in the technical literature. And today's sleeper agents not very sophisticated. So an example of a sleeper agent is that if this, if the AI sees that it's The kind of the any most most of the time, the AI would just write kind of reliable code as instructed. But if the AI sees that the year is twenty twenty six, then the AI will write code with vulnerabilities. So there's this kind of trigger that that triggers this malicious behavior. And, you know, normally, if you were, you know, interacting with the AI, you really wouldn't be able to tell. Because it would just always be, you know, acting in a as you'd intended.

• 23:50Speaker 01

  But, you know, because it's just one trigger, it's very hard to find. But then you do get this malicious behavior. Now, I'm not worried about sleeper agents today. Because to to cause real harm, we would need a sleeper agent to be very sophisticated. You know, it could never accidentally trigger or very, very rarely accidentally trigger. And it would have to be able to do, you know, very intense, complicated tasks like, you know, building a military robot and ensuring that that robot was actually had a backdoor. You know, that's very, very, you know, far beyond what AI today can do. So, you know, I think that sleeper agents provide a basic proof of concept that it's possible for a malicious actor to kind of gain illicit control over a system

• 24:31Speaker 01

  and then have that system be deployed in the rest of the economy and potentially without people noticing. But they're not yet scary. And then the kind of secret loyalties is just what I call the kind of scary situation where you now have a very sophisticated AI system. It doesn't just have any old, it's not any old sleeper agent. It's a sleeper agent which is specifically loyal to one person trying to help them seize power.

• 24:53Gus Docker?

  Yeah, so what we're imagining here could be, for example, a a US-based AI company integrated into the into the US military. The CEO of the company wishes to to ultimately be in control of what happens. And so he engineers or he instructs perhaps AIs or or human engineers to create a sleeper agent in these systems that can be activated as at his command, such that the US US military officials think they're in control of the systems.

• 25:26Gus Docker?

  The systems behave in ways that they approve of throughout perhaps a quite a long period until there's some way, until the sleeper agent is activated in some way. And perhaps that would be more sophisticated than changing the date or giving it some phrase. But you can imagine advanced versions of sleeper agent that could actually behave in this way. Do you think that's realistic? Do you think sleeper agents can become that advanced?

• 25:56Speaker 01

  Yes, I do. I think, you know, we can, one analogy is to human spies. You know, human spies, they're basically, most of the time, they're kind of doing their assigned job as expected. And it's not like one simple trigger phrase or one year makes the spy do something covert or malicious. They are just fully aware of their surroundings. They're kind of constantly processing their inputs and they choose strategically exactly what the kind of ideal time is to, you know, steal some information, send an illicit email. And so, you know, in my mind, that's by far the most scary sleeper agent. Not one that's triggered by a password, but one that is kind of holistically making a decision about

• 26:41Speaker 01

  how and when to act out. I mean, the password ones are actually quite fragile because, you know, if you were the military and you're deploying your AI system and you're worried there's a password, what you can do is you can scramble all the inputs. You can kind of paraphrase all of the instructions it gets. And that might just mean that the password, if ever, ever someone tried to insert it, would be kind of re-scrambled and would just never actually come up. So I'm not actually worried about the kind of simple password triggered sleeper agents. But again, they're a basic proof of concept. And I think that as AI has become as smart and smarter than humans, that there's a reason, there's a strong reason to think that it would be possible to build much more sophisticated ones.

• 27:22Speaker 01

  One, one thing I will briefly say is that, you know, people often talk about misaligned AI scheming. And, you know, this is just the same idea where, you know, in fact, the argument for secret loyalties being worrying is much stronger. Where, you know, misalignment, you know, there is evidence misalignment. We don't yet have, you know, strong evidence of, you know, really sophisticated scheming emerging accidentally. But if, if humans and a human team of engineers or an AI team of engineers were specifically trying to build a system that, that was kind of covertly thinking about when, when, when, when, when to kind of act out, then it's much more plausible that it, that it could happen.

• 28:04Gus Docker?

  And then you have exclusive access, which is different from singular loyalties or, or secret loyalties. Why is that its own category?

• 28:14Speaker 01

  Yeah. So in my mind, the kind of singular or overt loyalties and the secret loyalties, both of those threat models go through deploying AI systems in really important parts of the economy. So, you know, in particular, government and military, what I focused on. But, you know, for those threat models, it's, you know, you, you actually need the rest of society to choose to deploy those AI systems and hand off a lot of power to them. And so I, I kind of have this third threat model of exclusive access to think about another possibility, which is that maybe even without people choosing to deploy AI systems and give them a lot of power, even without that, maybe AI systems can be powerful enough to help a small group seize power.

• 28:57Speaker 01

  So that the prototypical situation I'm imagining here is, you know, there's a kind of one AI project, which is, you know, somewhat ahead of the others. And maybe it goes to an intelligence explosion, whereas by which I mean kind of AI can automate AI research. And then AI quickly becomes super, super intelligent compared to humans.

• 29:20Speaker 01

  And then, you know, that project maybe has a few senior kind of executives or senior political figures that are kind of very, very involved and have a lot of control. And they might just be able to, you know, siphon off, you know, one percent of the project's compute and say, okay, we're now running these, these, these, you know, super intelligent AI systems and saying, how, how can we best seize power? And then there, there's kind of millions of them, they're doing, you know, every single day, they're, they're, they're doing a month of research, every single week, they're doing a year's worth of research into, okay, how can we, you know, how can we gain this political system? How can we, you know, hack in, in, into these systems?

• 30:01Speaker 01

  How can we, you know, ensure that we end up controlling the military robots when they are deployed by hook or by crook? And I think that, that, that, that model could start to apply earlier in the game. That could start to apply before anyone even realizes there's a risk, because, you know, this is just essentially all happening on a server somewhere. But actually, it's possible that the game could be won and lost by the massive advantage that a small group get by, by being able to kind of co-opt this huge, huge intellectual force. And so I think it's worth tracking that threat vector independently. But it does, you know, it does definitely interact with these other, other, with the singularities and the secret loyalties.

• 30:40Speaker 01

  Cause no one's strategy that your kind of army of super intelligent AIs may come up with is, oh, one, you like use the fact that you're head of state to like push for the robots to be loyal to you and like, here's how you could buy off the opposition, so confusion. Another strategy might be, oh, one, I just help you put back doors in all this military equipment so that then you could use it to stage a coup. But there might also be other ways, you know, maybe, maybe it's possible to very quickly create, you know, entirely new weapons which you can use to overpower the military without anyone knowing or.

• 31:14Speaker 01

  Maybe it's possible to gain power in other ways.

• 31:18Gus Docker?

  Yeah, yeah. I mean, one thing that would make this kind of future hypothetical situation different from today is that today it seems that there are leading AI companies, but over time capabilities kind of emerge in second tier companies and in open source. And so there's not that much of a gap between the leading companies and what is broadly available and perhaps what is publicly available. That's something that would change in the scenarios you imagine. So perhaps explain why the gap in capabilities between the one leading project and all of the others is so important.

• 31:58Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 32:04Speaker 01

  A few factors there. So in terms of why it's important, it's just what you've said. I mean, a lot of these, a lot of these threat models kind of exacerbated if there's one, one group of people that has access to much more powerful AI than other groups. If, you know, if open source is pretty much on par with the cutting edge, then everyone will have access to similarly powerful AI. I will say that even if open source is kind of on par, that doesn't mean we're fine because we could still choose to deploy AI systems in the military and the government and still choose to make them loyal to the head of state. When we're choosing to hand off control to AIs, it doesn't matter if there's 100 AI companies, you know, we're only handing off control to some AIs and maybe, you know,

• 32:49Speaker 01

  the government will ensure that they do have particular loyalties. So I will say, you know, this risk doesn't go away if we have, you know, lots of different AI companies and open source close to each other. But it does, it does become lower because the kind of exclusive access point where one group has access to superintelligence and the other group doesn't have access to much, that goes away. And I think it's a lot harder to pull off secret loyalties if everyone's kind of roughly equal to each other because it becomes a bit more confusing why your systems in particular end up controlling so much of the military or were so widely deployed and it becomes confusing how no one else was able to realize you're doing the secret loyalties when they were kind of equally able to do it or equally technologically sophisticated and potentially detect your

• 33:31Speaker 01

  secret loyalties. So I do think it makes a big difference. In terms of why I think it's plausible that there's the much bigger gap between the lead project and other projects, that there's a few different factors. The most plain and simple one is that the cost of AI development is going up very quickly. We're kind of spending about three times as much every year on developing AI. And that's just going to get too expensive for many players. Yeah, if and when we're talking about trillion dollar development projects, which I do expect, then both you can afford that. And also, there's just only so many computer chips in the world. If you want to have, you know, that currently, the number of kind of computer chips produced each year is less than a trillion dollars worth.

• 34:14Speaker 01

  So if if we get to a world where, you know, the the way to go to the next level of AI is to spend a trillion dollars, then, you know, only one company will be able to do that. I mean, maybe maybe we stop with it earlier. Maybe we just stop with, you know, there's two companies both doing half a trillion. But, you know, it would be really kind of kneecapping the level of progress if if we stopped long, long, long before that. And there would just be strong incentives for companies to merge or one company to outbid others in order to like, you know, really raise the amount of money that's being spent on AI development. You know, this is all assuming that we can build really powerful AI and it is economically profitable, which for for me isn't all in the background of the scenario. So that that's the first kind of straightforward reason why I think we'll we'll see a kind of a smaller number of projects.

• 34:57Speaker 01

  And we'll see kind of big gaps because when when you're spending a hundred times less on development, then that's that's going to be a bigger gap.

• 35:06Speaker 01

  That's the first reason. The other reason I've already talked about the the idea of an intelligence explosion. When we automate AI research, even if companies are fairly close, maybe that one is a few months behind. The company that's a few months ahead automates our research in that next three months, they make massive progress. So then there's actually like a really big capabilities gap, even though it's still just a three month lead. So there's the question whether they can use that kind of temporary speed to kind of get a more permanent advantage. And then the last big reason is just kind of government led centralization. It's already been talk of Manhattan Project and Sun for AI. I know, I think there's there's there's reasons to do those projects. They can help with safety in some significant ways. But they would, you know, exacerbate this risk.

• 35:50Speaker 01

  Because, yeah, if you pool all the US or all the United States computing resources into one big project, this can be way ahead of any other project and you pull over all of its talent and all of its data, then, yeah, you'll you'll see a really big gap. And that that would definitely make make it a lot easier for a small group to to. To do an AI enabled coup.

• 36:09Gus Docker?

  Yeah, you're you're kind of putting a big a big prize out there for someone who who's interested or who's considering a coup, right? If you're if you're concentrating all of the power, all of the resources, all of the talent into one project, then well, that's where you gotta go if if you are if you're a coup planner.

• 36:30Speaker 01

  Yeah, and just to be, I don't particularly expect that anyone is planning any coups. In fact, I'd be very surprised. I more think it's, you know, you you wanna be powerful, you wanna be a big deal, you wanna be changing the world. So, yeah, obviously you wanna lead the main lead the main project and then you don't want anyone else to come in and mess you mess it up. So obviously you wanna protect the fact you're leading that project. Don't want anyone else to, you know, misuse AI. I think it's kind of step by step, you you just kind of head down that road of more and more power and then, yeah, you know, often in history, that that road does end in just consolidating power, you know, to a complete extent.

• 37:02Gus Docker?

  And I mean, it it it can be, so what we're imagining here are times in which AI is moving in at incredible speed, right? The the pace of progress is insane, there's a bunch of confusing information, people are acting under radical uncertainty and perhaps in those situations, it's tempting to think that you are the person that can that can lead this project. And perhaps you're doing this out of supposedly kind of altruistic reasons, you're thinking that I need to do this in order to prevent other people that would perform worse than me at at at this project. And so you're kind of slowly convincing yourself that it it will be the right thing for you

• 37:45Gus Docker?

  to do to to take over in perhaps a forceful way.

• 37:50Speaker 01

  Yeah, you know, I don't think Xi Jinping or Putin think that they are the bad guys. You know, I think that they have.

• 38:01Speaker 01

  You know, probably sophisticated justifications for what they're doing.

• 38:04Gus Docker?

  Perhaps here is a good point to talk about the possibility of one state or company outgrowing the entire world. This, this, this relates to the, to the problem of exclusive access because if you have one company or one government outgrowing, outgrowing the entire world, then you have that company or government with exclusive access to advanced AI. And so how could this happen? How likely do you think it is that growth could be so incredibly fast that one company would outgrow all of the others?

• 38:41Speaker 01

  Yeah, so there's two possibilities we could focus on. The one I think is, is pretty plausible is that one country could outgrow all of the other countries in the world. So what that would mean is, you know, today, US is 25% of world GDP. But this would be a scenario where it is leading on AI.

• 39:01Speaker 01

  It is, this is already the case, but you know, it maintains its lead, it maintains its control over compute.

• 39:08Speaker 01

  And then when it develops, you know, really powerful AI, it prevents other nations from doing the same. You know, it's this already beginning with export controls on China.

• 39:19Speaker 01

  And that kind of, you know, embeds its lead. And then it, you know, uses the AI to develop powerful new technologies and, you know, it's in control of the, of those technologies. It uses AI to kind of automate collaborative labor throughout the US and maybe worldwide.

• 39:38Speaker 01

  And, you know, countries that don't use its AI systems will, will be really hard hit economically. And so we're kind of massively centralizing power in, in the US. And if the US is able to

• 39:51Speaker 01

  maintain exclusive control over, over, you know, smarter than human AI, then it seems pretty plausible to me, you know, very likely that the US would be able to rise to, you know, a strong majority, you know, more than 90% of.

• 40:08Speaker 01

  World GDP. And there's a few different dynamics that are driving that. First is that labor currently, human labor receives about half of world GDP.

• 40:21Speaker 01

  You know, just half of GDP is paid out on wages. AI will ultimately, and robots will ultimately be better than humans at kind of all economic tasks. And so if

• 40:34Speaker 01

  if the US controls all the AI companies that are replacing human labor, then that half of, that kind of 50% of GDP which is currently going to human workers will ultimately be reallocated to paying to whoever controls and owns those AI systems, i.e. US companies.

• 40:51Speaker 01

  There's a wrinkle there because some of that is physical labor and the US doesn't currently have a lead there. Physical robots, in fact, China's quite far ahead. But in terms of at least the cognitive aspects of our jobs. So we're talking a significant fraction of GDP that would just now be reallocated to US companies that control AI. So that already gets them from 25% to above 50%. Then we've got this further dynamic which

• 41:18Speaker 01

  is the dynamic of super exponential growth. So this relates to kind of previous work I've done on how AI might affect the dynamics of economic growth. But you know, kind of very potted summaries that

• 41:34Speaker 01

  it's often quoted that over the last 150 years, economic growth has been roughly exponential.

• 41:41Speaker 01

  And what that means is that if two countries are growing exponentially and one country starts off, you know, maybe twice as big as the other country, then at a later time, still, one country is twice as big as the other country. So let's say, you know, the US economy is 10 times as big as the UK economy. Then if they're both growing exponentially at the same pace, then, you know, 10 years later, again, the US will still be 10 times as big as the UK. So that's exponential growth. That's what we've seen over the last 150 years. If you look back further in history, we see super exponential growth. That means that the growth rate itself gets faster over time. So, you know, an example would be that 100,000 years ago, you know, the economy wasn't really growing at all.

• 42:23Speaker 01

  If you think what's growing, it was maybe doubling every 10,000 years or something in size. Very extremely slow economic growth.

• 42:33Speaker 01

  Then going from about ten thousand years ago, it seems more like ballpark. There's a, there's a doubling of the economy every thousand years, still incredibly slow economic growth. You zoom back in and kind of fourteen hundred, you can begin to detect, you know, okay, more like, you know, every three hundred years or so, the economy is doubling. And then in recent times, we've seen that the economy is doubling every, every thirty years. So essentially, you know, the growth rate is getting faster, the doubling times are getting shorter. That's super exponential growth. And there, there's, there's various reasons, economic reasons, theoretical reasons, empirical reasons to think that A I. And robotics, when it can replace humans entirely, will go back to that super exponential regime that, that has been at play throughout history.

• 43:17Speaker 01

  Well, that means that the growth is getting faster and faster over time. And the reason, the reason I'm, I'm saying all this, the reason this is irrelevant is that, you know, go back to the example of the U S and the U K. The U S is currently ten times bigger than the U K. If the U S is on a super exponential growth trajectory, its growth is getting faster and faster over time. And that means that even if the U K is on that same super exponential growth trajectory, as they both go super exponentially. The U S will pull further and further ahead of the U K. Because, you know, maybe the U S is doubling in ten years because it's already, it's already bigger, it's already further along the curve. Whereas the U K, you know, is, is still doubling only every twenty years.

• 43:57Speaker 01

  And so that means that the U S, you know, is now, rather than just ten times bigger than the U K, the U S is now, you know, going to be twenty times, thirty times bigger in size than the U K.

• 44:07Speaker 01

  So if the U S is able to.

• 44:11Speaker 01

  To kind of, if, if there is super exponential growth and the U S is able to kind of.

• 44:18Speaker 01

  Be, be bigger to begin with and therefore be further progressed on that super exponential growth trajectory, then. Then that's another way that they could just, you know, continue to increase their, their size of the economic pie and ultimately, you know, come, come to completely dominate. World G D P. So, you know, just, just to sum up everything I've said, today, the U S is twenty five percent of world G D P.

• 44:43Speaker 01

  If it controls and develops A I, that, that could easily boost it above fifty percent.

• 44:49Speaker 01

  I'd be very surprised if it didn't. And then from that point, you know, it's already bigger than the rest of the world combined. If it's able to then go on the super exponential growth path.

• 44:59Speaker 01

  Then it will go fast and faster over time and pull further and further ahead of the rest of the world that, you know, may be able to grow super exponentially if they can also develop AI, but, you know, we'll, we'll still be falling further and further behind because, because of the nature of super exponential growth.

• 45:14Gus Docker?

  Yeah, this, this actually seems quite plausible to me and not very sci fi. The, the thing that seems quite sci fi is the notion that perhaps even one company could, could grow at such a speed that it would outgrow the rest of the world. How, how, how like is that?

• 45:31Speaker 01

  Yeah, great question. I think it's a lot harder, but it is, it is surprisingly plausible. So you know, that first part of the argument I gave about how 50% of, you know, the world GDP is paid to human workers, you know, if that went to AI, that'd be a big, a big chunk. It is possible that one company could get a monopoly on, on kind of really advanced AI. So I, we already discussed some of the dynamics there.

• 46:00Speaker 01

  Where again, the simplest one is just a combination of an intelligence explosion giving a company a big advantage and then they're kind of buying up all the computer chips that the world is able to produce and outbidding everyone. If, if a company does that.

• 46:13Speaker 01

  And already, you know, it's, you know, seems to be outbidding other companies on, on compute or the, although Google also, also has a lot.

• 46:24Speaker 01

  If, if a company is able to do that, they could end up just one company in control of literally all of the world's cognitive labor, you know, because human cognitive labor will, will, will somewhat be kind of dwarfed by AI cognitive labor. So at that point, that one company could be getting, you know, all of the, all, all of GDP which is currently paid to kind of cognitive labor, which is a large part of the economy. As I said, you know, may, maybe as high as 50%, but you know, certainly as high as 30% of world GDP.

• 46:53Speaker 01

  If all that, all that would then, you know, seemingly be going to this one company that, that controls the world's supply of cognitive labour. So though I think that would take time and obviously it's going to take a long time to automate all the different parts of the economy, there is just a basic dynamic by which one company can now be controlling, you know, double digit percentages of, of world GDP. And there's obviously questions, would a government allow that? Would they step in? And that's where we get into the, you know, the, the, these dynamics of like, well, this company has all these super intelligent AIs on its side. Maybe it's able to lobby, maybe it's able to do political capture to avoid the state stepping in. Maybe it's able to be like, look, we're providing like economic

• 47:37Speaker 01

  abundance for everyone. Can you step in? Like, you know, that, that, that, that might not happen, you know, we're, we're underpinning your nation's, you know, economic and geopolitical strength. And if you try and, you know, remove, you know, step in and nationalize, then, you know, that's not going to happen. We're going to move to another country. You know, so you can, you can imagine, maybe, maybe they convince the, the head of state to kind of support, support them. And there's some kind of alliance there. But, you know, it, it's not completely obvious that, that the company would be shut down. It would, it would have certain types of serious bargaining power. So if a company was able to maintain this position as kind of sole provider of cognitive labour, it would be able to get a significant fraction

• 48:20Speaker 01

  of world GDP. And then it's then possible that from there, it could, it could bootstrap. And this is where it gets a bit harder, but, but the tactic it would, it would need to pursue is it already controls most of the cognitive labour, pretty much all of it. The thing it doesn't control is all the kind of physical machinery and all the raw materials that are also needed to create economic output. But it could pursue a tactic of kind of hoarding its cognitive labour so that no one else can ever have access to that. And then kind of selling it at kind of really kind of monopolistic rents to bless the world because there's, there's no one that can match it. This is, you know, it's offering everyone by far the best deal they can get, but just, you know, skimming off, you know, 90% of the value

• 49:04Speaker 01

  add from, from companies using its, its AI systems. So if it's able to do that, then it can kind of, it can kind of reap, you know, by far the, the majority of the benefits of trade. And then maybe you can kind of buy, increasingly buy up. Physical machinery and raw materials from the rest of the world, design its own robots, buy its own land. You know, imagine like a kind of big special economic zone in Texas or something, where this company is kind of unconstrained by

• 49:34Speaker 01

  kind of bureaucracy and then it's also now, you know, got a big arm somewhere in Siberia and in Canada, it's kind of creating these big special economic zones by doing deals with specific governments.

• 49:47Speaker 01

  And I do think it's a bit of a stretch that this all goes ahead without, you know, various other powerful political and economic actors pushing back. But like the kind of basic economic growth dynamics are like surprisingly compatible with a company, you know, ultimately coming to control most of the cognitive labour and most of the kind of physical infrastructure that its AIs have designed using all the parts that's bought from the rest of the economy.

• 50:17Gus Docker?

  Yeah, and do you think this is a risk factor for AI-enabled coups then? Just because you're concentrating all of the power and all of the resources into either perhaps one country or one company even?

• 50:29Speaker 01

  Yes, I definitely do.

• 50:33Speaker 01

  The more realistic path is that a company kind of starts down this path of outgoing the world, gets kind of huge economic power,

• 50:43Speaker 01

  increasingly controls the country's industrial base, its kind of physical infrastructure, manufacturing capabilities. And then from there, it's in a much stronger position to seize political control because it's got massive economic leverage and it can also increasingly gain military leverage because as it, you know, as it increasingly controls the country's broad industry and manufacturing, that will feed in to military power. So, you know, some of the, you know, possibilities I discussed earlier, where, you know, you could potentially have your AIs be secretly loyal, that ultimately design the military systems, or you could just instruct your AI systems to start making, you know, a military

• 51:27Speaker 01

  that is not legally sanctioned, but, you know, because the government doesn't have much to threaten you with.

• 51:37Speaker 01

  It kind of, you you get away with it. I mean, it it gets a little bit tough. You probably need to do that in secret, otherwise the existing military could um could could prevent it. But yes, I do think that, you know, being very rich helps with lobbying, it helps with all kinds of ways of seeking power and then controlling, yeah, controlling a lot of industry can can potentially give you military power.

• 51:59Gus Docker?

  You mentioned these special economic zones. That's that's one way in which companies could kind of bargain with states in order to have favorable regulation and to and to be able to carry out their projects without intervention, basically. Another way for them would be to collaborate with non-democracies that are perhaps controlled by a a single, a small group, or or perhaps even a single person.

• 52:28Gus Docker?

  And in that way, it seems like perhaps it's easier to get something done in a non-democracy. And and that is a way to to grow fast. And and so perhaps there are incentives for companies to place more resources i- in non-democracies. What do you think about the the prospect of non-democracies outcompeting democracies when it comes to AI?

• 52:53Speaker 01

  I think it's a really great question, and it's tricky, because I think I agree. Like democracies have lots of checks and balances. They have a lot of bureaucracy, a lot of red tape, and that will disincentivize AI companies from investing. And then additionally, if there are people really trying to seek a legitimate power, that will be easier to do in non-democracies because they're less less politically robust. So there there are these various forces pushing towards, you know, this new supercharged economic technology being disproportionately deployed in non-democracies. Um, and I I think that is scary.

• 53:30Speaker 01

  My my my own view is that probably we should democracies should should should to kind of do everything they can to to to avoid that situation, make it much easier for AI robotics companies to to set up shop in in democracies, remove the red tape.

• 53:58Speaker 01

  Try and use export controls like are already happening to prevent technologies being deployed in non-democratic countries and that that goes beyond China. There's obviously lots of countries that are not allied with China but are also non-democratic here. And the US, you know, the US is in a strong position because it does have the stranglehold on AI technology at the moment. So I do think it can be done. But yeah, in my view, like it will be really important to to to kind of work very hard to to find a kind of a non-restrictive regulatory regime and it will also be very important to really try and pursue innovative innovations within the democratic process itself.

• 54:43Speaker 01

  Where, you know, democracy is great in many ways. It really distributes power and it and it has been very good at ensuring good outcomes for its citizens, but it's very slow.

• 54:54Speaker 01

  And often, you know, kind of nonsensical because you have competing interests that are kind of stepping on each other's toes and the result in legislation is just, you know, a garbled mess. And so, you know, AI can potentially solve those problems. You can have AIs negotiating and thinking much more quickly on behalf of the the the kind of human stakeholders. You can have AIs nailing out agreements that aren't a garbled mess, but that like really gave everyone what they truly wanted out of the legislation. And you can still do all of that really quickly so that you're not falling far behind the autocracies that have just got one person immediately saying what to do. And I think if we did that, we, you know, democracies could out-compete autocracies because, you know, the big thing that often screws over autocracies is that

• 55:38Speaker 01

  one person is flawed, often makes big mistakes, people afraid to kind of stand up to them.

• 55:44Gus Docker?

  Yeah, that that would be more of my assumption. That I would assume here that perhaps democracies with market-based economies have an advantage just because you can you can do kind of bottom-up knowledge discovery, you can try different things out, you can see what works, you can have competition between companies and so on. And perhaps in in non-democracies, well, I mean, there you can you can have one person or a small group stake out a direction for what the country should do, but if that direction is what is wrong, it's probably difficult to change course.

• 56:14Speaker 01

  Yes, I I think you could be right. I should have I should have, you know, given more weight to that that that advantage of of kind of democracies in terms of the free market being, you know, in many ways much smarter.

• 56:27Speaker 01

  But in terms of autocracies that are good at harnessing free market dynamics,

• 56:32Speaker 01

  my worry would be that the AI helps them more than it helps democracies because

• 56:41Speaker 01

  AI will be able to kind of replace, currently, you know, one person just can't think that hard, can't really figure out a good plan. But if, if, if that one all powerful leader has access to loads of AI systems that can kind of think things through and investigate lots of different angles, then, you know, that if they're, if they're following its advice and they could get advice which, you know, lacks the flaws that, that today, today's systems had and, and they could potentially move much faster. But I, I, I think, you know, you're right that kind of economic liberalism is, is still going to be important even after we get powerful AI systems that could give, that could give democracies an advantage.

• 57:24Gus Docker?

  This is a bit of a tangent perhaps, but I, I'm thinking whether, so if you have a, a leader of a country that has a lot of power, perhaps complete power over that country, and that leader is equipped with AI advisors advising him and, and, and kind of laying out kind of the landscape of options for him to choose from, wouldn't his decision making still be, in a sense, bottlenecked by the fact that he's a human, by the fact that he has these, these flaws that we all have, the biases that we all have. So even with fantastic advice, I think it's, it's quite plausible that, that he would still make the same mistakes that we see leaders make today.

• 58:04Speaker 01

  I think that's true. I, I think it's also true in democracies, unfortunately, that, you know, there's ten negotiators and they each kind of still have biases and still refuse to listen to the wise advice they're getting from their AIs, that could still gum up the, the system. And yeah, it does depend on how much humans come to trust and defer to their AI advisors. There, there's a possible future where the AIs are just always nailing it, they're always explaining their reasoning really clearly, and we are just like increasingly convinced and happy to trust their judgment. If AI is aligned, I think that would be a great future, because I do think humans have all these very big limitations and biases, which if we can solve the alignment problem, AIs don't need to have. But there's also another future where humans just, you know, want to be the ones

• 58:48Speaker 01

  making the decisions, have these kind of pathetic.

• 58:53Speaker 01

  Kind of motivations that that that that they're still kind of influencing their decisions. And that, yeah, that the kind of that that continues to to to to to limit the quality of decision making.

• 59:06Gus Docker?

  Seeing things from above, right, from kind of like ten thousand feet. How should we think about mitigating the risk of coups here? Is it is it about removing people that would use AI to commit coups? Is it about kind of finding those people in the militaries, in the governments, in the companies perhaps? Or do we have ways to reduce the

• 59:34Gus Docker?

  returns to to seizing to seizing power?

• 59:38Speaker 01

  Yeah, I mean, from real ten thousand feet up, the way I would characterize it is

• 59:45Speaker 01

  create a common understanding of the risks,

• 59:50Speaker 01

  build coalitions around preventing them. And then the existing balance of power can self propagate forward. You know, it's in everyone's interest to prevent a coup. Currently, no one small group has complete control or close to it. And so, you know, if if everyone can be aware of these risks and aware of the steps towards them and kind of collectively ensuring that no one is going in that direction, then we can all kind of keep each other in check. So I do think, you know, in principle, the problem is solvable. And it doesn't require, you know, solving the risk of misalignment does require solving some tough technical problems. This doesn't in the same way.

• 1:00:33Gus Docker?

  Yeah, you have a bunch of recommendations for mitigating the risks, both for AI developers and governments. And perhaps, you know, we don't have to run through all of them, but you can talk about the most important ones for AI developers.

• 1:00:49Speaker 01

  I might characterize this, I might kind of talk about it by going back to those three threat models we discussed earlier. So the first one was singular royalties or overtly loyal AI systems. Where again, you know, the main risk there is AI deployed by the head of state and the military and the government that's loyal to the head of state. And so the main. Countermeasure that currently appeals to me is faster, you know, figure out rules of the road for these deployments, you know, obvious things like AI should follow the law, AIs deployed by the government shouldn't advance particular people's partisan interests, but should only do like, you know, official state functions.

• 1:01:31Speaker 01

  AIs in the military shouldn't be law to one person, you know, they should, different groups of robots should be controlled by different people. And, you know, head of the chain of command can still be head of the chain of command via instructing other people that instruct those robots, but they shouldn't all go directly to head of chain of command because that centralizes military power too much. So, you know, fleshing out basic rules of the road of that kind and then building consensus around them because, you know, companies might want to say to governments, yeah, we don't want you to deploy our systems, you know, if, if, if, if they're willing to break the law, but if the government, you know, if, if, if the government will have a lot of bargaining power, the executive in the United States can, can, you know, it's hard for companies to stand up to them.

• 1:02:17Speaker 01

  So what we want to do is, you know, establish these rules of the road and then get broad buy-in from Congress, from judiciary, from, you know, other branches of the military, from many parts of the executive. So then it's very then hard for, say, the president to say, yes, let's like make this robot army law to me. And everyone's like, obviously not, we've all like agreed, that makes no sense, you know, and then the president doesn't even bother trying because it's just clear that it would be a no, no go, that, you know, their mind doesn't even go there.

• 1:02:44Gus Docker?

  In some sense, this is about kind of implementing the procedures and the transparency rules that we know from democracies today into how we use AI, both in governments and, and in companies, I think.

• 1:02:58Speaker 01

  Exactly, yeah.

• 1:02:59Gus Docker?

  Do, do you worry here that when, so the government is seeing, is looking at these companies from the outside and they don't have full insight into what's going on? Do, so, so there are kind of protections for private companies that, that mean that they can, they can do things in secret without the government knowing, at least as, as things stand now. Is that something that would evade these mitigations you're, you're thinking of?

• 1:03:28Speaker 01

  So I mean, this for this first bucket, the singular loyalties bucket, it's mostly the the kind of heads of state that I'd be worried about. So, you know, it actually is probably good for the government, or at least for the, you know, the head of state themselves, not to have full insight into literally everything the company is doing, because that would give them too much power. But, you know, actually having different parts of the government having insight into what the lab's doing, I think is very good. I'm a big, big fan of transparency. And, you know, we do have a good set of, you know, government checks and balances from different government bodies that we can deploy to kind of keep the lab in check, using these other bodies, but also not allow like, you know, the executive branch and the and the president to get excessively powerful. So that that's

• 1:04:12Speaker 01

  the mitigations, the kind of singular loyalties. In terms of secret loyalties,

• 1:04:20Speaker 01

  the key mitigation is is what I'm increasingly calling system integrity. That is, you know, using established cybersecurity practices and machine learning security practices, i.e. kind of preventing sleeper agents and backdoors in machine learning models. Using all of that to ensure that your development process for AIs is secure and robust, and that no malicious actor, be they an employee in the post-training team at a lab, or be they the CEO of the lab that is either malicious or is being threatened by the Chinese government to kind of tamper with model development, that no person or no small group is able to significantly tamper

• 1:05:02Speaker 01

  with the behavior of AI models. And no group is able to get illegitimate access to AIs that would help them seize power. So that's that's this idea of system integrity, which is, you know, essentially a technical project, which does just draw on existing practices, but is not yet implemented in any of the top labs.

• 1:05:29Speaker 01

  I will quickly shout out for non, non lab. For kind of people listening that aren't working at labs, I think there's a lot of really good technical research that could be done on kind of investigating the conditions under which you can insert a sleeper agent without a defense team knowing. And there's, there's just loads of research that, that could be done in terms of the different settings there for attackers and defenders, which could then inform what, what parameters we need to be in place to achieve system integrity, you know, if it turns out that it's, you know, it's very hard to make a sleeper agent accept in the final stage of training. That's really useful to know because then we can focus our efforts within labs at, you know, that final stage, just as a hypothetical example.

• 1:06:15Speaker 01

  So that, that's the kind of key mitigation in my mind for the secret loyalties. And then I'll quickly do for exclusive access.

• 1:06:23Gus Docker?

  That one seems more difficult. I don't know, just, just from, from, from me reading and preparing for this interview, that, that one seems like a, a difficult one to handle where this is, this is, this is in a some sense, a, a deep trend in history and in, in the kind of history of modern economics that you do see faster growth rates and you do see concentration into bigger and bigger economies, both in, in countries and in, in companies. So are you, are you in some sense pushing against underlying trends if you're trying to mitigate exclusive access to, to advanced AI from, from one actor?

• 1:07:04Speaker 01

  I think you can, you can do this in other ways. So you can have the law require that AI share, that AI labs share their powerful capabilities with kind of other organizations to act as a checking balance. So that, you know, labs should share their R and D capability, AI R and D capabilities with evals organizations.

• 1:07:28Gus Docker?

  Here you're thinking about giving insight into what they're capable of, not, not actually sharing those capabilities. That would be too big of an ask, I think.

• 1:07:36Speaker 01

  I mean, I do mean API access.

• 1:07:40Speaker 01

  So.

• 1:07:42Speaker 01

  You know, if a lot of the work in developing and evaluating systems is now done by AIs, then we want an evaluation organization like Apollo or Meta to also be uplifted. And so we want them to have access to really powerful AI that can similarly stress test how dangerous the frontier systems are. If they're only using human workers, then that's going to be a big disadvantage. So no, I do want API access to powerful capabilities for other actors. You know, for example, cyber security teams in the government and in the military should have access to the lab's best cyber capabilities. And again, that should be a requirement by law.

• 1:08:23Speaker 01

  So, you know, generally, like even if there's a natural tendency towards centralization of power in one organization, you can still require that that organization share its systems with the checks and balances. That's one thing. And the other thing is

• 1:08:39Speaker 01

  kind of preventing anyone at this organization from misusing the powerful AI systems. So the biggest thing on my mind here is that today we still have helpful only AI systems where you can kind of get access to the system and then it'll just do whatever you want. No holds barred. I don't think there should be any AI systems like that. I think you should always have, you know, at least a classifier on top of the system, which is, you know, looking for harmful activities and then kind of shutting down the interaction if something harmful is detected. And if you have a special reason to use cyber offense, you know, for your job, or you have a special reason

• 1:09:20Speaker 01

  to do potentially dangerous biology research, you can have that classifier allow certain types of activity. But you should never have anyone accessing a system where, you know, anything is allowed. You know, no one, no one has a legitimate reason to access an AI that will literally do anything. So what I want to aim for is a world where, yes, if there's a specific reason why you need to use a dangerous capability, absolutely, you can, you can use that system. But that system will just do that one dangerous domain. It won't kind of do anything you wanted. Because that, you know, that's a, that's a very scary situation where there's, you know, there's a hundred reasons why the CEO could ask for access to a helpful only system. You know, maybe the guardrails are annoying. Maybe, maybe he wants to kind of.

• 1:10:04Speaker 01

  You know, do do do something which which the model is reluctant to do. But today, when you asked to remove some guardrails, you're removing all of the guardrails. And now there's no holds barred. So instead, you know, we should we should be flexibly adjusting what guardrails are there, you know, by the use case and just, you know, never have a have a situation where where there's no guardrails. I think that I think that could go a long way towards helping if if that was that was robustly implemented.

• 1:10:31Gus Docker?

  With all of these mitigations for both secret loyalties and exclusive access and singular loyalties, you would worry that they would be disabled by the group planning a coup, right? Say say that, for example, you are the CEO of an AI company and you're giving API access to to evaluations, organizations, testing your model, trying to see what they're capable of. Maybe you just cut off access before you get to the really powerful model that could actually be the model that that helps you conduct a coup. Do we have ways of making sure these mitigations are entrenched before?

• 1:11:15Gus Docker?

  In in in such a way that they can't be removed by the group planning a coup?

• 1:11:20Speaker 01

  This is a great question. It is. It is pretty tricky. CEOs by default have a lot of control over their organizations. And similarly, heads of state, you know, including US president, has a lot of control over the military and over the government.

• 1:11:36Speaker 01

  So yes, there's a risk that one of these powerful individuals realizes that maybe they want, you know, more influence by gaining control of AI and notices that there's these kind of pesky little, you know, processes that prevent that. And it's like, OK, well, let's remove them. I can give, you know, easy, say, you know, productivity reasons to prevent them, red tape reasons.

• 1:12:00Speaker 01

  And you know, if they can make a plausible argument, then it could be hard to oppose them. So I do think it's a big issue.

• 1:12:11Speaker 01

  But I'd say a few things. Firstly, something I mentioned earlier, I don't think that anyone is today planning to do an AI-enabled coup. The way I think this works is that people are faced with their kind of immediate local situation, something they want to do over the next month. And the blockers that they're facing to doing that specific thing. And you know, what tends to happen is people, you know, tend to want more influence that that helps them get get stuff done. And so people will kind of bit by bit kind of move in the direction of getting more control over AI. But they won't be kind of thinking, yes, I need to make sure that I remove this whole process because that will allow me to do an AI-enabled coup. That that's kind of unrealistically galaxy brain.

• 1:12:54Speaker 01

  And so what we could do is we just set up a very efficiently implemented and very reasonable set of mitigations. That doesn't really prevent CEOs from doing what they're they're trying to do. Um. And so the CEO doesn't find in their day to day that wanting to kind of like remove these things that are holding them back. But because these mitigations are here, the CEO never gets to a place where they're anywhere close to being able to do a coup. Or or there's any kind of pathway in in their mind to be able to doing a coup because they're they're constantly prevented from getting access to kind of really powerful AI advice that might point out ways in which they could do this. Because they're like surrounded by colleagues that like, you know, strongly believe that these mitigations are sensible and reasonable.

• 1:13:38Speaker 01

  And in fact, they are well implemented and, you know, there aren't many downsides. Maybe an environment where they kind of get kudos for the fact that they've said, yeah, I obviously I'm not going to get access to helpful only systems. That's crazy. And then that's kind of like something, something that that makes them.

• 1:13:57Speaker 01

  Seen good.

• 1:14:00Speaker 01

  So that that's one thing to say. Another thing is, again, going back to this point that there are currently checks and balances, and there is not currently a situation where one person has power.

• 1:14:12Speaker 01

  You know, if the if if the entire board of a company and other senior engineers recognize the importance of the mitigations, know about this threat model, then they will notice if the CEO is is moving that direction.

• 1:14:26Speaker 01

  And, you know, similarly, similarly, within the government, there are checks and balances. And and they they could be activated if people people are looking out for it.

• 1:14:36Gus Docker?

  Do you think these traditional oversight mechanisms, like a board being in in control of of the CEO, being able to fire the CEO, or the possibility of Congress or the Supreme Court kind of overruling or constraining the US president? Do you think those will persist in environments where AI is moving very fast and it is is AI capabilities are growing at a rapid pace?

• 1:15:05Speaker 01

  So, great question. Here here's one story for optimism. Today, things are moving fairly fast, but those checks and balances are somewhat adequate, at least to preventing really egregious situations.

• 1:15:22Speaker 01

  By the time the AI is moving really quickly, we'll have handed off a lot of, you know, the implementation of government, the implementation of things in the AI companies, the research process will have handed it off to AI systems. And when we do that handoff, we could program those AIs to maintain a balance of power. So rather than handing off to AIs that just follow the CEO's commands or AIs that follow president's commands, we can hand off to AIs that follow the law, follow the company rules, report any suspicious activity to, you know, various powerful human stakeholders. And then by the time things are going really fast, we've already kind of got this whole layer of AI that is is maintaining balance of power, like the whole AI government bureaucracy,

• 1:16:07Speaker 01

  the whole AI kind of company workforce, they are like better than humans today at standing up to misuse potentially.

• 1:16:17Speaker 01

  They are like less easily cowed and intimidated and they could actually make it harder for someone in a position of formal power to to kind of get excessive influence. So this is like the flip side of, you know, the singular loyalties where you potentially deploy these AIs that are like explicitly loyal. You can actually kind of instead get kind of singular law following and balance of power maintaining AIs that you deploy. And so the hope is that by the time we, really, things are beginning to go kind of crazy and we're really seeing speed ups from AI, we've already kind of set ourselves up in an amazing way to maintain balance of power. And there's this critical juncture where we are handing off to AIs. And it's just, you know, what are those

• 1:17:01Speaker 01

  AIs, you know, what are their loyalties? What are their, what are their goals? And, you know, I think we can gain a lot by making sure that those AI systems are maintaining balance of power, reporting, you know, illegitimate suspicious activities and are not kind of overly loyal to any one person.

• 1:17:20Gus Docker?

  How do you think the risk of AI-enabled coups interface with kind of more traditional notions of AI takeover? So just a misaligned, highly capable or advanced AI system taken over in kind of contrary to the wishes of the developers or the governments?

• 1:17:41Speaker 01

  Yeah, I mean, there's some close analogies. Perhaps the most analogous case is the case of secret loyalties, where, you know, you've got these AIs that have been told by the CEO to have the secret goal of seizing control and then handing control to the CEO. That's just very similar to AIs that wanted to seize power for themselves secretly. And, you know, all the same stories could apply where the AIs kind of make military systems and then they control the military systems and the robot army and then they seize power. And the only difference is, were they seeking power because it just kind of accidentally emerged from the training process, which is the misalignment worry, or were they seeking

• 1:18:23Speaker 01

  power because the CEO programmed them in that way? You know, but that's the kind of seed of the power seeking. But then with the secret loyalties to that model, the rest of the story is, you know, pretty similar. I mean, there's still differences, you know, in the secret loyalties case, the CEO might be doing more to help the AIs along. With their plan, you know, maybe even in the misalignment case, the AI has managed to kind of manipulate the CEO into into doing similar things. So that that's the case where it's.

• 1:18:53Speaker 01

  Most most analogous, I think the. You know, and another difference that that's said into me is that if there are lots of different AI projects, then. A coup seems an AI-enabled coup seems a lot harder because you'd need like lots of different humans to kind of. Coordinate to kind of seize power together. Which seems, you know, while while I can totally believe that um. One, one, one person might might try and seize power does seem less likely to me that there'll be loads and loads of.

• 1:19:26Speaker 01

  Humans that would want to to do that from lots of different labs. Um, whereas for the misalignment story. It it is, you know, more likely the case that if one of these labs has misaligned AI, then maybe, you know, lots of them have misaligned AI. And so then it's it's more likely that you would have, you know, maybe ten different AIs colluding. And then seizing power and taking over.

• 1:19:51Speaker 01

  And so that kind of collusion between multiple different AIs is is more likely in the case of misalignment than in the case of an AI-enabled coup.

• 1:19:59Gus Docker?

  Just because i- if there's one misaligned AI, then there's something about the training process for AI systems that that are causing a misalignment. And then it will be a common feature among among many companies.

• 1:20:11Speaker 01

  Exactly, whereas just the fact that one CEO instructed a secret loyalty would not to the same extent. Make you expect that other CEOs have done the same.

• 1:20:23Gus Docker?

  So you mentioned this possibility, but but do you think it's? Yeah, what do you think of the prospect of? A president or a CEO of a company being duped by a misaligned AI into conducting a coup on its behalf? So you can imagine a president or or a CEO kind of thinking that. That he's conducting a coup to remain in control, but he's actually acting on on behalf of a misaligned AI.

• 1:20:48Speaker 01

  Yeah, I think it's an interesting threat model and, you know, some people who think about. AI takeover threat models take it pretty seriously. And it's just, you know, it's just a case where we're just completely mixing these two threat models together, you know.

• 1:21:04Speaker 01

  People who are worried about AI takeover for this reason should be very supportive of the kind of anti-coup mitigations I'm suggesting. Because if we implement checks and balances that prevent any one person from getting loads of power, then that AI will not be able to convince them to try because they just won't be able to succeed. So, you know, I see, I see this as like, you know, an additional reason to worry about AI-enabled human coups and to try and prevent them is that, yes, even if no human wants to do this normally, you know, misaligned AI might, might, might make them try.

• 1:21:41Speaker 01

  In terms of how plausible I find this threat model, you know, honestly, I think that if a human tries to seize power,

• 1:21:50Speaker 01

  the main reason is that that human wanted power. Like, this is just something we know about people. We know it about, you know, heads of state today. You know, it's very clear that, that, that, you know, many heads of state in the most powerful countries in the world are very power-seeking.

• 1:22:06Speaker 01

  We know it about CEOs of, you know, big tech companies, you know, we know about, you know, about, about, about some of those leading, leading AI companies that we, we do know that they're very power-seeking, their CEOs. And so

• 1:22:20Speaker 01

  I don't think we need to theorize that, like, they were massively manipulated by the AI and convinced to become power-seeking. Like, I think, I think it's more likely that if they seek power, they just, they just did it for the normal human reason. I, I do think AI will get, ultimately get good at persuasion. Um, I don't particularly expect it to be hypnotic-level persuasion, though, though, you know, obviously there's massive uncertainty here.

• 1:22:47Speaker 01

  And, yeah, I do, I do think that, like, a very smart AI, where there's a human that's already kind of, kind of interested in seizing power and it already kind of makes sense for them to maybe do it, the, and a misaligned AI could totally nudge them in, in that direction. And then could implement that in a way that actually allows the AI to seize power later. I think that is that is very plausible.

• 1:23:10Gus Docker?

  When we're thinking about distributing power and kind of having this balance of power, we can imagine the models being set up via post-training, via the model spec, via various mechanisms to have obey the user unless unless what the user instructed to do is in conflict with what the company is interested in. And perhaps obey the company unless what the company is using the model for is contrary to what the government kind of permits. But when we set it up in those levels, you ultimately end up with the government in in control in some sense.

• 1:23:52Gus Docker?

  And I guess that exposes you to to risk of of a government coup then. If you have at the ultimate top layer of the stack, here's what the here's what the models can and cannot do according to to the government.

• 1:24:08Speaker 01

  Well, I'd say a couple of things. First is that the government isn't a monolithic entity. And so that government decision of what the balance should be could be informed by multiple different stakeholder groups. Then ideally, you know, it's ultimately democratically accountable. I do think that democratic accountability becomes more complicated in a world with where there's massive change in a four-year period.

• 1:24:30Gus Docker?

  Just for for the simple reason that there there's no election during a period where where a massive change is happening. So so the feedback loop is too slow.

• 1:24:40Speaker 01

  Exactly. You know, I think the risks of AI-enabled coups will probably emerge and then, you know, be decided within a four-year period as in like it will be resolved whether or not it happens or doesn't, all without any like intermediate election feedback. That doesn't mean that democracy can't have an effect because politicians anticipate, you know, what what future elections will will will find and want to maintain favor throughout their terms, but it does it does pose a challenge.

• 1:25:10Speaker 01

  But but sorry, I was I was kind of saying, e- even absent that, there's there's many different stakeholders in in the government. And so, you know, there would it would have to be a large group of government employees that were kind of trying to do a coup. And then they would kind of, the companies would know that they were setting these these odd restrictions on on on the on on the kind of behaviour. And so the companies would know and they have leverage and power. And then, you know, it could go public. So I I do, I don't. I don't think it would be that easy for the government to do coup.

• 1:25:40Gus Docker?

  First, there's a difference also between allowing the government to set restrictions on what the models can do and then allowing the government some kind of access to to commanding the future AI systems in certain directions. So it's kind of setting setting limits versus versus steering the systems.

• 1:25:58Speaker 01

  Yeah, exactly. I mean, the distinction I was going to highlight was between specifically making AI systems loyal to, for example, the head of state and they're setting very broad limits. Where there's just like, you you can pretty much do whatever you want, except for these obviously bad things, where that second option doesn't really enable anyone to do a coup. It just enables everyone to do whatever they want. And then you kind of blocked out all of the kind of coup-enabling possibilities through through those limits. You know, as as long as you haven't made those systems loyal to a small group. So given that there's this obvious option to just put in these limits that block coups but don't enable coups. And given that there's, you know, a wide range of stakeholders that could potentially feed in to

• 1:26:44Speaker 01

  what what what the AI's kind of limitations and instructions are, I think it's very, very feasible to get to a world that that where there's robustly not central centralisation of power. Um, there there's obviously a big uncertainty over whether we will actually get our act together and and get those limits put in place in the right way.

• 1:27:03Gus Docker?

  When do you think these threat of the threat of AI-enabled coups materialise? I- is it at some specific point i- in AI capabilities? Or is it simp- does it simply scale with the systems getting more advanced? When do you think the threat is at i- is at its peak?

• 1:27:22Speaker 01

  It's a good question. For the threat models that I primarily focused on, they require pretty intense capabilities. So that, for example, the secret loyalty threat model more or less requires AIs to do the majority of AI research. So we're talking about, you know, fully replacing the world's smartest people in a very wide range of research tasks and coding. That's that's that's pretty intense. And then a lot of the threat models that I focus on root through military automation, that is AI and robots that that can kind of match, you know, human boots on the ground. And that's, you know,

• 1:28:00Speaker 01

  that's that's pretty advanced. Again, that said, I I I think, you know, you can probably do it with with with less advanced capabilities than that. So like we, you know, drones today are already pretty good, already already providing, you know, making a big difference in in some military situations. So it it's, you know, not out of the question that, you know, more limited forms of AI and robot military technology could could be enough to facilitate a coup. It's a bit harder because if if they're limited, then there's a question of why the existing military doesn't just kind of seize back control after a bit of time. And so probably that scenario also has to involve things like, you know, maybe

• 1:28:45Speaker 01

  the current president, you know, supporting the coup and therefore like pressuring the military not to intervene, or some other source of legitimacy for for the coup beyond the kind of the the AI controlled drones. And then there's also kind of more kind of typical types of backsliding, um, you know, like like is already been happening in the US, that I think, you know, could be exacerbated through AI enabled surveillance and, you know, AI kind of increasing state capacity in other ways. And I again, you know, that that that backsliding doesn't require, you

• 1:29:26Speaker 01

  know, super powerful AI. You know, you could probably do a lot of monitoring, a lot of kind of content moderation on the internet, a lot of surveillance with today's systems.

• 1:29:38Speaker 01

  It doesn't get you all the way to one person having complete control, where they can just crush any resistance with a robot army and replace everyone in their job with a, with an AI and so no one has any leverage. So I think, you know, to get to that real intense, this is like the most intense form of concentration of power via AI. That requires really powerful AI, but to just kind of significantly exacerbate existing trends in in political backsliding and, you know, to make it easier to do a military coup, I think, you know, more limited systems. Would suffice.

• 1:30:15Gus Docker?

  Yeah, we we discussed earlier the the possibility of of one country or one company outgrowing the rest of the world and kind of concentrating power into into those entities. I now you mentioned one person, do you think that's actually a plausible scenario in which you have, say, one CEO of one company being being the person in in control of the world by a a concentration of power and then a coup?

• 1:30:42Speaker 01

  A hundred percent, yeah. I mean, yeah, the the story I told earlier about secret loyalties, you know, meaning that now we backdoored wide range of military systems. I mean, you can seize power, that's that's one route. And then, you know, again, there's this other with the company and masses, you know, masses amounts of economic power by kind of having a monopoly on AI cognitive labor. And then you're leveraging that leveraging that to get more economic power, more political influence. Yeah, I mean, I do, I do think it's possible. You know, again, there's this big shift once AI can fully replace humans where today, no, one person can never have absolute power. They have to rely on others to implement their will.

• 1:31:24Gus Docker?

  And this is what makes kind of current exists currently existing dictatorships unstable, where there's always a threat of kind of internal revolt or outside factors threatening the dictatorship. But this this could potentially change, yeah.

• 1:31:40Speaker 01

  Yeah, there's always a threat of revolt and then to to guard against that threat, the dictator needs to share their power to some extent, has to compromise. But yeah. You could get it all concentrated in one person with sufficiently powerful AI.

• 1:31:55Gus Docker?

  Do you think we we move through a period of increased threat of AI-enabled coups and then reach some kind of stable state? Or do you imagine that there is a a a constant kind of risk of AI-enabled coups in the future?

• 1:32:09Speaker 01

  I think we move through it.

• 1:32:12Speaker 01

  Yeah, it's it's it's this point about once we have deployed AI across the whole economy, the government, the military, if those AIs are maintaining balance of power, then we could fully eliminate the risk of AI-enabled coup.

• 1:32:28Speaker 01

  You know, it'd just be as if, you know, our whole population was just, you know, so committed to democracy, would never seek power, you know, never help anyone else who who who wanted to undermine any democratic institution. Um, you know, you could, we already have strong norms, you know, favoring democracy, but, you know, they're far from perfect and they have been eroded over recent decades. But you could you could just get rock-solid norms that they're programmed in, you know, they cannot be removed except by the will of the people. I mean, there's a there's a bit of a question because you still want to give the human population the ability to change the AI's behavior and its rules. So the human population could always choose to move to an autocracy. Um, so I suppose

• 1:33:12Speaker 01

  I shouldn't say that we could fully eliminate the risk because, you know, we we we could, we could, we will always have that, you know, democracy, there's always the this point that democracy could, you know, vote to stop being a democracy. But I do think, you know, we could get to a point where it absolutely cannot happen without most people wanting it to happen.

• 1:33:33Gus Docker?

  And so you would get to a point in which kind of future AI-enhanced societies, you could say, are more stable than than current democracies and they're more, they're less at risk of coups than or democratic backsliding than than current democracies.

• 1:33:50Speaker 01

  Much, much more. Yeah, you you could get much more.

• 1:33:55Speaker 01

  Robustness there. I mean, there's this constant dynamic in today's societies where people care about democracy, but they also care about a host of other things, you know, their own achievements, various other ideological commitments. And so, you know, depending on how dynamics play out, depending on how technology evolves and what people's incentives are, sometimes people push against democracy, you know, that that's what the Republican Party has been doing. In some ways, that's what Democratic Party has done as it's kind of increasingly put, you know, pretty ideological people in powerful institutions.

• 1:34:26Speaker 01

  So with AI, you can, you get much more control over those dynamics because you can just, you know, you can just make it much more the case that that that democracy is not being compromised.

• 1:34:40Gus Docker?

  Are there any ways for us to, are there any kind of risk factors we can look at if we are interested in predicting coups? Do you think there's something we can, we can measure or something we can track to see whether we are at risk of, of, of an AI enabled coup?

• 1:34:57Speaker 01

  It's a great question. I don't think I have an amazing answer, but some things that come to mind.

• 1:35:04Speaker 01

  The, the capabilities gap between top AI labs and then the gap again with open source.

• 1:35:14Speaker 01

  The degree to which AI companies are sharing their capabilities with the public and if not with the public, then with, you know, multiple other trusted institutions. You know, like sharing their strategy capabilities with kind of US political parties and

• 1:35:35Speaker 01

  parts of government.

• 1:35:38Speaker 01

  The, the extent of economic concentration, you know, how much, how, how much, what are the revenues and net worth of particular companies, you know, particular AI companies.

• 1:35:55Speaker 01

  So another one, what is the extent of government automation and military automation by AI systems? And when it, when that automation is happening, how robust are the guardrails against breaking the law and guardrails against other forms of illegitimate power seeking?

• 1:36:17Speaker 01

  Um, how much transparency does the public or the judiciary or Congress have into how

• 1:36:27Speaker 01

  dangerous AI capabilities are being used by, by AI companies and by the executive branch? So, you know, take the example of military R and D capabilities, that is, you know, really smart AIs that can design super powerful weapons.

• 1:36:41Speaker 01

  It's scary if companies can just use those military R and D capabilities without anyone knowing. It's also scary if a small group of people from the executive branch can use those capabilities without anyone else knowing how they're using them, because they could be designing powerful weapons and making them loyal to a small group, you know, so transparency in, into these like high stakes capabilities and how they're being used by a, by a broad group, doesn't have to be public, um, probably shouldn't be public, but you know, we have checks and balances already. So, you know, another, another kind of question is, as, as, as these high stakes use cases start occurring or they become possible, do, do we know that there's transparency

• 1:37:25Speaker 01

  requirements in place, you know, as we, as we increasingly see AI companies contracting with Palantir and other military contractors, we can kind of begin to see they're making increasingly powerful, um, weapons. Is there, is there a process of oversight? Do we know that if someone was trying to, you know, make AI military systems that loyal to them, that it would be spotted? That, that, that's another indicator we, we can look at.

• 1:37:55Speaker 01

  You can look at, you know, all the kind of standard democratic resilience indicators that kind of the social scientists have come up with. There's various things about kind of free and fair elections, about civil society, about freedom of press, that, you know, have been getting worse recently in the US. But there's various indicators here. You can look at the degree of government censorship of a freedom of speech or what's on the internet

• 1:38:25Speaker 01

  and the degree of surveillance that the government's doing.

• 1:38:30Gus Docker?

  If you, if you take all of these things into account, yeah, how do you think about the, the risk of an, of an AI-enabled coup in the next, you know, thirty years, say?

• 1:38:41Speaker 01

  Next thirty years?

• 1:38:45Speaker 01

  I, I think it's high. I think the risk is high.

• 1:38:50Speaker 01

  I'd guess it's ten percent or something.

• 1:38:54Speaker 01

  And that, you know, to be clear, you know, if, if it was just existing political trends, ignoring AI, I'd be, you know, maybe a few percentage, maybe on like two percent or something, you know, there is definitely a risk of that. And I'm thinking about the US here.

• 1:39:10Speaker 01

  A big, a big part of my, my current worries are not about the indicators, but it's about my expectation that AI capabilities will keep, will keep increasing quickly and even more quickly. And then the kind of absolute lack of interest in regulating AI companies right now in the US and the, the difficulty that we will have of constraining the executive

• 1:39:40Speaker 01

  under the current situation where, you know, President is using sophisticated legal strategies to increase their own power and is, is succeeding on many fronts. Um, the US is, is not doing a great job at constraining the executive. So, you know, companies are unconstrained. The executive is poorly constrained. There, those are the key threat actors here. So, you know, with fast AI capabilities progress, plus that lack of constraint, lack of transparency, you know, the default is that a lot of those indicators I said get worse. And, you know, none of the indicators get better, like transparency. And so that, that, that makes me think this is this is very plausible.

• 1:40:19Gus Docker?

  Yeah, I mentioned thirty years, but, but what about five years?

• 1:40:23Speaker 01

  Five years, that's tough, isn't it? It's really tough. I mean, yeah, I I think there's a risk. I I wouldn't think there was a risk if it wasn't for the AI research causing an intelligence explosion angle, but AIs are a lot better at coding and cognitive kind of research related tasks than they are at, you know, for example, you know, controlling robots and stuff. And so even if the threat model ultimately comes through robots or comes through, you know, crazy levels of persuasion.

• 1:40:59Speaker 01

  It's just you you you really can't rule out a scenario where, yeah, AI research is is is automated in three years' time. Then in four years' time, we've got super intelligent AI controlled by a few few people. Maybe it's got secret loyalties. Maybe it's being deployed in the government and being overtly loyal to the president. And then, you know, a year later, you know, it's it's backsliding or it's political capture or it's, you know, robot soldiers.

• 1:41:25Gus Docker?

  Yeah. How how do you think about the badness of the outcomes here? How how much does the badness depend on the ideologies of the people who are who are conducting the coup or yeah, what should we look out for? If because, I mean, I I guess we can rank queues by badness, which is not an exercise I think we should actually attempt, but we can we can kind of talk about the factors involved about what would be the worst kind of coup and what would be a a slightly better kind of coup, slightly less bad kind of coup.

• 1:42:01Speaker 01

  Yeah. So we could, you know, we could let's imagine it's one person that sees power.

• 1:42:07Speaker 01

  Actually, no, that's the first distinction to draw. If there's a group, then even ten people is better than one person.

• 1:42:13Gus Docker?

  And and why and why is that?

• 1:42:15Speaker 01

  Yeah, so so ten people, you get a diversity of perspectives, so more kind of moral views represented and there's more kind of room for compromise between those perspectives. It's more room for kind of reasonable positions to win out as there's kind of some some deliberation as as actions are decided upon.

• 1:42:35Speaker 01

  There's slightly less intense selection for psychopaths than if it was just one person.

• 1:42:43Speaker 01

  So, yeah, if it's just one person, that's bad. That's particularly bad. Ten people is still very bad. You know, a hundred people is still pretty bad, but you know, it's that that that there's big differences there, big differences. If if if we're now just thinking about, you know, one person or the average person in the group, then we could think about how competent they are. And then we could say something about kind of how how kind of virtuous their motivations are. Um, well, I do think competency is important. Like, I think it's probably underrated in most political discussions, how important it is to just be really, really, really competent. You know, thinking about something like responding to Covid.

• 1:43:25Speaker 01

  Or thinking about something like, you know, trying to de-escalate a conflict. Russia-Ukraine or trying to de-escalate Israel conflict, like actually just being very competent and very good at getting things done is important. And

• 1:43:42Speaker 01

  as we as we mentioned, if you're just willing to rely on AIs and you, you know, align those AIs in the right way. Anyone could be really competent, but you know, that's not guaranteed. You know, people, you know, may may really want to to cling to their current views without without changing their minds. You know, let's let's take the example of Donald Trump. If a really smart AI assistant told him, look, tariffs are definitely bad for the US economy, they're definitely bad and won't give you what you want. Would would he change his mind? I would I would guess no. So, you know, lots of lots of smart people have already been saying that and

• 1:44:20Speaker 01

  you know, him and his supporters. Yeah, I I don't actually know like the economic details here, but like my understanding is that most people think that they're pretty bad.

• 1:44:28Speaker 01

  And you know, it'll still be the case that, you know, Trump will be able to find people telling him that what he thinks is is good. And he'll be able to program his AIs to keep telling him that if he wants to. So there's no guarantee that that that he will become super competent or that whoever sees power comes super competent.

• 1:44:43Gus Docker?

  So there's this kind of like, there's a, there's a form of loyalty that actually undermines competence. Just because you're, you're not, you're loyal to such an extent that you're not providing feedback that's, that's useful. Because, you know, negative feedback feels bad to receive. And so there's, there's, there's that kind of loyalty. I mean, maybe this is a bit contrived, but do you think there's a sense in which in the singular loyalty scenarios, the AIs could be so loyal that they are, they're kind of undermining the competence of, of the person that is, that, that they're singularly loyal to?

• 1:45:20Speaker 01

  Yeah, it's a really great question. I haven't thought about this, but yeah, in a way, the most extreme version of singular loyalties. We'll just agree with whatever the most recent thing that, you know, the, the dictator has said. It's like, you know, it's a version of sycophancy, which we already see without questioning. And we'll do that even when it's not in that person's interests. Because that's the kind of type of loyalty that's demanded, where there's a more kind of sophisticated type of loyalty, where you're still completely loyal, but you're also willing to challenge them when, when you think it's in their, in their best interests.

• 1:45:59Speaker 01

  So that, that's a really nice distinction. And yeah, I suppose one way of, you know, thinking about competence is thinking about what kinds of loyalties

• 1:46:09Speaker 01

  the dictator would demand from their AI systems. Um, another, another way of thinking about it is how much they would listen to the AI advisor. Like even if the AI has the kind of sophisticated type of loyalty and is trying to tell the dictator what to do, the dictator could just ignore them.

• 1:46:27Speaker 01

  And, you know, you see that again, you know, AIs are fairly sycophantic. They will also challenge you sometimes and then, you know, it's up to you whether you listen.

• 1:46:35Speaker 01

  So that's all the competence bucket, which I think is really important. And I do think there are differences between potential coup instigators on that on that front, which which could be significant. Well, yeah, I guess my expectation would be that that kind of lab CEO coups would be would be more competent than than heads of state. But, you know, even even within lab CEOs, there are some that are more dogmatic than others. And I think that dogma would go in the way of competence.

• 1:47:08Speaker 01

  That's competence. And the other thing I mentioned was kind of kind of broadly, what are your goals? What are your values or kind of more character? And here, you know, what I one thing I think is really important is being open minded, being willing to bring in lots of different, you know, diverse perspectives into the discussion and empower them to, you know, really represent themselves and grow and flourish. So I think a very bad thing would be, you know, a particular person becomes dictator. They implement their vision for society and of much better would be, you know, empower all the different kind of ideologies and ideas to kind of, you know, become the best versions of themselves.

• 1:47:53Speaker 01

  And then, you know, we can kind of collectively grow and improve our understanding of how to how to run society. So, you know, sometimes people focus when they're thinking about values on like, okay, are you are you at this type of utilitarian? Or, oh, no, I hope you're not, you know, the ontologist. Or, you know, it can get very kind of specific and finger pointing. You know, my view is more that, you know, we don't really know what the right answer is. And the most important thing is is is being pluralistic and, you know, letting a thousand flowers bloom.

• 1:48:25Gus Docker?

  So we we discussed the possibility of getting to a stable state in which we've avoided an AI enabled coup. And now we have, say, we have aligned a super intelligence kind of that where the risk of coup is is very low. Do you think this is something that happens for one country and then that one country is is? In control of the world to such an extent that it's that it's this is not a process that other countries are undergoing. To be more concrete here, for example, if the US goes through a risk of of queues of AI enabled queues and but managed to to kind of stay to to remain a stable democracy, is it the case that Russia or China will go through a similar

• 1:49:11Gus Docker?

  period of risk of queues?

• 1:49:13Speaker 01

  It's a great question, and it will depend on the US's kind of posture towards the rest of the world geopolitically, and it will also depend on, you know, whether the US has gained a huge military and economic advantage, you know, like outgrowing the world or just developing powerful military technology,

• 1:49:35Speaker 01

  as we were discussing previously. But you know, you can imagine one scenario where

• 1:49:40Speaker 01

  you know, the US isn't that that much more powerful than the rest of the world yet, and isn't that like kind of inclined to to intervene,

• 1:49:52Speaker 01

  which has been kind of the recent trend. And then, you know, China develops similarly powerful AI a few years later, and Xi Jinping uses it to cement his control over China. So then you now have one kind of AI enabled dictatorship that is extremely robust. And then you have the kind of US, which, you know, has has avoided that risk. And now they're kind of, you know, maybe they're competing against each other and kind of, you know, the Cold War three, I'm trying to kind of Cold War two, sorry, trying to outgrow the world, or

• 1:50:28Speaker 01

  may maybe they're striking deals because, you know, they they recognize this, it's it's not good to compete. And, you know, China just kind of indefinitely remains a dictatorship.

• 1:50:42Speaker 01

  And, you know, that that's just a permanent loss for for the world. But but you could also imagine a different scenario where the US is very far ahead, and maybe, you know, it just wants to really secure its position geopolitically. And so it, you know, it instigates AI enabled queues in other nations where it's really putting kind of US representatives.

• 1:51:07Speaker 01

  Up on top of those nations, that could be through secret loyalties, it could sell systems to, you know, sell AI systems, let's say to India that are secretly loyal to to US interests, or it could give some particular politicians in India access, exclusive access to super intelligent AI to help them gain power. So you could, you know, you could apply those same threat models we've discussed, but with the kind of US pulling the strings. Or you could have the US just kind of taking control of other nations in more traditional ways, you know, just military conquest and kind of really leaning heavily on kind of

• 1:51:48Speaker 01

  extracting economic value out of other countries as they are the world. So, you know, yeah, kind of wide range of options here, really.

• 1:51:59Gus Docker?

  Yeah, yeah. As a final topic here, perhaps we can talk about what listeners can do if they want to help try to prevent AI-enabled coups, and specifically where to position themselves. Should they be in AI companies? Should they be in governments? Should they be in perhaps eval organizations? Where's the position of most leverage?

• 1:52:24Speaker 01

  Great question. I think being at lab is a great place to be. I talked about system integrity, kind of robustly ensuring that AI is not of secret loyalties and behaves intended. That's something that companies need to implement. So if you have interest or expertise in sleeper agents, or backdoors to AI models, or cybersecurity, then I think being part of lab and helping them achieve system integrity is an amazing way to reduce this risk. Another thing you can do at labs, if you're, you know, if you're interested

• 1:53:05Speaker 01

  in, if you're worried about kind of the risk of, you know, heads of state deploying law AIs and seizing power, is you can help labs develop terms of service where when they sell their AI systems to governments, they have certain mitigations against misuse that maybe, you know, one way to frame this is, look, we're using really powerful AIs, and we can't guarantee the safety of those AI systems unless we have some degree of monitoring.

• 1:53:35Speaker 01

  To ensure that the AI systems aren't doing anything unintended.

• 1:53:39Speaker 01

  That monitoring could then be sufficient to allow for the prevention of coups. Because you're being monitoring not only for kind of accidental misaligned AI behaviour. But you know, that will also thereby mean you're monitoring for, you know, a bad human actor giving them illegal instructions. So that, you know, labs will be drawing up contracts with governments, terms of service. They will be thinking about, you know, the guardrails, if any, that go on a place on the systems that they sell to governments. But I think, you know, there's very careful work to be done thinking through, okay, how can we structure those guardrails? How can we explain them in a way which is, you know, very unarguable?

• 1:54:23Speaker 01

  And it doesn't seem like we're kind of trying to like, you know, constrain the government, you know? Private companies don't, you know, it's not really legitimate for them to kind of constrain the government. But, you know, I do think there's an important, an important thing to be done here in preventing AI-enabled coups. So kind of threading that needle. There's another thing you could do in a government, but you could, in a lab. But you could also do that kind of work for a think tank or for a research organisation that's kind of interlinked with the government like Rand. I think potentially do some of this kind of work, thinking about, you know, what should, you know, be in the terms of services between,

• 1:55:01Speaker 01

  between labs and governments.

• 1:55:06Speaker 01

  Let me think, I, I think another big thing is that for, for, for this system integrity. Yes, we want labs to implement it, but we also want there to be some external organisation that can certify that.

• 1:55:19Speaker 01

  Currently, no external organisation is working on this. And Metis not working on it, Apollo's not working, you know, I don't think any evaluations organisation is. But there will be at one point in time, a very strong demand for, for, for guarantees that the systems that these companies have developed don't, don't have C P agents. Because if you're deploying a system in, you know, critical infrastructure and, and, you know, important government roles and the military, you want to know that it does what it says on the tin. And so I think that, you know, a company can, can make a wise investment by developing that capability. The ability to understand how to insert secret loyalties, how to detect them.

• 1:56:01Speaker 01

  Being able to assess a company's process to see whether it's sufficient.

• 1:56:06Gus Docker?

  How how close would that work be to kind of interpretability work? Isn't isn't there overlap there kind of like looking into what is the system actually doing? You know, find kind of reverse engineering some some process to see

• 1:56:23Gus Docker?

  how things how things work. Is detecting sleeper agent like a specific constraint problem?

• 1:56:31Speaker 01

  Yeah, it's a great question. It's a specific type of

• 1:56:36Speaker 01

  problem. So you could use internals work like interpretability to detect sleeper agents. That's not the only method you could use. You could use behavioral tests. You could look at the training data. I think a mixture of those is useful. It's also it's a specific adversarial setup. So interpretability would just often interest in all kinds of kind of fax back the the model's behavior and its cognition.

• 1:57:02Speaker 01

  With with secret loyalties, we're we're specifically concerned with one red team trying to produce a secret loyalty. And then a blue team trying to detect it. And so it's it's adversarial in that, you know, the red, the the blue team has a tough task. The red team will be purposely trying to hide the sleeper agent. And so that that that, you know,

• 1:57:26Speaker 01

  that will raise a whole host of new considerations and important things to dig into. So you can imagine being like allowing the red team, you know, full control over all aspects of the development process. Can they create a secret loyalty? Maybe that maybe they can. Maybe it's pretty hard to test. Okay, but now you only allow them to intervene at one part of the development process. Or you allow the the blue team to look at all the data that they use for training, or you allow the blue team to look at all of the code that they wrote. And you could get potentially quite a sophisticated understanding of the conditions under which the blue team win, conditions under which the red team win. This research doesn't need to be done in a lab. It it could be done by an external organization. I think it's, you know, big, big, missing, missing focus of of of today's technical work.

• 1:58:10Speaker 01

  And ultimately, that that would inform the assessments of of the labs' attempts to do system integrity. So for technical researchers out there, I'd I'd I'd really highlight

• 1:58:22Speaker 01

  that possibility.

• 1:58:25Speaker 01

  Another another kind of piece of work for the right person would be beginning to

• 1:58:32Speaker 01

  understand the kind of existing military thinking around autonomous autonomous systems that this is already obviously a live issue for militaries. They are increasingly deploying AI. It would be nice to marry up that existing expertise with these kind of risks about more powerful systems enabling coups and kind of get to a consensus within that military community of basic principles like law following, like distributed control over military systems.

• 1:59:05Speaker 01

  And, you know, figure out a kind of a military procurement process, which is both practical, but also robustly prevents this kind of stuff. So, you know, if there's anyone listening that that hasn't has a way in, I think, I think that's that's potentially pretty

• 1:59:22Speaker 01

  pretty valuable, although there's also a risk of poisoning the well if it's done badly. So, you know, see with some with some care.

• 1:59:31Gus Docker?

  Yeah, yeah. Perfect. Thanks for chatting with me, Tom. It's it's been great.

• 1:59:36Speaker 01

  Yeah, real pleasure. Thanks so much, Gus.

• 1:59:40Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts where experts talk technology, business, economics, geopolitics, culture, and more, which is now a part of A16Z. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 2:00:25Nathan Labenz?

  And finally, I encourage you to take a moment to check out our new and improved show notes, which were created automatically by Notion's AI Meeting Notes. AI Meeting Notes captures every detail and breaks down complex concepts, so no idea gets lost. And because AI Meeting Notes lives right in Notion, everything you capture, whether that's meetings, podcasts, interviews, or conversations, lives exactly where you plan, build, and get things done. No switching, no slowdown. Check out Notion's AI Meeting Notes if you want perfect notes that write themselves. And head to the link in our show notes to try Notion's AI Meeting Notes free for 30 days.