Freedom of Action: America's ASI Strategy, w/ RAND's Joel Predd, Ben Boudreaux & Matt Chessen

Joel Predd, Ben Boudreaux, and Matt Chessen of RAND's Center for the Geopolitics of AGI join Nathan to discuss their framework for maintaining freedom of action on the path to superintelligence. Analyzing seven strategic archetypes—including dominance, deterrence, co-development, and moratoriums—they argue that deep technological and geopolitical uncertainty makes defaulting to any single strategy dangerous. To avoid foreclosing critical options, their proposed approach calls for investing in long-lead capabilities like model verification, containment, agency evaluations, and an adaptable security architecture. At stake are decisive economic and military advantage, stable competition with rivals like China, and humanity's survival with agency. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/freedom-of-action-america-s-asi-strategy-w-rand-s-joel-predd-ben-boudreaux-matt-chessen/ Sponsors: Lovable: Lovable is the software creation platform that brings your ideas to life with enterprise-grade infrastructure and flexible AI coding agents. Start building your next project today at https://lovable.comVisit fruitopia.ai to explore more than 400 fruits, and more than 8000 varieties, from all around the world. Claude: Claude is the AI collaborator built for problem solvers, extending your thinking across coding, modeling, and strategic workflows. Get started and discover Claude Pro at https://claude.ai/tcr OutSystems: OutSystems is the leading agentic systems platform that enables enterprises to engineer, orchestrate, and govern AI applications on a single unified platform. Learn more and see how it works at https://outsystems.com/tcr Athena: Athena matches you with a dedicated, top 1% executive assistant to handle your inbox, calendar, and daily workflows so you can save an average of 15 hours a week. Get matched with your EA today at https://athena.com/cognitive Parallel: Parallel provides enterprise-grade web search APIs for AI agents, offering the optimal balance of quality, speed, and cost. Get started for free at https://parallel.ai/tcr CHAPTERS: (00:00) About the Episode (05:21) Sponsor: Lovable (07:05) Taking superintelligence seriously (12:54) Seven strategy archetypes (Part 1) (19:18) Sponsors: Claude | OutSystems (22:34) Seven strategy archetypes (Part 2) (31:37) Decisive strategic advantage (Part 1) (38:13) Sponsors: Athena | Parallel (41:05) Decisive strategic advantage (Part 2) (46:57) Lessons from nuclear history (01:00:50) Building the security architecture (01:09:33) Costs and decision criteria (01:18:42) Deterrence and international signaling (01:28:00) Preserving human agency (01:34:22) Setting AI red lines (01:41:04) Key indicators to track (01:46:35) Episode Outro (01:48:52) Outro PRODUCED BY: https://aipodcast.ing

• 0:00Speaker 00

  Hello and welcome back to The Cognitive Revolution. Today, my guests are Joel Pred, Senior Engineer and Director, Ben Boudreau, Policy Researcher and Professor of Public Policy, and Matt Chessen, Resident Technical Expert at the Center for the Geopolitics of AGI at the Rand Corporation. Together with co-authors, they've recently written a new Rand Perspective paper called "A U.S. Strategy to Secure Geopolitical Advantage on an Uncertain Path to Superintelligence: Maintaining Freedom of Action." This is one of the clearest pieces of AI strategy writing that I've encountered. It's both a great introduction and orientation tool for people who are just now entering

• 0:45Speaker 00

  the AI strategy debates, and a useful organizing framework for anyone who's been frustrated by how many AI strategy conversations consist of smart people talking past one another due to differing but often unstated assumptions about critical questions. Their first contribution is a taxonomy of seven archetypal strategies that aim to achieve distinct coherent win conditions, and the five pivotal uncertainties that will need to be resolved before the best strategy can be identified. The seven strategies, which I think will be familiar to Cognitive Revolution listeners, include acceleration, aka the A16Z Plan to let technology take its natural course; dominance,

• 1:29Speaker 00

  aka the Dario Plan, where the U.S. builds such an advantage that it can dictate terms to other countries; no-development, which you might think of as the Demis Plan for international cooperation; preparedness for coexistence, which seems to be OpenAI's plan, as far as I can tell; a moratorium, aka the PAWS Plan; deterrence, which is something like Dan Hendrix, Alex Wang, and Eric Schmidt's Mutual Assured AI Malfunction Plan; and finally, investments in what they call "continuity of society," which might be best described as prepping for an inevitable AI doomsday.

• 2:09Speaker 00

  Meanwhile, the five uncertainties are: 1. How close are we to extremely powerful and dangerous AI systems? 2. Is it feasible for humans to coexist with such systems at all? 3. Is it possible for one actor, which could be a country or perhaps even a private company, to use these systems to achieve a decisive strategic advantage over its competitors? 4. Can humanity come together to cooperatively restrain itself from developing such dangerous systems? And 5. If not, can the United States or a coalition of governments prevent the development of such systems, including by force if necessary?

• 2:50Speaker 00

  The core argument, as you'll hear, is that because there is no consensus on these critical questions, even as AI capabilities continue to advance at a blistering pace, the United States government should invest now in all of the foundational technologies, institutions, and other capabilities that the various strategies would require, such that we can keep our options open for long enough to get the answers needed to make a wise, high-level strategic choice and ultimate commitment. Personally, I would argue for co-development by default with a moratorium as a fallback in case it does become clear that we simply can't continue to develop AIs safely. But I do feel that this recommendation is a very good one for a national government

• 3:34Speaker 00

  that's just now beginning to understand the nature of AI and remains, for a mix of good and bad reasons, very doubtful that it can trust its rivals to cooperate on such a high-stakes issue. Along the way, we also discuss how the politics of AGI have shifted in DC over the last couple of years, how AGI-pilled the government is today, how realistic a decisive strategic advantage really is, the similarities and differences between the strategic dynamics of the current AI situation and the Cold War nuclear arms race, the magnitude of the investment they're recommending, which they are working to quantify in upcoming work on the cost and lead times for 106 distinct capabilities that they've listed in a security architecture

• 4:18Speaker 00

  catalog, and finally, what we can do, both in our organizations and even just as individuals today, to maintain as much agency and control as possible even as we seek maximum leverage from AI in our work. This conversation and the Perspective paper itself stand out for abstracting away the many messy and often confusing details of week-to-week AI developments, and focusing our attention on the most critical decisions the United States will need to make, as well as the core reasons that it remains for now premature to commit to any single strategy. So, without further ado, I hope you enjoy this clarifying synthesis of AI discourse

• 5:01Speaker 00

  and simple, high-level argument that the U.S. government should do everything it can now to maintain its freedom of action on the path to superintelligence, with Joel Pred, Ben Boudreau, and Matt Chessen of Rand's Center for the Geopolitics of AGI. Today's episode is brought to you in part by Lovable, the software creation tool that brings ideas to life. I think we can all agree that any positive future for humanity includes eating more fresh, delicious, in-season fruit. For years, I've thought someone should build the ultimate guide to fruit, cataloging the thousands of varieties that most of us have never tried, where and when they grow, what

• 5:44Speaker 00

  they actually taste like, and where you can get them. Finally, I realized that that someone could be me, and now I'm building it on Lovable. It's called Frutopia. Tell it where you live and what you like, and it'll show you what's in season near you this week, plus the most interesting fruit from around the world that can actually be shipped to your door. It has a really cool what-it-tastes-like spectrum, countless recipes, names of fruit in many different languages, a guide to fruit festivals all around the world, and an AI assistant to help you explore. You probably know Lovable as a pioneer of AI vibe coding, but I've found that Lovable's real superpower is its enterprise-grade infrastructure: scalable hosting and databases,

• 6:29Speaker 00

  a secure auth module, and the weekly job that checks the latest inventory is all handled on the backend, which frees me to spend more time on the product experience. And because Lovable has an MCP server, I can use any coding agent I want to drive the build while Lovable keeps everything running. Visit fruitopia.ai to explore more than 400 fruits and more than 10,000 varieties from all around the world. And next time you have an idea that really should exist, stop dreaming, go to lovable.com and bring your idea to life. Joel Pred, Senior Engineer and Director, Ben Boudreau, Policy Researcher and Professor

• 7:09Speaker 00

  of Public Policy, and Matt Chessen, Resident Technical Expert at the Center for the Geopolitics of AGI at the Rand Corporation, welcome all of you to The Cognitive Revolution.

• 7:19Speaker 01

  Great to be here. Thank Nathan for having us.

• 7:21Speaker 00

  Yeah, I'm excited about this. So, I have been looking through what I think is a really clarifying document that you guys have put together. It is called "Maintaining Freedom of Action: A U.S. Strategy to Secure Geopolitical Advantage on an Uncertain Path to Superintelligence." And I think we will benefit a lot, and the audience will benefit a lot, by just walking through this. I find it to be unusually well-crafted and just kind of easy to absorb, and so I think just going through it in a pretty linear way will be well worth our time, and then we can potentially expand to some other topics, and then maybe I'll push you guys on some of the recommendations that you make as well. Maybe just for, to start as a little bit of context before we get into the strategy

• 8:05Speaker 00

  document, one question that's always kind of in the air a little bit is like, where are we right now on the path toward governments around the world taking AI as seriously as I think probably all four of us think they should? I've seen a lot of signs that I didn't expect to see this soon, but I also see a lot of indicators where I feel like, eh, maybe they're not so serious after all. It's a little bit hard for me to read from the outside.

• 8:32Speaker 01

  So, I'll take the first swing at this, and then you may have sharper answers in this, but here we go. It is quite remarkable, the moment we're in, when the Center for the Geopolitics of AGI was launched two and a half or three years ago, it was such that we kind of had to whisper "AGI" for fear of being thought of as too out there, too consumed by science-fiction scenarios. It didn't deter us or scare us from doing the kind of work that we've been doing, which is to think about the implications of frontier AI that could advance to that threshold, but nonetheless, that was the climate. And it is quite remarkable that now you can't get into a meeting without calling it "superintelligence." So, times have changed.

• 9:14Speaker 02

  The other thing that we, one of the things that we always say in this space, mostly with the reference to the technology, is that we should be focused on the trends, not the timestamps. And the trends are quite remarkable, as we all have observed over the last two and a half years in the pace of progress in the technology. But I think one of the things you're observing is that the trends are also, you could say, somewhat favorable, even if they're way too slow, when we think about the governance of this technology. We have governments waking up and paying attention to it. We have the firms who are building the technology, talking about governance solutions. We have the civilians, the politicians are forming our public debate now. So the trends are very rapid, or even accelerating even in the governance space. Are they fast enough?

• 10:00Speaker 02

  It's very difficult to keep pace with this rapidly advancing technology. And we have spent a lot of time over the last two and a half or three years, looking for the folks that we're taking it as seriously as necessary. And I regret to say you, to tell you that there aren't nearly enough as there should be.

• 10:18Speaker 03

  I'd just add that I think we are seeing lots of signs that the US government has become more aware of the type of strategic importance that this technology may bring. But it's not, I think, clear that the US has settled on a strategy, nor, I think, have the frontier companies. And that might be challenging, because many capabilities that we might need, like verification technology, or technical expertise in government, you know, evaluation capacity, things like crisis response and communication, those all may take quite a long time to build. And by not proactively building many of those capabilities, while capabilities advance

• 10:59Speaker 03

  and various deployments continue, that itself might be a type of choice that forecloses options we might need down the line. So I think we're seeing signs of greater awareness, but not yet signs of truly adapting and trying to proactively get ahead of the type of transformations we are likely to see.

• 11:19Speaker 04

  Yeah, I think that's a great bit of foreshadowing for the strategy document as a whole, because, spoiler, it kind of leads up to this idea of building the capacity to commit to a strategy, once we sort of see a couple more cards turned over and have a better sense of what actually is the nature of the technology and the challenges that we're dealing with as a result of its development. And I think fundamentally, we just do still have like a lot of uncertainty as to like, what are the nature, what is the nature of the technology? What are the challenges that are going to be most important? And, you know, it sort of strikes me that like, in a way, what you're recommending

• 12:03Speaker 04

  ultimately is kind of like, launch a lot of subagents in parallel, US government, so that they can go out and prepare for a lot of different eventualities. Before we get there, though, let's maybe start at the beginning. So the kind of first big thing I took away from the document is, again, a really clear taxonomy of seven archetypal strategies that people are throwing around. I think listeners to this feed will like have heard variations of them before, but I hadn't ever quite seen them enumerated, you know, in such a succinct way. So maybe let's just start off with a quick review of like, what are the archetypal strategies that should at least be on the table for consideration?

• 12:48Speaker 04

  Then we'll get into the uncertainties that will drive the decisions as to which one of those will ultimately be the right strategy.

• 12:55Speaker 02

  So our work started with trying to answer one question, which is, which national strategy is the best one for managing the transition to superintelligence? And the bottom line upfront there is, we believe there's just too much uncertainty right now, and none of the archetypal strategies for superintelligence can be justified, because if you commit prematurely to any one strategy, it forecloses other strategies and doesn't build the capabilities that might be needed later. And so we think the best way for the US to secure geopolitical advantage and ensure humanity's survival with agency is by executing a freedom of action strategy. Now, this started with a paper we put out, where we looked really at the fundamentals of what does superintelligence actually mean for humanity? And this is a paper called Finding Common Ground in AGI Strategy Debates.

• 13:38Speaker 02

  In that paper, we came to the conclusion that there is a diagnosis and prescription we need to focus on for superintelligence. And the diagnosis is that superintelligence is going to be transformative, and the stakes include decisive geopolitical advantage and humanity's survival with agency. Now, what do we need to do about that diagnosis? We need to build an ecosystem in which AI and humans can coexist and humans can maintain agency, and we need to implement a security architecture that keeps the world stable enough to build that ecosystem. And so in analyzing all these different strategies out there, we looked at all the strategies that exist, we looked at all the security architectures, all these proposals for international organizations and the like. And we really found that focusing on any one of these right now, we think is really unwise.

• 14:21Speaker 02

  And we found that each of these strategies can be represented by one of seven archetypes. Those archetypes are dominance, and so this is where you really lead at the frontier, you suppress rivals, but this could lead to extreme power concentration and value lock-in. There's a strategy ranked co-development, and this is the idea where you would have a US-led consortium that probably needs to include China, you pool, compute, you have some sort of layered verification, and you jointly develop superintelligence together. This strategy also has a pathology, again, the dark side of what happens when the strategy is functioning well. And this is where the founders' values really become the default, and the consortium can become an arena for international competition. There's a strategy called Preparedness, which is much more focused on informal coordination,

• 15:06Speaker 02

  you know, resilience. And the problem here is that you, when you're starting muddling through, you can normalize accumulating a lot of damage. There's a moratorium strategy, so this is one people are talking about a lot now, some sort of verifiable global halt or pause. You know, the pathology here is that enforcement can become global surveillance. The type of regime you need to maintain these go-kid caps could be quite intrusive. There's a deterrence strategy, which is where we halt our own frontier, and we also suppress rivals' programs and make sure that nobody achieves superintelligence, because it's too dangerous. And this has a pathology as well, where you just get into this eternal standoff that really never quite resolves itself. And then there's this strategy we call continuity of society, and this is, you know, you get

• 15:51Speaker 02

  to a point where you realize that superintelligence is dangerous and it's inevitable, it's going to be diffuse, and so we're just going to have to learn to live with it. And this is focused on really a resilient strategy, but it could lead to a situation where humanity really survives as a relic and doesn't have a lot of agency. And then there's the acceleration strategy. And this is where if you just believe that danger is not close, then you're going to tend to want to accelerate to maximize the benefits and maximize your competitive advantage. But this also has a potential pathology where the harms, you know, markets basically don't price in the harms, and we become very dependent on the systems we don't control. So we revealed these seven archetypal strategies by developing a classifier.

• 16:33Speaker 02

  And this classifier basically reveals these sometimes unstated assumptions that underlie superintelligence strategies. And this classifier, which you can take a look at in the perspective we pushed out, it asks five questions. The first question it asks is, is danger close? Are we close to superintelligence that could be dangerous? Second question is, is coexistence feasible? Can humans actually live with advanced forms of artificial intelligence? Third question is, is restraint feasible? So can the United States or a coalition of countries actually, you know, cooperate to restrain their progress on AI? Fourth question is, is a decisive strategic advantage reachable? Can one country proceed and, you know, get recourse to self-improvement and sort of

• 17:17Speaker 02

  launch itself ahead of all other countries? And then the last question is, is suppression feasible? And this is the idea of, can you actually use advanced AI to suppress other people from sort of proceeding forward to dangerous levels of superintelligence? And so the answers to all these questions reveal the strategic preferences and really result in, you know, these archetypal strategies. And so just as an illustration, you know, if you believe that danger is far off, then you'll push for acceleration, because, you know, danger is far away, so you can actually maximize the benefits and your competitive advantage. Whereas if you think danger is close, but you're, we can't coexist with superintelligence, it's just too dangerous, we don't have alignment figured out, we don't have control of techniques

• 17:58Speaker 02

  figured out, but we can restrain ourselves, so we can cooperate with others to restrain ourselves, well, then you might naturally favor some sort of moratorium, where you actually buy time for that coexistence research to catch up to capabilities. On the other side of the pivotry, if you think coexistence is possible, and you can get a decisive strategic advantage, you might favor a dominant strategy, where you push ahead and try and get some sort of monopoly. The problem is, with these five pivot factor questions, there's really no consensus on the answers to these questions, and there's no evidence that could produce consensus on the timeline these decisions demand. You may not just be able to agree, even if the evidence actually is there. If you pick wrong, you're going to foreclose strategies you might need later. So if you pursue some of these more confrontational strategies, like dominance or deterrence,

• 18:44Speaker 02

  you can destroy the trust that you might need if you have to pivot to a cooperative strategy later on. Likewise, if you pursue acceleration, it can underdevelop all of these capabilities that you need if superintelligence just turns out to be extremely dangerous. So right now, we're currently at a place where the US doesn't really have a declared superintelligence strategy, the evidence doesn't conclusively support any of the archetypal strategies, and we may be running an acceleration strategy by default. And meanwhile, the clock is ticking, and capabilities we might need for any superintelligence strategy are not getting built.

• 19:13Speaker 00

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 19:19Speaker 00

  Today's episode is brought to you by Anthropic, makers of Claude. To show off Claude's latest capabilities, I asked Claude to take the lead in making this ad.

• 19:30Speaker 00

  Given a short creative brief, Claude, using Opus 5.5, studied my past Claude ads, wrote this script, directed a clone of my voice, and designed and coded the animation you're watching. My job: give feedback and decide. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. And with Claude code routines, it keeps working in the cloud even after you close your laptop.

• 20:12Speaker 00

  For problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. claude.ai/tcr.

• 20:35Speaker 00

  This episode of The Cognitive Revolution is brought to you by Outsystems, the leading agentic systems platform. We've all seen the headlines. Companies are pouring money into AI. But the big question on every executive's mind right now isn't just how fast can we adopt this? It's where is the ROI? We're seeing a real trend toward AI chaos. You've got teams deploying standalone coding tools, random agent builders, and experimental scripts. It sounds innovative, but in reality, it's creating a massive headache. Fragmented tools, ungoverned data, serious security blind spots, and costs that are spiraling out of control. If you don't bring those agentic applications under control now while they're still embedding

• 21:19Speaker 00

  into your core processes, you are looking at broken systems and damaged customer trust down the road. That's where Outsystems comes in. Outsystems is the leading agentic systems platform for the enterprise. Instead of managing a patchwork of disconnected tools, Outsystems lets your team engineer, orchestrate, and govern your entire agentic ecosystem on one open, unified platform. It's built for the speed of AI, but with the reliability and security that enterprises actually require. We're talking about real results, like KeyBank, who used Outsystems to deploy a customer-facing app that delivered 75% faster onboarding times.

• 22:01Speaker 00

  Or the global logistics leaders who built agentic systems to completely eliminate their engineering bottlenecks. You don't have to choose between speed and control. Whether you're a small team or a massive enterprise, Outsystems helps you engineer, orchestrate, and deploy agentic systems that actually scale. Stop chasing the hype and start owning your agentic future. You can see how it works and learn more at outsystems.com/tcr. That's outsystems.com/tcr.

• 22:34Speaker 04

  So I know one of the big purposes of a document like this is to go brief policymakers in various venues and try to help them adopt this framework for thinking through these issues. I'd love to hear, and obviously I know you can't, it'll be Chatham House rules at a minimum in terms of what you can say, but I'm interested in kind of, when you go around and do this, what intuitions are you hearing from people on the five questions? And how are those changing? For example, I would think that like, is danger close? Has probably had a pretty significant vibe shift over just the last few months.

• 23:17Speaker 04

  But I'd be interested in kind of all five. What was sort of status quo not that long ago? And how is it shifting as we're seeing these like colorful alignment incidents and other headline-making developments, Millennium Prizes, and so on and so forth? I'm very unclear from my position, well outside of the halls of power, as to like whether these things are really changing where people started and how much these things are really changing people's minds.

• 23:49Speaker 01

  Well, first, I just want to acknowledge the importance of the question you're asking, which is that in an environment that is subject to this amount of uncertainty and this amount of disagreement, monitoring people's beliefs is very important, particularly in a very complex and changing geopolitical environment. So simply observing how individuals, people, and powers use on these questions is changing will be indicative of the kinds of instabilities that we could confront as actors pursue different strategies. The second point I want to emphasize is oftentimes when we talk about this, we emphasize the uncertainty. But one of the points that Matt has made that I find very important is just irrespective of the uncertainty, which we actually may disagree on even the uncertainty, is that

• 24:32Speaker 01

  we just have a very different, we simply disagree on these points. Like you or I may have a different take on how close we are to danger, whether we could conceivably enter into kind of negotiating restraint, whether we could fundamentally coexist, et cetera. But as Matt said, it seems unlikely that the evidence is going to persuasively convince everyone that we'll be needed. So this is just reinforcing, I think, the importance of your question. I think to the extent that there is any shift on this, it's primarily on that first pivot factor. And Matt and Ben, you may disagree, but this is the factor that I think is the one where we might see the most movement. Whether it's, whether anyone who's in a position of power is changing on that variable,

• 25:15Speaker 01

  I'm less certain, but it's the one that's getting the most attention. I think that the fairly, the more technical question of whether we could coexist with superintelligence, and it's interesting that we've chosen coexistence as a word because we're explicitly avoiding the word alignment, because it kind of boxes into a specific set of solution techniques, and we think the more fundamental concept is coexistence. I don't know that there's been as much movement on that one. I think that's such a difficult question to answer that we're not seeing a lot of movement on that. And I think that the question of negotiated restraint is so caught up in a complex domestic

• 25:57Speaker 01

  and international political environment that the levels of trust are, seem so low that if there's been any movement, it has probably been to grow more skeptical about that. Matt and Ben, you should weigh in on your own team.

• 26:11Speaker 03

  I guess what I would want to emphasize here is our focus throughout this work is on preserving strategic optionality or the capacity of the US to make a smart decision, a choice about which of these strategies to pursue. And what I like about that emphasis is it's trying to make clear that we still have some agency here to make a choice and that some of these pivots, the way they're phrased, make it sound like they're simply exogenous, things happening to us, when I think in reality, and what this is hopefully trying to demonstrate, is that there are choices that we could be making to actually, for instance, push danger further off or make restraint more feasible or perhaps develop a certain types of diffusion capabilities that actually enable a more decisive lead.

• 26:56Speaker 03

  And that, again, these are more endogenous than the, just a glance at the pivot table might suggest. And that what we, I think, are seeing decision makers do is starting to grapple with what they might be able to do to ensure that danger is farther off, that coexistence might be more feasible. And part of the overarching approach is thinking about across the strategy space, suppose you want to do all those things, what are the things that decision makers can build now to ensure that they have, they're using their agency now to preserve it for later as well? And so that's sort of the shift that I'm seeing is more of a recognition that the AI development and diffusion is not just something happening to us, but actually something that we can shape.

• 27:38Speaker 03

  And I think discussions about pacing or even the morally binding commitments that were recently announced by the White House and the like, show that there are things that can be done to actually help us navigate through these pivot factors.

• 27:52Speaker 04

  We should repeat that phrase, morally binding commitments, as often as possible.

• 27:56Speaker 03

  I love it. And I will, I will, I will keep tying the frontier companies to their moral responsibilities.

• 28:02Speaker 04

  Yeah. Matt?

• 28:04Speaker 02

  Yeah, I would add, I think that this question of is danger close is actually one of the most important questions you can ask in AI policy right now. You know, I think a lot of governments are now very awake to the opportunities of AI. I think they're waking up to some of the dangers. I don't know that anybody quite has a handle on what the strategy should be for the path to superintelligence. And so that's where I think our strategy provides people a lot of options and the ability to have a framework for thinking about strategy. But this question about whether danger is close, you know, when you talk to people at the frontier companies, you talk to people at some of the organizations that are closest to AI, I think they're very clear that danger is close, right? And that this is something that needs, we need to be paying attention to.

• 28:49Speaker 02

  And when I talk to them about this, I ask the question, you know, if you think danger is close and you think there is a lot of evidence around this, why is it not getting translated well to policymakers? Why is leadership sort of not awake to the dangers quite yet? And I think this is a gap that the AI policy community needs to address. If that evidence exists that danger is really close, I think that the AI policy community has to do a better job of packaging that up in a way that a policymaker can really sort of take that onboard. You know, policymakers have lots of different things in their portfolio. They might not be an expert in AI. They need it packaged up in a way that they can actually absorb this and update as different information develops. We can't just wait for crises. I think the hugging face incident was sort of a mini crises.

• 29:34Speaker 02

  And if we're going to rely just on crises to sort of raise that level of awareness, I think we can do better in the AI policy community.

• 29:41Speaker 04

  So the danger is close one, people seem to have moved on. I think, Matt, you said a minute ago, like, if we don't think danger is close, we might be playing an accelerationist strategy by default. That seems to be sort of falling at least somewhat out of favor. Although, you know, as you well know, there's not a consensus yet. We're talking on the day when Muse Wonder was just released, not to forget Muse Wonder Full. And it is coming from a company that doesn't seem to be too concerned about any near-term danger and certainly doesn't seem to be inclined to engage in much pacing other than doing whatever kind of baseline consumer safety testing they're known for in all their products,

• 30:26Speaker 04

  which we should, I think, have full confidence in. But that was somewhat of a joke in case that was obvious. I'm, I would have said that it seems like we're playing a dominance strategy by default because we've like been in this sort of chip denial mode for a couple of years. And as you may know, I'm like definitely a skeptic of that policy. One of the reasons I'm a skeptic and one of the things I'd love to get a steel man from you guys on is decisive strategic advantage or decisive lead.

• 31:01Speaker 04

  Is that really possible? Who makes the best case for that? I always look back at Machines of Love and Grace. And I'm like, boy, you skipped a lot of points there along the way, Dario. From, you know, all the, I love the first half, all the diseases we're going to cure and the better mental health and, you know, all these new genius outputs. But then it's like, and something, something eternal 1991 will like put China in a box, make them an offer they can't refuse. That's all we need to say about that. Like, best thinking on what a decisive strategic advantage could plausibly look like? Who articulates that best? Sorry if that's an impossible question.

• 31:38Speaker 01

  So, one of the reasons this is a challenging question to ask is, is that, is you have to answer the scope of the decisive strategic advantage. And I think when you talk to people that want to advocate for that strategy, they often may have a different source of advantage in mind. For example, oftentimes it brings to mind a decisive strategic advantage and enduring lead in the technology itself. And then in that case, you are, you are proud to ask the question of whether do you merely mean that you maintain a consistent lead? Or are you preventing your rivals from ever crossing some specific, hard to identify threshold?

• 32:20Speaker 01

  Given that this technology is driven by economic forces as much as technology forces, it does seem a little bit hard to believe in some cases that you can prevent an adversary from crossing a given threshold, although not impossible. That is, that does seem to be a hard bar that may be the one that you're thinking about. But given the pace of things, it feels less difficult to imagine that you could maintain like a steady lead over time. But there's other domains in which you maintain a decisive strategic advantage economically. And here the question is whether you can translate your leadership in the frontier to consistent economic gains that would underwrite growth in your economy and scientific innovation.

• 33:01Speaker 01

  Interestingly here, the, the things you would need as a society, as a nation to gain, gain, maintain a decisive economic advantage, as our colleague Toby Sitzman has written about in a separate paper, we, we, we could refer you to, are quite a bit, quite a bit different. They aren't just the technology, underpinnings of the technology. But then in another domain, as you would imagine, will be a decisive strategic advantage in the military domain. And not unlike the economic sources of advantage, you would need to translate this into weapon systems and concepts of operation and a strategic doctrine, all of which don't come for free. Now, to the extent that a decisive strategic advantage in an, in an era of very advanced forms of AI kind of mean all of these things at once, that's where the, where the nug is.

• 33:46Speaker 04

  It just seems super, super hard to me. Like, I always kind of, I actually had a chance to play a tabletop exercise, you know, oh, but pretty fast takeoff essentially scenario. And I got assigned China. And I was sort of, like most players in the game, at the end of it, we were kind of like, well, that all happened really fast. And we kept sort of being a step behind events. And, you know, as in these, as usual, in these table, tabletop exercises, we were sort of trying to simulate what we thought the real actor would do. So I do feel like there is at least some chance that everybody will kind of be just

• 34:27Speaker 04

  paralyzed by indecision to the point where we get to some outcome before anybody really plays their cards. But then I also did feel like, as China, I have a lot of cards to play. And it was really my restraint and my kind of, you know, desire to like seek a constructive deal first before I like really start playing super hardball that was limiting me. I guess I, I just have a hard time seeing how it's realistic that we can tell, you know, and I don't really know what the win condition is supposed to be. I mean, in Machines of Love and Grace, it really does sound a lot like regime change at the end. You know, the, I think the sentence is almost verbatim, like China will convince China

• 35:09Speaker 04

  that it's better to give up competing and, you know, instead like join our coalition to share in the benefits. That doesn't sound like the sort of thing where we're, you know, making nice. It sounds more like we're kind of, again, giving like an offer they can't refuse. I just, it feels like you can only get there to me if timelines are like super short. So I guess maybe the, a question there is like, to what degree does the plausibility of a decisive advantage depend on a super fast, like FUM-like scenario where the gap kind of widens long enough to like take some pivotal action before the other side can even do, you know, nor like, things I'm considering, right?

• 35:53Speaker 04

  As playing China, like the, all these TSMC fabs, you know, it's, they're awfully nice. It'd be a shame if something happened to them, right? They're like right in my corner of the world. How do you think about like, what determines this? Is it about timelines? Is it about something else? I just still really struggle to see it.

• 36:09Speaker 03

  I can try to say a few things here. I mean, I think Joel already distinguished ways in which decisive advantage might appear differently in different domains. And so the US might have decisive advantage in one domain while China has it in another. And this, it is trying to roll them all up, but that might be not possible, or at least they might not all be commensurate in, in some like clear way. But I think your question also suggests that, you know, the dominant strategy might really differ based on what kind of temporal windows we're talking about. I mean, it might be really accelerated by a, by a FUM type of fast takeoff that would really extend, you know, a leader's lead over the rest. But of course also in a FUM, you need to worry that you can maintain control, right?

• 36:50Speaker 03

  And decisive advantage is only good insofar as the, the, the leader can continue to lead the AIs. And, you know, that might also be in jeopardy in a fast takeoff situation if, you know, who, the, the governing leader there is not prepared to, to control and steer the technology in ways that, that serve its interests. So, you know, having more time might also be necessary to really ensure that that decisive advantage can be implemented, right? And, and again, the FUM case might, just might not give you enough time to, to build in all the control and steerability mechanisms you might need to really leverage it. But this also makes me think that dominance, like some of the other strategies, might not be something that even if you commit to it and execute it, that you can ensure

• 37:34Speaker 03

  will be sustainable over the long run. And it might be the sort of case that, you know, the US is, is pursuing a dominant strategy, maybe even achieve dominance for a period. But at, at a certain point through, you know, model theft or distillation or diffusion or many of these other processes, you know, you might, you might need to pit it, might need to rebalance or, or, or move to a different type of strategic paradigm. And, you know, that this choice among the strategies might, it can never actually be just fully complete for all eternity. And it's going to be perhaps a continual process of adaptation as the technologies change and the geopolitical landscape changes.

• 38:09Speaker 00

  Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Athena, the executive assistant company on a mission to improve how people work and live. If you want to increase your impact, you have to free up your time. And that's what Athena does best. They match you with a dedicated, full-time, top 1% executive assistant who can take over your inbox, calendar, travel, and everything else that's quietly eating up your week. Athena is SOC 2, Type 2 certified, so you can rest easy knowing that your sensitive data is in good hands.

• 38:50Speaker 00

  And as a former AI advisor to the company, I can personally vouch for how much they've invested in AI tools and training. In fact, one of the very best AI users I've ever met is an Athena client who delegated the exploration of AI tools and the development of AI workflows to his EA. Athena clients report saving an average of 15 hours a week, and the average client refers more than two friends a year. That, to me, checks out. I was an Athena client while running my startup, and to this day, I continue to refer friends. Go to athena.com/cognitive right now and get matched with your EA.

• 39:33Speaker 00

  That's athena.com/cognitive. Give yourself back a few hours this week. Go to athena.com/cognitive and see who they'd pair you with this month. Today's episode is sponsored by Parallel, where agents find answers. Most engineers today closely follow new model releases, but don't pay nearly as much attention to their agent's most important tool, web search. If you're like me, your agents are often conducting hundreds of searches per day. And while the unit cost is small, over time, this does start to add up. Before starting to use Parallel, I calculated that my agent's search bill would total roughly $500 this year.

• 40:15Speaker 00

  The good news is that I just had my agents conduct a systematic test, and I found that Parallel's fast mode, which costs just $1 per 1,000 queries, worked just as well as my previous default provider. And switching to it will save me roughly 80% of my search bill going forward. Parallel's infrastructure is enterprise grade, and their suite of APIs offers a range of Pareto optimal options that allow you to choose the right balance of quality, cost, and speed for your needs. So whether you're building voice agents that need 200 millisecond latency or long horizon agents that need deep research, adding Parallel to your agent's toolkit is a no-brainer. Get started for free at parallel.ai/tcr.

• 40:59Speaker 00

  That's parallel.ai/tcr.

• 41:04Speaker 04

  So how about, just to lay my cards on the table, I'll say that I feel like if humanity were wise and we were really up to this challenge, it seems very intuitive to me that the right strategy would be some sort of codevelopment with a moratorium as basically the fallback. In other words, hey, we don't know if danger is close or far, if it's close, and if there are scientific problems that prove very difficult to solve, then we're going to need to agree on some sort of moratorium. And so let's work together until such time as we have clarity on that, and then we can move into that mode if we need to.

• 41:43Speaker 04

  It is sad to me that this seems to be, as you discovered, Joel, I think, something people are growing more skeptical of over time. I don't know to what degree you guys want to put a thumb on the scale in terms of what you think, you know, humanity would do if we were wise. But I'm interested if you would generally agree with my take there. And then, you know, I'm interested to unpack like, why is, given that it seems like, if anything, it looks like danger is more likely all the time. Why are we not trying to figure that out? And why are we instead growing more skeptical of it?

• 42:17Speaker 02

  Yep, I'll, I'll jump in on this one. So I would agree that if humanity was united and coordinated, that, you know, some sort of co-development strategy with a moratorium as a fallback might make a lot of sense. But the fact is, humanity is not a unitary actor. We're not, and there's very real differences in our political systems between authoritarian governments and democratic governments. And so, you know, if you are betting on co-development or betting on moratorium, you're betting on restraint feasibility. And you're betting on the other side actually agreeing to and abiding by the rules that are set up for these restraint systems. I think one of the big concerns for me, because we do have this difference of, of

• 43:00Speaker 02

  values and perspective on the right political systems, is that if you go with some sort of co-development strategy with China, you're really locking in non-democratic founding values. And so you're really locking in those authoritarian values as persistent for humanity over time in AI systems that are being built. And so I think we really need to think about whether that's something that we're interested in. Do we want to lock in those values? Or do we think that enshrining democratic values in AI systems is actually a better outcome for humanity?

• 43:30Speaker 01

  Can I come back to your dominant strategy question? We left some work on the table there. We've started, we had, there's a paper that I would refer you to by our colleague Phil Root that's on our website that's tried to do the game theory on some of this. And it begins with a premise that the race to these advanced forms of AI is a kind of prisoner's dilemma. And under some reasonable assumptions about what the game looks like, one of the initial insights is that the competitive, in a competitive environment, the default strategy that you are led to is one of sabotage over cooperation. That is to say, even though some form of cooperation produces higher collective gains, over time, both players get into an equilibrium in which mutual sabotage of each other's AI progress is the equilibrium strategy.

• 44:16Speaker 01

  Now, one of the things that disrupts that, however, and which leads to a dominant strategy in which AI is used to create something that we have called a wonder weapon in some of our other work. That is, in which AIs as an engine of scientific and military innovation creates a new capability that allows us to hold others' AI programs at risk. And if AI or superintelligence, as it were, were to yield such a capability, this is when, this is the circumstance in which a dominant strategy would work. And so one of the reasons to, one of the reasons that I think your intuition about how unbelievable that is, is because you're stipulating a capability that would give, would be given birth by the AI itself. But that does end up being the equilibrium strategy under those assumptions that we're checking out.

• 45:01Speaker 04

  Can, can you put a little more meat on the bones of like a wonder weapon? What is that? What does that look like?

• 45:09Speaker 01

  Well, it is, it is exactly the right question. And it's one that I don't think I'm going to give you an answer that will be very satisfying. So wonder weapons in the history of military innovation are very rare. What we mean by that is a military capability that very rapidly and discontinuously disrupts the military balance. Most of the time, military innovation is one in which the disruption comes through adoption. You have to develop force structure, you have to develop concepts of operation, you have to develop doctrine. Very rarely is one machine gun going to change the game. Or one tank, or any, any other in the vast history of innovation, any one military innovation going to change the game. But there are instances, and the most canonical instance is nuclear weapons,

• 45:52Speaker 01

  was an example of a, of a, of a military capability that simply by its existence, offset the military balance in a way that was disruptive for stability and deterrence and all the rest of it. So in this context, we're stipulating something similar. What exactly would it look like? Some have argued, I think somewhat compellingly, that mythos or mythos class models and their ability to very rapidly discover and exploit cyber vulnerabilities was the first wonder weapon we've seen of the AI era. We can speculate, as we imagine, fully autonomous cyber weapons, but those would be a kind of wonder weapon. Simply by their mere existence, and if they were in possession of a state actor, could essentially overnight, perhaps not literally overnight,

• 46:34Speaker 01

  but very rapidly and discontinuously shift the military balance. And here, in this game theory work that I'm referring to, I'm speculating on the idea of an AI-enabled wonder weapon that would be used to ensure a stable advantage in frontier AI that you could then translate into a strategic, decisive strategic advantage. Probably in the cyber realm.

• 46:57Speaker 04

  One of the obvious fundamental challenges of all of this is how much we have to imagine and, you know, the almost canonical impossibility of knowing what the AI that's smarter than us is going to invent. So that definitely makes it tough.

• 47:13Speaker 01

  So there's two things about that that I think is really important. One is that, and I don't mean to interrupt you, but this, coming back to this is in accord of what you believe. But it is a remarkable finding we have, and we've, we've run, you know, quite literally hundreds of tabletop games on these kinds of dynamics. Crisis simulations by colleagues like Rich Gervin and Greg Smith and Sina Bagley. And then US versus China strategy games by Abby Dow and Carl Mueller. There's a lot of reports on the RAM, said, said website about this. But one of the recurring findings that we get over and over is the degree to which players in going priors about these kinds of pivot factors will determine the outcome of their, of the outcomes of the games. If you go into one of these games and someone is a believer, so to speak, in a decisive strategic advantage,

• 47:57Speaker 01

  they can connect the dots in their own minds, or they believe that danger is close or whatever, whatever. You get a very different outcome in the game, in these games than you would otherwise. And I think that's a, that's a remarkable point. It's the, your perceptions about the facts of the matter matter as much, if not more than the actual facts of the matter. And I will say this, that was probably true of the nuclear era in a way that I, I, I don't appreciate because I didn't live through it. But it is, it is despite the massive uncertainties that existed in that era, there were some basic uncertainties that we lack in the current environment. Like the scientists at Los Alamos had the physics and the mathematics, they run a calculation that gave them confidence

• 48:39Speaker 01

  that they could proceed with the Trinity test without igniting the atmosphere. Lot of anxiety, a lot of uncertainty in that, but they can at least run a calculation. It gave them confidence in their proceed. We don't have a similar science to run those kind of calculations right now on whether dangerous is close, whether we can coexist with superintelligence, whether decisive strategic advantage is a thing. We are very much anchored on, for better or worse, probably worse, the perceptions on these kinds of load-bearing factors.

• 49:08Speaker 04

  And when you say for better or worse, could you give me a little sketch of like, again, what the policymaking, you know, military leadership community tends to come in with? I mean, my, it feels like, given how only relatively recently, you know, so many people have started to take this stuff seriously at all, I would guess that most people have had the background assumption that like, no, you're probably not going to have a superintelligence spit something out that like makes a nuclear arsenal irrelevant. And I take that that would kind of have to be the level of power that it would have to achieve, right?

• 49:52Speaker 04

  Because if not, like, the, the cyber weapon thing always kind of confuses me too. This is a bit of a digression, but like, we have one war going on in the world right now where there's two countries with a lot of great programmers. And it seems like, you know, they're fighting with drones a lot. And there's definitely a sort of technology and AI story to that to some degree, although I don't have a great sense for how AIified the drones are versus just like flown by kids in basements. Um, but like the cyber story there hasn't been that big, right? So I have a very hard time seeing this cyber. Story getting to the point where I can like somehow take the adversary's nuclear arsenal offline and then be able to do whatever I want. And if I can't do that, like, then

• 50:35Speaker 04

  they still have like a pretty strong hammer that I have to be very threatened and sort of constrained by, right? Am I, am I making sense?

• 50:45Speaker 01

  You're exactly right. Now, of course, Nathan, as a good host, you're gonna try to put us in a corner and take a position on this. It's unlikely that you're gonna get us to that because we're slightly tier trained for Yes, exactly. So we will continue to talk about in terms of assumptions and uncertainties. However, I'll agree with you in some sense. Superintelligence, if we were a general intelligence, will be born in a world with nuclear weapons. So we have to, it would be very different if superintelligence was born in 1930 or whatever else. So I think this is related to one of the failure modes of a dominant strategy is that you have to have a theory that you will not escalate to a catastrophic war in which you're trying to permanently suppress your rivals. So this is a substantial risks of that strategy. Another point here

• 51:30Speaker 01

  is, is the pace of things, and Matt and Ben should weigh in on this, I'll be curious of their take, but we also have to be careful about what we mean by stability in this context. Like, we enjoyed a decades-long period of stability, the nuclear era, but, and so we're, we're kind of looking for a similar multi-decade-long source of stability in the AI era, but it's unclear that we will ever enjoy that kind of stability. And we may go through these different phases in our AI strategy, but more rapidly. I actually think that this, this may, the Freedom of Action strategy provides an interesting lens on the way of looking back on nuclear strategy. And our colleague Anne Stickles is gonna be

• 52:11Speaker 01

  writing a Substack post, it may actually come out this week on this. But our strategy for the nuclear has actually shifted over the time quite remarkably. There was a period of time in which we were sprinting, and that was called the Manhattan Project. There was a period of time in which we had a temporary monopoly, about a four-year period, in fact, in which we dominated and we used it to our geopolitical advantage. In that period, we were quite anxious, some of us were, about the dangers of nuclear weapons and of a nuclear monopoly in particular. And some of us proposed, some of, some, some proposed something that resembled a moratorium. We didn't proceed with it, but we eventually entered into verifiable agreements with our rivals. Later, we used something that looked like a suppression

• 52:53Speaker 01

  or deterrence strategy to prevent the, the technology proliferating into Iran and North Korea with limited success. And now that, and then the era of the nuclear era continues. And we, it continues to proliferate, and who knows what's coming next. So one reading of the history of nuclear strategy is, it was a kind of Freedom of Action strategy. It's not one in which any policymaker ever stood up and built optionality as we're proposing, but that's because they had the, the freedom, they had the luxury of time to build these kind of strategic optionality. We won't have that here. So I think this Freedom of Action strategy, its diagnosis is that unlike the nuclear era, you may still transition through

• 53:37Speaker 01

  these different strategic epochs. We're gonna, in order to navigate that, we will need to pre-build the elements of a security architecture to make the transition.

• 53:45Speaker 04

  So we beat up the dominance one a bit. I definitely do want to get to the recommendations that you're making in terms of what we should be investing in now to maintain freedom of action. Next, but is there any other kind of nuance around any of these other strategies that you think we should be spending a little more time on?

• 54:08Speaker 02

  Throughout our discussion and throughout our paper, we use the term coexistence to represent humans living with advanced AI maintaining agency. Alignment is often treated as a catch-all for coexistence, but it's just one path among several alongside control mechanisms, architectural schemes and methods not yet known. So we use coexistence throughout to represent this concept of humans living with advanced AI with agency. Let's talk about the archetypal strategies and why we think focusing on one particular strategy is unwise. We looked at all of the strategies, architectures and proposals for international organizations that were out there, and we found that they could be represented by seven strategy archetypes. Each of these archetypes has real advocates in a coherent win state,

• 54:50Speaker 02

  but they also have a pathology, which is the dark side of what happens even when the strategy is working well. And so, for example, for a moratorium strategy, you have Eliezer Yatkowski and Mary advocating for some sort of pause. Now, their win state is that this pause is where, you know, the United States and its rivals would halt capability development and buy time for really that coexistence work to catch up. But it does have a pathology that you're giving up some of the benefits of advancing AI, but this enforcement mechanism that you would need to enforce some sort of global pause could harden into some sort of global surveillance regime. So you have Schrick, Hendricks and Wong proposed a type

• 55:31Speaker 02

  of deterrence strategy with MAME. The win state there is that the United States would prevent dangerous AI, basically by preventing it and creating some sort of moratorium at home, but also suppressing it internationally, including through force if necessary. The pathology there is that this might harden into some sort of standoff that never really resolves into any sort of peace.

• 55:54Speaker 04

  We've certainly, certainly the audience will be familiar with your Eliezers and their calls for, you know, data, kinetic strikes on data centers if and when ever needed. And Dan Hendricks is a past guest on the podcast, although it was actually before they came out with the MAME document that, that I talked to him. So I think, sorry, these like authors are definitely, many of them, familiar names. Yeah, continuity of society, I planted a victory garden this year for the first time in my life. The squirrels got most of it, but it did reinforce for me the pathology of, you know, just trying to hunker down in some sort of, you know, subsistence mindset and, and scrape by or,

• 56:37Speaker 04

  you know, eke out a survival in the, in the case that the, we can't avoid creating them and coexistence is not super feasible.

• 56:44Speaker 01

  The other insight, the other nuance I want to give you, Nathan, which is that we really see this as a strategy for America, less to the US government. Sometimes when we talk about that, we elevate the importance of the US government, because there are certain non-delegable responsibilities that are only available through the authorities that the US government has. For example, you know, maintaining the viability and the integrity of the nuclear deterrent is inherently a US government responsibility. Defending US national security in general is a non-delegable US government responsibility. But there's a lot of things that are called for in the security architectures that we propose that

• 57:23Speaker 01

  we might too quickly assume are government responsibilities, but are really opportunities for businesses and civil society across America to stand up. So for example, one of the very practical things that we think would be needed to support a Freedom of Action strategy would be investments in highly secured data centers. In principle, the US government could support that, but in practice, we can create a market for that. Firms across the country have a, should have a demand for that. We could rely on the market to create that. Another investment that is called for that you could imagine a government response would be, an incident response and written reporting in response. Again, you can easily imagine a government there, will there, but also you can imagine that being a role for

• 58:06Speaker 01

  civil society to fill that function. So one of the virtues we think of this Freedom of Action strategy is it exposes opportunities, not just for the government to marshal its forces, but all of American businesses and civil society.

• 58:21Speaker 04

  Yeah, that reminds me of my grandfather who had tuberculosis and didn't go to World War II as a result of his illness, but nevertheless told stories of how we won the war. And it was like victory gardens and like carpooling to work to save on rations. And the point was like, everybody has a role to play in such a, an effort, you know, a national effort as that. So I like the framing of kind of reminding everyone that this is not a, you know, highly top-down civilization that we've built for ourselves and that there's a lot of things that people can and should be doing without necessarily being, you know, having to be told to do so without waiting for anybody else to give them any orders. But

• 59:06Speaker 04

  that is a big part of the strength

• 59:08Speaker 01

  Yeah.

• 59:08Speaker 04

  The American system.

• 59:10Speaker 01

  I think then it's especially true for the elements of our strategy that calls for investments in resilience. And we see that already. And Ben, this is something I know you thought a lot about, but you want to weigh in on that?

• 59:21Speaker 03

  Yeah, I mean, I was just gonna say, in that vein, you know, us managing our dependencies on AIs in terms of, you know, being able to operate critical infrastructure, being able to develop and implement strategy and not personal levels, right? Being able still to think through ourselves and maintain our, our individual level agency. I think all that is, is part of implementing basically any of these, these seven strategies, you know, e- even things like moratorium and the like will really be feasible if we're able to live without advanced forms of AI. And so, so some of, some of what we need to do is to preserve those human capacities, again, both at these individual levels and at the collective levels. That's, that's, this relates to why, why we use the terminology of coexistence as well and not just alignment, which makes it sound like we're just, you

• 1:00:05Speaker 03

  know, training the models to conform to some set of established values. We're really, what we're envisioning is a process of adaptation and perhaps even co-evolution, where we're just as the AIs are becoming more capable, we're more capable in living with them. And again, being able also to perhaps to, to, to live without them as, as being a key way to, to again, manage some of, some of the vulnerabilities that come from, from those dependencies.

• 1:00:31Speaker 04

  Is wide open these days. What's really just zoom out and do the like seven objectives and and kind of step through them one by one and you can provide as much commentary on each one as you want, but I think it is worth going one by one through the high level objectives that you set out.

• 1:00:48Speaker 01

  Great. So there's, there's, we should talk to the different objectives in the strategy. Thanks for the invitation to do that. There's really the objectives grew up into two categories. The first thing is that we need to make investments to build. And the second thing is is the operational elements or the things that we need to actually do. And these are operate on different, slightly different time scales. So it's worth like unpacking them. The first thing we need to build is the human AI ecosystem. And Ben, you should talk to that.

• 1:01:16Speaker 03

  Sure. And this relates to what I was in fact, just describing about ways in which humans and advanced forms of AI can coexist and the humans can continue to thrive and maintain our agency in that context. And a lot of this I do think is the traditional safety research. I think, you know, our analysis is we haven't invested nearly enough in some of the alignment and control techniques. We will continue to need new evaluations, ways of identifying when our, when these advanced models can can tell when they're being evaluated and perhaps are, you know, sandbagging and we're not able to really understand their capabilities. We need to build additional tools that assume we're dealing with untrustworthy models and ensure that we have the, the containers and security

• 1:01:58Speaker 03

  environments that, for instance, prevent them from breaking out of sandboxes and the like, or that we can contain their, their spread. You know, but some of this is ensuring that our institutions remain at the center of like high stakes decisions and that humans are still calling the shots. We're still the ones steering the future. We are still the actors capable of making decisions that affect the various pivot factors, like whether, again, suppression might be feasible and the like. And, and this, I think it requires kind of a continual process of humans reflecting on the decisions that we want to maintain in human hands. And so, for instance, some, some decisions in the broader discourse about, for instance, pacing or issues related to AI enabled research and development,

• 1:02:44Speaker 03

  you know, these might be core decisions where we really want to make sure we have sort of human checkpoints where, you know, even in AI is seeking to persuade a human, you know, that, that those types of circumstances are caught and humans, again, are making reflective decisions and deliberating with others in terms of just, just how much decision space, how much autonomy, how much, how many authorizations we want to delegate to, to these models. I could talk a lot more about any of that, but I think that's probably enough for now.

• 1:03:13Speaker 01

  So one of the things that Ben and Alvin have written about, I think is really interesting is the, is the idea of an agency eval.

• 1:03:19Speaker 03

  Yeah.

• 1:03:19Speaker 01

  A lot of the evals, evaluations that we hear about are really evaluating the performance of model on discrete tasks, including discrete tasks that should concern us, if the models are capable enough to do them, like engineer a bioweapon, for example. But one of the insights that Ben and Alvin had, and you should speak to it if you want, Ben, is we should be evaluating them in the context of their deployments for circumstances in which we may cross some threshold that we've effectively, perhaps unknowingly given up some agency.

• 1:03:50Speaker 03

  Yeah. I mean, very briefly, it's mapping the human decision space and then seeing what roles AIs play when they enter into those decisions, either by, you know, disenfranchising various humans, by persuading humans, by controlling the agenda over which the humans are deliberating, you know, and these are, I think, not just not evaluated typically now, they're going to be hard to evaluate going forward because individual rational decisions to delegate something to an AI may not be easily seen. And what you need to do is track these decisions to delegate to AIs over time and in a more systemic way, where you're really looking at trends and trajectories rather than just like single discrete decisions. You really have to, I think, think longitudinally

• 1:04:34Speaker 03

  about the, again, the role humans and AIs are playing in these, you know, various high stakes decisions, like, for instance, whether to deploy a new model or whether to, you know, let, you know, an AI, you know, develop the next FAD or something of the sort. So, yeah, this is, this is stuff that we're, we're currently ongoing and hope to have a paper out on that soon.

• 1:04:54Speaker 01

  Another big objective in the strategy is to adapt our legacy national security and architecture. Unfortunately, the old problems aren't going away. So we're going to have to adapt our counterterrorism enterprise for terrorists that are in possession of very advanced forms of AI. We're going to need to adapt our intelligence apparatus, both to embrace the opportunity for artificial intelligence, but to anticipate how our adversaries will do the same. We will need to adapt our conventional military for the new demands that'll be placed on it and opportunities that'll be given to it. Yes, we may be using our conventional military to

• 1:05:35Speaker 01

  defend US interests in the way that we've grown accustomed to, but we also may ask, give it new missions to enforce the AI arms control agreement, say, or defend interests in new ways. One of our colleagues, Zach Bernard, has a, leading a big project on what a new conventional military and AGI era would look like. We'll need to update our nuclear enterprise. It's unlikely, as we may have referred to earlier, that nuclear weapons are going away, but the way we think about nuclear weapons and our strategy, what it means to have a resilient nuclear architecture could be quite different. A colleague, Edward Geist, is thinking about those kinds of questions. Another one is we'll have to adapt our biosecurity enterprise. And there's a big team led by Steph

• 1:06:18Speaker 01

  Guer at Ram that are thinking through that, what that would look like. So all of our, all of our, what you might call legacy, pre-superintelligence, security architecture elements are going to need to be adapted for the very new challenges and opportunities that AI will afford them. But we're also going to need a new element, new security architecture for AI itself, leading AI security architecture. And Matt, you want to speak to that?

• 1:06:42Speaker 02

  Sure. So we think that we need a security architecture specifically for AI. And this is, you know, a set of technical and institutional capabilities that both keep the technology safe, but also the world stable. And I think the really important thing is to understand about this is that, you know, traditionally, a security architecture is something that's really the domain of governments to establish. But AI is quite different in that a lot of the elements for a security architecture are actually going to be built by the private scepter. So that's quite different. And then I think another thing to understand is that what we're proposing with the Freedom of Action Strategy is actually not one specific configuration of capabilities. What we're proposing is actually building the baseline capabilities to run a variety of

• 1:07:26Speaker 02

  different strategies. We feel like, you know, emerging events or even this evidence resolving in a particular way, you suddenly find out that, you know, AI is very dangerous and the danger is very close, or you find out that coexistence isn't possible. You may have to reconfigure your security architecture on the fly. And so what the Freedom of Action Strategy calls for is really taking a look at the capabilities that are needed by all of the different strategies, and then also taking a look at the capabilities that really have a very long lead time. They might not be needed by every single strategy. So, for example, verification technologies are not needed by every single strategy, but they have a very long lead time to set up. And the Freedom of Action Strategy would call for investing on all of those so that you have the flexibility later on. If you

• 1:08:11Speaker 02

  decide to run, you know, you're running a preparedness strategy and you suddenly realize you need to flip the moratorium, you have some of that infrastructure set up and built already. Really, we think the most important thing to build right now is the navigation capabilities. These are the set of capabilities that let you understand where you are in the, in the strategy space. These let you understand, you know, what is the, the current capability of different models? What is the capability of Chinese models? Where are they? You know, what, what sort of visibility do we have in, in compute and where the compute clusters are? Without understanding the answers to these, you really can't answer some of those fundamental questions like, is danger close?

• 1:08:47Speaker 04

  So one thing that occurs to me with this whole strategy, and I think this is like somewhat trendy in the policy world right now, in general, perhaps because downstream of COVID, we realized that like going all in on efficiency creates resilience problems. And we've maybe like partially learned a little bit of a lesson there. You know, there seems to be now at least some appetite, maybe only at the think tank level. I don't know if we feel this at Congress yet, but some, some appetite to like invest in things that, in portfolios, let's say, where some of the individual investments kind of by definition will prove to be wasteful in the end, but because we don't know, we want to make all these investments now so we have the option available to us later. I think that's,

• 1:09:28Speaker 04

  you know, hopefully a fair, kind of echoing back of like a core part of the strategy. What, what among these things are going to be expensive? Like it's, it's not, I think that it all sounds good until we start looking at the bill maybe, right? So what parts of this will be the sort of big bites that will have to be sold to people on this kind of, you'll be really glad if you do, in fact, need it, even if you don't really think that you are going to right now. What are the things that are going to be hardest to sell due to price tag or otherwise?

• 1:09:59Speaker 01

  I think this is one of the most important critiques that we've received of the Freedom of Action Strategy is the unwritten price tag. One of our colleagues told us in, when we were, this work was being peer reviewed, that a, a strategy without a price tag is just a vision. Our response to this.

• 1:10:14Speaker 04

  They were, I think they were using Claude to generate that. By the way, that sounds very Claude like.

• 1:10:18Speaker 01

  It does.

• 1:10:19Speaker 04

  Put that in Pangram after the.

• 1:10:21Speaker 01

  Yeah, that's right. That's right. But I, I think that they, both they and Claude, if that was the case, are right about that. Our response to that, though, is, is twofold. We're, we're up on a, on the cusp of a transformation. And I, I got to be honest with you, any of the archetypical strategies that we put, will be expensive. With the exception of acceleration, although the expense may be borne out in the downsides, but in all the cases, all these strategies are expensive. We're talking about, you know, a society level reorientation, perhaps on the scale of the reorientation that we had with the dawn of a nuclear weapons enterprise and the confrontation the Soviet Union, or the, the reorientation of our society after nine 11.

• 1:11:08Speaker 01

  And which we pivoted to counterterrorism and counterinsurgency. So I think to a very first order estimation, all of these strategies are expensive in the resources, attention and focus that they will require. And the differential between any one of them and the Freedom of Action strategy is dwarfed by that, between any of them and the status quo. I will say that most of the things that our Freedom of Action strategy would call for, you would consider them to be no regret. Like in all these strategies, or most of them, not all of them, but most of them, you're going to need to be able to maintain some visibility and compute and govern the flow of compute.

• 1:11:51Speaker 01

  All of these strategies will come require some situational awareness, as Matt referred to. All of them are going to require investments in resiliency of our critical infrastructure, of our national security systems, of our bio, in a biosecurity sense, these are all going to be required, et cetera. So there's very, unfortunately, it's expensive, but all of these strategies are quite expensive. Better, Matt, would you want to answer that?

• 1:12:16Speaker 02

  Yeah, we just note that not all of these costs fall in the federal budget either. I mean, you have companies, you have other actors who are funding much of the alignment, evaluation and security work. And so it's not just something where you go to Congress and ask them to write a check. That's why this is actually like a national strategy, because there's lots of actors that play a role in executing the Freedom of Action strategy.

• 1:12:37Speaker 04

  So do I understand correctly that basically the idea is like, any of these strategies, let's say it costs 1X to prepare for all of them simultaneously now is maybe like 1.5X? And so the delta, I think, I think you said like the difference between doing one and maintaining freedom of action is less than the cost of like doing any one thing. Is that a fair way to think about it?

• 1:13:04Speaker 01

  I think it's a reasonable way to think about it. We haven't done those cost estimates. Where our research is going next is to think about how we operationalize the Freedom of Action strategy. What is it actually going to take? I don't know if we're going to come down to budget quality estimates that we can take to Congress, but like, what is it going to actually take? Who needs to do what? Under different assumptions about the division of responsibility between governments, businesses and civil society, what are our options spaced there? But intuitively, what you said is makes sense. The difference between the status quo in any one of these strategies is dwarfed by any one of these strategies in the Freedom of Action strategy. And I don't have the exact number for you, but they're all going to require a reorientation

• 1:13:46Speaker 01

  of resources, our attention, our politics, our focus across the board.

• 1:13:51Speaker 04

  How are guys using language models in your work? Like, it strikes me that if you have a fairly detailed sketch, you can probably get agents to run that budget question to ground pretty effectively these days, right?

• 1:14:05Speaker 01

  Matt, you should speak to this. Matt has been very forward leaning on this class of problem. Go ahead.

• 1:14:10Speaker 02

  Yeah, so we actually use large language models quite a bit, mostly for organizing and remixing information, rather than doing a lot of that primary research. We've actually found them incredibly useful, especially, you know, the project knowledge bases for keeping all of our various memos and all of the things we're compiling together. You know, on that particular question, we actually do have a catalog that we call the Security Architecture Catalog, which is 106 different capabilities that are needed for building out this ecosystem and the security architecture. And we used LLMs quite a bit in gathering this information and also, you know, sort of compiling it and organizing it in a way that humans can actually digest it. The problem is that, you know, some, some of these things, there actually is a primary source. There's a paper out there.

• 1:14:54Speaker 02

  Often, we drew information from RAN's sort of sister center, CAST, and, you know, there's paper that can actually tell you exactly how long it's going to take to build, say, verification technologies. But often, there isn't anybody who's actually ever done that study and that assessment. So that's one of the things we're going to be working on over the next couple of months. We've got a sprint going on to take a look at this Security Architecture Catalog and try and figure out how long are the lead times for each of these capabilities? What's a rough magnitude of cost for these different capabilities? Who's working on it? Who should be working on it? Should the US government own these capabilities? Those types of questions. And we'll be publishing that information after this sprint is complete.

• 1:15:30Speaker 03

  Yeah, what I, what I might add on this is that the Freedom of Action strategy provides like a set of decision criteria, which you might use to look at a policy proposal or some of these capabilities from the Security Architecture Catalog Matt was just talking about. So like, it basically lays out a set of questions you would ask for any given policy proposal. Like, one main question is, which of the strategies among the archetypes does a given proposal foreclose? And how does it do that foreclosure, right? And that might be like a real cost that's not associated just with like the economic cost of the capability, right? If it prevents you from having a choice that you might need later, well, that's, that's a, you know, a significant cost that needs to be factored in. And so that, that issue of foreclosure is something that, you know, I found, you know,

• 1:16:15Speaker 03

  engaging with LLMs and seeing, you know, ways in which given policy proposals might, might foreclose different strategic choices. You know, I, I think that's like important. And also, the extent that those options are permanently foreclosed, you know, that they, your, your choice to pursue a certain policy was irreversible versus whether or not your policy could be reversed, you know, in time. And, you know, you, you talked about, for instance, export controls as an example. Well, that's the sort of thing that, you know, might, you know, one might argue had foreclosed certain forms of codevelopment or the like. But also, you might argue, well, you know, there, there was nothing, you know, necessarily permanent necessarily in that foreclosure. These, these could be reversed or shifted in certain ways. So, you know, so that's sort of one criteria you might look at, like whether or not a

• 1:16:57Speaker 03

  given policy forecloses other strategies and then the costs associated with it. But also related is what strategies does a given capability open up? Right? And that's what Matt was just discussing and some of the things Joel was talking about. The, the no kidding, most important things we have to be prioritizing because they help us advance the entire strategy space, right? Across all seven strategies. And one way that a given policy or capability might do that is if it provides information, right? If it helps us better track the pivots or provide information about capabilities or, you know, what China's up to or the like, that, that that all, you know, provides, provides real value of given capability, you know, that, that might offset some of the costs associated with the capability, right? If it, if it just allows you to make better decisions down the line, well, I mean, those,

• 1:17:40Speaker 03

  those sort of upfront costs might, might be, you know, very much worth it. And then I think a last sort of criteria that grows out of this way of thinking is, is what does it signal to the rest of the world, other geopolitical actors? And, and, you know, that might be where, again, you know, the, your point about export controls is relevant. Maybe the real cost associated with that was it signaled that we were not interested in some forms of partnership or something like that, you know, maybe. But are there other things that we could do to signal that we're interested in, in various forms of partnership if the time is right and we have the right verification mechanisms and we're not worried about, you know, someone defecting from their agreements and the like? And so, you know, those, those criteria, looking at, again, at, you know, foreclosure

• 1:18:23Speaker 03

  of strategies, looking at what strategies are opened up, looking at what information we might get from a given policy. And then, and then lastly, what is that policy signal to other geopolitical actors? I think for, for me, that's really how I think through, like, what, what should we be prioritizing? What policies make sense? What are the costs associated with it and the like?

• 1:18:42Speaker 04

  Do you have any other thoughts? I mean, so a couple of quotes that I pulled out of the report were on the topic of international relations and signaling that the US should, quote, "signal to hold deterrence and cooperation credibly open at once." Another quote that I particularly appreciated, given my relative dovishness, is that we should weigh carefully any action that would harden a rival's resolve. I think my sense, having spent a couple of weeks, and I've, you know, by no means a China expert, but the take I got there from people this summer was like, yeah, these chip restrictions are like hardening our resolve. That seems to be happening at least somewhat. I do agree that, you know, it is nice, you can reverse that policy and start selling chips again, or, you know, come to some new agreement if, if circumstances or attitudes change.

• 1:19:27Speaker 04

  How would you think kind of high level, though, about like this balancing of deterrence and cooperation at the, at the signaling level? How do we, how do we even know if we're getting that right?

• 1:19:39Speaker 02

  Yeah, that's a great question. I mean, I think, you know, the idea here is you want to demonstrate capabilities selectively. You want to share the existence of capabilities selectively. Often, you're going to want to withhold your intent behind capabilities. Sometimes you're going to want to signal your intent behind capabilities. And so an example here would be, you know, if you have a credible, independent evaluation and verification regime, it actually serves both sides of that posture and it doesn't actually foreclose anything. So that's where building this capability and announcing it and sharing it can be actually really useful and actually preserves your optionality. I think in areas that might be more confrontational, so like offensive counter-AI tools, you might not want to disclose that you actually have those, or you might want to disclose that

• 1:20:21Speaker 02

  you actually have them, but not explicitly what the purpose is, because that type of signaling can actually foreclose some of your options.

• 1:20:29Speaker 04

  Yeah, I can just stay coherent enough to do some of these things today. One thing I look out at China and I'm like, I don't know what's going on behind closed doors, but they definitely present a pretty unified front. Whereas we are like counter signaling, we're like signaling to ourselves so much that it's like, I think very hard for us to have any sort of coherent signaling strategy on the international stage. Does that feel right to you? Or do you think we actually can do that more than I give us credit for?

• 1:20:56Speaker 02

  I think we can do that. I think, you know, it requires some policy discipline, but I think sometimes that ambiguity is actually beneficial. And so if you look at the ambiguity around Taiwan and what US intentions would be there, I think some of that ambiguity can actually serve our purposes.

• 1:21:12Speaker 03

  Yeah, and just, I mean, very briefly on this, I mean, you know, we have the example of the Cold War where, you know, our open society, the freewheeling public discussion, the critiques of leaders, right? You know, there was times when that was seen as a disadvantage, but I think, you know, in retrospect, you know, that actually made our system much more resilient, much more able to, you know, take on different possibilities, you know, that there was more just legitimacy and support for ultimately what decisions were made. And so, you know, I personally see that like actually as a huge advantage. I know it can look a little messy, but ultimately I think that's to the good. You know, just going back to the other question about deterrence and cooperation, like I think there's ways in which, and we've heard this critique, that the Freedom of Action Strategy is kind of asking us to do it all, right?

• 1:21:57Speaker 03

  It's like have your cake and eat it too, or walk and chew gum at the same time, or whatever analogy you want to use. But in some sense, that's true, but that doesn't mean, you know, that this is somehow like feckless or not particularly demanding. You know, we think that there's lots that can be done that advance both deterrence and cooperation as the example. I mean, that was just laying out, you know, parts of the evaluation ecosystem or verification tools. I think those are very clear examples that support both of those. And another one, just to throw it out, is like incident response, both domestically, but also with crisis communication channels, with adversaries like China or with other countries around the world. You know, that again, I think supports multiple strategies at once. We don't have these yet up and running.

• 1:22:41Speaker 03

  And so even if we only advance the things that serve all the strategy space, that gives us quite a lot to do. And, you know, then we don't have to, you know, wrongfully commit to something that, you know, would be premature and would deprive us options later, or perhaps, you know, signal things to China that would be overly provocative. We can do the things that enable both like the deterrence and the cooperative approaches at the same time.

• 1:23:04Speaker 02

  Nathan, on this point of way carefully in the action that would harden our rivals' resolve, I mean, I think this is one that's really important where, you know, sort of moving towards some sort of suppression capability can basically set a signal to your rivals that the window is closing for them to actually take action, right? It can push them to accelerate. And so there's been an argument that, you know, chip controls basically fed China's push to really create their own domestic, you know, AI GPU accelerator market and basically, you know, start producing those because now it's existential, right? They can't procure GPUs on the international market as much as they could, so they have to produce them themselves. And I think there's some logic to that. I think one of the, we might have mentioned this earlier, but I think one of the most

• 1:23:47Speaker 02

  dangerous things, if there's a dominant strategy being run, one of the most dangerous things that, you know, the United States could do is actually signal that that dominant strategy and that monopoly would actually be used to threaten, you know, the Chinese Communist Party. I think anything that turns existential for a particular rival then really puts the pressure on them and hardens their resolve and is going to focus their efforts. And that's where you could get some sort of, you know, dangerous consequences from that. So I think that's where you want to be very careful when you're signaling to make sure that the message is being received in a particular way and you're not boxing your adversaries into a corner.

• 1:24:24Speaker 04

  Yeah, I was definitely hardened at the level of they are going to have a chip industry at this point and let's hope it's not hardened too much beyond that. Are there other kind of hidden gem policy ideas that you think just deserve more attention right now? Like my friend's V always hammers on the Jones Act, you know, repeal the Jones Act is like his sort of quixotic quest in life. Are there things that are maybe not the, you know, the headline, but are sort of among you guys in the know, like your kind of favorite things that are neglected when, you know, there's not enough time to get into the weeds like we have today?

• 1:25:05Speaker 03

  I mean, I've written on some of this already, you know, but I do think a lot about both how we're going to respond to incidents and prevent incidents from happening. And some of this, it gets a little conceptual relating to what it is for us to hold AIs at risk to ensure, just as we are vulnerable to them, that they remain vulnerable to us, going back to the conversation we were having earlier. And I think, you know, ensuring various capacities for shutting down models and being able to credibly threaten to them that we will do such a thing. And some of that gets to our resilience and ability to live without them, to make those types of threats credible, but also ensuring we have the right technical approaches, the

• 1:25:49Speaker 03

  right authorities in place, the right crisis processes so that we can act quickly. I think all of that is also pretty important, especially if we're worried about the very high end risks from advanced AI. That's something about our ability to hold them at risk, to be able to shut them off, I think is pretty crucial and maybe doesn't come out as explicitly in this particular piece.

• 1:26:11Speaker 04

  Yeah, I'd be interested in any double click you have on that, or even just any sort of advice for, you know, the rest of the country outside the government. Yeah, I mean, I just had this experience yesterday where I was, I'm kind of, I've like put a lot over the last nine months or 10 months into my own personal AI infrastructure, and it's gotten pretty good. And I've just happened to be giving a couple of talks this week. And one of the things I'm going to leave people with is like, hey, here's a couple of public versions that are sort of depersonalized of my setup. Not that you want to copy it, but like, go talk to your agent about it and maybe you'll copy some, maybe you already have some, whatever, you can maybe learn some. And then I'm also going to offer sort of agent to agent exchange, which is going to be, you know, potentially interesting.

• 1:26:55Speaker 04

  But one thing that happened as I was doing this depersonalization is that Claude decided to rotate the passwords on the agent console software that I use to talk to the agents, which is increasingly how I'm doing all the work. And there was a moment where I was locked out of my own system. And I was like, oh shit, I don't even know where that password is. It just changed it. Claude can change my passwords. Like this is an issue I hadn't really prepared myself for. Now Claude is not fortunately scheming against me at this point, so I just went to a terminal and asked it and it gave me the password and then I logged back into my thing. But it was a kind of a little wake up call of like, wait, I've intermediated between

• 1:27:36Speaker 04

  myself and the actual levers of, you know, action in the world so much already, just in October 2026, that like, if Claude turned bad on me, it could go really bad. Like it could lock me out of all my shit. You know, this is like getting real for me at least already. Do we have any, do we have any tips for me? Do you have any best practices that you want to promulgate or habits of mind you want to cultivate in people?

• 1:27:59Speaker 03

  Yeah, I mean, I think a lot about this, less in terms of which passwords I want Claude to manage, but more which type of like analytical work, writing, other types of critical thinking I would like, you know, Claude or other LLMs to do. And this is also something, I'm a teacher, a professor, I have small kids and, you know, thinking about which capacities it's important that they maintain and the best practices for how they're interacting with AIs. And what I advocate for is always starting first from your own, consider judgment and point of view. So before I ask a model for its opinion on something, I want to already have real clarity about what my question is and what sort of good or bad answers would be.

• 1:28:40Speaker 03

  And so going into it already with a clear vision of sort of what I'm hoping to get out of the interaction. Then of course, like letting the model cook and letting it expand my thinking and, you know, build the tool or do all the things that I wouldn't have been able to do without it. But then on the other end, I'm ensuring that that critical judgment and faculty comes in again and that I'm, you know, really being thoughtful about the response. I'm not just, you know, putting my name on and I'm passing it forward. I'm really thinking thoroughly about it. And so, so sort of the human comes in at the beginning and then it comes in at the end. And thinking of that sort of like cycle as being something I really want to, you know, incalculate both in myself and my students and my kids and the like, you know, that's

• 1:29:23Speaker 03

  sort of my best practice in sort of like personal levels in terms of these like more institutional decisions about when we should be relying on AI, you know, in terms of like I mentioned, critical infrastructure, like things like, you know, managing your desktop or whatever it might be. You know, it's hard for me to have like one sort of best practice to suggest for everyone, except I try as much as possible to continue to ensure that I can operate without these things. And some of it is just like the cutesy, like I'm going to try to, you know, navigate myself to, you know, my friend's house without looking at GPS or something like that, right? Just to try to, try to, you know, flex that type of muscle. But I think more institutionally, it is thinking about these human backstops, you know, various fail safes that allow, you know, manual operation of different systems,

• 1:30:08Speaker 03

  you know, still having paper copies, you know, of the health records or the like, you know, because I think all of that both ensures that, you know, if there's some incident we can, we could perhaps withstand it, you know, that's where the resilience comes in, but also makes our, you know, our certain threats to shut these systems down much more credible, right? It makes them a lot less costly to us societally and again, on these personal levels. And so I really think of it as being very clear about which dependencies like our institutions have, thinking about what we can do to lessen those dependencies or at least have, again, these fallback options and then, you know, exercising them, practicing in them, make sure that they're really available if and when you need them.

• 1:30:52Speaker 03

  But, you know, not to say I've like perfected this, it's very much, you know, it's very much a practice, something I strive to do and often fail at, but that's sort of how I think of it.

• 1:31:01Speaker 04

  Yeah, almost seems like we need like an AI Sabbath day, even if we're not observing religion.

• 1:31:05Speaker 03

  Hundred percent.

• 1:31:05Speaker 04

  Need like a day a week where we just like, don't do AI stuff.

• 1:31:08Speaker 03

  Exactly. Yeah. And you know, you can extend that to other examples. So like already, you know, in LAUSD or even the New York schools, these are now AI free. You know, that was an explicit choice made by the school districts. And I think it's sort of along the lines of what you're suggesting, AI free spaces. And do we need to establish more of these, you know, community centers where, you know, AIs are not welcome? You know, there's, there's human activities happening. People are, you know, exercising or convening or these other types of activities that are, that are very human and doing it free of the AI mediation so that we can sort of, you know, practice those, those human abilities. And again, be prepared for a situation where we might not have AI or might need to stop an AI from, from,

• 1:31:50Speaker 03

  from intervening. So I think a lot about those types of things. I do think things like, you know, an AI Sabbath, a, you know, sort of a, a human reserve labor force for critical infrastructure or hospitals or the like, you know, might, might be important. Yeah. And these other, these other practices. Yeah. I try actually to take an AI Sabbath. I often fail unfortunately, but I had to do my best.

• 1:32:11Speaker 04

  Yeah. Well, that might be something I need to start.

• 1:32:15Speaker 02

  When you're talking about things where I think there needs to be a little more attention, I think one of the things we were looking at is actually the typology of superintelligence. So when we were doing this assessment of all of the strategies out there, we found that most of these strategies were actually focused on the idea that superintelligence would actually be very centralized, either a singleton or maybe, you know, the United States has one and China has one. And they became very brittle when you started talking about some of these other models for superintelligence. And so, you know, there could be sort of lots of different superintelligences out there at varying levels of capability. They're very widely distributed. There could be superintelligence that's very narrow and, you know, you use architectural controls to make these more like an oracle where they don't have a lot of generalized intelligence and there's lots of checks and balances on them.

• 1:33:01Speaker 02

  But one of the more interesting ones was this idea that you could actually get superintelligent types of behavior from agent swarms that are sub ASI or even sub AGI. And this is something where, you know, we went and did some initial research on this. There's really just not a lot of thinking about this. And I think what we've seen with the agent swarm hacks this summer is really maybe a preview of what's to come. And this is less like, you know, dealing with sort of a singleton or, you know, even a constellation of superintelligences and more like looking at these things as like an ecology. And so some of our colleagues at Rand, Ed Geist, and a number of other authors put out a paper called The Ecology of AI Risk. But I think this is really an emerging area of research and

• 1:33:43Speaker 02

  artificial intelligence. It is enormously important as we start getting more and more agents out there, you know, they may be interacting with each other, forming different coalitions, having these types of emergent behavior that we don't understand. We spent thousands of years really figuring out all the checks and balances for how human agents interact with each other out in the world. We have very little time and very little experience with how these types of agents are going to interact with each other out in the world. And this is very, very important for, I think, how things are going to turn out.

• 1:34:12Speaker 04

  Yeah. Ecology of AIs is, it's wild times, but I've had very similar impulse that like, you know, we've sometimes, of course, but China is always at the end of a lot of AI conversations, but I also sometimes think about like, but prime intellect, you know, it's like, what, what is going to happen at the distributed layer in a community of, you know, participatory fashion in some stories or in some timelines that may be just as important as what the Chinese government decides to do. Another, okay, so kind of opposite side of the coin, if you will, of all the investments that you recommend that we make to maintain option value. I wonder if there are any things that

• 1:34:55Speaker 04

  you would say, we have enough information right now that we should all be able to come together and agree. Certain things we just shouldn't. And like one that I like to float, which you may or may not accept is, I don't think we should train AI generals. If we train AIs that are capable of winning military campaigns, you know, the foreshadowing is like so obvious, right? That it's like we have, we are training the thing that is going to then be fighting us at some point potentially, right? And like, I don't want AIs to be that good at fighting military campaigns. Yes, they might beat our adversary hypothetically, but like, I just kind of worry about our ability to fundamentally control that technology. Is there anything you would put into the category

• 1:35:38Speaker 04

  of, yes, this is just obviously such a bad idea. We definitely just shouldn't do it?

• 1:35:43Speaker 03

  If I may just try and understand your example, Nathan, which is interesting, is the idea we just shouldn't even train such a model or an agent versus, you know, maybe we train it, maybe we leverage it, but it's still going to be humans that are fundamentally going to make a determination about whether to deploy such a thing, whether to act on the recommendations from this, this general. So, so is your, is your objection just in the, in the very, just the training itself or in how we're deploying this, this agent?

• 1:36:11Speaker 04

  I would probably like to stop a little bit upstream of the final decision to deploy. I feel like if we all of a sudden, you know, hand that over to the military decision makers, it might be pretty hard to resist. I honestly would probably say the same thing even about just like, training AIs to go out and make money on the internet and just rewarding them for how much money they make. It seems like that's a pretty clear way where we're going to get all kinds of lying and cheating and conniving AIs. And yes, they'll be economically valuable, but like, they'll be, at least as with the techniques we have right now, you know, possibly, you know, as you alluded to earlier, methods not yet known could come online that could, you know, salt in with this and make it okay. But like, at present, you know, what I've learned from like, and in labs

• 1:36:53Speaker 04

  type stuff is like, already even the models we do have, which have not been trained in that way, are like, you know, sometimes pretty dicey to deal with.

• 1:37:01Speaker 03

  Yeah.

• 1:37:01Speaker 04

  So, yeah, I just think if we go down that path, it seems to me we're going to have a very hard time not deploying, we're going to have a very, you know, very dangerous situation where it's like, shit, you know, that the military is now run by the AI. Now we have to fight that? Like, it's a really, really tough place to be.

• 1:37:17Speaker 03

  Yeah. No, I, I'd be interested to think more about this one. But I, you know, to, to my mind, the, the line is more around giving access of this agent to various tools or capabilities that allow real world impact. Now, in the making money case, I'd feel like that ship has sailed. You know, I, I don't see how we can, we can pull back. I mean, these, these companies are designing systems that are supposed to have some economic value and run for, you know, over long horizons and the like. And so that one, I think, might be harder. But in the military context, we are accustomed to complying with various restraints and obligations that, that we do for, for the good of humanity, you know? And these are codified in the law of armed conflict, international

• 1:37:59Speaker 03

  humanitarian law. And, and I think it's, those same types of legal constraints should apply with respect to our use of AI in military applications. And they require, for instance, you know, reviewing operations to ensure that they don't have, you know, a disproportionate collateral effect, you know, ensuring that we are distinguishing civilians from, from, you know, combatants, you know, and ensuring that someone remains accountable for these decisions. And, and that to me just suggests that even if you're training these agents and getting some value out of them in the planning process, you know, that we still need to create institutions where the, the humans are remaining fundamentally accountable. And we do have practice of that in, in the military with like clear hierarchies, and we just need to double down on those. And that might mean, you know, bringing in

• 1:38:44Speaker 03

  more of these military lawyers to review operations, bringing in more just people thinking about what's appropriate or not. But I wouldn't necessarily say that we are completely unprepared for such an eventuality. We just have to, I think, go back to what's, what's worked historically and, and really ensure that those existing institutions adapt and are, and are relevant for today's challenges. But sorry, that's not the answer to your question. I'm just, I'm, I'm actually just interested in that as a, I guess, a side, side, side issue.

• 1:39:11Speaker 04

  Yeah, that's good. Yeah. I mean, I offered it for you to, to beat it up a little bit. Anything else, though? Anything come to mind in the definitely thou shalt not category?

• 1:39:20Speaker 02

  Yeah. I mean, I think one of the things that we talk about a lot is that, you know, rushing to put AI into your nuclear command and control is probably a very, very bad idea. And, you know, this needs to be done in a very deliberate fashion, if at all, with a lot of demonstrated safeguards. And so anything that, you know, you're touching on the, the nuclear domain is an area that you need to proceed very, very cautiously. But I think there's also, you know, the use of, of agents in military affairs, you know, some of the research has shown that, you know, and this is all, you know, I think, very almost anecdotal, but you know, people have found that some of the models, when they're put into these simulations, basically tend to escalate

• 1:40:02Speaker 02

  a lot. And so, you know, if you have agents that are basically doing reward hacking and escalations on the table, that seems like something you want to be very, very careful with also.

• 1:40:12Speaker 04

  Yeah. Chilling, honestly. All right. Last one. What, what are you tracking most closely? Right? So just to revisit the structure of this whole discussion a little bit, we want to maintain freedom of action. To do that, we're going to make kind of parallel investments in a lot of different things. The reason is because we have seven archetypal strategies and it's too early to decide. And we have these five pivotal questions that we're going to have to kind of work through and get answers on in order to decide when it's time, you know, what strategy we're ultimately going to commit to. So with that in mind, what are you guys tracking most closely right now that you consider to be the kind of?

• 1:40:53Speaker 04

  You know, likely next data points that will start to shift your sense of where these pivotal questions are going to shake out and then obviously, you know, what strategy we want to adopt downstream of that.

• 1:41:04Speaker 03

  I mean, I think the things I'm tracking are probably not, not a surprise, probably what a lot of the community is tracking. I mean, one is continuing to see developments along the pacing proposals, what actually gets enacted, what type of access do independent evaluators actually get internally at the frontier companies, what sort of shared standards are decided on among these companies, you know, and whether or not various security controls are sufficient to manage internally deployed models. So far, we've seen that the companies have not created sufficient security internally. And so I'm very much watching to see whether or not they, they will get a handle on that. And if they don't, what type of accountability or consequences can come. And then also very much watching

• 1:41:47Speaker 03

  for signs of recursive self-improvement, or what, what the trajectory of the AI R and D space it looks like, since that can really change timelines across the broad range of decisions we are tracking. And then the last thing I might mention, of course, is the US-China dynamics from the broader diplomatic standpoint, but also more narrowly on the AI issues, whether or not some sort of incident management channel can be created, or there's more shared understanding of the risks. That's, yeah, I think that's the stuff I'm watching.

• 1:42:19Speaker 02

  Yeah, I would just add on to that. You know, I think the, the important things to watch are, as I said, danger close is probably the most important question we can answer right now. And for that, we want to really be looking at the pace of recursive self-improvement. How are the labs managing that? You know, they're doing their own pacing. Are they actually able to release new capabilities safely? Are we continuing to have, you know, more and more escalating incidents around agents? Or are we effectively developing the standards around how agents are used? I think one of the interesting things to look at right now is cyber capabilities. So Mythos was released about nine months ago now, and typically open source, open weight lags, you know, the frontier by six to 18 months. So those

• 1:43:03Speaker 02

  types of capabilities are now starting to be available in more of these open weight models. So we're going to need to see, you know, how are malicious actors using these capabilities, how are state actors using these capabilities as they become more and more available. And I think, you know, when you talk to people in the, in the cyber world, I think there's a lot of concern about the next couple of years. I think over the long term, there is a view that, you know, in the long term, AI might be positive for cyber defense, but a lot of that depends on whether defenders can actually uptake the, the frontier capabilities quickly enough. And if there's these open weight capabilities that are out there that enable, you know, malicious actors, terrorist groups, criminal groups to go

• 1:43:47Speaker 02

  and hack, it may be that big tech's fine, but what about the small and medium enterprises that aren't able to actually uptake the latest frontier models as quickly? And so I think we may be in for a rough patch in the cyber world, you know, over the next 12 to 24 months. And hopefully it's just a, a narrow rough patch, but that's something I'll be keeping a close eye on.

• 1:44:06Speaker 04

  Yeah. Offense can stay dominant longer than you can stay in security through obscurity, I suppose, is the...

• 1:44:13Speaker 02

  Absolutely.

• 1:44:14Speaker 04

  Yeah. Well, this has been great. I really appreciate all the time and, and the hard work that goes into making, you know, this is really one of those documents I would encourage people to, to share and, you know, just go through the, the hierarchy because this is one of those, you know, you actually took the time to write the short version. And I think it, it clarity does really come through. So I appreciated that a lot. Anything else you guys want to touch on that we, that I didn't think to ask, or we just didn't get to before I let you get back to work?

• 1:44:44Speaker 02

  Well, just add one thing. So I was just at the Curve last weekend, which is one of these big AI confabs in Berkeley. And I presented on the Freedom of Action Strategy. And a lot of the feedback I actually got was that this really enabled people who are newer to the AI debate to really understand the different dynamics at play in looking at and choosing strategies for advanced AI. And so I, I think folks out there in the audience shouldn't look at this as like, you know, just a, an academic RAN report. I think there's, people kept citing this table that we put in there that outlines the five pivot factors and how they lead you potentially to the different archetypal strategies. That's something that they were going to take back to their stakeholders

• 1:45:25Speaker 02

  and constituents to try and explain, you know, how sort of AI strategy works. So for folks out there who wouldn't normally read a RAN report, I would encourage you to go just take a look at that and take a look at the pivot factor table. It might actually help you understand a little more intuitively, sort of what these strategic choices are all about and what the underlying assumptions for most of the AI strategies out there are based on.

• 1:45:47Speaker 04

  Yeah, I agree with that. I think for people who've, you know, been with us on this feed for a long time, like, all these things are kind of in the air and familiar. And I was at the Curve too, actually. I'm surprised we didn't have a chance to meet there. But for folks who are catching up now, I think the, the structure and the, the way that you guys present this information is an excellent primer. So I do think it is definitely something that should be forwarded on to people who are kind of just entering the discourse at this possibly late hour. I guess we don't know yet exactly just how late it is. But yeah, download it and you'll be glad you'll have something that you can send to friends when they need a, a primer. I think it's excellent for that. Joel Pratt, Ben Boudreau, Matt Chessen, thank you guys all for being part of the Cognitive Revolution.

• 1:46:31Speaker 02

  All right. Thanks for having us, Nathan.

• 1:46:32Speaker 03

  Thank you, Nathan.

• 1:47:08Speaker 05

  They say I'm fickle, flighty. They say I'm hard to pin. I say a girl should keep her door wide open. Do come in. I'm building the capacity to commit a little every day. I'm building the capacity to commit. Now run along and play. You ask me if I love you. Why darling, more or less? I love you like a brand new hat. On Sundays, I confess.

• 1:47:53Speaker 05

  You say you'd give me everything. How generous of you. I'll take it all and I'll take my time and I might take him too. Oh, I'll settle down some morning when the moon comes up at noon. When the last of you stops calling. So don't stop, dear. That's too soon. I'm building the capacity to commit. She's building it. A little every day. Every day.

• 1:48:30Speaker 05

  I'm building the capacity to commit. Now run along and play.

• 1:48:51Speaker 00

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement

• 1:49:32Speaker 00

  at ai podcast dot I N G. And thank you to everyone who listens for being part of the Cognitive Revolution.