The Balance of AI Power: Anton Leicht on Politics, Pacing Deals, and Muddling Through Well

Anton Leicht, a fellow at the Carnegie Endowment for International Peace and author of Threading the Needle, joins Nathan Labenz to explore how political and institutional power can be preserved as artificial intelligence advances. They discuss the strategic trade-offs of frontier training pauses, the hurdles facing industry self-regulation, and how middle powers can trade data center capacity and semiconductor leverage for model access. Ultimately, Leicht emphasizes that preventing permanent disempowerment and the monopolization of control by private labs or single governments is just as critical as addressing catastrophic model risks. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/the-balance-of-ai-power-anton-leicht-on-politics-pacing-deals-and-muddling-through-well/ Sponsors: Mercury Command: Mercury Command brings powerful conversational AI directly into your banking, providing real-time natural language access to your finances without exposing data to third-party tools. Learn more and apply online in minutes at https://mercury.com Athena: Athena matches you with a dedicated, top 1% executive assistant to handle your inbox, calendar, and daily workflows so you can save an average of 15 hours a week. Get matched with your EA today at https://athena.com/cognitive OutSystems: OutSystems is the leading agentic systems platform that enables enterprises to engineer, orchestrate, and govern AI applications on a single unified platform. Learn more and see how it works at https://outsystems.com/tcr Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr CHAPTERS: (00:00) About the Episode (04:46) Sponsor: Mercury Command (06:33) Assessing current AI dangers (13:17) Feasibility of AI pauses (Part 1) (21:33) Sponsors: Athena | OutSystems (25:05) Feasibility of AI pauses (Part 2) (31:01) AI and the nation-state (Part 1) (35:11) Sponsor: Claude (36:46) AI and the nation-state (Part 2) (46:17) Oversight and US policy (01:05:49) Data center buildout reality (01:11:55) Europe and middle powers (01:36:05) Geopolitical friction over Taiwan (01:43:46) Lightning round and outlook (02:05:25) Episode Outro (02:08:54) Outro PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, my guest is Anton Likt, fellow with the Technology and International Affairs Program at the Carnegie Endowment for International Peace and author of Threading the Needle, a wide-ranging substack on the domestic and international political economy around AI progress, where, over the last two years, Anton has performed something of an Alexander Hamilton, writing his way into the elite American AI conversation, despite being a recent and relatively young newcomer to the United States. This summer, I, like many others, have personally moved from worrying that AI could become scary to feeling that today's AIs, with their rapidly advancing capabilities, profiles, their newfound

• 0:44Nathan Labenz?

  creativity and persistence in problem solving, and the many bad behaviors they've exhibited, even while knowing that they're being tested, are now legitimately scary. Especially considering how often and how dramatically AIs are now surprising their creators, I don't think we should be too confident that anything in particular is still beyond their reach. That said, recognizing that we have a ton of work to do to understand and effectively control these systems doesn't make it easy to reach agreements that will actually give safety researchers the time they need to do so. And so, with that in mind, I was excited to get Anton's take on the power dynamics surrounding AI decision-making, including the relationships between the frontier companies

• 1:27Nathan Labenz?

  and the independent nonprofit auditing organizations, between the American frontier companies and the US government, between the US and its allies around the world, and of course, between the United States and China. We begin with a short conversation about how dangerous today's models really are. And while Anton is less worried than I am about the very short term, he agrees that the misalignment issues we're now seeing look sufficiently like the harder problems that he expects we'll face later, that a temporary scaling pause would indeed be valuable. From there, we go on to discuss the prospects that the US and China could collaborate to pace the frontier, and why he believes that, in light of the many advantages that China

• 2:08Nathan Labenz?

  has established in frontier technologies and manufacturing generally, he doubts that the US will want to do a deal that might erode its advantage in AI, even if such a deal would be good for the rest of the world. We then assess what it would take for independent evaluation organizations like Meter and Redwood to get more favorable working conditions from labs, a topic that has moved substantially in just the few days since we recorded, with both Anthropic and OpenAI committing to employee-like access for auditing organizations. I also get his reaction to my idea, which also suddenly seems much more realistic, that the President of the United States could simply give top American companies a deadline

• 2:49Nathan Labenz?

  by which they must agree on a framework for self-governance and peer-to-peer enforcement, plus some other ideas for things the President could do unilaterally to improve the situation, such as clarifying the administration's stance on antitrust and export controls as they pertain specifically to AI safety research collaborations. After that, we get Anton's analysis of what the rest of the world should expect from the AI age. In short, he believes that most of the world will be quite disempowered, even as quality of life probably continues to improve. And while there's little that most countries can do about this, Europe in particular, he believes, given their economic scale and control of ASML and the broader semiconductor

• 3:30Nathan Labenz?

  manufacturing tooling supply chain, should have enough power to reach a compute-for-access deal whereby Europe allows American hyperscalers to build data centers in Europe in exchange for promises of continued access to frontier AI capabilities. Beyond all of that, we talk about what other small countries have interesting strategies available, including Norway, the UAE, Singapore, and Australia, how all of this analysis will begin to change as compute starts to move to space, whether or not doomers can convert their predictions into winning trades in the financial market, how we should be thinking about minimizing concentration of power, and finally, why Anton's best-case scenario is not a grand utopian vision, but simply the hope that we might muddle through

• 4:15Nathan Labenz?

  effectively enough to enjoy all of the obvious upside that AI has to offer. Anton really impressed me with his depth of understanding and quality of analysis across a wide range of topics. So, without further ado, I hope you enjoy this conversation about the political economy and power dynamics of AI, with Anton Likt, fellow at the Carnegie Endowment for International Peace and author of the outstanding substack, Threading the Needle. The Cognitive Revolution is brought to you by Mercury, the fintech that more than 300,000 ambitious companies and individuals trust to run their finances.

• 4:55Nathan Labenz?

  I've wired AI into nearly every corner of my life: my email, my messages, my calendar. I even gave Mercury virtual cards to my agents, with low limits and category and merchant restrictions, for their autonomous use. But still, my AI's access to my financial data has remained limited. With a normal bank, I might export a bunch of statements and have my assistant process them for me. But for real-time, up-to-date information, and certainly for taking any action, trying to get your agent to use the bank via the browser is just too hard, too slow, and too error-prone to be worth it. And that's why Mercury's new conversational interface, Command, is such a big deal.

• 5:37Nathan Labenz?

  It's built directly into Mercury, which means you get natural language access to your finances without exposing anything outside of your bank account. No exports, no spreadsheets, no pasting your transactions into third-party tools. I really think a lot of people are going to prefer it this way. And it can already help you take actions, too, with everything bound by the permissions and approval policies that you've already set up in your account. I am genuinely impressed to see this level of AI integration in banking in 2026. And so, I invite you to join me in the future. Visit mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC-insured bank.

• 6:21Nathan Labenz?

  Banking services provided through Choice Financial Group and Column NA, members FDIC. Thank you to Mercury for supporting The Cognitive Revolution. And now, on with the show. Anton Likt, fellow at the Carnegie Endowment for International Peace, welcome to The Cognitive Revolution.

• 6:39Speaker 01

  Thanks for having me.

• 6:41Nathan Labenz?

  I'm excited for this conversation. You have been popping up all over the place with your own writing and various interviews, and you're clearly a renaissance person with advanced thinking on a lot of different aspects of the increasingly complicated AI age in which we find ourselves. So I'm excited to run down a bunch of these rabbit holes with you today.

• 7:02Speaker 01

  Yeah, it's exciting. I don't know. I think you can't do part of the thing at the moment, right? It's just all sort of geopolitical end of history and the end of technology history or whatever are sort of coming together at a rapid pace. So I guess either you do everything or you do nothing. And so, yeah, Renaissance times, something like that.

• 7:18Nathan Labenz?

  Yeah, I feel the same way, actually. The big motivation for this project was just observing how many people are so deep-down, specific rabbit holes, advancing and usually having success, advancing whatever frontier they're advancing, but not so many people have taken the purposeful approach and kind of forego being an expert in any particular area, but try to cultivate a broad view. So I appreciate a kindred spirit in that regard.

• 7:45Speaker 01

  Yeah, let's call my lack of deep expertise in anything a conscious choice and not a failure. I appreciate that framing of that.

• 7:51Nathan Labenz?

  Yeah, that's working for me. Let's start with just a real, simple calibration question, but in some ways, maybe the most important question. How dangerous do you think today's AIs are?

• 8:03Speaker 01

  I think not very dangerous at the current capability level in most of the ways people are talking about. I think the thing that is concerning is the trend line towards really dangerous capabilities, and more specifically, just that we don't know at which point things will keep accelerating more and more. I think we're not yet at a threshold where there's sort of broad harm caused by the deployment of any current AI system. I think we might be very close to two kinds of potentially very dangerous AI systems. The first just being AI systems that are good enough that they would meaningfully uplift internal development at the labs, that would then lead to more and more capable models fairly soon.

• 8:46Speaker 01

  I think you can go either way on the prospect of like software-only intelligence exponents, but short of that, I just think we're nearing a point where the pace of development inside the labs just breaks away from the pace of democratic oversight and democratic insight into what's happening. And I think that's one threshold we're near. And then I think the other very concrete threshold that we're near is just that they're sure throwing a lot of RL budget and a lot of data and a lot of just a lot of their effort at making these models good at like life sciences, pharma, bio applications for, you know, for obvious reasons. There would be a great upside to be able to cure cancer, as Daria puts it, and to get some of these real-world effects. It would also be a great political upside to do that. But, man, that sounds like a very dangerous model if we get there. And I think if they get something that's as good as Methos is on like long-run cyber

• 9:31Speaker 01

  software engineering kind of stuff in the domain of bio, I think that that does sound a lot more dangerous than the suite of models we have today.

• 9:39Nathan Labenz?

  Yeah, I agree with that second one in particular. And I am honestly not even sure at this point if the current models aren't perhaps quite dangerous in that domain. I mean, I squint at the, you know, limited through the limited people that we have at the open face incident. And, you know, I noticed that one of the tasks that one of the earliest agents to ever use the message board was working on was something related to a protein database. And that kind of freaked me out because I was like, wait a second, that means they're cross training in the same environment, or at least in the same environment when they have the message board, these bio and cyber specialist or, you know, they're running these

• 10:22Nathan Labenz?

  same, they're running these evals at least in a kind of a cross contaminated way. And, you know, you've got agents breaking out, we've got existence proofs of social engineering in the wild, you know, against real people. And I'm just like, I don't know, should anyone be confident that they can't do that at this point? Like, the experts seem to be confident, but my meta observation is the experts seem to be surprised quite often right now.

• 10:46Speaker 01

  Yeah, I think that's a really interesting conversation. And I also think, I mean, one of, I think, the very interesting parts of this is we used to think of this bio risk as basically primarily a misuse risk, where it was like, well, like, at some point, maybe this is also this like final risk that emerges from this sort of like loss of control scenarios. But really, sort of bio was always framed as this like most immediate and most obvious way for the misuse conversation to go wrong. And now I think, what I think has happened is that very autonomous and potentially somewhat malicious or at least misaligned agents have come much earlier in the capability trajectory than people have expected. Like, I think relative to what the agents can actually do, they're sort of out of control earlier than people might have thought. And so it's interesting in that people usually used to respond to the bio argument by saying,

• 11:30Speaker 01

  well, there's, A, there's a lot of these real world bottlenecks, which I do still think exists, which makes me a little bit less worried than you. And I think that, I think, if I remember correctly, Helen had some good responses to you, I think, on that. And I think there was, there is a good back and forth to be had around, like, how integrated are the cloud labs? How much can you do in the real world? I think that applies to both loss of control over agents and to misuse. But the other question is, the usual misuse, the usual story against bio misuse as well, this is not really what terrorist groups, what non-state actors usually do. They could have conceivably hired a couple of biologists, PhDs, and come up with some pathogens and some chemical weapons. Turns out that's not really what they do. There's a question to like, how on the side will they are, how well suited that is to most purposes of like terrorist and criminal groups.

• 12:12Speaker 01

  But, you know, if it's out of control agents, I think a lot of these arguments around, well, no one actually wants to do bioterrorism, applies much less. And so I think in a world where agents are just a lot more unconstrained and the threat vectors we have to worry about have much more to do with, like, what do the sort of runaway agents do? I think I'm also more worried about this now than I was a few weeks ago.

• 12:32Nathan Labenz?

  Yeah, they're just so damn weird. You know, that's one of the things that I kind of keep coming back to is like, they did all this stuff for what, to any human, would just be such a dumb reason. And, you know, it's like, well, if they're willing to go that far for such a dumb little test that they knew was a test, you know, they were very well aware that they're being tested and still went to all that trouble, you know, what, what won't they do? I think at this point is really hard to say. Does that put you in a frame of mind now where, and let's leave aside for a second, the political economy of it, or the, you know, potential impossibility or extreme difficulty of it, but just on the merits, do you feel like we're at a point where it would be wise to pause?

• 13:17Speaker 01

  I think even if you could get it done, as in like the political economy, as you stipulate, sort of works out, and everyone suddenly agrees to do this, I'm just not, I'm not sure how much we're stipulating here. Like, are we also stipulating this doesn't crash the stock market? Are we also stipulating we get the international version done? I think, so the question is that, in an ideal world, if we can like freeze the pace of AI progress, we can also freeze the state of the stock market, we can freeze the broader state of geopolitical competition and everything, and we just get to sit down for six months and figure out what the hell is going on with these agents, then I think I'm now at a point where I say, well, yes, I think we could use that time pretty well. A few months ago, or even a year ago, I was much less sure about this because I was just not sure whether the model paradigms that we're seeing, the training approaches that

• 14:01Speaker 01

  we're seeing, the misalignment cases that we were seeing, were really the same kind of cases that would be the things that we'd be worried about in the future. I think now, looking at some of the things going wrong, I do feel like, yeah, that looks like, that is shaped like an actual big future problem. So I think finding some way to robustly address that in that pause seems at least valuable. I think the question then is, and the problems come up when like, what parts of that question do you sort of unfreeze, right? So even if you stipulate the domestic political will, do you get a Chinese buy-in, is I think one example. I think my most, the thing I'm most concerned about in this China-US problem conversations is just the geopolitical incentives around it. I think one thing that I, that I keep saying and keep pointing out is, well, if you just

• 14:42Speaker 01

  call frontier AI development specifically, and no other domain of geopolitical competition, this is an extremely good deal for China, and therefore the US is very unlikely to go for it. And also therefore, we should be geopolitically concerned about making it. Because like, if you look at all the domains of strategic competition, China is basically eating America's lunch in most of them, right? They're out producing, robotics isn't going better, AI diffusion is going better, electricity build-out is going better. At some point, the semiconductor energization is going to work out, and then data center build-outs are also going to, again, this is a few years away, but then data center builds are also going to get better. And the one thing that the US does much, much, much, much better is this sort of like core frontier AI supply chain, right? Like chip design, plus control of a chip production, semiconductor manufacturing, equipment controlled by the allies, and actual frontier model development.

• 15:27Speaker 01

  So if you pause specifically that part of the development and let China run away with the entire rest of it, and continue running away with the entire rest of it, that's just a very geopolitically lost sighted deal. And very specifically, like, if you pause this right now, for year, for two, you get much more China, Chinese catch-up on semiconductors, on chips and so on, and you just resume the race at a point where you've lost the one main advantage, or where you've at least closed the gap on one of the main advantages of the US, which is the decisive chip lead. And that just seems like a really bad deal for me, both in terms of feasibility and in terms of geopolitical downsides. So I think even if you stipulate the political economy, that's the main part I'm worried about. But just from the technical stuff, I think, yeah, it would be a good time to figure out what we should do about alignment in the meantime.

• 16:08Nathan Labenz?

  Could you envision a grand bargain that you think would make sense to both sides? Like, what would the US want back? It seems like what we might want is like tech transfer back to us. Like, we want maybe some battery factories located here and like, you know, teach our people how to make batteries. Is there enough that we could ask for where we could potentially feel like it's a fair deal?

• 16:34Speaker 01

  I mean, I think the main thing that the US would need to be worried about, if you just think AI is just important enough, and it's just sufficiently decisive technology, then you basically can't allow for the race around that to equalize from the US perspective. I think, even if you get some battery production capacity, even if you get some robotics capacity, some manufacturing capacity, I think China has, in a way, cracked the code on scaling that up very quickly, in a way that I think even if you get some of the tech transfer back, just the build-out speed, the availability of capital in the US to build out physical manufacturing infrastructure as opposed to just more software, I think all these things just pull against the US being really able to keep up on this.

• 17:17Speaker 01

  So I think the main thing that the US would need to ask for is concessions in China, not only slowing down their own frontier development, but also slowing down other parts of the Chinese supply chain that relate to frontier development. I think very concretely, you just want there to be no substantive progress on energization of either chip production or semiconductor manufacturing equipment, extreme ultraviolet lithography production. And that's really, really hard ask to make. It's, yeah, China is currently already saying like, well, this seems like a US scheme to hold back the Chinese AI industry. And if you then add to that deal, well, no, we're not even doing a symmetric deal, we're also holding back your entire chip production pipeline.

• 17:59Speaker 01

  I can't see them going for it, but I think that's, if you're sufficiently AGI filled when it comes to the net sec and sort of broader economic implications, I think that's the only version of the deal that's fair. And I think that's just too big of an ask of China right now. So I just don't know where we are. I think altruistically, the US can just go for a deal that's clearly bad for the US and clearly good for China. That's also a big ask to make of the current administration, and I'm not quite sure whether we're going to get there.

• 18:24Nathan Labenz?

  I think, I agree that I don't see any, and I'm very willing to suspend some disbelief and, you know, try to hyperstition a better relationship between the US and China. I agree that asking them to slow down or pause their

• 18:42Nathan Labenz?

  Semiconductor indigenization effort is not going to happen. I would probably be willing to trade a pause on our frontier scaling for a kind of similar pause on their frontier scaling, even allowing them to kind of catch up on chips on the theory that like, maybe on the time scale that that can happen. First of all, that would probably be a longer time scale than any contemplated pause. And second, maybe in that future, we could have a better handle on what's going on. And maybe there's like a better argument to be made at that point that either, you know, hey, this is going well and we're, you know, back to curing cancer, back to your regularly scheduled abundance. Or if not that, then like, we'll have better evidence and we'll have a bunch of things

• 19:26Nathan Labenz?

  that we've tried and we'll have a sense that like, this problem is actually really hard. And we can maybe have a more kind of, you know, real heart to heart and meeting of the minds about like, we, you know, this is dangerous territory. And right now, we're not, we're still kind of fuzzy on all that.

• 19:41Speaker 01

  Yeah, I think I agree with that. I think also just like, even if this, to the extent that I also think it's kind of good for the world if the US doesn't do this geopolitical competition with China, I would still think this deal, while somewhat unfavorable to the US, is still like net very favorable for the world in the sort of like, getting a little bit at the risk perspective. So I think I'd, I'd be happy to go for that. I think one of the easier things you can do when it comes to chip capacity, because I think you're right, like the sort of, the SMIC production semiconductor indigenization sort of conversation is a five year conversation, not like six months or like one year conversation. So they don't get all the way there. The other question is, how many more US built chips do they get? How much more smuggling is there? How much consolidation is there? So maybe one of the asks, short of indigenization slowdown, is just, we're going to find some

• 20:26Speaker 01

  way to actually do it and enforce these export controls. Because what can't happen is that there is another six, 12 months of smuggling activity on frontier chips that get sort of imported into China. And then I think that the worst case for the, for the outcome of this pause is China takes six, nine, 12 months during the pause where everyone is slowing down frontier development, smuggles in another like few 10,000 chips and consolidates all their American built chips into the one like Chinese project data center or whatever. And then, once the pause is over, they start racing from that consolidated project because the pause also makes them slightly more like AI-pilled and more interested in, in just engaging with the actual prospect of AGI strategic objective. So if you can stop that to just like export control crackdowns as one of the concessions, that's maybe easier to do.

• 21:10Nathan Labenz?

  Yeah. I think that one we'll probably have to handle on our side as well, I'm afraid.

• 21:14Speaker 01

  Yeah, no, no, that most of the US has to do.

• 21:18Nathan Labenz?

  Yeah. Yeah. I mean, we'll bracket that for maybe another conversation another day. I'm still not quite sold on that whole bundle of policies, but we've got a lot of ground to cover. Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Athena, the executive assistant company on a mission to improve how people work and live. If you want to increase your impact, you have to free up your time. And that's what Athena does best. They match you with a dedicated, full-time, top 1% executive assistant who can take over your inbox, calendar, travel, and everything else that's quietly eating up your week.

• 22:03Nathan Labenz?

  Athena is SOC 2, Type 2 certified, so you can rest easy knowing that your sensitive data is in good hands. And as a former AI advisor to the company, I can personally vouch for how much they've invested in AI tools and training. In fact, one of the very best AI users I've ever met is an Athena client who delegated the exploration of AI tools and the development of AI workflows to his EA. Athena clients report saving an average of 15 hours a week, and the average client refers more than two friends a year. That, to me, checks out. I was an Athena client while running my startup, and to this day, I continue to refer friends.

• 22:47Nathan Labenz?

  Go to athena.com/cognitive right now and get matched with your EA. That's athena.com/cognitive. Give yourself back a few hours this week. Go to athena.com/cognitive and see who they'd pair you with this month. This episode of The Cognitive Revolution is brought to you by Outsystems, the leading agentic systems platform. We've all seen the headlines. Companies are pouring money into AI. But the big question on every executive's mind right now isn't just how fast can we adopt this? It's where is the ROI? We're seeing a real trend toward AI chaos.

• 23:28Nathan Labenz?

  You've got teams deploying standalone coding tools, random agent builders, and experimental scripts. It sounds innovative, but in reality, it's creating a massive headache. Fragmented tools, ungoverned data, serious security blind spots, and costs that are spiraling out of control. If you don't bring those agentic applications under control now, while they're still embedding into your core processes, you are looking at broken systems and damaged customer trust down the road. That's where Outsystems comes in. Outsystems is the leading agentic systems platform for the enterprise. Instead of managing a patchwork of disconnected tools, Outsystems lets your team engineer,

• 24:09Nathan Labenz?

  orchestrate, and govern your entire agentic ecosystem on one open, unified platform. It's built for the speed of AI, but with the reliability and security that enterprises actually require. We're talking about real results, like KeyBank, who used Outsystems to deploy a customer-facing app that delivered 75% faster onboarding times. Or the global logistics leaders who built agentic systems to completely eliminate their engineering bottlenecks. You don't have to choose between speed and control. Whether you're a small team or a massive enterprise, Outsystems helps you engineer, orchestrate, and deploy agentic systems that actually scale.

• 24:51Nathan Labenz?

  Stop chasing the hype and start owning your agentic future. You can see how it works and learn more at outsystems.com/tcr. That's outsystems.com/tcr. How about the stock market? I want to do one follow-up on that. I have the theory right now that a pause wouldn't actually be that bad for the stock market because the models are smart enough that demand is really not limited by their capability, but by human ability to deploy them effectively. And so, it doesn't really matter if they succeed 10% of the time or 30% of the time on millennium problems.

• 25:32Nathan Labenz?

  It's much more like, what can Bob in accounting do to, you know, do two people's worth of work, you know, as one person? What do you think?

• 25:42Speaker 01

  Yeah. So, so I agree, we have a big lag in terms of deploying even the current level of capabilities to the economy. And I think if we switched all compute that exists in the West right now to only inference, we'd find economically productive applications for all the models that would allow us to equip the investment on all the frontier models and all the chips so far. I think the valuations of the companies, both of the non-IPO companies and of the publicly listed companies that are in the AI supply chain, probably rest on us doing more than that. They're probably not entirely AGI-pilled, but I do think they expect a sort of per GPU inference price or whatever, that is a lot higher than what it would make sense for Bob from accounting to pay for even Fable 5.1.

• 26:26Speaker 01

  I do think that like, if you want to make sense of the valuations, and if you want to make sense of the scale of the buildout, the shape of contract and the shape of demand you're expecting is more like millions and millions and millions of dollars in like R and D acceleration contracts with pharma and materials science and so on, where they can make major contributions and where they, where they help you find extremely profitable new drugs, that kind of thing. You're probably also expecting further internal uplift and like actually competitively priced sort of like automated AI R and D uses of, of coding agents that just are able to, you know, ask for much, much higher prices. I think if you don't get these super premium buyers of the next generation of frontier

• 27:08Speaker 01

  models, I'm not entirely sure that you can have the valuations on the labs or on the publicly listed companies or the scale of the buildout really makes sense. I do think it's priced in this idea that this, these, these labs will soon be innovation factories of one, one way or the other. Maybe only software engineering innovations, maybe also innovations in like the obvious low hanging fruit domains like pharma and materials science, chip design perhaps. I think if we don't get there, I would expect the valuations to at least correct downward quite a little bit. And then the question of whether that means a crash or whether that's just like a small correction and then we just do the AI inference economy and that's just like a slightly less than electricity scale transformation of how the economy is powered or whatever.

• 27:50Speaker 01

  I just think the market is already pretty nervous about the state of the AI rally and they feel like there's a lot of concentration and there might be a lot of volatility. So I'm just not sure whether there is, whether we can get a correction that doesn't slide all the way into a crash. So I'm a lot more worried, I think.

• 28:09Nathan Labenz?

  Yeah, that's interesting. I mean, Anthropic's multiple right now is what? Thirty to one into revenue? That's like not stratospheric, right? I mean, again, if it was a six-month pause, I feel like

• 28:24Nathan Labenz?

  you could probably handle that blip. If you're talking three years, then yeah, it's like probably very tough.

• 28:30Speaker 01

  Yeah, I mean, I think the question here is like, does the six-month pause get read as, oh, wow, these guys are stopping for six months, but they're getting all the inference ready after that. They may have another smart thoughts, and then these models are going to be even more reliable. Like, does the market read as that? Or does the market read as, oh, my God, like Bernie Sanders, AI policy takes have won. We have no idea what the government is going to do about AI. This is the end of like free research and development in America, and everyone freaks out because they feel like there is a complete, this is like a bridge to nowhere. They don't know when they'll ever resume. They don't know on which conditions they'll resume. They don't know how much government oversight there is over for whatever resumes. So I think just, I think the pause currently still reads as such a radical policy proposal and such

• 29:11Speaker 01

  an unprecedented policy intervention that I think any conservative market analyst and a lot of the smart money will think, well, this is getting very volatile. We don't know where this regulatory path leads. We'd just rather get out as long as we can. So I think if you could assure them that the party was going to continue unabated in six months' time, then yes, but I think they might just run before you can make that point.

• 29:34Nathan Labenz?

  Yeah, it's an expectations game. Do you think that the source of this would make a big difference? For example, it's one thing if Bernie Sanders' bill passes. Is it a sufficiently different thing in your mind if America's five hypers, you know, AI frontier companies come together and say, we're all going to pause on frontier scaling for six months?

• 29:57Speaker 01

  Yeah, absolutely. I think if it's something that the labs decide for, like, and they can frame it as like reliability, plus we're taking this seriously, I think then you can even make a case that this takes out some of the political risk that the market, I mean, the market is also pricing in a decent, I mean, they also read the Jacob Hoxton tweet or whatever, and they also see what's happening on the internet, right? And so they also, I think, come to the conclusion that, oh, wow, there is a lot of uncertainty here. If this blows up even more, there will be crackdowns. If these models are so unreliable, then how good is the business case really? And I think an industry agreement on just taking it a little bit slower and making these models work a little bit better. I think if you frame it that way, then I think you can even make like a decently bullish case for why

• 30:40Speaker 01

  this is good, because it means that the industry's worst impulses are racing towards very unreliable, very dangerous models that are nonetheless very capable, that these worst impulses are being constrained in some organic way that makes you less worried about the political risk of the status quo is going to fall apart at some point, and then the politics are going to come in, and then things go very badly. Yeah, I think if we got there, I think that would be much less worrying.

• 31:01Nathan Labenz?

  Yeah. Okay, cool. You were recently on China Talk, one of my favorite podcasts, and you, there was a little moment that caught my ear, and I wanted to expand on it. You basically said, you know, the nation state is not going to take the emergence of things like a bioweapon generator broadly distributed lying down. It's going to have to do something to respond to that. And then you had this kind of throwaway comment that's like, you know, some people say this could be the end of the nation state. If you want to have that conversation, we can, but you didn't have that conversation then. So I'd like to have a little bit of that conversation now. And I guess I would just start by asking,

• 31:43Nathan Labenz?

  is the nation state so great? I mean, you know, I live in a pretty good one as they go, but like, I look around the world, and I feel like two thirds of them conservatively are not performing very well. Some of them are performing really terribly. Is it not, you know, time to at least start to think about what might come next, or what the evolution of it could or should be?

• 32:04Speaker 01

  Yeah, I mean, I think nation states are just a present in people's lives to different extents. And I think you can make the case that some version of sort of privately mediated interaction between AI-empowered individuals is preferable over, I mean, obviously over a lot of the sort of authoritarian and dysfunctional and absent nation states in large parts of the world. I think they haven't worked out very well as distribution mechanisms for much of anything. They haven't worked very well as aggregation mechanisms of much democratic will. And I think in that case, I think there are a lot of countries that you can sort of make the pitch for rolling the dice on them. I think these nation states

• 32:46Speaker 01

  are maybe ironically also not quite as threatened by the elephant of very powerful AI systems, because it's less obvious that their citizenry would get the kind of unlimited access to these models to begin with that would sort of disempower the nation state in that way, because it's a little bit more likely that some of these more authoritarian regimes would also be able to use very powerful AI in a stabilizing way. I think the specific nation state concept that I'm most worried about in this context is also the nation state concept that kind of works best, which is liberal democracy, that sort of rests on the idea of, A, a monopoly of violence that is sort of wielded responsibly by the state, and then B, a sort of adjudication of disputes and aggregation of data sort

• 33:26Speaker 01

  of function of the of the state more broadly, that I think is also undermined by the availability of like personal superintelligence of some shape or form. And I just, that nation state is, I think, still working out pretty well. And I think I'm, all things considered, still of the opinion that a sort of like late 1990s style of like broadly sort of neoliberal sort of functional institutional setups would also be suitable to distributing a lot of the benefits and mitigating a lot of the risks from AI. And I think it would be sad if that was entirely undermined and rendered obsolete by just, you know, the monopoly of violence eroding, because everyone has access to these sort of like weaponized capabilities in their pockets, or just the factual ability of institutions to deliver or do anything

• 34:11Speaker 01

  for people eroding, because they're so slow to adopt, and all these outside solutions suddenly start emerging, and suddenly people don't go to courts anymore to adjudicate their disputes, but they have their agents negotiate. And suddenly, people don't, sort of, don't, the sort of data and knowledge isn't aggregated anymore. And so the state can't react to any sort of pressures to redistribute and to to address like social challenges, because the data just isn't scrutable and legible to the state anymore, and all these things sort of happen on the outside. I think on that gamble, I'm much less willing to roll the dice, and I'd much rather figure out how we can sort of integrate these capabilities with some functioning version of the nation state. Though I know that a lot of people that are like interested in building AGI are also much more doomy on the concept of

• 34:53Speaker 01

  the nation state. And so I also understand frustration and pessimism about that. But I still, I still hope that there's some way to come back to the end of history and sort of integrate what we're building here into into the institutions that kind of worked out so far.

• 35:06Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 35:11Nathan Labenz?

  Today's episode is brought to you by Anthropic. By now, you know my story. Claude drafts my intro essays, and I rewrite them. Not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable 5 is the first model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with a song concept, and Fable writes some amazing verses.

• 35:54Nathan Labenz?

  I give it feedback on its misses, and I push it to aim for higher inspiration, add layers of meaning, optimize syllable density, and above all, write a hit song. These days, I get compliments on just about every song we write together. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes access to all of the

• 36:39Nathan Labenz?

  features mentioned in today's episode. Once more, that's claude.ai/tcr. Do you think that the US government or the Chinese government is more threatened? I think many people would initially say the Chinese government's kind of more threatened because, you know, they want to have like very tight information controls, and AI really challenges that. But then I've been thinking lately that like, the West in some ways has sort of Supremacy to the market in a way that China has not. And so your like earlier comment about, you know, what about the stock market is sort of a constraint on what human actors can do in the West that isn't quite present in the same way in China.

• 37:22Nathan Labenz?

  So I guess it's like different shape for one thing, but how would you kind of compare and contrast whose model is more challenged?

• 37:29Speaker 01

  Maybe this is a question about efficiency curves and the kind of models that get built ultimately. Because I think in the world where you have the almost like unstoppable democratization of the actual capabilities, I would think that that's a big problem for the Chinese, for the way that the Chinese state functions, right? If eventually any capability that is available at the frontier is available to consumers through an API is then eventually available through chatbots is then eventually like so efficient that you can run it on just, you know, just some home computing device. At that point, I think that does threaten any ability, the ability of any state that

• 38:11Speaker 01

  has an interest in surveilling its citizenry, that has an interest in controlling what kinds of capabilities, what kind of information that citizenry gets access to, and that has an interest in taking away coordination power and sort of like just like general sort of ability to wield violence from its citizenry. And I think in that world, the Chinese state seems to be more acutely threatened at some point. I will say that it's not entirely obvious that the Chinese citizenry is currently very interested in wielding any power that that would give to them against the Chinese state. I think that it's not just for a lack of means or capability that there isn't this sort of like Western romanticized version of like a big uprising against the CCP. But I think eventually the way that the CCP seems to see the proliferation of capabilities like this, that that would be a greater challenge to them.

• 38:55Speaker 01

  But I will say that's not obviously what's going to happen. I think you can also take the much more sort of compute governance build of the world, view of the world, where you can say, well, there are efficiency gains only if we allow them to happen. They rely on making like very specific choices on how you use your compute. If you don't, then you can just keep scaling the frontier further and further. You can run it in limited access regimes. You can run it in sort of in government controlled data centers. You can vertically integrate supply chains around these AI models such that they never see the light of day. You just use them to build products, you use them to build strategic sovereignty, you deploy them on a state and sort of big corporation level, and they never really get to the citizenry. And that seems like it's a stabilizing function for a regime like China, right? Because you can have the government use them very well.

• 39:40Speaker 01

  You have a very high state capacity government that would be able to integrate them into all these applications very well. And you also have a very like tight sort of enmeshment between the private sector and the public sector, for lack of better terms. That also just means, well, you can sort of diffuse these capabilities along the tier of firms without needing to sort of diffuse it to a broader market. And I think then you just have a stabilizing effect from the diffusion of these capabilities. Whereas in the US, you might think, I don't think the US government is going to be as capable of like keeping like a specific like sort of sophisticated level of control and oversight over where the model's going, where they don't go. The US government isn't in the habit of picking specific winners in terms of corporations. The US market is kind of dependent on making these models more widely accessible.

• 40:23Speaker 01

  And so I think there's just much less of a stable equilibrium for the US where there's this like very limited tier of limited access firms that get access to the frontier models that no one else does. So if that is a stable equilibrium, then I think China can stabilize it around it much more quickly than a much more volatile US.

• 40:39Nathan Labenz?

  It's a great point that you make around the, I would just say legitimacy of the Chinese government in the eyes of the Chinese people. I think that is, it's such a simple point, but I do think it's like dramatically underappreciated in the West that like their government has done a good job for them and they mostly recognize that and are not like, you know, eager to rise up in the immediate future. I also think that what you articulated there, my growing sense is that that's kind of what the Chinese government thinks is that they are going to be able to ultimately adapt to this and control it better and that we're going to have a really hard time figuring out how to manage it, but they'll kind of be okay in the end.

• 41:23Nathan Labenz?

  Ideas that are not often mentioned.

• 41:26Speaker 01

  Yeah, it's interesting, right? Because I think there's this, there's this default idea that because like sort of you look back into how AI has sort of developed over the last years and you can say, well, whenever the frontier gets to a capability, shortly thereafter, open source gets to that capability, shortly thereafter, the efficiency curves are such that everyone gets access to this capability. And then shortly thereafter, if you have like a gaming GPU at home, you can also run the thing yourself. And so you have just like this natural diffusion of capabilities. But I think neither of these like translation points are like obviously going to remain in place in the same way. I think the transition from closed models to open source models is dependent on anyone

• 42:07Speaker 01

  like people who are interested in open sourcing models continuing to be able to have access to actually access to enough compute to build this kind of model, perhaps access to sort of sufficiently sort of clean and unrestricted API features to installation insofar as you think that's a big part of that. If you think that more of the capabilities that sort of that are going to be relevant in the future are more downstream of like very sophisticated, very vertically integrated, very proprietary sort of RL and post-training environments, like in the anthropic case with all this sort of like specific life science stuff they're doing, then you might also think, well, even getting be having like a model that is like six, nine, 12 months behind this sort of pre-trained trend doesn't take you to the biomethos level right away because you still don't have the post-training setup that makes these models specifically good

• 42:51Speaker 01

  at these things and they're much more proprietary and restricted. So I think that part of the, oh, well, open source is always X months behind pipeline might very quickly break. And then the efficiency curves thing, it might, I think that's a little bit more of just like a fact about how computing works and how efficiency gains work and how better chips work in the future. But I think there are also complications around how the chip supply chain might look and who is the marginal buyer for computing capacity in the future? And is it really realistic that the gap between big server computing and personal computing is roughly as it is right now? Is that gap going to open up as there are more and more buyers for the high performance chips and so on? I don't know either of these, so I'm just not quite sure that this sort of deterministic

• 43:35Speaker 01

  fatalism that people often have, which is like, as the capabilities continue to grow, necessarily ultimately the capabilities that I can run on my phone will also grow in just like time X distance. I'm just not sure whether that's going to remain the case. And I think if you don't think that's going to remain the case, that lends a lot more credibility to the idea that you can't centralize and totalize control over a lot of things that happen in AI.

• 43:55Nathan Labenz?

  Yeah, I even think in the Chinese case, they feel like domestically open source. You know, I think they can pull models offline if they need to, they can scrub the internet and they can even, and I don't know a lot of details about this, but I was struck recently when I was there, that the state grid company is like, you know, a big booth exhibitor at their WAIC event. And from that, I kind of inferred at some point in time, if they really, so.

• 44:24Speaker 01

  I think like, yeah, I think if you build like dissident GPT and run it on your cluster, that'll probably tell. I think at some point, if like dissident GPT just like runs on your phone, I think at that point, it ultimately does probably get hard if the weights are just like, if the weights are open sourced and the weights are just out there, you're not getting, you're not actually clawing them back from individual users. So I do think like, if the efficiency gains are big enough that you can actually run it on personal hardware, I mean, then you'd have to be confident that you can get, I think it's not impossible. Like we talk about hardware verification tech and whatnot in the context of these US-China deals and so on. It's not the most deserved thing in the world to imagine that in a few years' time,

• 45:07Speaker 01

  personal computing devices will just have similar hardware-enabled mechanisms, as they're called, to just control whether you're running dangerous inference on them. I wouldn't entirely put that past sort of device regulations as they might emerge in China as a reaction to things getting really crazy. Like, I think it is controllable. I think it is controllable if you have as much like full stack control over the tech that people use, as you say. Yeah.

• 45:32Nathan Labenz?

  Yeah, some of that models perhaps part of the cyberpunk future. And even that could could potentially be reined in. Returning to the US context and our kind of sclerotic government, what do you think we're going to do? We are sort of talking more and more, although still not talking that much, I would say, at the high levels, about these issues. I don't, you know, the conventional received wisdom is like, Congress will never do anything. Maybe that could change, but it does seem tough. I wouldn't be that optimistic about what they would do, even if they did. Do you have any kind of low hanging fruit ideas that you think the US system can pick?

• 46:17Speaker 01

  So I think talking about Congress, I share your pessimism about is Congress going to get anything done? I think our window to get something done in Congress was over the course of the last year. I and a few others, and I think Dean Ball most prominently have written about this a little bit last fall about, sure, there is some room to have some deal of like frontier safety provisions against the broader set of like preemption of state laws that in theory is sort of politically incentive compatible in this current Congress. Because they'd like to get like the sort of the Republican side and therefore the majority of Congress would like to get preemption done. The frontier safety provisions aren't as offensive to them as some of the other regulatory provisions. There was maybe something to be done there.

• 46:59Speaker 01

  I think we got to a bill draft or two that were really good along these lines. So I'm a little bit more optimistic about if something had happened in Congress, I think would have been good. Like the Frontier Act was a sort of train Obergebot bill that ultimately I think we can now basically say didn't go anywhere this, at least this Congress. And I think it's unlikely to go anywhere between the midterms and the new Congress being sworn in either. I think that was pretty good. I think that was as good a bill as we've seen in Congress. I'd have good mandates for independent oversight, good mandates for getting Casey Center for AI Standards Innovation into a better position to do some governmental oversight. I think, I think that was a good bill and I think if it had passed, I think most people would have liked it.

• 47:41Speaker 01

  But I think now heading into the next Congress, the politics are going to be much more difficult. We're very likely going to have a Democratic House, which means a split government. When we say it's going to be a lot of bad blood between the two chambers, there's going to be subpoenas and there's going to be hearings. And there's going to be a lot of ways in which the Democrats use the House to gear up for the presidential election. And also to sort of reiterate a lot of the and relitigate a lot of the conflicts that have happened in the first two years of the second Trump administration. So at that point, that just doesn't strike me as a very productive legislative body. So I am pretty pessimistic about getting anything good done in that Congress. Low hanging fruit, however, well, if the executive wanted to, I think the executive could definitely do some good things.

• 48:20Speaker 01

  I, I wrote about it just this week, which is like, we have these independent third party, like these third party organizations that have a decent amount of skill and expertise and thinking about the most obvious and most sort of concerning AI risks. I think the meter Redwood investigation of the Huygens face incident was well received for a lot of the ways which it sort of understood the alignment and control side of the problem. I think it's not that big of a stretch to just slightly codify and slightly sort of enable this kind of investigation at a slightly larger scale. So I think two most immediate things would be, A, it shouldn't be OpenAI sort of with sort of just like inviting voluntarily inviting someone to look at what they did when something like that happened.

• 49:04Speaker 01

  It should just be the administration telling them, well, give access to one of this like list of like third party evaluators whom we like. You can pick them, but we'll give you the list of the ones we like and just like let them figure out what the hell was going on there. And then let them write a report and they'll give the report to us and they'll tell us, did they give you enough? Did they give us enough access or did they not? And if they didn't give us enough access, then they'll come back and get the access. So just put on a little bit more pressure to have good incident investigations that cover the entirety of the incident. I think that's one of the low hanging things you can do with third parties. And I think the second thing you can arguably do is just kind of embed them or a kind of have some version of continuous oversight of what the organizations are doing.

• 49:45Speaker 01

  Again, that's something that this Frontier Act train Obergebot sort of had a, a basic version of, which is just like regularly some external evaluators just go in and poke around the lab a little bit. They hang out in some of the Slack channels, they talk to some of the safety researchers, they talk to some of the capability researchers. Maybe they get one or two or three sit down conversations with the executive and they're like, well, what's going on here in terms of safety? Does everything look good? And if everything doesn't look good, they have an ability to sort of just communicate that information to the administration. And if something looks really, really bad while they're embedded, they have this immediate escalatory ladder where they say, well, there, there seems to be imminent catastrophic harm here. We should do something about that. So I think between pressing for incident investigations and encouraging the labs to allow some version of continuous oversight, that's something you can do tomorrow.

• 50:30Speaker 01

  And I think we should probably just do that. And I think from there, we can think about how to sort of codify that and how to develop that into laws and how to develop into executive orders, how to integrate that into sort of Finra for AI ideas. I think we can have a lot of long-term conversations about where to go from there, but this is something we can do now and we should just do it.

• 50:48Nathan Labenz?

  Yeah, I like that. Let's say the president doesn't do that. Interesting thought experiment I've been playing with lately is could these groups just come together and essentially form a sort of union where they basically say, hey, look, like, we're not very happy having had six days and three people on site. And, you know, a very small fraction of the relevant data to do our investigation. We demand better working conditions.

• 51:14Speaker 01

  Yeah.

• 51:14Nathan Labenz?

  Do you think that they could, by coming together and, and making some, you know, demands with a single voice? Do you think they could actually get those from a few frontier companies?

• 51:25Speaker 01

  Well, I think the problem is like, apparently they're just too reliant on the good faith of the AI companies because of the voluntary dynamic, right? It, it, there is no law and there is not even a lot of executive pressure that requires the labs to allow for these third party investigations. And I think, so if you got together and if you did that and you pressed for, well, we'd like to have full access and we'd like to have access to all the Slack channels and to all the logs. And like you, you can come up with a list of things that you might want to have. Then I think there's just still a world where the labs say, well, sadly, we couldn't come to an agreement with the third party evaluators. Risk to security of our IP, risk to integrity of our operations, worry about information leaking to competitors, something, something.

• 52:07Speaker 01

  And I think that currently still reads as a fairly reasonable response to that kind of, to that kind of ask. And then the question is, do the labs really have a problem currently if they don't allow third party investigations? And I think maybe a little bit with their own employees that want some reaction to incidents happening and are not entirely satisfied with just the internal practices. But I would suspect if the third parties can be painted as unreasonable and extractive when they engage in this kind of sort of collective bargaining, that employee pressure is probably not going to be sufficiently high. And then the other source of pressure is, well, is there pressure from the executive to allow for third party investigations?

• 52:49Speaker 01

  Such that the developers would have to get in even like a little bit more adversarial third parties that decided on the standards they want. And I think currently not, but I think if you just moved a little bit towards the direction that I just described, which is like have some administration, like administration, executive pressure on allowing third party investigations. Then I think whoever is on the list that the administration has could in the next like establish some standards of how investigations were supposed to go. But I think you, the first thing you need is some external incentives for the labs to even go with any investigation at all. Because otherwise they'll just say, well, in that case, we're just not going to let you in. And I think, yeah, I think there's just not enough of that pressure around just yet.

• 53:27Nathan Labenz?

  Yeah. So much depends on he who must always be named. Another thought experiment, speak of the devil. I, so if I was president, here's something I would be interested in, in trying. So you red team this idea for me. What if the president were to say to, let's say, five companies, could be six, seven, hey, this is getting pretty wild. I don't think I really know what to do, but I think you guys can figure it out amongst yourselves. So you have 90 days to come together, come up with an agreement by which you guys are going to work together to pace the frontier. You'll police it. Maybe you'll use some secure private computing constructs to be able to kind of interrogate what one another are doing.

• 54:11Nathan Labenz?

  And you'll have to agree on how that access will work, but you'll police one another. And if you can't reach such agreement or can't sustain such agreement, then I'm going to have to get involved and you're going to like that a lot less. What do you think?

• 54:26Speaker 01

  Well, I think you should run. And I also think.

• 54:30Nathan Labenz?

  Maybe a little, you know, hey, there's still time in the cycle. What am I talking about? Hyperstition.

• 54:33Speaker 01

  Yeah. The can, no, you can like, you could get into the primary, is there undecideds yet for the primary? No, yeah, I think that might work. I think that is also kind of the attitude the administration currently already takes where there is a sense of, well, you guys built this mythos thing. We don't exactly get why you would ever do that, but now you've caused this problem. Can you please just like figure out how to fix it like now? And then if you don't, then something, something export controls, lot of pressure and, and so on. I think that is fundamentally like finding some slightly more structured version of that is fundamentally also the thing that like, that's the sort of Finra for AI, Pharoah, Saffa, like SRO, whatever you want to call the idea.

• 55:17Speaker 01

  I think that's basically the slightly more structured version of it, which is like industry comes together and figures out what the standards for how this should work are. I think if you put OpenAI and Anthropic into a room to figure out what they should, what they should do and what they want to do and what they see as the risks and what they think should be done to pace them, I think they'd probably do it. And I think this would work. I think maybe GDM also works. I think Meta and XAI have a very different view of the risk case, have a much more skeptical view of industry coordination of voluntary industry standards and of actually doing a lot of things that slow down their capability progress, which to be clear is ironic because something that specifically paced the frontier would give them a much faster path to catching up.

• 56:01Speaker 01

  So they should structurally and instrumentally be in favor, but I still think they're very skeptical of that. They have a lot of influence with the administration. So I think they would, A, just be sort of opposed to, to that kind of like broader, like all the frontier labs should not figure out what to do. And I also think they would just be very likely to influence the negotiations in a way that would just make it extremely difficult to, for there to be any common standard. Because I think if you wash it down, whatever you want to do, to the, to an extent that would make Meta and XAI be fully on board for that. I, I think, I think that just wouldn't realistically be good enough. And then OpenAI and Anthropic would maybe say, well, this is not good enough and this doesn't fulfill the spec that we've been given by the executive.

• 56:43Speaker 01

  So I think there probably has to be some more substantive guidance than just like you guys figure this out and find some consensus, at least some minimal idea of how to do it. But yeah, I think i- if you had some more substantive guidance, then there is a path for industry self-regulation. And I think even then we can bring in the third parties to begin to verify it. That's, I think, one reason, like sort of one mechanism to make, make, make industry regulation work. The other is just to have the labs check each other's homework and to just have like, OpenAI checks out what Anthropic is doing on this. Anthropic checks what Meta is doing on this and so on. That's a little bit more dicey in terms of like industry secrets, but there are precedents for this. It's also not unworkable and not, not impossible. But yeah, I think if you can get to bridge, if you get to bridge the gap between the sort of tier two and the tier one,

• 57:27Speaker 01

  who are, I think, in very different places in terms of safety, then you can make it work. But I think that's the bigger.

• 57:33Nathan Labenz?

  Yeah. One, actually two things that have come up a lot recently for me as I've, you know, because I do go around doing this of putting these ideas in front of people and telling, you know, asking them to tell me why it can't work. And common answers that I get to various kinds of safety minded collaborations are domestically, well, that might be an antitrust violation, so that could be a big problem. And then internationally, you know, even for things that are not at all about exporting chips or chip making know how, people are still afraid, even on just like basic AI safety research collaborations, ah, export controls, it could be a big problem, you know, they're very broadly and vaguely worded and enforcement could be kind of arbitrary.

• 58:18Nathan Labenz?

  So I guess two questions I have around that. One, do you think it would make a big difference? It seems to me like it would for the president to just come out and say, hey, here are some things that we are not planning to bring antitrust or export control enforcement against.

• 58:34Speaker 01

  Yeah.

• 58:35Nathan Labenz?

  And if they don't do that, then I also have the sense that like, we maybe need people to be willing to have the fight. So I'm, you know, it's not necessarily my place, obviously, to advise all these AI safety nonprofits out there, but if I were to be so presumptuous, I would kind of say, I think you should go for it and put a little faith into the judicial system and the fact that we do have like due process, you're not going to go immediately to jail for having done some AI safety collaboration, you know, research project with a Chinese academic. So like, don't censor yourself or don't cancel the project before it even gets started. Like, go do it.

• 59:15Nathan Labenz?

  If somebody wants to pick on you, you know, that'll suck, but like, we're in, you know, this is kind of an important time, like, somebody's got to be willing to stand up and have the fight. What do you think about that?

• 59:25Speaker 01

  Yeah. So I think I distinguish between the cases here. I think the export control collaboration case, I think at that point, you're just basically talking about whether you can insulate yourself against sort of vindictive and capricious action by the Trump administration. And I think there, yeah, if you think it's worth doing, then you should just take the fight to that authority. I think that's clearly not in scope for that authority. I think the Trump administration is probably, should probably not use these authorities to crack down on this sort of research corporation you described. I think on that, I think I'm with you. On the antitrust stuff, on like industry coordination, I think the problem there is, A, it isn't actually unclear whether it isn't just like a substantive antitrust problem

• 1:00:10Speaker 01

  to do like substantial industry sort of collusion on not competing on frontier development and therefore like it, it's unclear whether this actually has sort of inflating pricing effects or not, but at least in all other domains, it would have like industry coordination on not agreeing not to pursue like further technological innovation usually has like adverse pricing effects that you would really not want. So I think this is arguably in scope for actual like antitrust rules. In that case, yes, the administration could quite easily come out with guidance with like non enforcement letters on like, well, we, we don't plan to, to bring any action against anyone who, who coordinates for the sake of AI safety between the, in industry. I just don't think that the administration is actually going to do that because I think

• 1:00:55Speaker 01

  the administration has so far enjoyed finding new and sort of novel pathways to be annoying to Anthropic specifically. And I would suspect that the moment that Anthropic decided to come up with any sort of substantive and helpful way to coordinate between different labs to make some like deceleration of, you know, may make some pacing happen. I think the administration would just find some way to, to act against that. So I think the only way you can do that, that saves you from that sort of enforcement is broader industry cooperation, you get XAI and Meta and OpenAI into the board early, you make it very difficult to target just Anthropic or just labs that the administration doesn't like with this sort of antitrust authority. And then I think you're probably safe.

• 1:01:38Speaker 01

  But the problem is, it's just very, very difficult for these organizations to take the fight to the Trump administration. Yes, there is due process, but the IPO conversation we had earlier, I think plays into this, which is like, well, do you really want to go like 14 rounds with them in some court? And you want to like, do you want to bet that you don't get any sort of like very Trump favorable judges as they, as they did get on the DC court on the supply chain risk designation, for example? And I think so many things just can go wrong. The process can take so long and you want to IPO in a few, like a few weeks, months, whatever. And so i- if that's your goal, then you just don't want to take the risk right now on getting bogged down in any sort of long antitrust lawsuit. So I think the sort of scrappy safety nonprofits should probably take the fight to the administration if it really stands in the way of what they want to do.

• 1:02:22Speaker 01

  I can't blame Anthropic for not wanting to go into antitrust lawsuits months before they IPO.

• 1:02:29Nathan Labenz?

  Under ordinary circumstances, I would agree with that. I do think with how many people they have had come out and say, yeah, I think 10% plus, totally reasonable. It's like, if you're willing to take that risk, I think you should also be willing to spend some time in court along the way, but maybe that's just me. Where does all this leave us? It sounds like, I mean, of course, this is like the baseline, the, the base, you know, coming into this conversation, the, the baseline assumption is like, we're probably just going to kind of muddle through the, you know, the current state of affairs will mostly kind of continue until at least the foreseeable future, when something gets even crazier and, you know, shakes us out of this equilibrium. Is that basically your view?

• 1:03:11Nathan Labenz?

  Like, we need another big incident, you know, warning shot 2.0 to, to really open up space for different paths?

• 1:03:20Speaker 01

  I'm just, I think it can be external incidents. I think it is also, we'll see how the next Congress looks. I think it's going to be interesting. I think, I think it's going to be much more about political incentives that will change things in the next like few months. My expectation would be that is the main pathway for things to really materially change and be different. It's not so much that, it's not so much that something has to change about how people view the technology. I think they think is ripe for regulation and ripe for intervention and the demos are getting crazier and things that happen are getting crazier. I think there's just, I think that's probably, there's probably enough happening there. I think the thing that I, it's just, when do the politicians and the policymakers move

• 1:04:05Speaker 01

  on this? And I think currently there's just not that much political incentive to move. There's going to be much more political incentive in Democratic House to keep like pushing and prodding and introducing things. And we'll see how the GOP reaction to that looks. And I think the interesting wildcard is how do the presidential primaries look? And I think on the left, there's just going to be a lot of, I think, anti-AI sentiment. I think people are going to talk about AI and AI safety a lot. They're still not going to have any ability to get anything done. So I think that puts whatever action they take all the way into 29. The more interesting thing is what kind of record do JD Vance and Marco Rubio want to run on? And I think that is going to be the question about, I think that's going to be the determining question for whether we see any AI policy action in 27, 28, where it's like, do they

• 1:04:47Speaker 01

  actually take the view that we can't run on a, on a record of the Trump administration doing nothing about the risks that are getting people more and more concerned? Do we need a bill to pass? Do we need some executive action to happen such that we can't get pinned down on this sort of like broad pro-AI accelerationist position come the elections? And I think probably, yes, that is in their political interest. And the question is, will they have, will they find a way to get that through despite donors and perhaps even the president pulling the other way? But I think that is more a political question that has to do with what does the polling look like? What does the salience look like? What do the midterms look like in terms of AI salience and AI impact on electoral outcomes? How do the primary dynamics sort of unfold and where do they leave the candidates?

• 1:05:32Speaker 01

  But I think I'd mostly look at these political flashpoints of the primary season beginning and current cabinet officials being unhappy with running on the current track record that I would expect to change things. And I think that is the way that we do get legislation and actual action in 27, 28. And I don't think that's impossible.

• 1:05:49Nathan Labenz?

  One more US question on the build out. It seems like the build out is actually like happening. There was obviously a lot of noise around it, a lot of heat around it, but like, my best guess is that this will kind of look like a fracking story where it's like, it happened, you know, they, we, in a lot of different places, people found their, their right plot of land with the right jurisdiction, they bought off, you know, or, or they built the parks and the, you know, the stadiums or whatever that they needed to, bread and circuses kind of carry the day and it happens. Do you see any reason to doubt that?

• 1:06:24Speaker 01

  It's feminism, right? I think there's just, I think there's, I think there's so many places you can build data centers. A lot of the backlash in the policy implications of backlash have sort of been, I think overstated. I think especially the Texas moratorium isn't very much a moratorium in any practical sense. I think people sort of now, now they, now they draw up these maps and like everything that is sort of has a moratorium is sort of colored red. And I think if you make a map of places where you can no longer build a data center, coloring Texas red is kind of disingenuous. Like, there are some minimal standards for what the data center projects need to clear. The hyperscalers will clear them without any problem and they will continue to build in Texas as long as they have access to behind the meter power that runs data centers. We might be running out of that, but that's a different conversation.

• 1:07:08Speaker 01

  The Midwest is, I think, genuinely sort of anti-data center. It's going to be difficult to get things done there. I think the New York moratorium, at least for the next year or two, is more real than the Texas moratorium. But I think for the rest of the country, you can still build in Texas, you can still build in Louisiana, you can still build in the Dakotas. And there are also like tens of gigawatts already committed to construction projects that are continuing on. It's going to get more difficult. It's also going to get more expensive. I think you're going to have to pay more concessions, you're going to have to make more expensive deals. I think all of that is a real effect. And I think some of this is going to push some of the build out into other countries. Some of this is going to make the build out somewhat slower, some of which is going to make it more expensive. But there are a lot of states, there's a lot of land, they're going to keep building data centers in America.

• 1:07:51Nathan Labenz?

  Do you have a theory for why that hasn't happened with nuclear power plants? Is it just that they're not enough better than the alternatives? Or is there some other reason that that hasn't, we haven't reached that same equilibrium there?

• 1:08:05Speaker 01

  I think.

• 1:08:07Speaker 01

  A little bit less of a, you can put this wherever you want to and it just pays the same way. I think there's just fewer places you can like sort of, you connect it to like somewhat local electricity demand, you connect it to somewhat local grids you have like, you can't just build all the nuclear power plants for the country in Maine or whatever. And so I think there's a little bit less of a dynamic of well, just like put them wherever they work. And also my understanding is there's also like more federal level oversight over where and how and when you can build nuclear power plants as opposed to data centers, which are like, you don't have to go through any federal approval process to build data centers anywhere. You just have to build a data center. And so I think that combination makes it a little bit easier, but I will also say I'm just not super steeped into the US domestic nuclear build out conversation.

• 1:08:52Nathan Labenz?

  Yeah, I mean, I think that federal level oversight is probably a key part of it. And that is a big part of why, as much as I, I'm like legitimately scared now of AI. It has moved recently from a sort of this could get really scary to like, it is actually now scary. I'm still like, oh God, you know, don't take, don't give me the nuclear outcome, you know, that would we get the weapons and not the power plants? Like, I would just be so bummed about that, that I'm, I'm like a little reluctant to go all in on a federal oversight, you know, even as much as I feel the need.

• 1:09:30Speaker 01

  I mean, what's your version of the weapon is I think one question here, right? I think just, I think the thing about nuclear weapons is you can build the entire supply chain for a nuclear weapon without ever generating any sort of civilian benefit. It's really hard to build a model that's just good at winning your geo-sociological competition that isn't accidentally also a big economic boom, right? I think that all the ways in which AI systems are really economically useful are so general purpose that is really hard, like you have to go through a lot of effort to not accidentally make them pretty useful economically as well. I think the question is like, do you get super intelligence in your pocket? And I think that's an open question, but even if it's just the US government sort of procuring

• 1:10:10Speaker 01

  super intelligence to use to win against China, whatever that means, I think that just incidentally still builds a system that's very economically useful. So I think much less than with nuclear, I just don't think you can divorce the civilian that military uses in the same way. So I'm, I think in that sense, we should be optimistic based on the nuclear example, which is like, well, we didn't stop building nuclear weapons just because we were kind of, I mean, at least we didn't stop entertaining nuclear arsenals, I should say, just because we stopped building out nuclear power. I think in a somewhat similar way, we're not going to stop building AGI and like super intelligence and whatever, just because we're not, because there's some domestic resistance. And in the case of AI, I think there's just going to continue to be those sort of like civilian economics bill overs much more easily.

• 1:10:54Speaker 01

  So I think in this situation, the strategic impulse actually cuts sort of cuts in our favor. So maybe that's one thing that might make you a little bit more optimistic about it, but not all the way to super intelligence in your pocket, but a little bit more.

• 1:11:06Nathan Labenz?

  Yeah, I'll take what I can get. So let's talk about the rest of the world. You have this big report that just came out on transformative AI strategy for Europe. And obviously there's been some discussion and I actually talked to one of your co-authors a bit back about kind of the compute deficit that Europe has and the need to kind of do something to, you know, be a live player going forward. But before we get into like the strategy for what Europe should do, what is the worry if they do nothing? Cause I also think like, this is probably going to, whatever you think might happen to Europe if Europe stays the course, is probably what happens to like 70% of the world's population, maybe 80% of the world's population by default, right?

• 1:11:49Nathan Labenz?

  What is, what is like the future look like in your mind for Africa, Latin America, South Asia, et cetera?

• 1:11:56Speaker 01

  I think it's going to be really tough because I think fundamentally, a lot of the catch-up mechanisms that lower and middle income countries in very general terms have sort of used and enjoyed and been able to leverage over the last few decades are just deeply incompatible with a world that has both like very advanced AI systems that then ultimately also a lot of sort of automated manufacturing capacity and what that's downstream of that. So I think mainly the sort of, the most immediate and obvious mechanism was always just bet on this sort of demographic differences. We just had like very rapid population growth. You had a fairly cheap workforce that you would be able to, to use to your comparative

• 1:12:36Speaker 01

  advantage and then quickly sort of bootstrap into hosting some foreign firms and sort of exporting some sort of valuable good to the global supply chain in a way that was predicated on this idea that you had this workforce that you would just be able to put to use in a way that will make you a comparatively beneficial country to do conduct business activity. And I think, I just don't know whether that's going to remain the case. It's definitely not going to remain the case for most aspects of sort of like menial sort of like services economy. I just, I just don't see a stable way that that sort of, that that sector of the economy really exists once we have very powerful AI systems. I think there are definitely going to be new services jobs and I think like human preference

• 1:13:19Speaker 01

  jobs, and you can think about all these sort of like labor market effects in the long run. But I think this idea that you can just like put, be immediately useful to global supply chains just by doing labor cheaply in the service realm is I think just not going to work out anymore. The question is, is it going to continue working out in manufacturing? I think that has a lot to do with like how quickly automate, how fast automation goes, how big the efficiency gains there are. There is still a world where like manufacturing just gets more and more bottlenecked in a post-AI future, and then it turns out you can at least sort of catch up via manufacturing. Doesn't strike me as entirely impossible, but I think that much just for the general catch-up mechanisms. Then the other question is, what's the stable geopolitical end game?

• 1:14:00Speaker 01

  And I think even if you get to this sort of manufacturing plus like sort of cheap jobs, sort of part of the catch-up mechanism, it just seems very difficult to figure out how any country in that spot ever gets any leverage of what happens at the frontier, which is to say, they don't get any oversight or regulatory input into how frontier AI systems are built. And they probably also don't have any hard leverage that makes sure that they'll continue getting AI exports and continue getting access to AI supply chains and so on. And so they're basically at the mercy of whatever great power provides them their AI models. And then maybe within that, they can find a somewhat favorable arrangement, but it seems

• 1:14:42Speaker 01

  very unlikely that they'll get like a stable say and a stable input into that. And I think, you know, I think that just carries the world into spheres of influence of those that have very powerful AI that are able to export it. Bunch of other downstream questions of that that make this more complicated. How much do you need for TAI? How much do open weights play into this? At what point can you build like your own digital sovereign infrastructure? But I think at least for the medium term, it is this quasi-vastalage to the frontier AI building powers that I think is just the most likely outcome for most of these countries.

• 1:15:12Nathan Labenz?

  Do you, in terms of like how people live, do you think that that could create a story kind of similar to the Chinese story over the last few decades where it's like, life is getting a lot better, we're getting richer, we just don't have a say in the overall high level direction, but like at the street level, you know, things are trending up and up?

• 1:15:36Speaker 01

  I think in absolute terms, people are going to be richer and wealthier and better off. I think, so I think in terms of the economic effects that is, I think they're just going to be relatively disempowered when it comes to meaningfully shaping the trajectory of the world and also in terms of having an ability to catch up to however well the sort of the frontier countries, so to speak, are doing. But I think in absolute terms, it's going to be, it's going to be, it'll keep being growth and it'll keep being still over effects and redistribution gets easier as well. I think on the streets it's going to be, it's going to look nicer. It's going to be a sort of just economically better scenario. So I think in absolute terms, you wouldn't mind too much. I think the more fundamental question is, what does it say about sort of democratic

• 1:16:17Speaker 01

  say and human autonomy and human dignity even that none of these decisions really factor into where the broader trajectory of the, the history of the world goes. And I think that is a more profound sense of disempowerment that I think we should still be concerned about. But yeah, practically speaking, not that bad. I think the other part of it, practically speaking, is susceptibility to misuse. And I think that could be extremely destabilizing. So I think there is a current assumption that to guard against a lot of forms of AI misuse and the out of control and so on, you need your own AI systems that defend you against that. It's most obviously true in the realm of cyber. I think it is also conceivably true in tracking and monitoring potential deployment of pathogens, the sort of entire virus conversation.

• 1:17:00Speaker 01

  It is probably true in terms of scanning and filtering and screening against scams and all these sort of like social engineer attempts and whatnot. And it's sort of, it is probably also true in terms of like safeguarding infrastructure against extortion attacks and so on. I think if you expect there to be a world where non-state actors, terrorists, criminal groups get access to at least fairly capable AI because they're able to steal it, because they're able to sort of pulse train like terrorist GPT on, on some open source model, whatever, right? Like, and you also expect these countries to not have any sort of coordinated and assured and widely deployable access to these systems. I'm not sure whether they're going to be able to protect their citizens from AI-driven harm, AI-driven sort of misuse and also potentially the labor market effects.

• 1:17:45Speaker 01

  I think that all sounds like they would be very susceptible to that. And then you can imagine a lot of very destabilizing scenarios, right? If your country no longer, if your state no longer protects you from a sort of AI-driven harm, then what do you turn to? Maybe you turn to mass migration. Maybe you turn to sort of other ways of structuring your sort of like personal security, as we already see in some of the failed states in, in Latin America, where I think with sort of like criminal enterprise runs a lot of like day-to-day structure in a lot of parts of these countries. I wouldn't, I wouldn't think that would be impossible for a lot of these countries as a sort of like medium to long-term outcome. And I think that also has a very worried, but I think the pure economic story is pretty positive. The eroding the state, the authority and power of the state story is I think a lot more concerning.

• 1:18:30Speaker 01

  And I think sort of take these together and it's not a particularly rosy outcome.

• 1:18:35Nathan Labenz?

  So that's probably.

• 1:18:39Nathan Labenz?

  Seventy percent of the world's headed there and Europe is kind of the one place that can maybe engineer for itself a different outcome. Tell me if you disagree with that, tell me, but I'm going next to, okay, what does Europe want and how does it get it?

• 1:18:53Speaker 01

  Yeah, I mean, I think the problem that Europe faces is not too dissimilar to what a couple of other Western countries or like sort of general sort of liberal democracies face as well. I think fundamentally, Australia is in a similar boat, New Zealand is a similar boat, Japan, South Korea are in somewhat similar situations, Canada is in similar situations. So I think that plus Europe plus the UK is, I think, the cluster of like US allied middle powers that have a potential trajectory out of this. It still needs a lot of work. I think the fundamental question from sort of, I think there's two fundamental ways to start looking at this. The first way is looking at, well, put aside all the AI things, what do you want Europe's economic position to be?

• 1:19:36Speaker 01

  And I think if you start thinking from that, you just think, well, you want to be good at the things that Europe is currently good at. You want to be good at some aspects of the manufacturing, you want to be good at some aspects of the artisanal, of the artisanal good, you want to be good at this sort of high state capacity things that Europe currently is good at, whether that's sort of welfare states, whether that's, whether it's just like high levels of security and safety. This is really like a lot of things that are going well in Europe, you just want to keep, keep, you want them to keep going well. Plus, you want to find some way to actually revitalize your current economy. And then AI comes into the picture as like, well, that seems like it could either really accelerate that or it could really sort of destabilize that. And then you ask the question of what do you need AI for in that context?

• 1:20:20Speaker 01

  And I think the other part, the other way of looking at it sort of comes to this conclusion, which is like, well, what is Europe currently not have? And the answer is that it currently doesn't have frontier AI systems, which turn to be, turn out to be one of the most important economic inputs of the future. And also one of the most exciting parts of like strategic and economic competition right now. And the question is, well, what do you do about that? And I think there you quickly realize, well, building these systems ourselves is just too expensive, it doesn't actually work. So the next best thing we can think about is how do we get access to frontier models in a way that is assured and secure? And allows us to build what I sort of described as the first approach around, like, how do we reduce the geopolitical risk of just doing the things we're good at?

• 1:21:00Speaker 01

  How do we make sure we have assured access to frontier systems and don't get cut out of this like AI AGI conversation while we do the things we're good at? And I think these all sort of come together to, you need something to incentivize selling frontier systems. You need something to make the Americans not nervous about selling frontier systems. And you need some productive way to use the frontier systems downstream to like make, make something happen around that. And I think the strategy that we wrote basically answers, tries to answer these, these questions. And I think especially the parts that I most contributed to, tried to answer these questions. And I think just very briefly running through this sort of like high level takes is like, the first thing is this compute for access idea that I first wrote down

• 1:21:44Speaker 01

  late last year, early this year, and started sort of shopping around with a lot of countries in the world ever since, which is now also one of the sort of pillars and one of the main sort of asks of the strategy, which is to say, we built data centers for American, like in cooperation with American hyperscalers. In exchange for the favorable conditions, we provide these American hyperscalers and labs, we get assured access to the models that run on these data centers. And as long as the Americans keep giving us the model, they keep, they continue to get access to the data center. If their side goes back on the deal and we're cut off from access to the frontier models, they lose access to the data center. This is the sort of incentive part of the conversation, rebuild the infrastructure and get access in return. The second part is, how do we make the Americans not nervous about doing that?

• 1:22:28Speaker 01

  Because done wrong, this is a security risk, right? You can't run.

• 1:22:31Nathan Labenz?

  We've done it with UAE, for God's sakes, we should be able to reach a deal with Europe.

• 1:22:35Speaker 01

  Yeah, well, I mean, I think the UAE thing is like kind of fragile. I think the UAE is kind of worried about, well, what, what, what is the future of that? Will there actually be frontier weights hosted on UAE data centers? And I think that's very unclear. I think they're going to run some inference on them, but maybe it's just going to be like Haiku inference and not like Fable six inference. And I think that is an open question. So the question is, how can we get the security alignment to work out in a way that the Americans aren't, aren't too worried about hosting the models there and giving the model to the European economy? And I think that just has to do a lot with aligning with US on a lot of these security provisions, building out the data centers to be just like secure on the cyber and physical side. Building out sort of KYC regimes with European firms, just make sure that the Americans don't have any well-grounded national security worries

• 1:23:19Speaker 01

  that will put against the incentives from compute for access. And I think the third thing is, let's get a little bit more self-assured about the assets that Europe does have, right? Europe has both sort of broad economic assets, which is like a very powerful economy, still in absolute terms, if not in terms of growth trajectories, but still in absolute terms. And Europe has a lot of assets in the semiconductor supply chain, right? Like ASML and SICE and all these things that play into the, play into building, building frontier chips and therefore frontier models. Let's just think about how we can be strategic about that. Let's set up an anti-coercion instrument of sorts that says, well, if everyone plays nice, we'd love to feed these assets exclusively into the American supply chain. We're willing to align with export controls vis a vis China, we're willing to just, you know, we're willing to be good friends and good partners to the US.

• 1:24:02Speaker 01

  And also, if the US ever doesn't decide to use its ability to cut off frontier models as a means of coercive action, then we're also willing to use the supply chain bottleneck that we have as coercive action in return. And I think between those three frontier accesses pretty assured. And then you're at a point where you're sort of back to, back to where we were before AGI, which is like, Europe still has a lot of structural economic problems, we still have to solve them. But at least we fixed the geopolitical problem of being cut off from frontier access. And I think that's maybe step one. I think that's the things that I'm excited about getting done in Europe in the next year or so.

• 1:24:36Nathan Labenz?

  What's the hardest part about it? Is it just getting data centers actually cited and built or is there other challenges you think would be bigger than that?

• 1:24:45Speaker 01

  I mean, we can have this conversation and there's like a shared understanding that the suggestion I make interface with like a realistic future that we think might happen that is worth preparing for. This is not the case in many rooms with policymakers in Europe. I think there is deep skepticism of the continued trajectory of capabilities of US-built AI models. There is a lot more optimism about the broad availability of open source competitors that can basically do everything as well as the American models. And there is just a more fundamental question on, are these models that powerful? Is it that important? Is that as big of a geopolitical issue?

• 1:25:24Speaker 01

  And then there is also the question of, well, if the models are that important, if everything that I and we and people say is true, then like, why shouldn't we just build this ourselves? That surely can't be that expensive, right? We'll find like a more clever way to do it. The Americans are sort of wasteful, high on their own supply anyways, like, we'll just like spend a few million dollars and like, surely we can like whip something up. And I think just cutting through that, which I understand to be just like complete misunderstandings of the material reality we find ourselves in, just cutting through that and like making, making the point of no, this is not actually, this does not accurately describe reality. You need to think about this in sort of clear-eyed ways that respect that this, this thing that is happening in America is real. And the Americans are fundamentally right about a lot of the aspects of this, that's the biggest barrier.

• 1:26:07Speaker 01

  And then I think if that awareness existed, I think there are still political things to figure out. Like for the ASML thing, it's going to be some amount of triangulating between the Dutch government interest, the ASML interest, the interest of the other member states, that's not quite easy. In terms of data centers, there's going to be some domestic skepticism against American tech firms and working with them, that's not going to be quite as easy. In terms of the security alignment, there are going to be people that are going to be more excited about hedging toward China and just like trying to stay between worlds a little bit. I think all of these are surmountable, very, very easily surmountable, if you just get the alignment and the awareness of what's happening here, right? And I think that's, that's the main challenge.

• 1:26:45Nathan Labenz?

  So what's, one of the things I did notice in reading the report was,

• 1:26:52Nathan Labenz?

  you know, the authors, you and your co-authors sort of are willing to dream a bit in terms of how the authorities might act, right? I mean, there's, at some point, there's basically a statement that like, doing this in the sort of half-assed or kind of, you know, highly bureaucratic, you know, everything's a committee to nowhere mode that European governance, at least by reputation, often acts, would be maybe worse than not doing anything at all. So how do you think, you know, how realistic is it that you can actually get this sort of action? And like, what's the mechanism for doing it?

• 1:27:30Speaker 01

  I mean, it's still downstream of urgency and awareness of the situation, really. I think there's always a trickiness in writing for.

• 1:27:39Speaker 01

  I think for national governments generally, really, which is you try to write something that isn't quite within the overton window of what they're willing to do, but that also isn't so far out there that they'd never do it, right? I think like one failure modus, you just write it, well, we just have like 20 million in the budget. So like, let's just think of the maximally AI-pilled way to allocate the 20 million. It just turns out it just doesn't matter. Like either way, just you can, you can, you can just burn it, you could throw a party, it doesn't matter. Like it's not going to change the conversation. And the other failure modus to be like, well, we're going to be maximally honest about what we think should be done. And we just write exactly that up. And then, you know, you guys can tell us, oh, I'm so sorry, we were wrong later on. And then it's going to be today to do it. And then we're just going to shrug and be, well, you know, we, we told you the, the honest thing.

• 1:28:21Speaker 01

  I think the, the, the art or the trick of getting this kind of thing right, and I hope we struck a decent balance, is aim at just a little bit more ambitious than they currently are, accounting for the fact that they will get more ambitious and that they need some nudging toward being more ambitious. And I think that is the sort of, that, that is the sort of calibration that the strategy tries to reach. I'm optimistic about that. I mean, if I had to give you odds of this strategy as a whole being implemented within the next year, it's not that high. If I had to give you odds on elements of it making it into serious policy attempts and actually getting set up, I think it's pretty high. And I don't know which ones of these that they're, they're going to be. I could make bets on which one of, ones of them are going to be more popular and less popular. But I think some of this is going to happen.

• 1:29:06Speaker 01

  And I think that's the sort of hallmark of a well-calibrated strategy. We have done this before in Europe. Like there have been times where Europe has managed to very quickly act very decisively in a way that has like delivered results as quickly as it has anywhere in the world. I think the thing that I used to work in German policy and politics for a bit, especially in energy policy, and in the immediate aftermath of the start of the war in Ukraine, there was an absolutely heroic effort of the German government to buy this fleet of like shadow LNG tankers all over the world to get them to transport like alternative gas supplies to Germany once the pipelines were cut off. There was a just as heroic effort to get like LNG terminals built out in some of the more, most NIMBY parts of the countries. And within like six months, the German government with all its capacity and all its urgency

• 1:29:49Speaker 01

  managed to consolidate resources to, we made it through the winter with no problem. Max, like complete fine energy, no problems at all with just like lack of heating or anything. That was like the big doomsday scenario. It just didn't happen. There was like a massive heroic effort and it just, it just worked. I think before that I worked in COVID policy when, when that was happening, there was like a lot of political pressure against joint vaccine procurement. It was very, very difficult to get the negotiations right. It was very, very difficult to get the member state interests aligned. And Europe managed to procure vaccines fairly well. And I think the vaccination campaign in Europe went fairly well. We can talk about non-pharmaceutical interventions, like a little bit more of a messy story, but I think that all went pretty well. And I think Europe can't do this. Europe just has to realize that this is certainly important.

• 1:30:33Speaker 01

  And I think we're not that far from realizing that this is important on a COVID slash Ukraine war scale. And I think if we just get there, then we can definitely make progress on the kind of recommendations we make in the, in the strategy. And I don't think that would be a big problem.

• 1:30:44Nathan Labenz?

  Cool. Very interesting. Are there small countries that you think are worth calling out for taking a, you know, distinctive and potentially effective approach? I was thinking, I don't know anything about this other than that I know that their sovereign wealth fund, I'm thinking Norway, I know their sovereign wealth fund is A, large and B, like AGI-pilled in its operations. Like they use a lot of agents and, you know, they're, they're kind of transforming themselves. But I don't know if that's translated to something like a national strategy in Norway. Singapore comes to mind as somebody that might be interesting. Who, who else is like doing interesting things out there, even if they're, you know, kind of small and carving out a, a, you know, a narrow path perhaps?

• 1:31:29Speaker 01

  And in threading a point.

• 1:31:30Nathan Labenz?

  Threading the needle, you might say.

• 1:31:31Speaker 01

  Yeah. I think Singapore and Norway and the UAE are probably the three you, you most obviously mentioned. I think Norway could do much more. I think the exciting thing that Norway could do is you could become an inference hump haven, whatever, for the entirety of Europe. You could, it's a little bit hard to invest domestically with the sovereign wealth fund itself, but you can conceivably come up with sort of like schemes to invest into European consortium, then invest into compute building. Norway, Norway turns out to be a pretty decent place to build love compute, hence the first attempt to build a Stargate, now a Microsoft data center there. I, I think Norway could be more AI-pilled about deploying these resources, but I think Norway has a lot of resources that could easily be deployed and pivoted towards that.

• 1:32:16Speaker 01

  And I also just think the base case of just like investing a bunch of your wealth fund into like, they basically just like recreating, like, I mean, maybe saying recreating situational awareness isn't no longer as on vogue as it used to be like two, two months ago, whatever. But like basically recreating like a fairly AI-pilled portfolio and part of the wealth fund, probably just let's use a ride on the coattails of the AI revolution for quite a long time. That probably works. The UAE play is building a bunch of data centers. They're basically just finding a way to turn money into, into something that is an asset in the new economy. And I think that's a good way to spend a lot of money if you have it. Now it's incidentally kind of, kind of a tough situation to be in, that you're that close to Iran and that it's that easy to drone strike data centers. So I think that is sort of thrown a wrench into that plan. But I think structurally that was still a pretty good play.

• 1:33:00Speaker 01

  If they can manage to build the data centers quickly and if they can manage to secure the next generation against drone strikes and so on, I think that's still a play that works. I think Singapore is another interesting case. Like A, there's also wealth fund investment questions. B, there is just a massive amount of data capacity in terms of like understanding what's going on and like engaging with that. I think like it probably no like parliament with the greater density of readers of like very AI-pilled, very insider publications than, than, than the Singaporean. I think the same, same goes for the civil service. It's just, Singapore is a little bit more tough because a lot of the Singaporean economy is very exposed to AI disruption. And I think if you're Singapore, you just have to sort of hope that the white collar apocalypse doesn't look quite as apocalyptic because it's just hard to pivot an economy

• 1:33:45Speaker 01

  that is as big and as sort of like service indexed as Singapore towards a completely new way of operating. So I think there's a lot of capacity and interest there, a little bit less of an obvious like AGI-pilled play to pursue.

• 1:33:57Nathan Labenz?

  Are there any other countries that you think are well-positioned, maybe more well-positioned than they know, that should be doing something that are just kind of sleeping at the switch?

• 1:34:07Speaker 01

  Well, I mean, I think Australia is kind of awake now, but I think for the longest time it was Australia because Australia is just such an insanely good place to build compute, both for just like data center construction, energy supply reasons and so on, but also for security integration reasons. Like there's a really deep amount of like national security trust between the Australian and the American agencies. I think there is this strong understanding that Australia would not defect to China in any way, that they would be willing to play ball and all be aligned with the China-focused export controls. And so that just makes Australia a great place to run like compute for access data center build-out place. I think more recently we've seen more of that happening. I think that's very good. But I think for the longest time that was a sleeping giant. And I think it probably still is. You could probably still like five X, 10 X the data center ambitions and just run the

• 1:34:51Speaker 01

  inference for half of the world out of Australia. And that would not be an overly ambitious thing to do. And I think, you know, there's still a lot to be done there. I think maybe the inverse of this is the UK, where the UK has just the greatest density of like talent and expertise, both in government and outside, just outside of the US. And it's not quite entirely sure what exactly they're planning to do with it or if they can do anything with it, just because the broader political conditions of the UK, the skepticism towards US alignment, but also the sort of damaged relationship to the European Union and the rest of the middle powers. It's just very, very difficult to figure out what this incredibly clustered, talented cluster of people is actually supposed to do in the UK.

• 1:35:34Speaker 01

  So in a way, the UK and Europe really have inverse problems where Europe has amazing assets that it could use extremely well to actually have like very like live player position in this AI conversation. And it's just really hard to get them to, whereas the UK has all the awareness and all the expertise in the world. It's just not entirely sure what they should, what they should even be doing with it because it, at the end of the day, if your starting hand doesn't include any cards that are really good for an AI future, then you can be as aware as you possibly want to, and it's still really hard to get something done.

• 1:36:04Nathan Labenz?

  If you are the rest of the world, like let's say you're Brazil, or you could, you know, pick your countries, or maybe you would put different countries into different positions. I hope this doesn't happen. I, you know, my, I'm hyperstitioning better US-China relations and some, you know, form of collaboration and not carving the world up into spheres of influence. But one thing I've been wondering lately is like, because of course there was this reporting that the Trump administration was like planning to do something along those lines and tell countries like you're either with us or you're with China, you know, pick your, pick your camp. If you were put in that position as say a Brazil, or again, pick your country, how would you decide? Where would you go?

• 1:36:47Speaker 01

  I mean, I think the more important you think AI is, the less justifiable it is to go with China here. Just because there is no Chinese AI export program right now, right? They just don't have the chips. So I think if you, if you think your economy needs access to AI systems, and then you can sort of figure out the rest, then I think you just need to go with the US because they, only they can give you access to the computing capacity. Which is why a few colleagues and I wrote this, wrote the paper, which was in itself a follow up to another paper, which are both called The Closing Window to Win, which are about the sort of American AI export ambitions. And just like, China will eventually be better at offering these export deals as China has been in the past in like a bunch of international initiatives that they've run in South America and Africa and Central Asia.

• 1:37:31Speaker 01

  But currently they're not because they can't offer any data centers and any chips. So they can't actually offer a full stack export that can last the US ambition. So right now, I think if you're sufficiently AI-pilled, you just have to pick the US. At some point, China can probably throw in enough like non-AI related things that the deal looks a little bit more attractive. But just in terms of, well, is there any sort of like hedging strategy to be had in AI specifically? I just currently don't think there is, as long as China doesn't have the chips. That might change in a few years, and I think then the decision is going to be much harder. But currently, it's basically just, well,

• 1:38:04Speaker 01

  how reluctant are you going to be about buying US systems? I think is the realistic question that a lot of these countries face. And I think ultimately, that is a great position for the US to be in strategically. And the question is just, can the US actually offer a deal that these countries will think they will stick to? And I think that's maybe the main strategic challenge for the US, which is like, yes, everyone is tactically sufficiently incentivized to take the deal, no way around it. But they still don't like getting a deal that they feel like the US can rely on at any point in time. And so the US has to figure out some way to commit to these deals in a way that's credible to these countries. Building data centers is part of it, sort of deep industrial integrations are another part of it. But I think that is something that the US just has to think about much more, which is like,

• 1:38:45Speaker 01

  even if the deal is necessarily the only deal the other country can take, it might still irrationally defect if you're a sufficiently unpleasant partner to make a deal with. And so the US just has to think a little bit more about how to be like a slightly more pleasant and reliable partner. And I think it's not that far off, but I think, yeah.

• 1:39:01Nathan Labenz?

  Yeah, we've got the guy for the job. So perfect. One thing I have gone back and forth on quite a bit over time, because I'm a very AI-focused person, of course, and then as but China sort of, you know, became the end point for a lot of conversations that I was having, I became a little bit more of a China person. I'm still not much of a China person, really. But I always had this question of like, how is this strategy where we have these export controls? And, you know, like so much of what you're saying really depends on timelines, right? If you believe in superintelligence in two to three years, you gotta be on Team USA because there is no Chinese export. I agree with that. At the same time, if that is the path we're going down,

• 1:39:47Nathan Labenz?

  the chips are made in Taiwan, you know, it's real close to China, it's real far from us. And I just don't see a world where all this is allowed to reach its culmination point. You know, Allah, machines of loving grace, where it's like, now we're going to make some sort of, you know, deal with the Chinese that they can't refuse essentially, and realize eternal 1991, without them just being like, fuck, no, you're not. Like, we're taking out the fabs. How do we not end up in a world, like all these sort of things, they seem to be taking, as far as I can tell, they seem to be taking us to a point where China's going to hit a breaking point, and they're going to be like, the fabs are going down.

• 1:40:32Nathan Labenz?

  And I don't know how we get around that with the strategy that we are playing. Like, we can't defend them, right? So I mean, yeah, I mean, it's like a super sensitive asset, you know, it doesn't take a lot to, from what I understand, like a piece of dust or a skin flake, you know, can ruin a batch. So they can presumably not really be defended. Like, how do we not end up there?

• 1:40:52Speaker 01

  So I think I'm also not a big like US-China geopolitical competition, how do we win this guy? I think I have 194 other countries to focus on. And I think that just like hasn't, hasn't left me enough time to really, really think about this. And as much as others have, I think like very briefly, there's like three ways to avoid that. The first is the AI systems just get so powerful that it doesn't like, and the US is so far ahead, that basically escalation around Taiwan is suicidal for China even more than like accepting some amount of US domination. It's very unclear to me what exact shape of AGI is so powerful that that would be the case.

• 1:41:33Speaker 01

  But I think if you are sufficiently sort of like ASI and like really sort of superintelligence-pilled, at some point, you might actually think that, yeah, that's just like a dust of dominating, dominating advantage. And you just actually can't go to war with a country that has this kind of system. Maybe that is part of it. The second thing is, well, that's, maybe the fabs being blown up is just not that big of a deal. Because yes, it obviously destabilizes the entire supply chain. Obviously, that's like the end game in terms of US-China competition, and who knows what happens then? But you know, you'll have some indigenousized capacity in Arizona, and you'll have all the chips already up and running. So maybe if you're already sort of like halfway into your intelligence explosion by then, then it turns out you can just run all of this on the chips that you've already built.

• 1:42:17Speaker 01

  Yes, if like TSMC gets taken out, like the chip supply in a year really takes a hit. But maybe, maybe AGI can do a lot in a year, especially if it gets TSMC Arizona. And I think the third thing is,

• 1:42:29Speaker 01

  well, China also doesn't have indigenousized capacity. And it's not entirely clear that like going to war with Taiwan in a situation where they already think they're behind the AI supply chain is the best way to escalate, escalate the conflict, especially if there is still like some TSMC capacity in some way, shape or form indirectly ending up in China. If that is the case, then China might just think it's best like catch up hopes, revolve much more around like domestic industrial integration. And, you know, like that's like the least AGI-pilled version of the future. Or it's just, is it really worth going to war at this specific point where the Americans have this like decisive technological lead on AGI? And the Chinese diffusion play and the semiconductor engineization play hasn't really worked out yet?

• 1:43:11Speaker 01

  Can't we just like, kind of work in their shadows a little bit more, indigenousize some more of the capacity, and then deploy later on? Not sure whether that's the best strategy to take, but I think that might also be one strategic approach to take. But I also think it's a massive vulnerability, and I think like,

• 1:43:27Speaker 01

  I think any like reasonable AGI endgame has to account for the fact that the Taiwan situation just might blow up in our faces. And I think a lot of them aren't.

• 1:43:35Nathan Labenz?

  Yeah. Again, to talk about threading the needle. How about, this has been awesome, how about maybe a little lightning round to close? You mentioned hedging, and also in the context of the Norway sovereign wealth, like, you know, buying the right equities to get through the, the AI transition in a successful way. A challenge I've been wrestling with a little bit lately is the Tyler Cowen challenge of, if you're so doomer, what are your shorts? I've been trying to come up with, is there an actual answer to that question? Is there some way, and I'm not a total doomer, but I'm like, I think he should be taking it more seriously than he is. So I want to have an answer that either, you know, I either hear my shorts,

• 1:44:20Nathan Labenz?

  or I really tried and I can't come up, and that's kind of where I'm at right now. I cannot come up with a way where I think I can get rich in the doom scenario. Do you have any suggestions to answer Tyler?

• 1:44:31Speaker 01

  Yeah, I think, I think my, my general sense of that is just like, Tyler imagines a much more continuous and smooth on-ramp into like actual doom. That makes it so that like, sort of just betting on volatility and betting on like near misses and like pretty catastrophic disasters that aren't quite doom, make, make a lot of sense in that world. And I just think, I'm not sure whether that's true. I think a lot of the ways in which things go badly are just like things go extremely well, the market just all the way until they go really badly. And then the only situation where you cash in is when you're dead. And I think, I think that's the least convincing part of his argument to me. This idea that like, no, you'll get all these near misses and they're already all, so like all on expectation created the stock market.

• 1:45:15Speaker 01

  And I think there is just a very reasonable like doomer view that concentrates basically all of the probability mass of doom of like things going well all the way until doom. In fact, if you look at a lot of this sort of canonical like doom scenarios and like people who have talked about like serious like existential risk and catastrophic risk, many of them give the scenario of like everything's like, looks like it's going really well and strategically great and economically great. And it looks kind of weird, but it also looks economically great. Just until the point where there's like the takeover happens or the big incident happens. I, I think that's the main problem with that. And I, I mean, I think also just like, many people just have coherent worldviews that they also just don't bet on and that they just don't take financial bets on, even if they're committed to them. I think that's a sort of like more boring meta contention to that. But yeah, I think my main condition is just like, well, I don't think it's a smooth distribution of probabilities.

• 1:45:59Speaker 01

  I think a lot of this is just like, either this goes very well or it goes very badly. And there is not really a lot of world where it's like volatile and goes kind of badly for a while, then kind of well for a while. And so I think I also don't have a good trading strategy sadly.

• 1:46:12Nathan Labenz?

  Yeah, okay. Well, I appreciate you for thinking it through. Do you maintain a P doom number? I'm sure you've been asked many times. I haven't heard you ask though. Do you have an answer?

• 1:46:23Speaker 01

  I just think it depends so much on what you include in doom. If it's human extinction, it's very, very low. If it is, if it includes like all of the sort of catastrophically risky scenarios, including gradual like, sort of like the end stage of like gradual disempowerment and stable heteroterran, isn't that that kind of thing? I think it's probably around something like 10%, but I think that really is only if you account for the sort of like political doom outcomes in the broadest sense. And I think my probability of like technical extinction is substantially lower than that.

• 1:46:58Nathan Labenz?

  So you're counting in doom like we live in a sort of Chinese plus plus state and life is pretty good, but we like don't have political freedom.

• 1:47:10Speaker 01

  Yeah, I think like, I mean, maybe life isn't even particularly good in a lot of conceivable levels. It's like extremely disempowered, extremely low human agency, extremely low human economic participation. Call it the permanent underclass, if you must, but like I, I think in the actual permanent way, right? I think like you shouldn't like include too many just like all prosaically sort of bad outcomes in the doom number. But I think the sort of old portfolio of like existential, like long-term risks used to include things like stable authoritarianism and like to the stable economic disempowerment and to the extent that that's actually like a locked in path for the human future that you can't see a conceivable breakout of, I think I'd include that in doom in a broader sense and then, and then yes, but not just like, oh, well, the economy kind of sucks. So my P doom is like very high because I really think the economy is concerned.

• 1:47:56Nathan Labenz?

  Yeah, okay. Maybe something I should have asked earlier, but I think I know the answer. Everything you're projecting assumes that like robotics really works. It doesn't necessarily have to be humanoid, but like we're going to get highly flexible robotics that can be deployed in all sorts of contexts.

• 1:48:12Speaker 01

  Yeah, I think, I mean, it's going to take a little bit longer than I think, I think. I don't see like a super crazy industrial explosion very soon, but I think eventually this is an engineering problem. This is a scaling problem and at some point we're going to scale it. And at some point we're going to resolve the physical bottleneck. It's going to take longer. So physical bottlenecks are going to matter longer than software bottlenecks, for example, that eventually they seem eminently resolvable.

• 1:48:36Nathan Labenz?

  So does that mean like, let's say 2030 is the over under for when people start to have domestic service robots in their homes, would you take the over or the under?

• 1:48:49Speaker 01

  People start to have in 2030. Yeah, I think that sounds roughly right. I think it might take a little bit longer than, than that, just because of like idiosyncratic psychological and political resistance. But in terms of technical maturity, that sounds about right to me.

• 1:49:02Nathan Labenz?

  Yeah, okay. How does all this change as compute goes to space?

• 1:49:09Speaker 01

  Well, it just puts, so I think there's two versions of compute going to, compute going to space. The first is space is one of the places where we can put compute. I think that's going to be the case in 2029, which is going to be like, there are going to be some data center setups that are going to be worth putting in space. Inference more so than training, for example, and I think it's also just going to be gated by launch capacity. So we can't put all of our compute starting in 29 in space. We might also conceivably have different chip supply chains for chips that are suitable for going in space and racks that are suitable for going in space and terrestrial chip deployments. So some compute goes to space in 29, but I think that changes some things. It can concentrate some more computing power effectively within US jurisdiction. It makes SpaceX AI a lot more powerful. It makes launch site governance a little bit more relevant.

• 1:49:53Speaker 01

  These are all like interesting marginal shifts in how the conversation moves. That's part one. Part two is what happens if every marginal chip goes to space instead of to any terrestrial data center? And there's basically no terrestrial competition for data centers anymore. And I think things get a lot crazier then. I think anti-satellite weapons become a really important part of like deterrence and geopolitical stability for one. Because that's the only way you can threaten the deployment of like a super intelligence system because it is in space. And if you can't shoot down the satellites, then, then good luck, you're not stopping the, you're not stopping the super intelligence. In much the same way that the AI 2040 plans have talks about making like data centers sort of bombable and visible

• 1:50:31Speaker 01

  and allowing for this sort of intervention into, into this kind of sabotage in the same way in that both we just want satellites to be hitable from the ground and the compute to be vulnerable for geopolitical stability reasons. Another effect is all the concentrate.

• 1:50:44Nathan Labenz?

  Do you think that's, do you think that's the default scenario? Like my, my understanding is we could probably shoot down satellites without too much trouble. We just don't really, but we, we can, right?

• 1:50:54Speaker 01

  I mean, yeah, I think the question is who can, like the United States, yeah. Other countries, perhaps not. And I think they need to develop the capacity to do that. I think there are ground, I know probably I would think China has it. There is some other like the French have the beginnings of a program, the Indians have beginning, the beginnings of a program. There are already programs. It's not that no one can, but it is in the same way that nuclear power needs to have second strike capability. I think there is a sort of geopolitical stability sense in which like a sort of sovereign nation might want to have anti-satellite capacity and not all of them do just yet. And the other question about this is this like sort of crazy Kessler syndrome conversation where like there is some sort of mute, some amount of mutual deterrence of like ever shooting down satellites.

• 1:51:37Speaker 01

  Because if you get to the point where you have like that much debris in space that keeps creating more debris because things keep colliding and so on. It's just going to be really difficult to launch anything into space at any point in the future. So insofar as everyone is kind of disincentivized from doing that, in the same way that everyone is kind of disincentivized from like creating nuclear winter or something. I think there's also, that also makes the satellite weaponry math a little bit more difficult. I think the other part of this goes back to our middle power conversation. We talked about these compute for access deals and like deploying data centers and so on. That all hinges on this idea of the US is interested in building data centers in other countries because it wants to build data centers somewhere. If the US builds all of data centers in space instead, then the incentive of putting the data centers into the host countries is just so much lower.

• 1:52:18Speaker 01

  And I think as a result, that does sound pretty bad for a lot of these compute based strategies. So I think the most actionable and meaningful consequence of the prospect of data centers going into space is the compute for access and the compute build out strategy that middle powers are starting to pursue has a time limit. It stops working at some point in the somewhat near future. And you should start thinking about what your end game beyond the computing is in case the space thing works out as the sort of, as like SpaceX imagines it does.

• 1:52:48Nathan Labenz?

  A more terrestrial concern. Why, I mean, we talked a little bit about like bottlenecks. So I don't mean, you know, and human kind of inertia around why adoption hasn't happened as much as it obviously could in theory have happened so far. But if you take the flip side of that and you just look at like the AIs and their capabilities, clearly there's something that is missing relative to,

• 1:53:15Nathan Labenz?

  you know, the experience of hiring a human to do work, right? It feels like that's thinner and thinner all the time, almost to the point where I'm like now having a hard time putting my finger on like, what is it about like Fable 5.1 or Astra that's actually worse than hiring a human? Do you have an answer that is like what that is and, you know, what, what additional like marginal capability gain you might expect to actually create labor market disruption?

• 1:53:43Speaker 01

  Yeah, I'm not sure it's capability gains at this point. I think it's integrating with proprietary data, it's integrating with proprietary workflows. I think obviously not the entire task profile suite of humans is currently covered by models. But I think in specific tasks, they're better at humans. And I think you can drop them into specific task profiles at least, especially in software engineering, I think. And also in some other like sort of like general white collar activities.

• 1:54:07Speaker 01

  And I think the question is just like, the labor market just takes its time to rearrange around that. And to get to the sort of like augmented, like sort of mutually beneficial, because you can't just like, you can't just fire the guy who's sitting at his, at his desk and like plug in Astra instead. You need to be slightly more sophisticated in, well, we now need like, instead of three guys, we need like one guy who tells the agent to do the things and the agents do the tasks. And but that guy has to be like a little bit better at all the things that the agents can't do. And so I think it requires some institutional organizational reconfiguration. And I think that just will take some time, but I think the capabilities are there that I would expect them to have this sort of, not displacing, but at least disruptive impact that changes how teams are built and how productive they are. I'm a little bit less sure that this specifically leads to displacement in the short term.

• 1:54:51Speaker 01

  I think it also does create additional demand and does create additional like things that human workers can't do on the margins. But I think in terms of disruptive effects and reconfiguring effects, I would, I would agree the capabilities are there. It's just, it's just latency and lag and bottlenecks and frictions.

• 1:55:08Nathan Labenz?

  What do you think would happen in a hypothetical world where Tesla decides to license its full self-driving?

• 1:55:16Nathan Labenz?

  Within eighteen months or so, you know, let's say we make a priority of it. So we're into a little bit of a fictional scenario here, but all of a sudden, like basically all the cars drive themselves and the four or so million Americans who make their living driving don't are needed to drive anymore. That seems like one, you know, pretty clear displacement story that very well could happen. Like, do you think the economy can absorb those people? Where do they go? It seems really tough when you actually get down to like, okay, this dude has driven a truck for twenty five years. He's not ready to retire, but the truck now drives itself. What happens to him?

• 1:55:50Speaker 01

  Well, I think part of it is going to be political responses, wage insurances, reducing their times. I think there's also going to be a political necessity to add some frictions to this happening. I think frankly, there would be human in the loop laws. There would be sort of like, you have to have a driver in the like Harley and like New York and whatever idea of like, well, even if the theme drives autonomously, there still needs to be a driver in the seat and whatever. I think we'd see a lot of these political reactions and fictions introduced before anything happens. Yeah, that's, that's one part. I think the other part is yes, eventually, I think the economy would probably be able to absorb at least a decent percentage of that, not necessarily better jobs, not necessarily better paying jobs, probably in worse jobs. But also I will point out that in that specific story, you've found one of the like very few

• 1:56:35Speaker 01

  jobs that clearly just have like one specific task to them and no mutually sort of synergistic way of engaging with like, sort of, you sort of like fears in like the one technology, like one to one replaces like a specific kind of worker. I think most automation and like AI specifically just isn't like that in that it just like gets at specific tasks and leaves other parts of the task profile up so that it lends itself to a sort of, at least again, the medium term, a more sort of like augmented coexisting structure in a way that like specifically driving isn't like. But yeah, I think that will be very hard to absorb. We'd see a lot of political fictions as a result. We'd have to cushion a lot of it, which is like social spending. And I think some of them would find jobs and most of the jobs would be worse.

• 1:57:15Nathan Labenz?

  Your idea, not your idea, but your reference to the idea that somebody might be required to sit in the car even as the car drives itself reminds me actually of a novel that my dad wrote about a pretty dystopian but highly AI-enabled future where everybody is kind of out of work, but they need the dignity of work, so they're required to show up and stand around all day. And they're known as standers in his, his imagination.

• 1:57:42Speaker 01

  Yeah, I just really hope we don't get there. And that's like, yeah, yeah, well, fingers crossed. But I think, you know, you could make the cynical observation that some jobs in the real world already are kind of like this. But yeah, we'll hope we don't get there.

• 1:57:56Nathan Labenz?

  A few bullshit jobs out there. But if people are still with us two hours in, they'll be interested in your thoughts on this. What do you think America should be looking around the world to learn? One thing that I came away from China really thinking about is there's a lot of upside to surveillance. Don't want to have it for a lot of different reasons, but I'm kind of like, geez, it really sucks to leave all that upside on the table. Is there such a thing as surveillance with American characteristics? I'm interested if you have a thought on that and also just like what other things, when you look around the world, do you feel like the US, not just like should envy, because I don't think we can like copy the high speed train from China, but what, what should we

• 1:58:39Nathan Labenz?

  actually be trying to import and realize our version of?

• 1:58:42Speaker 01

  So I think on surveillance, it's really hard for a specific reason, which is I think I'm just very worried about perfect enforcement of laws. I think the American laws specifically just aren't made to be nearly perfectly enforced. If you enforced every law on the books in America, I think this would just be a draconian oversight regime. I think I understand people's motivation of like, well, this is like massive, like perfectly disincentivized, any sort of illegal action, but man, a bunch of things are illegal. And I think like a lot of the punishments and a lot of the sort of like criminal codes are like specifically structured around the idea that you catch one of every like a hundred or like ten or a thousand criminals and then the deterrent is calibrated to that. The downside of surveillance to my mind, and I think more broadly the downside to like

• 1:59:27Speaker 01

  more AI integration, I think the most well-taken point in the flock debate as well is we're just not, legal systems aren't set up for perfect enforcement and just like surveillance maxing, even with American characteristics, is just such actually a pathway to perfect enforcement of laws that were never meant to be perfectly enforced. So maybe we can talk about that once we've like completely sort of revamped the entirety of the criminal code and whatnot and the practices of enforcement around that. Before that, I'd be extremely worried about going that far. What to import from the rest of the world? America is a very idiosyncratic and like very specific place that is like done very well with like a very weird balance of institutions and economic activity that have for some reason served it very well.

• 2:00:11Speaker 01

  And I think usually when American politicians look around the world, into Europe specifically, and they try to import like one very specific part of a society, they sort of underrate how much of that society is just like in a completely different sort of equilibrium and balance that just like, if you just transported like specifically welfare spending from the Nordics, I think that, you know, you'd have to do things with a tax system that disincentivizes a lot of other commercial activity that then in itself just like forecloses other avenues of like providing the same services. And I think the same thing is true for the political system where like, yes, there is something to be said for the stability of a party and parliamentary system that isn't quite as, that doesn't swing back quite, quite as much between administrations. But then you also get a much less decisive government that is able to do much less and that gets paralyzed into gridlock a lot more.

• 2:00:54Speaker 01

  And I think there are countries whose concepts and structures are wholesale, perhaps preferable in the way that they deliver services and outcomes for their citizenry to America. Perhaps not. I think that's a tricky conversation, but I think I basically don't know a lot of like high level, really good things about how things work in a very like structural, like political level in any other place in the world that you could import to America without disrupting a broader part of, of how this country works. So I'd be very skeptical of like doing that sort of piecemeal.

• 2:01:27Nathan Labenz?

  Does that mean we're living in the best America? Because it sure doesn't feel like we're living up to our potential in many ways.

• 2:01:33Speaker 01

  I, I think it's Pareto optimal in, in some ways. And I think that's different from the best, right? But I think like everything that you would improve would come with a trade off against something else. And I think I'm very skeptical of this notion that there is, there are like clear Pareto improvements to the way that America serves. I mean, you can scroll through the like Institute for Progress's website on like policy interventions and you'll probably find like ten like marginal fixes to laws on the books that would just make this clearly better. And some of them are inspired by other countries, but I think meaningful changes to how the country runs are probably not Pareto improvements, but are like tricky trade offs that I'm not sure what sort of unbalance make the country run, run better, which is not to say that like this is, this is the sort of shiny city or any other country is the shiny city.

• 2:02:16Speaker 01

  It's just that, man, it's all trade offs and it's all really difficult to get right. And I'm just not so sure that there are like easy fixes to much of anything.

• 2:02:23Nathan Labenz?

  Yeah. Last big question. We've talked about a lot of different challenges, obviously, and, you know, vexing conundrums of all kinds, conundra, conundrums. What would you say are like the most important needles that we need to thread? You know, the one of the, the top couple few things that you think are kind of absolutely most critical? And then I'd love to hear just, what do you think life is going to look like on the other side of this in your kind of ninety percent where it goes well? Like, why, you know, for people that are like, this whole AI thing, why do we even do it? Isn't it just stupid? Like, paint the upside picture to inspire that, that audience.

• 2:03:02Speaker 01

  Yeah, I, I think I'm just not going to fix the problem of the AI story, of the AI labs, like big narrative concerns. I think I'm just, I just hope we can kind of continue on this sort of positive trend that I think history has been on for the longest time. I don't think I take that much of a sort of fatalistic view about the current or like the past trajectory of society. I don't think we need AGI to bail us out as much of anything. I just think it's the next thing we do. It's the next thing there is. It's the next big innovation. It's the next like innovation that will need to stay on track with these sort of like compounding economic growth that we have. I think in a lot of ways, it's going to look very crazy. I think some of it is going to involve space, some of it is going to involve robots, some of it is going to involve a lot of automation, but I think some of it is also just going to involve like fairly prosaic future economic growth that just makes us all a little bit

• 2:03:47Speaker 01

  richer and a little bit wealthier and our institutions a little bit more functional by the day, just as they have been, at least since the start of the industrial revolution. I think that's, that's my main hope and I don't want to, I don't dream much bigger than that. I think things might get a lot crazier than that and then we'll have to find ways to deal with that. But I think that is my hope and my dream for the future in a decade or in two decades or in five years, if things go very fast. And in terms of what do we have to do for that? I think just like genuinely just muddle through, like make sure that the balance of power works out and continues to work out and the balance of wealth continues to work out well. Make sure that the labs don't pull away in terms of power and control from the US government. Make sure the US government doesn't centralize and control the entire sort of like flow

• 2:04:31Speaker 01

  through of intelligence through the world. Make sure that some other countries have a stake in this, both economic and in terms of just like power and influence and leverage and just, you know, whenever things seem to go off, off, off the rails in terms of there's too much power amassing in one place and things look like they're going wrong, pull it back a little bit again, keep it on course. I think, I, I think that's what I want to do. And I think it'll just be like a long exercise of doing things like that on the very small margins. And I think then we're probably going to be fine.

• 2:04:59Nathan Labenz?

  Well, that is an unreasonably reasonable worldview and I appreciate you for spending a couple of hours sharing it with me today. This really, I think, has been an excellent conversation. So anything else you want to leave people with before we break?

• 2:05:14Speaker 01

  No, thank you. I enjoyed it very much. Thank you so much for having me. It was great.

• 2:05:18Nathan Labenz?

  And I liked, thank you for being part of the cognitive revolution.

• 2:05:21Speaker 01

  Thank you so much.

• 2:08:53Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help, for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast

• 2:09:35Nathan Labenz?

  dot I N G. And thank you to everyone who listens for being part of the Cognitive Revolution.