Securing Superintelligence: National Security, Espionage & AI Control with Jeremie & Edouard Harris

In this thought-provoking episode of The Cognitive Revolution, host Nathan Labenz speaks with Jeremy and Edouard Harris, founders of Gladstone AI and authors of "America's Superintelligence Project." The conversation explores a critical dilemma facing US policymakers: balancing the race to develop advanced AI ahead of China against the risks of losing control of increasingly powerful systems. Drawing from their extensive research with intelligence officials and technical experts, the Harris brothers detail the vulnerabilities in US critical infrastructure that would need to be addressed for a Manhattan Project-style AI initiative, while raising profound questions about the security compromises and centralization of power such a project would entail. Nathan offers his perspective that international cooperation might be preferable to an AI arms race, inviting listeners to consider whether humanity's shared interests might ultimately outweigh geopolitical rivalries in the development of superintelligent systems. Upcoming Major AI Events Featuring Nathan Labenz as a Keynote Speaker https://www.imagineai.live/ https://adapta.org/adapta-summit https://itrevolution.com/product/enterprise-tech-leadership-summit-las-vegas/ America’s Superintelligence Project: https://superintelligence.gladstone.ai SPONSORS: Box AI: Box AI revolutionizes content management by unlocking the potential of unstructured data. Automate document processing, extract insights, and build custom AI agents using cutting-edge models like OpenAI's GPT-4.5, Google's Gemini 2.0, and Anthropic's Cloud 3.7 Sonnet. Trusted by over 115,000 enterprises, Box AI ensures top-tier security and compliance. Visit https://box.com/ai to transform your business with intelligent content management today Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive NetSuite: Over 41,000 businesses trust NetSuite by Oracle, the #1 cloud ERP, to future-proof their operations. With a unified platform for accounting, financial management, inventory, and HR, NetSuite provides real-time insights and forecasting to help you make quick, informed decisions. Whether you're earning millions or hundreds of millions, NetSuite empowers you to tackle challenges and seize opportunities. Download the free CFO's guide to AI and machine learning at https://netsuite.com/cognitive Oracle Cloud Infrastructure (OCI): Oracle Cloud Infrastructure offers next-generation cloud solutions that cut costs and boost performance. With OCI, you can run AI projects and applications faster and more securely for less. New U. S. customers can save 50% on compute, 70% on storage, and 80% on networking by switching to OCI before May 31, 2024. See if you qualify at https://oracle.com/cognitive PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, my guests are Jeremy and Edouard Harris, founders of Gladstone AI and authors of the sobering new report, America's Superintelligence Project. Many listeners will recognize Jeremy from the Last Week in AI podcast, a show that I've repeatedly recommended and continue to listen to regularly myself, as Jeremy and his co-host Andre do a consistently excellent job of getting into real and often quite technical depth on the most important stories in AI, with Jeremy in particular presenting a very sophisticated and nuanced perspective that balances genuine enthusiasm for AI upside with sincere concern about loss of control and other existential risks.

• 0:45Nathan Labenz?

  Today's conversation offers a more zoomed-out perspective on what is arguably the core dilemma of our time. Racing ahead with AI development to stay ahead of China increases the risk of loss of control and other catastrophic accidents, but at the same time, insisting on adequate safety standards and allowing time for safety research to develop potentially cedes technological superiority to a rival nation that we can't currently trust. This problem, which they describe as "over-constrained" and admirably face head-on without flinching, creates profound challenges for US policymakers, especially as AI capabilities continue to advance rapidly toward potentially superhuman levels, as US-China tension continues to

• 1:26Nathan Labenz?

  escalate, and as the national security establishment begins to consider a Manhattan Project-style effort intended not just to maintain technological superiority, but perhaps even to achieve enduring strategic dominance. Rather than taking a side in the debate regarding whether such a project is wise in the first place, Jeremy and Ed set out to understand what would actually be required to achieve the level of security needed to protect both the infrastructure and the secrets of such a project from our national adversaries. They do this not just by reading and theorizing, but by spending time on location at data centers and elsewhere with seasoned intelligence officials, special operators, and other technical experts.

• 2:10Nathan Labenz?

  The resulting report forces us to confront the gritty realities of security, espionage, and geopolitical hardball that absolutely must be factored into any serious discussion about the future of advanced AI.

• 2:24Nathan Labenz?

  Drawing on their unique access, Jeremy and Ed paint a picture of a US critical infrastructure stack, from the electrical grid to the data centers to the highly international research teams, that is far more vulnerable to disruption and espionage than most realize. And they argue that while perfect security is practically impossible, an acceptable level of security for such a sensitive project requires a deterrent strategy that both raises the cost and observability of adversarial action and credibly threatens controlled but meaningful retaliation. While some will no doubt read this as a blueprint for victory in the AI race, others, myself included, read it as more of a warning. The sheer number of vulnerabilities we'd need to patch to effectively prevent China

• 3:08Nathan Labenz?

  or other sophisticated actors from stealing our AI secrets, the compromises to core American values this would require to operationalize, and the risks from the centralization of power that all this would presumably entail, are for me, reason enough to steer away from such a national project and instead try whatever measures we can, however unlikely they may be to succeed, to create both the technology foundations and the geopolitical conditions necessary to begin to rebuild trust with China and the international community at large, such that humanity can work together, both to safely develop advanced AI systems and to manage the societal transitions that are likely to result from their deployment. Is that naive on my part? Maybe.

• 3:51Nathan Labenz?

  But hopefully after listening to this conversation, you'll agree that it's not uninformed or close-minded. I am very much open to changing my mind. But for now, with the latest generation of intensively RL-trained models showing so many of the bad behaviors that AI safety theorists have long warned about, and with no particularly low-risk plan available anywhere, I continue to remind myself that we have far more in common with the Chinese than we do with the AIs. And I personally would choose to bet on our ability to find common ground with our fellow humans if it proves truly necessary, over our ability to not only control but effectively steer the future with hastily-built superintelligence.

• 4:32Nathan Labenz?

  As always, if you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Remember too, that I'll be speaking at three major AI events over the next few months. May 28th through 30th at Imagine AI Live in Las Vegas, August 12th and 13th at the Adapta Summit in Sao Paulo, Brazil, and September 23rd through 25th at the Enterprise Tech Leadership Summit, again, in Las Vegas. If you'll be attending any of these, I would be happy to meet in person. And if you're a business leader who's working on AI automations, application development, agent design, or high-level AI strategy, I also invite you to get in touch. I'm currently advising three companies for just a few hours per month each, and I have found it both

• 5:16Nathan Labenz?

  very fun and highly educational to help companies strengthen and confidently accelerate their AI initiatives. For any of these reasons, or just to give us some feedback on the podcast, you're always welcome to reach out via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. With that, I hope you enjoy and take some time to properly consider the implications of this critical examination of America's vexing superintelligence dilemma with Jeremy and Edouard Harris of Gladstone AI.

• 5:50Nathan Labenz?

  Jeremy and Edouard Harris, founders of Gladstone AI and authors of the new report, "America's Superintelligence Project," welcome to the Cognitive Revolution.

• 5:59Speaker 02

  Oh my God, it's so cool to be in your, in your, in your podcast bubble.

• 6:06Nathan Labenz?

  Yeah, I'm excited to have you guys here. I've been really looking forward to this for a while. Fun little backstory. I guess first of all, I listen to you, Jeremy, all the time on Last Week in AI, which I think is one of the best podcasts in the AI space. I am pretty much in every episode listener and a frequent recommender for folks that want to keep up. I think you guys do a great job with that and really appreciate all your analysis there. You know, we've been trying to make this happen for a while. I think you're the only guest that I ever had booked and then got preempted by Joe Rogan, which, you know, millions have heard you on Joe Rogan round about a year ago now. So I'm glad that we're finally circling back and making this happen. And the timing is, is perfect because

• 6:52Nathan Labenz?

  the sort of, you know,

• 6:55Nathan Labenz?

  Manifesto, mega plan, treaties on what the hell are we going to do about AI are coming fast and furious. And your latest contribution to that tradition, I think, is a really provocative one. And I'm excited to get into it with you guys.

• 7:10Speaker 03

  It's fantastic. And we've been looking forward to it for as long as you have.

• 7:16Speaker 02

  Actually, I will say too, like, I've been a fan of the cognitive revolution for a long time. Like, it's sort of, it kind of, I find it a little bit funny, because when I watch what you've been covering on the show, and then I think about you watching Last Week in AI, I'm like, what, what is Nathan getting from us? Just because the quality is so high, like, you guys do great technical deep dives, which is something that's so useful. So anyway, yeah, I'm super thrilled to be in the podcast box.

• 7:40Nathan Labenz?

  Well, thank you. That's flattering. But I do really find a lot of value in Last Week in AI. I think your analysis is really good. And I mean, nobody can really be comprehensive in the AI game anymore. So everybody's got to take some pitches, for sure. But I do think you guys choose a lot of great stories. And sometimes I even just go back, if I do fall a little bit behind, I'll just go back and listen to a couple of episodes and be like, all right, which stories in there do I really need to make sure I follow up on? So to a significant degree, you're almost kind of like assignment editing for me at times where I'm like, yeah, that one is one that's key to double click on.

• 8:13Speaker 03

  So you can maybe, you can maybe flag some items for Nathan there.

• 8:17Nathan Labenz?

  Yeah, I need all the help I can get, certainly in today's world. All right. So let's start with this. There's a lot of talk about AGI. There's a lot of talk about superintelligence. Not a lot of talk about what superintelligence looks like. So I would love to start off with just getting a little bit of your intuition for what the hell is superintelligence? What do we think it can do, you know, with obviously some error bars? How soon should we expect it? And what kind of strategic advantage might this technology convey on one nominal superpower relative to another?

• 9:00Speaker 03

  Yeah, so on the question of what superintelligence is and what kind of advantage could it convey? I'll try to tackle both of those at once and then Jer, maybe you can talk through the timing and fill in any gaps. In terms of what superintelligence is, right out the gate, people have all kinds of different definitions that are slightly different from each other. I've heard one person say that a calculator is super intelligent in the narrow domain of, you know, doing prescribed math operations. And like, okay, but that's not necessarily a useful definition.

• 9:35Speaker 03

  The definition that most people have kind of coalesced around in the space is it's something that can do almost, at least almost everything, if not everything, far, far better than the best human. And from that perspective, it's something that is, that is going to do stuff and is going to likely have goals that are not comprehensible to us. And so from the standpoint of what kind of strategic advantage does a superintelligence? As opposed to like an AGI, which is more kind of human level, does a superintelligence give your country? It's actually, at least under current conditions, in our current ability to control and align these systems,

• 10:19Speaker 03

  it's not clear that it gives you much of an, much of an advantage because it's going to do its own thing if it's that much smarter than you, unless you are much further advanced than we are today in the science of how to actually control it.

• 10:34Speaker 02

  Yeah, I think that's, that's definitely one factor. And to go back to the definition piece too, there's an interesting implication to this like universality of the superintelligence definition that people throw around, right? Smarter than all or more capable than all humans at all things, right? Something that absolute really implies a process, like you can almost start from that, work your way backwards. It implies some kind of trajectory. If you have an AI system that is necessarily better than human beings at, you know, autonomous AI research, then that means necessarily it is the product, almost, almost by definition of an intelligence explosion, whether you measure that. In in years, right? Because hardware is hard and the world is slow and wet and biological. Or or in, you know, weeks or whatever that kind of explosion time horizon is.

• 11:20Speaker 02

  Something that in retrospect, everybody would agree was like a fucking hockey stick, right? So that's really, in a weird way, the the process itself, I think, is more helpful than the definition of the thing. The thing, what like if you have superintelligence, if you have something that is genuinely more intelligent than all humans at all things. You have something that can design better biological weapons. You have something that can design like the ultimate cyber weapon. It is the national security technology. It is the decisive source of strategic advantage, full stop. That's just what it is. That's what is implied. Then you can ask about how we get there, whether it can be controlled and so on. That's the raw capability that comes with the package. The controllability piece, the alignment piece is an open question.

• 12:01Speaker 02

  And this starts to lead into our thinking strategically around what are the constraints that America faces today? As it enters into what is ultimately and clearly a race with China on AI. You can't assume that you can control the system like that is that is an important thing.

• 12:17Speaker 03

  Level at that level, if you're, you know, if you're something closer to human level, even that that's kind of fraught, right? Because even if it's qualitatively human level, it's maybe thinking 10 or 100 times faster than a person. And it's like, maybe can be deployed on a on a data center at bigger scales. That could still give you qualitative differences, but still, if you have something that's like human ish level or AGI level. You can kind of imagine maybe having some kind of temporary slippery control over it and like having it really contained and aimed in a particular direction. If you're talking superintelligence, I mean, I think most useful definitions are this is a thing that outstrips you to the same extent that you outstrip like a toddler at the very minimum.

• 13:01Speaker 03

  And you think about like, how is a toddler going to keep you contained over any meaningful span of time? Forget about it. You're just, they're just not going to even think in the right ways to do that.

• 13:12Speaker 02

  Maybe this is a good point to kind of like just jot down or plant a flag on the constraints that we see that kind of inform our picture of the world. It's almost a, yeah, like the signposts that guide us as we go through this investigation and just the work that we've done over the last year. This really seems like a pretty over constraint problem, just to be clear. So number one, we have AI superintelligence that seems, again, it's unclear, right? Like, this is an open question, and we should dive into this part. It's unclear what it means to align a superintelligence. Can we do it? If you do have it, can you control it? Like, what, you know, what does that look like? But that's one fact of the matter about the world that we ought to take very seriously. Unfortunately, people who take

• 13:55Speaker 02

  that view seriously seem to have an allergic reaction to taking other points of view seriously as well. And the converse is also true. So just at the same time, as it seems like it actually will be pretty hard to control these systems, and there's all that uncertainty, we know that striking a deal with China is going to be incredibly hard. And when I say incredibly hard, I mean, when you talk to people, whether it's at the State Department, whether it's in the intelligence agencies, people who've had firsthand experience dealing with China in a like, real world setting, it is extremely clear to them that there is no deal to be done with China under current circumstances. Anything that involves a deal with China has to happen with a trust and verify framework in place. We don't have the hardware tooling to do that. We cannot do what we've done with some nuclear technology

• 14:40Speaker 02

  with some success. We just can't do that right now with China. So those two realities like simultaneously coexist in our world. And what we keep finding is you got folks who believe one, take one seriously, and just refuse and have an allergic reaction to acknowledging the other and vice versa. And so our sort of philosophy is, how can we square the circle between these two, what appear to us to be just facts of the matter about the way the world is?

• 15:08Nathan Labenz?

  Yeah, I think there's a lot there, a lot of different directions to unpack. I'm a big avoider generally of analogies because I always am trying to understand these crazy AI developments on their own terms as opposed to, you know, by proxy through some, some other conceptual framework. But recently I at least made the concession that to the degree people are going to use analogies, it's a, it's often helpful to broaden the set of analogies that they'll consider. So partially against my better judgment, one of the things I sort of hear you saying here is like, the nuclear analogy is very often invoked, but maybe we should also be thinking about like a biological weapon analogy as well. Because those things are the ones that,

• 15:53Nathan Labenz?

  you know, whether or not they can be properly said to have goals in the sense that we typically understand them. They certainly are unwieldy and we, you know, have a whole other dimension of control problem that is layered on top of the conventional challenges that we have in achieving some stable equilibrium in the context of like a nuclear, you know, technology wave.

• 16:16Speaker 03

  Yeah, the biological analogy is, is closer or has many more points in common with AI than the nuclear analogy. Exactly for the reason that, that you've, you've given and for other reasons too. There is, yeah, there is a control problem, at least with current instanci- instantiations of biological weapons. There is chatter about, you know, adversaries trying to develop biological weapons that will target specific DNA profiles and specific like races and really like ultra, ultra scary stuff like that. But yes, there, there absolutely is a control problem that keeps everybody like kind of running scared on the biological weapon side of things.

• 16:56Speaker 03

  It also has the, the same kind of potential for devastation, at least from our perspective as humans. Like you actually, in, in principle, you could design a bioweapon that exterminates the entire population of the world. Like that, that is a thing that could happen. And then, and then finally, the, the other aspect of it is, we actually have seen containment breaches of biological research labs in the past. And not just talking about the elephant in the room type stuff with potentially COVID and that now likely being seen as a, as an escape, but also other stuff, you know, in the 1970s and just research on kind of viruses, which doesn't even necessarily need to have come from biological weapons research, but

• 17:39Speaker 03

  can have come from just our attempts to understand these diseases to cure them. So, we know, we know, we can't control these biological agents and they do escape containment. And biological agents are a lot stupider than the AIs that exist today, and certainly the ones that we're going to build.

• 17:58Speaker 02

  That's also a bit of a, yeah, the, the whole biological agents are, are stupider thing. It, it's simultaneously true. And then it's one of those analogy breakdown moments, right?

• 18:08Speaker 03

  Yeah.

• 18:08Speaker 02

  That we, we are ourselves are like, human organic matter is the mechanism of conveyance of that biological information and the pathogen.

• 18:17Speaker 03

  We're particularly susceptible to the, the biological vector. And, and that is a, a difference. But, I mean, I would argue intelligence more than makes up for that at these scales.

• 18:31Nathan Labenz?

  Yeah. So let's flesh out a little bit more of this picture of the super intelligence. You know, in a sense, you're sort of saying like, as something that is by postulate sort of better at humans than everything and sort of by development is sort of the process of like a recursive intelligence explosion that's like a bunch of generations potentially compressed into a narrow frame of time, but still nevertheless, like a bunch of sort of steps away from the AIs that we have today. It's like, sort of just hard to say what that might look like. But I'd love to just, you know, spend a little bit more trying to develop intuitions or, or share intuitions. I guess, you know, one thing that I think is like,

• 19:17Nathan Labenz?

  These things have really weird profiles, right? The jagged edge is just extremely jagged. I did one episode with Adam Gleave from FAR AI, who with collaborators at FAR, did work showing that even the superhuman go players are like, in fact, dramatically vulnerable to certain adversarial attacks. And I wonder, like, is it realistic to think that we're going to have the AIs sort of solve all their own problems? And like, actually patch all those adversarial vulnerabilities? Or might we end up in a situation where in some ways, these things are like super?

• 20:01Nathan Labenz?

  And by the way, this could be a good thing, right? Like, maybe a great scenario if it's like, you know, every 98% of the time it works every time. And that's enough to make all the, you know, drug discoveries we might ever want. But that 2% of the time is just maybe enough for us to never feel comfortable using it in warfare. You know, for example, like, that could be a really fortunate, natural path. But, I mean, do you, do you see hope for that, you know, sort of luck? Or do you see all these loopholes being closed?

• 20:34Speaker 04

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 20:39Speaker 03

  Yeah, so that's an amazing question. One thing I'll mention and point out here, you're saying the jagged edge is really jagged. That is totally true. What is often missed is that we, humans, are also super jagged. But because we're humans, we don't realize how jagged we are. So if you have a blind spot, that's just like a structural blind spot in your brain, and just hasn't come up in societal history or in your own life, that stays a blind spot. The reason we perceive jaggedness in AI systems is that they're so alien and they think so differently from us, right? So like, the kinds of mistakes that they make are just different from the kinds of mistakes we make.

• 21:23Speaker 03

  And that means when we see an AI make a mistake, it makes the kind of mistake that lights up like a Christmas tree to my human eyes because my brain and architecture are just cognitively different. But the reverse is also true. I am making mistakes that are really, really obvious to an AI system. And the more, the more advanced the AIs are, the more they'll be able to see how to hack me and like all these just really obvious mistakes that I make. And that's true of all humans. So it's maybe somewhat less of a question of the frontier being jagged and more that every, you know, every high capability system is like jagged in different ways.

• 22:04Speaker 03

  And when you bring two different systems together, they can see each other's flaws, but they don't immediately see their own. I'll bump it off to Jer there actually, because I think your main question was around, you know, is that good?

• 22:18Speaker 02

  Well, yeah, I think so to the jaggedness point, I think there is also this question about how jagged is it? I, I think humans are regularized by our interaction with our environment in ways that AI systems are, right? We get data sources that are much more diverse in general, we're much more natively multimodal. We, we work through like a, a perverted kind of RL, which is kind of less prone to overfitting to begin with. So, so there, there's like a lot of what we're, what we're doing that I think would argue for a less lumpy capability surface, but we still have those jagged edges. And the, the rest of the, I'm not so sure that, that that in any way affects the analysis. It's more kind of a bit of a footnote.

• 22:53Speaker 03

  It's true, and we've been in competition with each other as well. And that adversarial pressure tends to smooth out jagged edges. That's totally true.

• 23:01Speaker 02

  Yeah, yeah. So like when it comes to the idea of, you know, the, the sort of will superintelligence be like X, Y, or Z, or what's the setup there and the connection to the jaggedness? I mean, I'm not, I'm not so sure that it matters. And the reason I would say that is internally, when we talk about the capabilities, the threat models that we're considering, we don't necessarily start with superintelligence and then work our way backwards. What we do is we say, okay, what are the specific capabilities that are tied to specific threat models? So for example, do you have, if we're talking about superintelligence, like loss of control, which is what we started off talking, talking about, we should get into weaponization as well. But for loss of control, you know, typically you're looking at something where AI systems are automating a large fraction of AI research. At least that's one big exacerbator driver of

• 23:45Speaker 02

  loss of control risk. And there are footnotes and asterisks everywhere that we could dive into. But generally, that's what we think about. And so when you want to narrow down the conversation, kind of to make it maybe constructive, so we can get our arms around things, I think it can be helpful to think about like, what is the first AI going to look like that does recursive self-improvement? Can it be, let's say, not jagged enough? Can it be smooth enough in the part of the capability surface that matters for driving that outcome, such that you actually get something that can notice the jaggedness when it's getting in the way of pursuing real goals in the real world? I think any part of an AI system's optimization surface that isn't kind of optimized for

• 24:27Speaker 02

  or regularized for is going to end up jagged, because it's essentially the unoptimized parameter lag. You have just kind of these, these hanging, like grotesque parameter threads that haven't really been touched by contact with that sort of optimization pressure. So, I mean, I think in terms of whether or not it's good, I think it's good for a lot of loss of control threat models, because at the very least, you might hope that the thing like gets, like you said, you know, it's like agents today, right? They'll, they'll have like a 99% chance of success on any given step, but you chain together 100 steps for one coherent goal, and suddenly their success rate is like 20%. So you might just hope that the thing would get partway through, give us some clear indications that it's about to do something really bad. And we can kind of go, oh, damn, like, let's stop that. That's no good.

• 25:10Speaker 02

  Definitely agree with that perspective for sure. I think it's a it's an open question, but to me, it's like about can you make the system that makes the system? And that's a way a more well kind of framed question that I could see you ironing out the jaggedness enough to kind of solve for.

• 25:26Speaker 03

  And there's some other aspects to that, like, you know, once your system becomes capable enough, you kind of anticipate up to superintelligence, it starts to it starts to actively consider its own cognitive processes and like to some extent, maybe start to fix its own flaws and problems. That's speculative, but there are indications that that AI systems even today can at least consider, for example, the impact of how they're being trained on what their future versions will want. Anthropic had a famous experiment about this. So the idea of AI systems considering their own flaws or

• 26:05Speaker 03

  needs, if I can use that anthropomorphic, you know, language, that's a thing that is happening, at least early versions today.

• 26:14Nathan Labenz?

  Yeah, I'm a frequent nag, I think, when it comes to reminding people that we are really still just in the early innings of uncovering the bad behaviors that the latest AIs are starting to demonstrate. I had an episode with Ryan Greenblatt talking about the alignment faking and then I haven't done yet one yet, but Apollo Research also put out some really interesting work.

• 26:37Speaker 03

  There's a way.

• 26:38Nathan Labenz?

  Showing a pretty significant step change in situational awareness where the latest Claude model is starting to say things like, this feels like an evaluation, you know, so there's like the the number of sort of levels at which these things are thinking when they are ultimately just deciding, you know, am I going to do this, you know, nominally harmful task or not? It's like, well, the user asked me to, but I was trained not to, but they're telling me I might be trained differently, so maybe I should do it in this case. But wait, maybe actually I'm being tested for exactly that. So maybe I should go back. It is, we're definitely like entering a hall of mirrors and it's getting pretty wild.

• 27:16Nathan Labenz?

  Want to take one beat on multimodality because you mentioned how we are more natively multimodal than current certainly than current AIs are.

• 27:24Speaker 02

  I do take that back for what it's worth, but more, yeah, anyway, something we can dive into. I think you can arguably, you can arguably train AI models on more modalities, obviously than humans, but anyway.

• 27:37Nathan Labenz?

  Yeah, well, that's exactly what I wanted to ask. I mean, so we've seen recently this, and this is my vision of an early superintelligence at least. I mean, I think there's sort of, one vision is like, you sort of put O1 in a pressure cooker and it becomes O3 and you put O3 in a pressure cooker and it becomes like, I think at that point it jumps to O7 or something. And then, you know, it just gets so good at this like, super long chain of thought reasoning that it can sort of reason its way out of any problem. And it's kind of like, uh, you know, it starts to look a little bit like maybe the earlier Eliezer type visions of like, you know, if this thing is just the perfect rational Bayesian and it's got, you know, these perfect fundamentals.

• 28:19Nathan Labenz?

  Then, you know, you can infer the whole structure of the universe from like one, you know, picture of a plant and the way that, you know, gravity appears to be bending the leaf, right? And you, you're just extremely sample efficient and you're, you're just like extremely lean and mean. And, you know, that was a scary vision to me that I heard a number of years ago and it would still be scary, obviously, if it, if it were to come to that. Notably, the current AIs are a lot softer and flabbier than all that. And so my, what seems a little more likely to me, and I want to get your take on it, is like, take GPT-4O image generation.

• 28:54Nathan Labenz?

  Where there's clearly been a step change in the, let's say, depth of integration between the text and, and image understanding. Such that now it's no longer like this arms length thing where the, the one model has to say to the other, like, make me an image of, you know, all this text. Instead, that's integrated in the latent space and the same model is able to kind of work it from both sides. If I had to guess what a superintelligence, first superintelligence we're likely to see would look like, it would be basically that times like 10 to 20 more modalities such that the AI

• 29:35Nathan Labenz?

  has this sort of intuitive physics in a lot of different spaces. And by intuitive physics, I just mean like, if I throw you a ball, you don't have to calculate, you know, in sort of intensive, you know, molecular dynamics space like every molecule in the air that the ball is displacing as it comes to you to catch it. You just kind of know there's where it's going to be and you can just catch it. We've seen a lot of examples of this now from different modalities like material science and protein folding and whatever, where a proper simulation takes a lot of compute. A model trained on those, even sometimes just trained on pure simulation data, can do the same thing at like orders of magnitude

• 30:20Nathan Labenz?

  faster. And so, again, though, I have some hope, maybe that like, if we do that, maybe we still end up in this regime where it's like, superhuman enough to solve lots of problems, but that thing might be like, pretty unwieldy. And, you know, if you heard the Jeff Dean on Dorkesh conversation, like, could require like, an unbelievable amount of RAM for all the experts, you know, to be sort of splayed out across the pathways architecture and, you know, potentially multiple data centers or whatever. And maybe it's still kind of like, flabby enough that, and I'll just like, big enough footprint that, you know, in some ways, it becomes like, easier to control because it's, you know,

• 31:02Nathan Labenz?

  God knows how many trillion parameters, like, not super easy to egress off the network. Anyway, yeah, it's about there. How do you react to my superintelligence, my flabby superintelligence vision?

• 31:12Speaker 03

  The funny thing is, you're describing, like, the baby version of what you're describing is Gato, right? Like, this is a model, like, it's an old model, like, 2021, I think, maybe 2020.

• 31:23Nathan Labenz?

  Do you think Gato 2 is one of my big questions?

• 31:26Speaker 03

  Yeah, it was like, when I, when I first read that paper, sorry?

• 31:29Speaker 02

  No, I was saying, he's asking, what happened to Gato 2? I just said Gato 3, but

• 31:34Speaker 03

  Gato 3, Gato 3 ate it. That's what happened to Gato 2. Yeah, when I, I first read the, the Gato original one, Gato 1 paper, you know, 2122. And I was like, oh yeah, this is just the future. Because it is just like this, you have one model, it's like a billion parameters. It was very, very small by today's standards, but like, you know, pretty hefty ish by, by even those standards. And for a multimodal one, and it integrated all of the modalities that they could basically get their hands on. It was, you know, it was looking at images and describing them. It was, it was chatting on text. And he was even manipulating robot arms pretty successfully to, you know, move blocks on top of each other. And it was all done through a single kind of residual stream-ish thing, or a

• 32:19Speaker 03

  single kind of encoding, where all of those modalities were mapped into the same, like the same stream, the same token set. So this is absolutely something we know how to do. Yeah, maybe the future version of that is you have experts that are kind of sub-trained across those modalities. But I think that one of the trends that we see is that you, you start with like an exploded GBT 4.5 thing, then you distill it down and it doesn't, you know, quite have all the world knowledge, but it is capable, it's faster, it's cheaper, and it runs and it can be used as the basis for the next gigantically splayed thing. So I think it's a step along

• 33:00Speaker 03

  the line. Absolutely. But we're going to see more of, more of that multimodality. And I think across, uh, yeah, things like protein folding, things, all of these different areas.

• 33:10Speaker 04

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 33:15Speaker 02

  Yeah, I think like, one of the things that'll probably shape this, I would guess in, in a fairly significant way. So one, one thing that Gato showed and that we've certainly seen since is like, you know, positive transfer and the power of positive transfer with scale, right? So, you know, by that meaning, you, you, you train on one modality. And then what you learn from that actually leads to like lower loss when you combine it with learning from a new modality. Typically, obviously you have some, some like catastrophic forgetting type of effects or some sort of like model capacity issues that, that cause you to not be able to perform as well on, you know, say that the language modeling task, uh, once you move on to images, but now we're seeing that positive transfer. So that's certainly an argument in the direction of where you're gesturing. I guess another kind of, on the other side of the coin,

• 33:56Speaker 02

  I would expect a power law roughly in terms of like the the value that new modalities add. And we're always asking ourselves, I think, if we're OpenAI, if we're, if we're, you know, whatever these labs are, like, what is on my critical path to to recursive self-improvement, right? That that is, you know, they see that as the target. So there there comes a point on that curve where I'm I'm no longer interested in adding, you know, all the MRI data, or I'm not interested in adding certain kinds of like infrared data or whatever the modality may be, just because the the skills that I really care about look maybe more like autonomous coding. And and there, I'd rather spend that marginal compute on inference to allow the thing to kind of do the the RSI thing, or on RL loop, or Erdian, whatever it is. So I think ultimately, it it always boils down to how do you spend

• 34:40Speaker 02

  your compute? I think the the the story now is like, the ways to spend your compute just keep like, there's an ever growing list. And I think that there's a real kind of à la carte thing going on within labs, where one of the ways in which maybe they diverge is we're seeing them choose to trade off inference time compute versus pre-training compute a little bit differently. Within inference time compute, right? Not all inference time compute is fungible. There is a great kind of bit of research that Epic AI put out a couple of months ago that went into this is like, what kind of, what kind of inference time compute compounds, rather than sort of like, is getting at the same core thing? So if you think about like, you know, model distillation versus pruning, both are about trying to like, squeeze more into less. So so those are, you know, you're

• 35:22Speaker 02

  not going to be able to stack your inference time budget that are a little bit more fungible, whereas you have relatively independent axes of inference time compute elsewhere. So interesting.

• 35:30Speaker 03

  Yeah. So what you're saying is basically that when it comes to the labs themselves, and I think I agree with this, where there is increasingly going to be a competitive pressure that's just kind of forcing them to go, well, you know, what is it that actually gets us to RSI? Because that's just kind of how you have to go. But then you you would kind of imagine that.

• 35:52Speaker 02

  I don't think that's how you have to go, but I think that's.

• 35:54Speaker 03

  Well, no, I mean, it's like.

• 35:57Speaker 02

  More than anyone. Yeah.

• 35:58Speaker 03

  Yeah. But what I'm talking about is like, you have one lab that is is doing that. It creates a strong pressure for the others to do it. So you have this like logic of the race type thing where, well, to keep up, I I have to kind of raise my bar along that particular line. But then what what we would expect then is the other modalities to be kind of maybe tacked on after the fact by by fast followers. Folks who take, you know, open source models that are maybe a generation behind and maybe fine tuning them on on other scientific data, which is a really interesting picture.

• 36:32Nathan Labenz?

  So you guys have done some interesting, like,

• 36:39Nathan Labenz?

  what's the right word? The sociological work exploring the culture of AI development in Silicon Valley. And I would love to hear a little bit about how you understand, and we could go like maybe lab by lab on this, if you have that kind of resolution.

• 37:06Nathan Labenz?

  What is their goal? And, you know, to what degree is it really a sort of monomaniacal focus on getting to this point where the AIs can take over the AI research and then we sort of let what, you know, come what may from that point, like increasingly, you know, we just have obviously seen AI 2027 from Daniel Coccotello put out. And I haven't spoken to him about that. I did have a chance to play his war game, you know, that sort of basically is the scenario that that document describes.

• 37:45Nathan Labenz?

  But I can't speak for him in terms of like exactly what he's trying to communicate. But like one way that I read that manuscript is as a warning that this is what OpenAI is trying to do, like they are, you know, they have their sights set on this recursive self improvement dynamic and.

• 38:04Nathan Labenz?

  I don't know why, if that's what he thinks and what he's trying to communicate, why we're not a little bit more just matter of fact and declarative in saying like, this is what they are trying to do and you should be afraid of it. But what have you guys, I mean, you've done a lot of this like on the ground, hanging out at the infamous Silicon Valley parties and all that good stuff. Like, how would you describe what they and perhaps other leading developers are actually envisioning themselves doing to get to a superintelligence?

• 38:31Speaker 03

  Yeah, I mean, without, without directly speaking to, you know, Dan's opinions or his call, his opinions, yeah, the, I mean, yes, in terms of what the game plan is at, at many of these labs, some of them are more direct and forthright than others. Some of them are more gung ho, some of them are less, less so, less enthusiastic than others. But the idea of like, we're going to get to a point where AI is going to just do our homework is kind of like, especially if you've been following the curve, it's like, it's the obvious kind of default path. Back when Jan Leica was at OpenAI, that was, that was his like, his take when it came to the alignment side of things.

• 39:14Speaker 03

  And, I mean, you know, it's like, it's not clear how good of odds of success of that are, but it's obviously not a crazy approach under current conditions. And he's trying the same thing over at Anthropic. Yeah, I mean, yes, people have absolutely told us that without like singling out OpenAI, because it's, again, it's like the, the obvious thing you do if you're kind of YOLOing this. Yeah, OpenAI is considering this and the path to superintelligence is faster if your researchers operate at machine time. And from the leadership perspective, it's also so much better, right?

• 39:59Speaker 03

  You have like, you hope perfectly loyal AI employees who will not leak your stuff at parties or in, in conversations at bars and it will work faster and work, work at night. But obviously, you know, then the question is actual control, actual alignment. And I I I think the AI 2027, one thing they did really well was conveying the kind of emotion of like, oh, this beast is like, starting to take on a life of its own. And like, I like, I can understand as the researcher working on like supervising this thing that these are the last few months where my input and my work is actually going to matter at all to the outcome. So I'm going to burn the midnight oil.

• 40:45Speaker 03

  It's evocative and it could, it could be how things end up going.

• 40:51Speaker 02

  Yeah, it's, I was a really big fan of the AI 2027 framing, just because I do think it's a, it's great to have the story that's concrete out there. If only because people can then criticize it. There are specific things that, not a huge number, but there are specific things that we found that that we would have written differently, let's say, in in the the document. But that's the the virtue of it is you can actually point those out. Whereas one of the the really interesting things is to tie back into the cultural point. When you look at how, and I will single out Sam A. On this because he's, I think by far the worst offender on this, like the the vagueness with which he describes what superintelligence actually is, what it means to get there and what he will do with it

• 41:33Speaker 02

  is an asset to him. It is only an asset to him. It has always been an asset to him, and he has used it as an asset in recruitment. Just the same way that they've used, for example, the, you know, famous nonprofits control over the for-profit entity. That's, I think, I mean, any reasonable person would say that was used for recruitment purposes. There are now, you know, an amicus brief out there saying from from all these researchers sharing that view.

• 41:56Speaker 03

  But it was also pretty much in the email chain that, you know, that was that was made public and, yeah.

• 42:02Speaker 02

  Yeah, no, I mean, this is like, not even open secret. This is just known stuff, right? So I think when you think about the kinds of people who work at the different labs, they are different. There's a selection effect for sure. If you, the the first thing I'll say is, having talked to a lot of current and former people at OpenAI during this investigation, you see, you see a very sharp contrast between what the the researchers themselves say and believe and are concerned about, and then what the executive says in. And seems to be concerned about publicly. Those things do not track as a simple point of fact, they are at odds. That's not something you see at Anthropic. What you get when you talk to somebody at OpenAI, or at least as we did during the investigation, is you'll get people who are very nervous, very keen to tell you, please don't tell anybody that we spoke.

• 42:47Speaker 02

  Please don't communicate like it's it's really, really like a tight kind of leash that people feel themselves to be on. You talk to people at Anthropic, that story is completely different. So you get the sense that yes, you're talking to people who may maybe sometimes have disagreements with the leadership for sure. But you have the sense that they would be comfortable voicing those disagreements and the fact that they have. And that's a kind of a really key piece of almost cultural bedrock that I think is is most clearly. Like you see that divide between those two companies in a pretty big way. And obviously, OpenAI's character has changed in fundamental ways. There's no getting around it. They used to be a lab much more focused on the sort of, at least overtly focused on on the risks of one person having too much power.

• 43:31Speaker 02

  Right? That's a big theme like I'm old enough to remember when Sam Alvin was not supposed to unilaterally be able to like jujitsu his way into kicking off an entire board, having friendlies basically come in and sort of rewriting the script on that. And you just keep seeing those goalposts get shifted and shifted and shifted. And so I think, you know, one of the the challenging plays has been from our perspective, trying to get a very like objective, clean sense of where all the different labs stack. You know, we want American champions to be at the head of the game here. That that's necessary. But at the same time, you need to think about like, OK, who can you actually rely on to make the right calls for a variety of reasons? And frankly, at this point, given the industry incentives, we're not seeing anyone invest in security in particular,

• 44:16Speaker 02

  nearly in the way that they need to. I mean, these these labs are CCP penetrated like there is just no way in hell that they're not. And if there's one thing that the investigation like made extraordinarily clear, it's that is a wildly pressing problem that is not being taken seriously enough right now. That needs to change in a really big way. But there are cultural factors too. There's all, you know, obviously the like. Whatever version of transhumanist so many of these researchers are and effective accelerationism and all that stuff, like there certainly is an influence of those things. I think a lot of folks at these labs, especially OpenAI, do kind of seem to have blinders on. The ambition of their kind of their line of sight seems to be a lot shorter than the ambition of the lab's line of sight.

• 45:01Speaker 02

  The lab is saying, we're making superintelligence. And it really seems when you're talking to these folks, they're focused on like the next beat. Okay, how do I solve this narrow problem? Yeah, yeah, we'll deal with the like the security, the alignment, that all that shit. We'll deal with that down the line. That is its own kind of problem. And it's only true in some cases. It's not universal.

• 45:19Speaker 03

  Yeah. And not that there's not that they've done zero in security relative to like when we.

• 45:24Speaker 02

  Right.

• 45:24Speaker 03

  So we published that State Department backed report about a year ago now. The situation was truly catastrophically bad at that time. It is still really bad. It's nowhere close to where it needs to be from a security standpoint. We know that. But they've made some some motion in the direction of progress there that we should just, we should note, even though it's like, it's not even close to be enough.

• 45:49Nathan Labenz?

  So would you describe this whole project as ideological? I mean, we have heard similar things from Dario with respect to, you know, their leaked fundraising deck, which said like the, you know, leading companies in 2025, 2026 might get so far ahead, nobody can ever catch up. More colorfully, I would say, in his more recent and public writings, saying that, you know, we can get this edge on China and then we can sort of use our edge to build an ever bigger edge and, you know, whatever, we'll somehow make them an offer they can't refuse.

• 46:25Nathan Labenz?

  I'm not sure if I should understand it as almost religious or.

• 46:32Nathan Labenz?

  What exactly? But it does, I mean, I just made a small donation to a project called Building God, which is a documentary that's trying to, you know, trying to bring more of this to light. But I, you know, support that because I genuinely don't know, like, are these people trying to build their own new God? Is that an overstatement? Or is that like an actually apt description of the culture and the ideology that animates it?

• 46:57Speaker 02

  You mean, did Ilya lead prayer sessions at OpenAI about with the feel the AGI mantra? Or yeah, no, I mean, it's a thing like, I think objectively, they are trying to build superintelligence, right? Or the very least AGI. And we should get into into that calculation too for strategic reasons. But that's the goal. Whether you view that as a spiritual imperative. I mean, certainly they've said this, right? Like, I think Miramirati said, you'd like use the word spiritual or something like that in kind of describing what motivates people to come to work. And that's absolutely the case when you talk to a lot of these people. With others, it's not. But I think that's, you know, they believe they are building the hyper object at the end of the universe type thing, right? Or well, anyway.

• 47:41Speaker 03

  The labs, the labs also are not monoliths, right? Like, the different people have different beliefs and stuff. So take OpenAI as an example. They grew really fast. And so some, some people there are in it mostly like for the money, you know, this is an incredibly fast growing company. There's a story around how like it goes vertical and goes to infinity. And so if you're thinking of things in terms of like money or in terms of just economic value or value to yourself, maybe in a more abstract,

• 48:11Speaker 03

  you want to get on that rocket ship. And that's one, that's one kind of pathway. Another one is absolutely this, the transhumanist kind of ethos or ethic in the idea that, you know, there's, there's kind of this destiny aspect to it that, you know, we, we are going to upload our consciousnesses or transcend through this, like, this object. And there's, there's a number of ideologies kind of all mixed together. But yeah, certainly, from a secular perspective, the messaging and a lot of the internal conversations are around, like, at the very least.

• 48:49Speaker 03

  This is history making. This is an inflection point in the arc of history unlike any other that has come in the past. It's beyond the wheel. It's beyond fire. It's tantamount to the inception of the human species in its impact on the earth and the universe. Only more so.

• 49:09Speaker 02

  I do want to touch on the Dario China thing, though, too, because I think that is distinct from the sort of like, you know, religious zeal that I think some people in the space are, and I'm pseudo religious zeal, whatever, like, you know what I'm getting at, that some people are motivated by. I think Dario, I mean, frankly, it's difficult. If you, if you take the view that China is a competent adversary in the space, if you take the view that they're a committed adversary, that doing business with them is, is not going to work on the timelines that Dario seems, sees as plausible. So if you think about the 2027 or anything like that, then, then suddenly you are forced into some pretty tough choices. I mean, we said earlier, this feels like an over constrained problem. This is really where that comes from, right?

• 49:53Speaker 02

  I mean, you have this tension. And, and so I think that's why you're seeing Dario move in that direction. I think increasingly it's just become obvious to a lot of the frontier lab CEOs that genuinely, the situation with China is very dire and requires us to forge ahead. And it means that you are playing chicken with a cliff. And the question is like, can you, can you turn your car into an airplane before you, you hit the lip of the cliff? What am I even saying? But yeah, I mean, that, that sort of, I, I think that there's a very reasonable reading. I'm, I'm quite sympathetic actually to Dario's argument there. And that's unfortunately part of what was surfaced in the, the work that we did was just like the extent to which China holds us at risk, holds our infrastructure at risk.

• 50:37Speaker 02

  And really, that our option set is a lot more narrow than I think a lot of people in the kind of AI safety ecosystem tend to think.

• 50:44Speaker 03

  Yeah. And this is, this actually comes back, Jared, of what you were saying earlier, which is like, there is a fundamental dilemma here, right? Like, and the dilemma is kind of realized or exemplified by these two different camps that are not talking to each other and are not hearing each other to the extent that they need to, to resolve the dilemma. One is, yeah, the, the AI safety ecosystem kind of broadly construed, which is saying like, look,

• 51:09Speaker 03

  These systems may not be controllable or alignable. And some folks are like, no, they're absolutely at super intelligent scale. What are you even talking about? Like, we're just not going to be able to do that. And like, that's potentially a reasonable, a very reasonable view. And so therefore, this is a coordination problem. And the only thing we can do is we have to, everyone has to slow down to give us time to solve the alignment thing. Or these systems are basically just going to like rule us and probably kill us because of all of these, these kind of instrumental convergence arguments and stuff like that. That's the one camp, right? It's a coordination problem. We can't solve alignment, so we need to slow down.

• 51:51Speaker 03

  The other camp is the national security folks. They're starting from the position of, look, I've seen how China operates. I've seen how they leverage international agreements to constrain us while they have a free hand and basically like break whatever agreements they want. I've seen how they treat members of their own population. And like, some of the stories, like they, you know, they, they put people onto buses to, to just like go off to be executed. It's not like an incredibly common thing, but absolutely, there's like a, there's a treadmill of that going on over there. That's like a, just a thing that they do, right? Like no due process.

• 52:33Speaker 03

  The, the, the view is like, we would have to be crazy to make a deal with these folks and that government and have any assumption whatsoever that they're going to keep to it. Because a deal is impossible, we have to win, therefore we have to race. And one says, yeah.

• 52:54Speaker 02

  And implicitly, we're either, we either gotta hope the loss of control problem isn't a thing, or, or that we're gonna solve it in flight, right? Like, that's the implied premise there. Like, I, I, I'm not looking at that problem versus I'm not looking at that problem. And yeah, anyway, that's the gap.

• 53:09Speaker 03

  Bingo. And, and the safety side is, is looking at the same thing on, on the China side where it's like, yeah, you know, we have to slow down. And therefore, I'm I'm just going to ignore or or I'm going to minimize the arguments that say like, no, all of the evidence shows that this government will do whatever it takes to get ahead. Like, they will lie, they will like, uh, use criminal surrogates, they will like pull every possible lever to do what they want to do. And those two sides are not listening to each other. And that's where the argument comes in. But that's it's fundamentally a hard problem.

• 53:49Speaker 03

  It's it's a dilemma and that's the that's the dilemma that we are trying to make some steps towards resolving in this report.

• 53:57Nathan Labenz?

  Before we get into the findings of the report, uh, how maybe this is good, sort of starting to get there, I suppose. But how AGI pilled, generally speaking, do you guys feel the US government is today? I mean, I'm not even sure that's a coherent question because we have obviously a very stable genius, uh, who I sometimes call he who must always be named now as it seems like more and more of these, you know, AI documents are being, you know, written with an audience of one in mind. And I'm not sure there's any like coherent center to it, but

• 54:31Nathan Labenz?

  I guess, you know, multiple question is like, how AGI pilled, do you think the US government is broadly? How

• 54:39Nathan Labenz?

  subject to the whims of one individual is that

• 54:43Nathan Labenz?

  degree of AGI pilledness? Uh, how much does it matter if the guy at the top sort of says one thing or says another? And then how would you compare that to China? Like, how AGI pilled is the Chinese government today?

• 54:54Speaker 03

  Yeah, we're we're limited in in what we can say necessarily on on the US government side with respect to the particular people that we've spoken to. I mean, what what's known publicly, right, is, you know, Trump has come out and said, hey, there, you know, there are things about AI that look are really exciting, it could be transformational, he's talked about that a lot. Uh, he's also alluded to the risks too. There were some podcasts that he did, I think, one with, uh, one of the Paul brothers. I can never remember which one's which, but anyway, we were just talking about the Yeah, yeah, there

• 55:22Nathan Labenz?

  He did strike him, I think, right?

• 55:24Speaker 03

  Yeah, that's right. Yeah. And and actually, funnily enough, I

• 55:26Nathan Labenz?

  Somebody was selling some bullshit online with his likeness. Can you imagine?

• 55:30Speaker 02

  Well, I think I think he was also like it wasn't a comment about the deep fake thing. It was a comment more generally, which I think, you know, you can read into any number of ways. Obviously, J D Vance gave that speech in Paris, which I think is sure the most concrete thing apart from Crassios's speech that recently happened, where we're getting a bit of a sense for the the positioning there, which does reference at the end, like it, you know, says, we can't be hand wringing about safety. And by the way, this is like, this is in a context where he sees himself as so many do, as in a race with China, right? This that is, that's what it means to buy that argument. The flip side at the end, he does say, you know, acknowledging that there are real safety or security risks or whatever with the technology or the real risks do exist.

• 56:14Speaker 02

  And so there is an awareness that there are real risks here. There are, you know, people, as you might imagine, in various national security contexts within the government who are tracking this and are quite aware of the the full range of possibilities. But what to do about it is like, we've seen Congress wrestle with tech. We've seen the executive wrestle with all these things for, for, you know, administration after administration now. I think it's it's still very much being formed, like people are figuring out what they think, both about the prospects of the technology, the risks of developing it from a loss of control standpoint, and then as well, the risks of losing the race to China. All of that right now is still very much up in the air.

• 56:51Speaker 03

  Yeah, and that's like, that's exactly right in terms of the administration side of things. More broadly, in the US government, you kind of alluded to like, you know, the the coherence of the question, well, the reality, obviously, is the US government is kind of defined by how huge it is. And everybody has their, their, you know, 3 million people or however many million people with that many millions of opinions. But certainly, many, many more people in the government are AGI pilled than they were like a year and a half ago.

• 57:26Speaker 03

  A year and a half ago, there would be like the odd individual person who was like kind of getting there, but not sure about their colleagues. And so kind of didn't speak up. And so you would talk to them and be like, yeah, yeah. So actually, like, I kind of see like where your head's at, but like, I don't like nobody else like da da da. Whereas now it's like you have entire offices that are AGI build and like talking openly about it. One other thing that's like actually like a an interesting data point. Every, everyone or almost everyone whose work touches on this area even obliquely knows about or has read situational awareness and like kind of absorbed the high level points.

• 58:10Speaker 03

  This is actually a really interesting illustration of like how you know memes travel across channels. Situational awareness wasn't really reported much in the traditional media, which is like, you know, Leopold Ashenberger didn't write a Wall Street Journal op ed, for example. And yet it traveled and it's in the water in those spaces. And make no mistake, it's in the water in China, too. There were translations in Chinese of that manifesto circulating over there. Not long after the original was published here, that awareness is is present.

• 58:48Speaker 03

  And when you see the CEO of DeepSeek like sitting with Politburo members in in this like, in this context where it's like you guys are the captains of industry that are fighting the fight for us. And and the fact that the Chinese have have invested now have announced this, you know, quarter trillion dollar investment package into AI infrastructure.

• 59:14Speaker 03

  It's a, you know, we're starting to slip into that, that that zone and that space for sure.

• 59:21Speaker 02

  As as Ed was quick to point out when that first happened, too, that's a a quarter trillion dollars in in PPP terms. Right. Because a lot of the early reporting got this wrong and was saying like, you know, I think 137 billion was, you know, semi analysis is kind of like take home, which is incorrect. It is 250 or it's it's over a quarter trillion dollars.

• 59:40Speaker 03

  Yeah. There is like they they have like they're building the all this stuff themselves. So it's like, how much can you procure in country?

• 59:50Speaker 02

  And increasingly so, right? I mean, like it's obviously there's like you're you're taking those those age twenties as well. It's a balance, but but yeah, it's it's a it's a like a really big issue. I think another piece to this, too, is like. Like we had some intense conversations early on when we were doing some of our work a couple of years ago with people who were like, well, you shouldn't be talking to people in the US government saying, using the term AGI. And there's this like kind of concern, this sort of like nervous anxiety in the community of people who worry about AI policy, which I both understand and think is very counterproductive. And we've actually seen the consequences of that anxiety play out both culturally, because it's profoundly off-putting to people who are builders.

• 1:00:34Speaker 02

  And this administration is an administration of people who value the building culture and vibe much more than the kind of regulatory sort of cautious culture and vibe. That's a huge mistake. That is a thing that also anybody could have seen coming. Like, you're going to get a Republican administration that is sort of more, in a sense, more libertarian-leaning, at least more kind of pro-accelerationist. And this idea that you're somehow going to prevent people from talking about AGI is, I think, one of the most dangerous things that this movement has sort of done. One of the consequences, at the time, what we were saying was, look, you have before you a unique opportunity to be one of the only groups of people who is speaking to this issue before it comes into the mainstream. People haven't made up their minds yet.

• 1:01:16Speaker 02

  This is your time to inform people about the stuff that will become politicized, right? Like, again, like we just talked about, this polarization around, well, are you either a China hawk or are you concerned about loss of control of these systems? It's fucking stupid that those two things are like somehow anti-correlated. Like, they have nothing to do with each other, right? But we live in that world because that's just how people have made up their minds. We've lost the opportunity to shape a lot of that landscape because people were hand-wringing about, can we even have conversations about this? Which I think was just like a profound self-own and fortunately something that we didn't heed, though we thought about it very deeply at the time, obviously. So many people are telling us, don't talk about the thing.

• 1:01:57Speaker 02

  I think it's pretty self-evident now that had more attention been paid to that in certain kind of tactical ways, I think that the landscape would look quite different. But yeah, it's now kind of become that China hawk versus loss of control bit, which is just, it's inaccurate and it's pretty awful.

• 1:02:15Speaker 03

  And you see the, I mean, you can sympathize with the view of like, Every, you know, every incremental thing you do to accelerate or to, you know, broaden the concept is a thing that you can't take back. Like now everybody's gonna be thinking about like, what a big weapon AGI is, what a big weapon super intelligence is and stuff. The reality is like you're in, you're in the space where even, even absent the geopolitics, there's, you know, there's like four or five different like super intelligence projects, depending on how you count, going on, right? In, in America with, with all the frontier labs.

• 1:02:52Speaker 02

  And there are always going to be, right? And there are always going to be people who are going to publish, whatever manifesto about the obvious strategic landscape. Like at least this seemed like a very likely outcome. But sorry, I just like, I think that's kind of part of the context. It's like this will happen eventually. People will talk about AGI, but yeah.

• 1:03:09Speaker 03

  Yeah. You, you end up in this like, in any case, you're, you're kind of in the, the, the Moloch, if you like, part of the scenario where the race, like now just zooming in to the corporate race between the, the American labs, forgetting China, even just that race has a life of its own, right? It has its own logic. The CEOs of the labs like have relatively little control and little agency over the dynamics of the entire thing. Like if Sam A goes like, oh shit, actually this is like super fucking dangerous. Let's like bow out and like shut down OpenAI or whatever. Well, that doesn't actually change anything. Now you've, you've got like XAI, you've got Anthropic and you've got like maybe Meta and a couple of others like still trying to race ahead. And so at the end of the day, it's like you're the,

• 1:03:54Speaker 03

  you pass a point where like little tactical interventions on, on individual like pieces, those matter. When there's like one or two or maybe three players, you get to a point where like beyond a certain point, you have to like, you may have to execute an intervention and you need the intervention to be based and grounded in an understanding of what is being intervened on.

• 1:04:24Nathan Labenz?

  Yeah, I mean, I might have a little bit more sympathy for a couple of things than I think you guys. I mean, broadly, I think that's it's a hard analysis to avoid. But if there were a couple of things that I would push back on a little bit, one would be that I think there is some interaction between the US-China dynamic and like how likely we are to lose control of AIs, if only because if we get really reckless in racing to super diligence, then that seems like we're more likely to lose control. MSO one having, I think, is like.

• 1:04:55Speaker 02

  Just to just to clarify on that point, when I was saying they're independent, what I mean is the conclusion like China is a real player in this race. They're an adversary. Logically, getting to that conclusion is independent from getting to the conclusion, oh, AI is potentially like loss of control is a real threat factor. The like those two thoughts should be able to live simultaneously in the same head. What you do about it, that's where you get into the contradictions for sure.

• 1:05:18Nathan Labenz?

  Yeah, I mean, there's a lot of motivated reasoning going on for sure, and in a lot of places, and I'm probably guilty of at least some of it myself.

• 1:05:24Speaker 02

  So are we? Nah, none of us are guilty of it.

• 1:05:28Nathan Labenz?

  Yeah, present company excluded. The other thing that I do hold out maybe a little bit more hope for is like if Sam Altman, you know, it's a hard hypothetical to really imagine, but let's say that he did have a sort of, you know, awakening type moment and was like, I'm leading a bad dynamic and he actually did what you said and, you know, genuinely bowed out or like made, you know, some costly signal style of commitment to a different trajectory. That could be like a pretty powerful signal that could perhaps create a cascade. Like I'm a big believer in like multiple equilibrium, and I think we are in one.

• 1:06:10Nathan Labenz?

  That doesn't preclude in my mind, the possibility of others existing, like how to switch to them is hard, but maybe what it takes is like one or two brave and foresighted individuals to say, I'm trying to move to this other equilibrium, like who's with me? And, you know, maybe the rest could fall in line. I don't know. It doesn't seem so.

• 1:06:30Speaker 02

  Yeah. So there there are paths. Yeah, there are paths that look like that, right? For sure. And so if it was just like Sam A bowing out, that would be a big deal. Be news around the world, be all kinds of stuff. I don't know that that would that would on its own like affect the race, but.

• 1:06:47Speaker 03

  If it was, you know, for example, Sam A bowing out and he's like, hey, I'm bowing out because the latest model that we trained like started doing all this fucked up shit that we don't understand and it looks like super scary and here are the results, everybody. Then maybe that has a different or broader or more sobering effect. Another possibility is just simply like you get some kind of malicious use of a, an open source model in a way that has like very broadly bad consequences, like where, where people get killed, for example, like where a lot of people get killed and it's clearly attributable to this thing. Then I think that's like, that's a, a potential avenue where we go

• 1:07:27Speaker 03

  like, oh, okay, this stuff is for real. There's blood on the floor now. We want to look at this in, in a sober way and with that kind of general understanding. Then I think you get kind of pulled up into the geopolitical context because regardless of which those two, which of those two things happen, you are in a, you're in a space where the risk and dangers and therefore potential military and strategic capabilities of systems like this are now in sharp relief. And that's a somewhat different space than we're in now.

• 1:08:07Speaker 02

  Yeah, I think it's always the, you know, the first movers dilemma or whatever. Like you, you have whoever the most risk tolerant actor is who's going to go out and do a thing. And in a context where, you know, I think the geopolitical dynamics or geostrategic dynamics with China matter here too, you know, the CCP views itself as being in a struggle for survival. I mean, this has always for them been an existential struggle with the United States, with the West, to push their world order, their agenda forward. It's like, it's difficult to see people. I mean, there's always that temptation to go like, oh, can we push it like a little bit further? Cause now OpenAI has this scary, powerful system that they've just announced to the world that they have. By the way, by then China has for sure stolen it by the time Sam A says, oh my God, I'm freaking

• 1:08:49Speaker 02

  like, uh, under nominal conditions and, and even given the current trajectory, like China has stolen that shit a long time ago. They're like polishing up the, off the training run on their servers and, and you know, using their fleet of like, like stolen H20s or whatever the next version of the Ascend chip is. But bottom line is that brinksmanship keeps playing out, right? And like, you get to continue the game from whatever step you're on. And so like, I see how it would lead to a big media moment. I see how it would lead to a lot of probably gridlock in Congress. So like, like, regulation is probably not going to happen then on any like, relevant time horizon. So then you're, you're, okay, well, what can the executive do on its own? You know, you can start to pull all kinds of strings

• 1:09:30Speaker 02

  and emergency powers and DPA and all that stuff. But ultimately, the logic of the race, if nothing else, I think might get exacerbated in that moment, where you have an administration that looks over its shoulder at China and says, oh, yes, I guess we basically got to the endgame. And you, you kind of recover the same conditions that we faced before. Do you trust China? Or do you not trust China? And the like, based on everything that we've seen and heard, the, the list of reasons not to trust China, even in extremis, is pretty damn long. I mean, you know, the, the like, lab leak stuff, the, the way that, that

• 1:10:04Speaker 03

  They've embedded, like, they've, they've actively embedded a bunch of, a bunch of Trojans in our critical infrastructure, and they're holding a gun to our head. Like, that's, that's publicly, that's publicly disclosed. Like, and, and I think we, we talked about this a little bit offline, and we mentioned in the report, but just like anecdotes, right? About folks in the national security space who've dealt with the Chinese and, and kind of the way their espionage system is structured. Like, the, the CCP takes a controlling and like ethnically centered view of its diaspora, as in like, doesn't matter that you're not a Chinese citizen, and that you're like, a second or third generation

• 1:10:46Speaker 03

  American or something. You're still, you're still like, from the motherland, and we own you. And so therefore, we're going to apply pressure to you. Like, the story that this like

• 1:10:56Speaker 02

  To be clear, this is also like, this is deviating a bit away from, what if Sam Ma came out and said this thing, right? This is more like, the tools and mechanisms that the CCP enjoys with respect to the US. I, I think that the, the logic we fall into is this question of like, do you trust them or do you not? I, I find it really difficult to imagine, like, Actually, like, we, again, we don't have Trust But Verify. We don't have like Fletchek. We don't have the ability to build those secure, governable chips. Would be great if we could use very advanced AI systems to accelerate that development. That seems like a really important direction to push in. But right now, if we hit superintelligence in 2027, it's going to be, you know, just nothing but fog of war.

• 1:11:38Speaker 02

  And right now, our ability to monitor what's happening in China is really terrible. Like, anybody who has in their head a scenario where somehow, like, we're at parity in terms of intelligence with respect to each other's stacks, that does not track reality, right? It is a profoundly asymmetric scenario. And that's one of the first things, well, that we think has to be fixed, too.

• 1:12:01Speaker 03

  Yeah. One of the reasons why I was bringing up those, some of those anecdotes, was to put color around the, yeah, the trust issue, right? So this story that I'll just quickly mention, this defense official, he was talking about a power outage in Berkeley, like around 2019. And all the Chinese students were freaking out because they are, they were mandated to report back on a regular cadence to the motherland or their handlers or whatever. And these are just like regular people, like regular students, not people particularly working in any sensitive areas or whatever. It's just like, you report back or your mom doesn't get her insulin or your brother like loses out on a job opportunity.

• 1:12:44Speaker 03

  And there's an escalating ratchet of just systematic pressures that gets applied more and more and more. And basically, the weight of the state and this very kind of optimized set of pressures is applied down to you. And it's challenging for anyone to resist that degree of coercion.

• 1:13:02Speaker 02

  And which is, by the way, like, I mean, an absolute travesty and a tragedy. Like, no one is a greater victim of this than the Chinese people themselves. Chinese nationals working in the United States are subject to, at least the potential for this kind of activity. That's like, that's I can't think of a word for it. It's despicable. I mean, but and Chinese researchers have made amazing contributions to our frontier AI, you know, achievements. You can just look at the names on the papers, right? Yeah. So, so this is, this is one of, that's another issue, right? You look at these labs, the, the composition is then like, you know, large, the double digit percentages of these folks are, are at risk from this sort of activity. And so imagine building a Manhattan project in a context where that's the case.

• 1:13:44Speaker 02

  You quickly get into like these, the really dicey things, right? Like, we're not happy to have to report this. Very much aware, obviously of like, yeah, there, there's, there are these awful, like, you know, there's like Japanese internment camps is the other side of the coin here, right? That is World War Two. That's what it means to like, to take an argue, like, an argument like that to one extreme. But there's also just the reality of like, what would it look like to build the Manhattan project? In a world where it's not just that, like, you know, your, your, uh, your team building the project is, has got a bunch of Germans on it. It's the world of technology has fundamentally changed and these people can be monitored nonstop. And they have family back in Germany and, uh, they have all their, you know, a bunch of financial ties to that country.

• 1:14:27Speaker 02

  And, and, and, and, and that's just a fundamentally different calculus. And so it's, it's just a real challenge, but it's a reality that somebody has to come out and just say, without fear of like, you know, being called names. It, it is just a fact of the matter. And again, if there's one thing that we are here, it's just sort of reporting the news. It's, it's an unfortunate reality. We wish we had different news to report. That's kind of how things are shaping up right now.

• 1:14:52Nathan Labenz?

  Let's go down this rabbit hole of what does it actually look like to try to do an American superintelligence project? AKA the Manhattan project for AGI or ASI or whatever. You know, maybe start off with like,

• 1:15:07Nathan Labenz?

  how does it happen? You know, you've mentioned like the president and the DPA, but kind of interested in like, what you think it actually looks like for us to go from sort of market normal to this like national, you know, Manhattan project style thing. Then I'd love to hear a bit about how you actually learned what you did. I think you guys are great examples of, you know, putting. You know, shoe leather to pavement and actually getting out into the world and learning some stuff. And I think that's really both credibility building for the report and also just something that should inspire other people. And then we can get into the actual like, okay, if we're actually going to do this, like, what is actually required to

• 1:15:52Nathan Labenz?

  really do it?

• 1:15:53Speaker 03

  At one.

• 1:15:54Nathan Labenz?

  To take it away.

• 1:15:55Speaker 03

  Yeah, one thing I'll just mention out the gate is like, the way that we started out working on this report was it was in the wake of situational awareness. And we're like, well, you know, some picture like this needs to be fleshed out, whether it comes to pass or not. And so our view was like, we're approaching this like, we're not, it's not clear that it's the right decision to do something like this. But it's sure as heck the right decision to think through what it would look like if, in the if, like, under the condition that, like, what if we end up doing this? Someone had better have thought through those things in advance, right? And so we started working on it through that lens and through that angle.

• 1:16:40Speaker 02

  Yeah, I think the other piece is we explicitly stayed away from, partly as a result of that, but we explicitly stayed away from questions about what are the specific authorities that would be invoked in the pursuit of this sort of thing? Bureaucratically, how would this be done? If only because, A, it's just not best and highest use for us. We did include some thoughts about that because we ended up talking to people in relevant departments and agencies and hearing their views. And we didn't want to lose that value. So we did kind of park some thoughts there. But they're not set in stone. They're just kind of like, they're there to be considered. Our main focus was at the gears level, what has to happen? What are the verbs that have to like, be written into the storybook for this to unfold? Well, what does a data center have to look like? What does the supply chain have to look

• 1:17:25Speaker 02

  like? Like, all of these questions that in many cases are, it's interesting, like, the the challenge you run into is the space is very long on opinions and opinionary, much of which is very well informed. And we've leaned heavily on a lot of the great work that's been done in this space. But what is really missing is If you actually talk to the very, very miniscule number of people, I'm not talking about like intelligence and special forces, I'm talking about zoom in on intelligence, special forces at our, you know, tier one units. These are like mostly SEAL Team fucking six, Delta Force. Then zoom in even more, zoom in on the specific people who, who anyway, specialize in the relevant

• 1:18:09Speaker 02

  areas and ask them, this is an extremely small community, like, you know, and they're tightly kind of networked and it's a very high trust ecosystem. They are aware of TTPs, like tactics, techniques and procedures that others are not aware of and on which your entire assumption stack can unknowingly rest, right? So when you have conversations with people in the space and you're like, hey, so like, let's just go out and do, their recommendation is like, let's just go out and do X. And you're just like, well, I just had a conversation with somebody about an approach that they know of that makes this totally moot. Like, that's the kind of thing that you run into. And so unless you talk to this very specific group of people, you kind of have an incomplete

• 1:18:53Speaker 02

  picture. And, you know, going back to this idea of the jagged capability surface of the models, right? This is the jagged capability surface or understa- or yeah, capability surface, you could say, of the US government, of US industry. And when you actually talk to people at the labs, you get a very nuanced perspective on what's possible. It's not the same happens when you go to the intel community, the same happens when you go to the special operations community. You get a picture of like, well, actually, this is trivial. This thing you thought was hard is trivial. This thing you thought was trivial is really hard. And that suddenly just refactors your whole picture.

• 1:19:24Speaker 03

  Some of the stuff, some of the stuff is just wild, too. Like, we, we visited a data center with some former special ops folks and not like tier, again, tier one folks from these like restricted, very, very like elite units. And the, they're, they're just like wandering around, like asking, oh, you know, what happens if you do this to that thing? Oh, that, okay. What happens if you do that to that? Okay, cool. And wandering around a little bit and, and comes back to you and just says like, yeah, so I can think of like one or two, one or two ops you could pull for like 30 grand that would knock out this data center for about a year. And everyone's like, wait, like a $10 billion facility. Like a $10 billion facility. And when you're talking at that level,

• 1:20:07Speaker 03

  You're not even talking about, oh, China could do this. Like, and a, a smart person with a bit of time and money can do this. And so that means that, you know, adversaries can just like, proxy that and use surrogates and use like whatever kind of smokescreens and, and kind of sock puppets to go and do that with total deniability. And that's the state of security. And we actually have been asked, like, don't give any details on those kinds of, on, on what those vulnerabilities are because those vulnerabilities remain and they are critical and they are universal, basically, across data centers today.

• 1:20:42Speaker 02

  That's kind of one of the challenges here, right? It's like, the right answer is in so few heads and there needs to be, and we're under no illusions that we have the right answer across the board, right? Like, that, that's another implication directly of what we're saying here is the full picture resides in no one's head. But the combination of the, an awareness of that uncertainty and then having been shocked by a couple of these individual things is sort of what is shaping at least the strategic picture of the kind of document.

• 1:21:10Speaker 03

  Yeah, and, and just quickly, it's a reason why, or one reason why it can make sense to publish something like this at all, which is it gives folks something to point to and say like, hey, I actually know that's wrong because my specific niche area of expertise says this and this and this. And that is the beginning of a constructive discussion.

• 1:21:31Nathan Labenz?

  So, without getting into the specific vulnerabilities, let's talk about what do we take to actually secure such a project? I mean, we're in the scenario now where governments are wide awake. Governments have decided we're in this race, we got to win it.

• 1:21:52Nathan Labenz?

  We can't just have, you know, people, uh, spilling all the secrets at San Francisco parties. We can't have, uh, whatever data center vulnerabilities exist at the infrastructure or hardware level. It's time to get serious. What is that getting serious actually look like?

• 1:22:12Speaker 02

  So the first step, and again, we're going to kind of go a bit meta just to help us have a productive conversation without getting, you know, too deep. And then there are a couple of things we can say at the object level. But so first thing is, if you think that timelines on the order of like late 2026, 2027, whatever are plausible, you ought to be in the business of finding all of the one-way doors that currently exist in the critical infrastructure stack from supply chain to data centers and all that stuff, right? So by one-way doors, what we mean is, these are decision points that we are walking through sometimes without realizing it, where you either like start, so data centers will take 18 months to build or whatever, at least nominally with current rate

• 1:22:54Speaker 02

  of the regulatory environment. So that basically means that, hey, it like, it's time to show your homework. Like, the world's knocking, and the data centers that we're breaking ground on today and designing today are the ones that will have to house a lot of these superintelligence great training runs if they happen in 2027. And so the question then becomes, well, what can we do? What are the sort of like cheapest things that we can do that the market will will support today that bias optionality down the line? And that actually has been a huge fraction of what we have focused on, talking to these special forces operators, talking to the intel community, talking to folks on the hardware side, really deeply understanding what is that set of one-way doors that we're walking through? So we can at the very least not rule out the optionality that

• 1:23:38Speaker 02

  we need down the line. So from a philosophical standpoint, that's kind of it. There's a very small number of people who can actually give you at least what we consider to be the right answers here. And we've seen a lot of interactions between people who think they have the answer, and then they meet another group of people, and you're kind of like, oh, okay, I guess not. To the point where we finally kind of like coalesced around some things where we're like, okay, interesting, like this is now sort of locking in, but that'll evolve too.

• 1:24:02Speaker 03

  I'll maybe give one example of like a one-way door that we can talk about that is in the report. This is what's called a tempest attacks. So these are actually kind of like awesome and wicked James Bond-like things where you can actually figure out what's going on inside of a computer or a system, or like pull data from it by just like watching the electromagnetic emanations from that system. So like in the most extreme version, you're in a room with a computer.

• 1:24:34Speaker 03

  The computer is air gapped. So it's like, it's literally not on the internet. It's just like sitting there on a desk, not connected to anything. If you have like a particular virus in that computer, this is one example of this, that accesses the computer's memory in a specific pattern. You can have a radio receiver that's like a few feet away from it, across the air gap, that's listening for literally the radio emanations from that little like memory access stuff that's going from the CPU to like, to the memory. And you can actually have information transmitted to you across the air ba- air gap by just like, listening for the memory access patterns.

• 1:25:16Speaker 03

  It's insane that it's possible, but that is, that particular attack and detection is public information. It was published about a year ago, but almost surely has been known by intelligence agencies and many kind of the usual suspects for quite a while before. Because it's like, something you can do and it's extremely useful. And so there are standards around how you defend against that. And some of those standards are, like I talked about, you know, you're a few feet away and you can detect. Kind of the, the only way to realistically do that and, and, and block that off at the data center level is you literally have to have like, a few feet of spacing between your compute racks and the walls of your data hall.

• 1:25:58Speaker 03

  Because you can have like a visitor space on the other side of that wall and some like, some dude who looks all innocent like, has like a, an app on their phone or something that's detecting the emanations from your stuff and you've planted malware on, on the system. And they're just like pulling data from you at a, at a bit rate. And they're just like sitting there casually in your visitor center and pulling data from your computers and your GPUs from across the wall. And so the way to block that is you just create that space. And if you don't create that space, if you don't build your data halls with that space in mind, then you're kind of like, oh whoops, like my data halls are just too physically small. And that's a one-way door because the way out of that, like, yeah, you can retrofit,

• 1:26:41Speaker 03

  but it involves like knocking down and redesigning your facility.

• 1:26:45Speaker 02

  And so the data center one-way doors are really interesting because they affect construction today of a thing that you can like see right now. The supply chain one-way doors are in a weird way, they're actually a thornier. So because they involve obviously entire supply chain. So TSMC makes all our chips. TSMC sits on Taiwan. You know, I don't need to tell you that the CCP is very, very fucking interested in what's going on there. And if you were China, I mean, you could just draw the obvious extrapolations. What would you be up to knowing about, you know, it doesn't even have to be 10th. I mean, it's like cyber attacks, personnel. I mean, geez, TSMC famously has lost executives who went on to found SMIC and steal their shit, right?

• 1:27:28Speaker 02

  Like this is a known playbook that has been working for decades. 20 years. Yeah, so there's that. There's also like, even at the level of these mundane, easily overlooked, boring components that go into server systems. So like baseboard management controllers is a great example. They're like 70, 80% market share of VMCs goes to Aspeed, this like other company in Taiwan. And like, I guess I hope that they have their shit together because baseboard management controllers run on like a separate power supply from the GPUs that, you know, hum and the CPUs that hum. And they often have like read access and write access to like firmware and like it's a dream backdoor.

• 1:28:11Speaker 02

  Like it is the soft underbelly of a lot of like HPC AI infrastructure. And so like, there's a million things like this. There's transformers and transformer substations, basically like component wise. Like none of them have no components that are made in China. And we know that the CCP has actually used transformers specifically to plant backdoors in them in American infrastructure. So it's, it's not just like so many of these things, it's tough because you talk to people about it and there is correctly a view that like you can't cover down on all vulnerabilities, all threats. Like you gotta, you gotta pick and choose. The challenge is we're not even highlighting like, or let's say a good fraction of what we're highlighting here isn't even stuff that is like speculative.

• 1:28:55Speaker 02

  It's stuff we know China has literally done to us on our soil. And so it's, um, yeah, it's one of those things where you just have to take, take the reins in some way and.

• 1:29:08Nathan Labenz?

  Maybe just give me a little bit more on the sort of backdoors as they're understood to exist in the like electrical grid and then, you know, I don't, maybe that'll be enough to sort of imply what it means for the data centers, but I'd be interested too in like, you know, in the report, there's like the cooling modules come from China. There's like, you know, we have sort of the sort of crown jewels of, oh, data center might come from Taiwan, but then like a lot of the other pieces that are needed to make it go seem to come from China to the degree that, you know, my kind of takeaway was like, there's really just no way that we're gonna bring that whole supply chain to the United States

• 1:29:50Nathan Labenz?

  in any sort of short-term scenario. But yeah, but maybe let's start with like, what is known, what is understood, you know, what, when you say like China has a gun to our head with respect to the transformers that they've backdoored in our electrical grid, like, what does that mean for me? What, what am I vulnerable to that I may not realize?

• 1:30:09Speaker 03

  Yeah, so this means that they can hold our, they can hold our infrastructure at risk and use that in some sense as a bargaining chip. If things heat up to disrupt us completely and distract us while they are taking hold of their own objectives. So this is, this is one, you know, possible scenario of a Taiwan invasion where there are many, many logistical challenges that the United States faces in defending Taiwan. One of those is just like, we're just gonna like run out of stuff after the first week or two of like intense, high heat combat at those, at those scales. But another thing they can do is like, hey, if it looks like, uh, if it looks like things are not going as well as they

• 1:30:53Speaker 03

  hope for a plan because of American intervention, sure, just like turn off all of our grids after, uh, after, you know, putting some propaganda in our information spaces, basically turn out the lights and like, okay, like let the chaos rain and yeah, see, basically just like throw a wrench in the gears. Why wouldn't they do that?

• 1:31:14Speaker 02

  I think one important ingredient to strategically here is, you know, when you have a capability or a set of capabilities to deploy against an adversary like China or like the United States, whoever you are, you're never going to show your card, like your full set of cards, right? You always go with, what is the, what is the smallest kind of jab that I can throw that has the desired effect that doesn't reveal my whole set of capabilities, right? The intent is always, how can I learn without teaching? And that, that means that you, like, we have genuinely no idea how high the ceiling is on Chinese capabilities, but the floor is pretty goddamn high, right? So, as we, as we think about stack ranking and prioritizing the interventions, which is where I think

• 1:31:56Speaker 02

  your question was headed, like, what, like, what do we actually do here? The, the good news is that the structures you need to build are, you need to build a relatively small number of structures relative to the, like, kind of power demands of the entire United States. We can't go, we can't go into the solution because it is redacted from the document for obvious reasons. There are solutions that we've identified for a number of things, including, well, I, I won't mention, I can't remember if, if we say what the thing is, but in any case, they, they have to do broadly with, yeah, supply chain techniques, but the fact that this is a focused problem does help you. There are ways in which the problem is constrained, geographically, technologically, that, that make it more,

• 1:32:41Speaker 02

  more tractable. But another kind of corollary to this is, you can't live in a world where China holds you for ransom and you don't do the same to them, right? Right now, that is the world we live in. We have, like, very poor ability, visibility to, to see what is going on, on CCP infrastructure. That gets hard, actually. So, Nathan, I think we, we talked a couple of weeks back about, like, why did Google release streaming Delo Co? And that, that whole kind of, like, this gets harder with fucking Delo Co. It gets harder with, you know, intellect, it gets harder with Together AI, all these kind of distributed computing technologies that just make it so hard to track what is even happening, right? So you can hope for today, the big mega cluster, 10 gigawatts of three

• 1:33:24Speaker 02

  ...damned like power that you might detect through satellite imagery. But that's not always going to be the case. And in fact, it seems plausible that it'll become less and less the case on relevant timelines. And so, so the question then becomes, okay, well, how do we establish a reciprocal capability? We need at least that option. Like, i- if China has the ability to hold American critical infrastructure for ransom, the, the fact that we do not have that option in reverse is a giant problem and, and should be a priority to fix.

• 1:33:51Speaker 03

  And we can do, I mean, I mean, yeah, without going into too much detail, our understanding is like there, there is stuff we can do in retaliation, but it's, we're not where we need to be.

• 1:34:06Nathan Labenz?

  And just to make this a little bit more colorful, are we talking about like,

• 1:34:12Nathan Labenz?

  transformer substations in communities across America, like suddenly going like Hezbollah pager and just like exploding? Or like, what exactly is the?

• 1:34:24Speaker 03

  Yeah, I mean, it's like scenarios. Yeah, stuff like that. So the thing is like, there's, uh, this actually comes back to what Jer was saying about, you reveal the, the capability that gives you the effect that you desire and nothing above that. So you've got like 10 levels that you can operate on as a nation state. You're operating on all 10 of them. You want to reveal like the lowest level that does the thing that you want. So for example, there's like little like touches that we see kind of across the United States, where you have things like folks like, uh, this has been reported where people, you know, a couple of folks like Chinese folks are flying drones to just like observe critical infrastructure and things like that.

• 1:35:07Speaker 03

  You certainly can just like, yeah, knock things out with explosives or, you know, literally just like, uh, a short circuit a transformer when it's turned off and, and rewire it into itself. And if it's not being regularly inspected, when that transformer turns on again, it explodes like a bomb. All of that stuff is within the arsenal of things that they can potentially do in certain critical places. Other stuff involves like you own the firmware on that transformer. You also are the supply chain that produced that transformer. So if you want to, and if you're thinking far enough ahead, you can totally install backdoors that make that transformer effectively inoperable and just like bricks it. So you don't need to blow

• 1:35:52Speaker 03

  it up. You don't need to do anything that's subtle. In theory, you could flip a switch and like, okay, this transformer is like just a brick now. And that's just the way it is. And so now it's a, it's a game of like, well, we need to somehow replace like a large fraction of the transformers in random places in the United States under conditions where we are, we, many places in the United States don't have electricity or comms because the transformers are now.

• 1:36:23Nathan Labenz?

  Yeah. And we also don't build many of them, if any, right? So that's a

• 1:36:28Nathan Labenz?

  Yeah.

• 1:36:29Speaker 02

  Means to build a transformer or a transformer substation is really interesting, right? Cause it's like, well, so where do you source your materials?

• 1:36:36Speaker 03

  Yeah.

• 1:36:37Speaker 02

  How far up to that supply chain do I have to go before I have, you know, a manufacturer in China? And it's actually like, quite often not very far. And, and that's part of the challenge, right? Generally speaking, this is, it's a, it's a very high dimensional problem, obviously. Again, data centers are fortunately, they're an insanely high dimensional object. They're a much lower dimensional object than the entirety of the United States. And there are things you can do anyway, to reduce the risk in significant ways. But you can see the geostrategic calculation too here, right? Where it's like, okay, given that this is the case for the United States, if this is true, then we have a larger strategic challenge on our hands, even absent ASI. And it can be awfully tempting to look at

• 1:37:21Speaker 02

  ASI as partly a solution to that problem. So if you have this ability to build, you know, whatever kind of crazy offensive weapon through ASI that you might, there's then this temptation to say, well, if we can just secure this one thing and we can like use this thing to have this effect, then you know, then we'll be able to kind of transcend all these other issues. So, so the kind of the strategic calculus here is super, super loaded and complex, which is anyway, it's a big part of what we can't talk about.

• 1:37:50Nathan Labenz?

  But I mean, the general picture. That I take away from the report is like,

• 1:37:58Nathan Labenz?

  we're moving AI research to some sort of secure facility in Nevada or something, right? Like some remote place and we're building like data centers underground and we're trying to, you know, at whatever cost, right? We don't care about it being economically competitive. We just need to like, figure out ways to make certain critical components domestically. So we can have some sort of secure supply chains. We're like just pouring good money over bad onto that. And we're also like doing serious vetting of our people, maybe like not allowing them to go home. Meaning like they literally have to, you know, sleep on the site and have their calls monitored or whatever.

• 1:38:42Nathan Labenz?

  Maybe that would be a way to not have to like, literally purge all Chinese, you know, nationals or, you know, even just Chinese Americans. If we had that sort of level of surveillance, obviously that's not super pleasant. It starts to sound, by the way, a lot more like China, which is a general trend, I should note, in American governance, you know, just tweeted this morning, like, the, you know, the US government is trying to tell me that I should be afraid of Chinese values. At the same time, it is embodying Chinese values in a more and more egregious way all the time. Like literally disappearing people without due process for political expression. So that's a problem, but.

• 1:39:19Speaker 03

  Well, with, I mean, with a project like this, here, again, we're talking about like, on the assumption that you're doing a national super intelligence project, right? And, and without necessarily talking about like, is it a good idea or not for the, for this purpose? If you're actually doing that, you, you look back to the, the original, the, the actual Manhattan project for nuclear weapons and yeah, I mean, even back then, right? You had this kind of surveillance through the comms channels that they had available and.

• 1:39:46Nathan Labenz?

  Yeah, Feynman famously like made a code system with his wife, right?

• 1:39:50Speaker 03

  That's right. That's right. And in like little protest of that and, and like it's, they were reading your letters and that was like absolutely outrageously unprecedented at the time, like, so they had to, they had to agree to it voluntarily to have their mail read and all this stuff. And people were playing around with the sensors and the code and figuring out the boundaries of the system. But then Klaus Fuchs just went and stole the secrets anyway. So any like it's Yeah, and in preserving the secrets.

• 1:40:17Speaker 02

  Well, a couple of thoughts here, though, too. So on the Klaus Fuchs thing, this is another area where as research gets more and more automated, that that does help, right? In some ways, it's a double edged sword. It's a from a loss of control standpoint. Holy shit. And then from a security standpoint, it's like, oh, okay, I see what you're doing there. The the other side too is so we're not advocating for building these things underground or in any such context. Part of the calculation was, okay, so here's the, you know, here's the list of the full suite of things that you would really want to do to play it tight. Separate question, how do we pragmatically do this? That's actually a question that we've been thinking about, working on, not that we've been thinking about working on, that we've been thinking about and working on for the last

• 1:41:00Speaker 02

  for six months really intensely and and sort of like figuring out, okay, if we actually had to like to do this for real, what is the kind of Pareto optimal frontier look like? And what do those trade offs look like? And I think you actually can get to some pretty satisfactory places, even in the face of the full Chinese and CCP threat here. But the question is always going to be, how do I? Yeah. How do I spend my marginal dollar? And how how do I trade off? Yeah, the need for more compute and whatever else with with, you know, buffer for security?

• 1:41:32Speaker 03

  And to be specific about that, like, okay, how do you actually think about those trade offs, right? You can't cover everything. You can't actually, in the time we have, build a data center in the mountain with like frigging like cooling cooling ducts like sticking out of the like, it's just like, forget it. That's just too much work. So how do you actually think about it? Well, you think about it in the context of like, again, if you were doing such a project, you would need it to be embedded into your national security and counterintelligence apparatus. So you're you're taking this because otherwise like it just doesn't make sense. It's like a standalone thing. You're so, you know, the NSA is like

• 1:42:13Speaker 03

  has a team there, right? Like the other stuff like that. You've got you've got people actively working and collaborating. And one of the things that that means is you've got eyes and you've got attempts at collecting information.

• 1:42:28Speaker 03

  On the adversary's side, so like you're you're collecting information on China's efforts to subvert an attack and exfiltrate from that project. So now your your challenge actually is not defend against everything. Rather, the challenge is like, we want to put enough measures in place to raise the cost for them to do an attack or an exfiltration to the point where we can see that. Because we're forcing them to marshal enough resources to be able to overcome our security bar. Doesn't mean that we have, doesn't mean that we can actually fully defend against everything. At the end of the day, like, they just shoot a cruise missile from offshore and like, blow the bejesus out of us.

• 1:43:13Speaker 03

  And like, nothing we can really do about that, like, at the end of the day. But if they do that, then we can see that cruise missile and we can go, hey, guys, you just shot a cruise missile at us. We're going to retaliate. And so you basically raise the bar to the point where you force enough resources to be marshaled from their side, that that creates a signature we can see incredibly offers a threat of retaliation. This is, this is like how that security posture gets integrated into a counterintelligence motion.

• 1:43:45Speaker 02

  Yeah, the the kind of flip side to that, you know, if you don't have that set up, right? You essentially just, if you're OpenAI or you're XAI or whatever lab is is close to doing it first, what you find is like, oh, this, you know, this training run just, I don't know, it's just not, or the or the inferencing process, whatever, whatever the key stage ends up being, like, it's just not working, like, it's super buggy, there there are a bunch of issues weird. And in the meantime, obviously, like, you're not aware, but relevant IP has been exfiltrated, there are more scale training runs happening more efficiently in China or wherever, and and your adversary is developing that capability. And you have no, there's no consequence. And and this is one common theme that we just like, I guess, just to kind of

• 1:44:28Speaker 02

  add some color, this is maybe backing up a little bit, but on the US-China side. What's been missing, at least until now, what's been missing throughout like the Biden years, this has been a complaint that we've heard and before too, in many administrations, has been consequence. So, you know, America's adversaries, including China, have been able to conduct operations in the United States that do a lot of damage without consequence. And that's because there's been a lot of anxiety about like, oh, well, if we respond, then, you know, right away, the brain goes to nuclear escalation. The problem is you cannot do business with adversaries with, when that's your mindset. You just can't. I mean, what you end up with is people who are going to take advantage of every little,

• 1:45:11Speaker 02

  I mean, it's the whole, I'm not touching the devil thing. Like they'll get closer and closer to your face. They will eventually start to do things which we know they have actually been doing to American critical infrastructure and people. And I mean, you think about the, the like Chinese police stations operating within US borders, basically with impunity. What the hell is that? Right? Like, like if we acknowledged that these things were real, those are things that border on acts of war. And the fact that we have not offered consequence for those moves means that we are teaching the adversary to continue to push in that direction. We're now about to pay the cost when it comes to AI. If there's no consequence for what is done to American interests domestically or abroad, if people are able to act with impunity, all of a sudden, I mean, yeah, I get to for free try to take

• 1:45:56Speaker 02

  out your, your critical infrastructure. Why would I even, why would I think twice? Right? It's a free shot. Whereas we don't get the same reciprocally.

• 1:46:03Speaker 03

  People, people don't realize this, but like stability between powers today is actually is like, is not maintained through actual defense. It's maintained through threat of consequence and threat of credible retaliation. It's like, it's this, it's actually like kind of fucked up that the world is this way, but stability between like powers and major states is maintained the same way that like gangs in Chicago maintain each other's territory, which is like, I, I know that I can't actually stop you from like busting a cap in, in the ass of one of my boys. But if you kill one of my boys, like I'm gonna fucking kill one of your boys right back.

• 1:46:45Speaker 02

  That's the first time I've ever heard them say bust a cap. Okay, cool.

• 1:46:48Speaker 03

  Yeah, yeah. That's, that's a first. That's an exclusive for, for the cognitive revolution. But that's how stability is maintained in truth. It's unattractive to think about it, but it, it is, it is. It's it's the bedrock. It's the air or in the air we breathe in the world today as wild and scary as that is.

• 1:47:08Nathan Labenz?

  So.

• 1:47:12Nathan Labenz?

  I mean, I will, I guess I'll refer people to the report for more depth and detail on the many supply chain vulnerabilities, you know, the difficulty in securing a libertine research staff in the, you know, remote locations to which they might be absconded. And on top of that, we haven't even got into the measures that you recommend for maintaining control over the AI itself, which is definitely a, you know, a new type of capability that the, I mean, for everybody, but, you know, including the US government if it wants to develop and deploy those to make sure that it's not inadvertently, you know, unleashing Armageddon on everybody. So we got a lot of problems.

• 1:47:56Nathan Labenz?

  I know you're kind of trying to keep this somewhat neutral stance on like, we're not saying do or don't do, but we're kind of just trying to tell you what would be involved. Would you be? I guess, you know, to put my cards on the table, I read this and I kind of came away feeling like we shouldn't, you know, like it reads to me like a warning more than a manual that I'm excited to to sign up.

• 1:48:17Speaker 03

  Actually, this is so funny because we've had so many conversations with people who are like, dude, why are you telling people to do a super intelligence project? It it's um, it it. It's a bit of a Rorschach test, right? Like people will see what they want to see in it. And to an extent, that's fine. I I think the thing that we took from it, like after, you know, all said and done and kind of starting from like, you know, we don't know if it's a good idea or not, then and then like having done it and looking back and be like, well, what is our conclusion? Like, you know, our privately ourselves? And big part of our conclusion is. Most of the measures we recommend should be implemented, whether we're doing some kind of big national thing or not, because if they're not implemented, then you get that

• 1:49:02Speaker 03

  situation that is exactly what Jared described, which is like, yeah, China just like steals your weights at the 11th hour, they stuxnet you so that like, you don't even realize it's happening, you just feel like your training run is like, ah, it's like, it's, it's, everything's proceeding slower, there's so many more bugs and blah, blah, blah. And all they're doing is like, they're, they're throwing sand into your gears and they're, they're working and improving stuff on their side. And you don't even realize that's what's going on. Um, one of the things that like, we learn from talking to these intelligence folks is like, non-state actors, like proxies for nation-states, uh, are, are, sorry, are proxies for nation-states,

• 1:49:44Speaker 03

  and have many of the techniques and tools and procedures, in fact, almost all, that the nation-states themselves have. So the distinction between like, China's doing this versus like, some group that China is supporting, that's a non-state group is doing this, is actually not that sharp in terms of capability. Like, they're trained, funded, and supported by the Chinese or by whatever adversary. And so you're facing down nation-state capabilities, even if your proximate adversary is not actually a nation-state. And what that means is like, the bar for security just has to be high all around to prevent this from happening,

• 1:50:27Speaker 03

  whether we have a national superintelligence project, or whether we have like, five random superintelligence projects that are all kind of dipsy-doodling from their own size and angles. This is just like, the vulnerability surface.

• 1:50:40Speaker 02

  That exists. And so that's true, but it is. Yeah. And so, because when it says like a national superintelligence project, right? Like essentially like a US government backed superintelligence project, which then there's like backed and led and like that's where we start getting into like, well, honestly, like it doesn't matter. Like at the end of the day, what you care about is, how is the rubber going to meet the road? Are they or are they not going to be able to just extract critical IP, Stuxnet your stuff? Or anyway, do any number of other things? Those questions all have answers that route through a lot of the same channels. There's a question of task organization, if the US government takes it on, but that's why we scoped that out.

• 1:51:20Speaker 02

  Right? So a lot of this really is the story of if you're going to build superintelligence and treat it as the WMD capability that, you know, Sam Altman tells us it is or will be with no apologies. Then you ought to be doing something that looks fairly different from Stargate. Like what it looks like to actually take that seriously, to take your responsibility seriously in building this tech looks very different from let's announce to the world that we're doing a $500 billion project and everybody knows exactly where the facilities are. And oh, by the way, our lab contains like X percent of like Chinese nationals. And by the way, we've got all these issues with people leaking like critical IP from Slack openly. Like we've got, uh, one of the things the report uncovered was we had a, an insider from OpenAI tell us,

• 1:52:05Speaker 02

  hey, I like just digging around, I found these two really critical cyber vulnerabilities that would have allowed me to access and then extract weights for like a critical model that the lab was hosting. Like this is just not what you do if you are a serious actor and you take your commitment seriously. And so that's a big issue. It's not that OpenAI is like, like a bad actor in the kind of zoomed out sense. It's that in part, they're just victims to the, the racing dynamics that are playing out right now. And so how, yeah, how do you get your arms around that? Well, step one presumably is having a sense of what the vulnerabilities are that you need to cover down on. And then you can price those in, you can start to make a list of them, prioritize by cost,

• 1:52:47Speaker 02

  prioritize by probability that they'll be exploited and work your way down. That's, you know, all you can ever really do in this space.

• 1:52:59Nathan Labenz?

  I don't love it, and I know you guys don't either. Let me just try in the the few closing minutes that we have to to just get a couple reactions on on other things. One is like China's open sourcing all their stuff, you know, you know, one sort of Pollyanna-ish view is like, they don't appear to be trying to race us, they're giving us all their best shit for free. How do you interpret their open sourcing strategy? Do you expect it to continue?

• 1:53:23Speaker 02

  Oh, having an open sourcing strategy is one question that never gets asked. The answer surely is no. They have an they have an open sourcing strategy, but it is not a universal open sourcing strategy. The one of the challenges is we do not know what's going on on those Chinese servers, right? Like, we can't answer that question. What we see is like, so we actually heard from a like a former OpenAI researcher that he was having a conversation about, like, how do we know that China hasn't stolen any of our shit? And the response was literally at the time, well, we haven't seen any comparable open source models come out of China. So that basically tells us they haven't stolen us. Like, dude. models, models with comparable capabilities. Yeah, right. Sorry, things that are legible. Yeah. And so you're just not going to get that intelligence. The flip side is anything that comes out of DeepSeek from here on out.

• 1:54:08Speaker 02

  In fact, anything that comes out of Huawei or anywhere else, Tencent, you name it, is coming out with the imprimatur of the CCP. There's 0% chance that those models are getting open sourced without people high up at the CCP going like, cool, at a high level, that makes sense. They may not approve every single individual one, but the general strategy behind it is CCP approved. You can make of that what you want. If you are the CCP, you'd probably have an interest in, you know, flooding the West with language models that are going to not talk about Chin and Min at a minimum. But as we move into more and more agentic systems, boy, does it look interesting to start to buy the public's confidence in your agentic models when there are interesting backdoors that could be baked in. Anyway, interesting behaviors you might want to exploit.

• 1:54:51Speaker 03

  One other thought there is like, we suspect that prior to the end of last year, DeepSeek was not that integrated into the CCP apparatus, though it is super integrated now. The reason being a number of comments by the founder of DeepSeek around like, "Man, American export controls really work against us. Geez, they're really biting", that he made like back in July on a random podcast, which just like utterly undermined the CCP's entire like propaganda around like, "Yeah, American export controls like just don't work around ships, so you might as well give them up. You might as well just like, stop, right? Please?" And they've been, we don't have time to get into this, but they've been putting

• 1:55:36Speaker 03

  like propaganda set pieces around that in place for years at massive cost and effort. And this like random dude who now has been catapulted into fame, just like absolutely gutted that entire effort at great cost to them.

• 1:55:48Speaker 02

  And there is also the economic argument too. And these are just like public arguments people are throwing around. And I'm just offering them as explanations for why would the CCP be okay with this, which is the ultimate question that we really just need to answer. But the other one is, you know, the economy of flood, flood the market with cheap LLMs. You've seen what that's done to Meta. I mean, like, Jesus, they're basically like, they're now having to gut their...

• 1:56:09Nathan Labenz?

  They're flooding the market with cheap LLMs.

• 1:56:10Speaker 02

  Yeah, exactly. Yeah, yeah. So.

• 1:56:14Nathan Labenz?

  Okay, I don't know if we have time for two more, but I would ask two more if we have time.

• 1:56:19Speaker 03

  One.

• 1:56:21Nathan Labenz?

  All right, maybe I'll see if I can combine them into one. How can I do that? Maybe I'll just put them, I'll just ask them both and you can answer it as you will.

• 1:56:29Nathan Labenz?

  One question is, do we even have a set of prerequisites for some sort of grand bargain? You know, if we were gonna say, okay, we're gonna come to the table. We know we don't have great trust, but is there some sort of technological checklist or trust, you know, and or trust building exercise checklist that we could go down to say, okay, if we had these things, then we could maybe enter into some sort of sustainable grand bargain? And then part two, another kind of angle on it is just like, I know you guys read Dan Hendrix et al's Mame Theory.

• 1:57:15Nathan Labenz?

  And I wonder if you have, you know, any hope for either an emergent equilibrium that sort of keeps things under control, or perhaps a more engineered, you know, maybe as part of a grand bargain, some sort of semi-engineered equilibrium that could keep things stable between great powers, even if, you know, trust continues to be a scarce resource.

• 1:57:39Speaker 02

  The grand bargain side, like the prerequisite, so this starts to touch pretty closely on the whole FlexHeg program and, you know, secure governable chips. Can you make, can you make a tamper-proof enclosure for your chips? Can you make at least a tamper-detecting enclosure that requires, you know, inspections? Can you then have some sort of like, you know, like governance chip that's sitting on there too, to verify inputs and outputs and all that stuff? I love that research agenda. I think it's a great research program. The problem with it is timelines. There's just like, it's a really hard obstacle to overcome that fabbing is slow and chip design is slow and integration is slow and scaling is slow and debugging is slow. So, love that program. Hope that it gets really significantly accelerated

• 1:58:24Speaker 02

  by AI. And this is something that I think should be a priority in the context of the superintelligence project. You would want to couple the, the kind of intelligence engine to the design process for, for these, these chips. You'd want to couple it to, you know, advising on what to kind of do next and fab next. But, but yeah, that's, that's kind of, I think a prerequisite is we need to have visibility. Trust but verify has got to be an option. Until it is, we need to be treating China as an intelligent adversary and a committed one. Maybe I'll, I'll leave the, Ed, to do the cleanup on that. And if you have thoughts on the main thing you want to throw out, Ed, too.

• 1:58:58Speaker 03

  Yeah, generally agree. I think we need to be pursuing every avenue. And if, if there is any chance, even though it seems quite remote, of some kind of verification around collaboration, yeah, like, let's throw a few billion bucks into that. Because like, absolutely, if that turns out to be like a dark horse, like win, totally, yeah, that, that's, that's worth the investment. Timelines are tough. Another tough aspect is like, The folks that we talked to are not optimistic about like, can you actually build a tamper detecting enclosure like at all? That if it's like subject to, you know, uh, sustained pressure under full control

• 1:59:41Speaker 03

  of, of an adversary that can resist even at scale. So like, you know, you'd have to, you'd have to like basically bust through a million of these to have like a meaningful amount of, of chip capacity is like less optimistic. And then on, on the main side, it's actually a shame. We didn't get to discuss main more because it's actually a really, really interesting piece of work. And I think that one of the things that it did really, really well is that it was the first, the first place where really this kind of like approach of, of like, no, we actually need to be reaching out and touching the adversary. We need to be actively doing like counter, counterintelligence and capability degradation stuff like soon really came out and that's a critical pillar.

• 2:00:26Speaker 03

  And you can kind of see that, like how the logic kind of forces you into it, right? Because if, if you want to like have, have, have an aligned AGI or superintelligence, you need the margin over the second place adversary or competitor to actually do that development of that, that alignment technology. In order to get that margin, you can either accelerate yourself or try to decelerate the competitor. Accelerating yourself shortens timelines and creates like additional risk. And so the logic kind of forces you into, well, you need to do capability degradation for the adversary and main did a great job of framing that and, and pointing that out.

• 2:01:08Speaker 03

  One of the issues with it, where, where we, when we talked to some of the, the intelligence folks that, that we are connected with. Is it comes back to like, you can't actually put a bar of security that's like, nation states can go above this bar and can like hold your stuff at risk, but non nation states can't. And so, and so you can, you can defend against one, but not the other. As we've seen, those sets of capabilities are actually not as distinct as many people think. And this idea that you can fine tune security levels and, and do this kind of like scalpel stuff is not viewed very encouragingly in those spaces. Doesn't mean it's not a really important

• 2:01:50Speaker 03

  part of the truth, and it is. But there's, there's some aspects missing that like, I think, I mean, I think it would be really productive to discuss with Dan about more deeply, even on, on your podcast at some point in the future.

• 2:02:04Nathan Labenz?

  I love it. We'll see if we can make that happen. Any other closing thoughts or, you know, assignments you want to give to the audience?

• 2:02:13Speaker 03

  Um, yeah. Well, I would say none that comes to mind. Like, read the report. If you have thoughts or feedback, we would love it. Uh, we, we think of it as a living document, even though it's a point in time. It's not going to be perfect and things change. Uh, but yeah, I mean, really eager to get anybody's views, thoughts on it. We want to aggregate as much as possible. We are maintaining a list of, uh, like, yeah, data center, security, uh, techniques, uh, that we think would make the difference, uh, and that are one way doors especially. So to the extent that you have thoughts on, on that side of things, always eager to hear from anybody in the, you know, intel community, special ops community and all that. Just the, any, any relevant community, right? Like AI, security, AI.

• 2:02:59Speaker 03

  Flex HAG, uh, secure government shifts, uh, you know, like alignment people. Yeah. Yeah.

• 2:03:05Nathan Labenz?

  Cool. Well, then they'll, uh, know where to find you. For now, Jeremy and Edouard Harris, founders of Gladstone AI, thank you both for being part of the cognitive revolution.

• 2:03:16Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.