Helen Toner: OpenAI Reflections, Adaptation Buffers, and AI in Warfare

In this thought-provoking conversation, Helen Toner, Director of Strategy at the Center for Security and Emerging Technology (CSET), discusses her perspectives on AI development, regulation, and international competition. Toner shares insights from her time on OpenAI's board, clarifying that reports about a "Q*" breakthrough influencing the board's decision were "totally false." She introduces her concept of "adaptation buffers"—critical windows of time between when AI capabilities are first demonstrated and when they become widely accessible—arguing that society needs this time to build resilience against potential misuse. Toner also explores the challenges of implementing effective AI governance, military applications of AI decision support systems, and the complex dynamics of US-China relations in the emerging AI landscape. Helen Toner's appearance on the TED AI show: https://www.ted.com/talks/the_ted_ai_show_what_really_went_down_at_openai_and_the_future_of_regulation_w_helen_toner Helen Toner's substack : https://helentoner.substack.com/ Additional recommended reads: https://helentoner.substack.com/p/nonproliferation-is-the-wrong-approach https://cset.georgetown.edu/publication/ai-for-military-decision-making/ https://www.lawfaremedia.org/article/ai-regulation-s-champions-can-seize-common-ground-or-be-swept-aside https://www.economist.com/by-invitation/2024/05/26/ai-firms-mustnt-govern-themselves-say-ex-members-of-openais-board SPONSORS: Oracle Cloud Infrastructure (OCI): Oracle Cloud Infrastructure offers next-generation cloud solutions that cut costs and boost performance. With OCI, you can run AI projects and applications faster and more securely for less. New U. S. customers can save 50% on compute, 70% on storage, and 80% on networking by switching to OCI before May 31, 2024. See if you qualify at https://oracle.com/cognitive Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive NetSuite: Over 41,000 businesses trust NetSuite by Oracle, the #1 cloud ERP, to future-proof their operations. With a unified platform for accounting, financial management, inventory, and HR, NetSuite provides real-time insights and forecasting to help you make quick, informed decisions. Whether you're earning millions or hundreds of millions, NetSuite empowers you to tackle challenges and seize opportunities. Download the free CFO's guide to AI and machine learning at https://netsuite.com/cognitive PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) Teaser (06:04) Introduction and Background (09:32) AI Timeline Perspectives (12:26) OpenAI Board Experience (17:04) OpenAI Culture Discussion (Part 1) (19:20) Sponsors: Oracle Cloud Infrastructure (OCI) | Shopify (20:33) Sponsors: Oracle Cloud Infrastructure (OCI) | Shopify (22:30) OpenAI Culture Discussion (Part 2) (24:03) OpenAI's Contradictory Messaging (26:15) Whistleblowing in AI (31:30) AI Transparency Requirements (Part 1) (36:30) Sponsors: NetSuite (37:58) AI Transparency Requirements (Part 2) (38:38) Adaptation Buffers Concept (44:06) Technical vs. Societal Solutions (52:13) AI Regulation Approaches (57:20) Iterative Deployment Challenges (01:05:50) US-China AI Competition (01:14:54) Military Decision Support Systems (01:20:57) Future AI Military Equilibrium (01:27:39) Outro

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, I'm speaking with Helen Toner, Director of Strategy and Foundational Research Grants at CSSET, the Center for Security and Emerging Technology, and author of a new Substack called Rising Tide. Helen is best known to the general public for her role as an OpenAI board member responsible for temporarily firing Sam Altman in late 2023. But she's been feeling the AGI, or at least the need for society to invest in preparation for the possibility of transformative AI since way back in 2016, when she started working on AI policy full-time. That's a full five years before joining the OpenAI board in 2021, when, it's worth noting, OpenAI had already launched GPT-3 as an API product,

• 0:43Nathan Labenz?

  already taken $1 billion in investment from Microsoft, and was increasingly recognized by those in the know as a leader in the generative AI wave. Certainly by that time, OpenAI had plenty of access to super-talented candidates for its board. With that context in mind, and again, remembering that her blog is called Rising Tide, despite what you might have heard elsewhere, it probably should not surprise you to learn that Helen is definitely not an AI D-cell or even especially hawkish on most AI safety issues. On the contrary, she argues in early posts on her Substack that nonproliferation is the wrong approach to AI misuse, and instead promotes the concept of adaptation buffers, the notion that society broadly has a critical window of opportunity

• 1:27Nathan Labenz?

  to adapt to new AI capabilities between the time when they're first demonstrated, typically at high cost in terms of both R&D and compute, and when they later become widely accessible, typically at much lower cost, as we've recently seen, with companies like DeepSeek dropping the cost of frontier reasoning capabilities. While her focus today is on other things, I couldn't resist asking Helen some OpenAI-related questions, and I appreciate her willingness to engage despite having addressed these issues in multiple forums already, including especially an episode of the TED AI Show, which we'll link to in the show notes. The only truly new detail that you'll hear in this conversation is her assertion that media reports suggesting that some sort of Q-Star breakthrough in reasoning had led

• 2:10Nathan Labenz?

  to the board's decision were "totally false." But nevertheless, I think it's important that Helen and other former OpenAI team members continue to speak candidly about their experiences with the company and its leadership. As Helen notes in another of her first blog posts, "Everyone's timelines are dramatically shorter than they used to be." What passes for long timelines in AI circles today would have been quite short not many years ago. And given this new short timelines consensus, scenarios like former OpenAI researcher Daniel Cucatello in team's recent AI 2027 reflect not just one of the shorter timeline forecasts, but if I'm reading between the lines effectively, a warning about how OpenAI leadership might

• 2:52Nathan Labenz?

  fail to act responsibly around the time of AGI by abandoning its principal iterative deployment, keeping the best models for its own internal use, plus maybe that of the US government, and aiming for a sort of AI takeoff via the automation of AI research. That's a warning, by the way, that's become a bit more credible this week with the news that OpenAI has indeed announced that GPT-4.5 will be deprecated from the API. All that's enough for me to feel strongly that it's important for Helen to use appearances like this to continue to remind Washington decision-makers that OpenAI's CEO was not consistently candid with its board, and also for me to applaud moves like the Amicus Brief recently filed to the Elon Musk versus OpenAI lawsuit by 12 former OpenAI team members

• 3:36Nathan Labenz?

  who argue that nonprofit promises were central to OpenAI's early hiring success and that the nonprofit should not cede control of the company at any price, a development that happened after I recorded with Helen and on which I hope to do a full episode soon. Of course, the stakes are only rising from here. With OpenAI and other AI companies seeking Pentagon contracts and special legal protections, Helen's latest research out of CSET with Rhodes Scholar and former Navy Aegis operator Amelia Probosco on AI for military decision-making is super important and is a sober attempt to map out how AI systems have been and are likely to be used, and how that may diverge from how they actually should be used, given their current limitations.

• 4:19Nathan Labenz?

  Among many other interesting details, I was amazed to learn that some nations, including most prominently Russia, currently have published military doctrines about AI, which seem to be fundamentally out of touch with current AI systems' lack of reliability and total lack of adversarial robustness. This, too, is something that Washington decision-makers probably can't be reminded of often enough as they seek to develop autonomous killer robots. As always, if you find any value in the show, we'd appreciate it if you'd take a moment to share it with friends, write a review on Apple Podcasts or Spotify, or just drop us a comment on YouTube. Of course, we welcome your feedback, too, as regular listeners will know. While I believe that there's probably some non-trivial

• 5:02Nathan Labenz?

  and irreducible risk associated with developing advanced AI at all, it's my sense that much of the extreme AI risk we face today, in fact, exists because key decision-makers, under intense and growing pressure, seem fairly likely to make some very bad mistakes. If this show can do anything to contribute to a positive future, I hope that it can help people start thinking about those critical but avoidable failure modes sooner and better, so that we can minimize the extreme downside risk and get to live in that age of AI-provided abundance that we've been promised. If you think I can be doing a better job, I encourage you to reach out, either via our website, cognitiverevolution.ai, or by DMing me on

• 5:43Nathan Labenz?

  your favorite social network. With that, I hope you enjoy this conversation, looking back on OpenAI and looking ahead to adaptation buffers and military use cases, all amidst shorter and shorter timelines to AGI, with Helen Toner from the Center for Security and Emerging Technology and author of the new blog, Rising Tide. Helen Toner, Director of Strategy and Foundational Research Grants at CSSET, the Center for Security and Emerging Technology, and author of a new Substack, Rising Tide. Welcome to the Cognitive Revolution. Thanks. Great to be here. Yeah, I'm excited for this conversation. We've got a lot of ground to cover. I think, you know, we all have crosses to bear in this life and one of yours is you're going to go on and do a ton of things in the AI space.

• 6:28Nathan Labenz?

  And yet, people are always going to come back and ask you questions about your tenure on the board of OpenAI. And of course, you know, everybody is somewhat familiar with how that ended. So I'm not going to be an exception to that entirely, but I do want to make sure we have time for a bunch of different things. To set the stage, one question I actually don't know the answer to at all, and I'm really kind of curious about is, how did you get involved with OpenAI in the first place? I mean, this goes back years to a time when like, there was no powerful AI. Most people would, you know, dismiss the notion as fanciful. And, you know, very few people were taking the whole topic seriously in any real way. But you obviously were. So maybe just kind of share like your backstory with respect to AI?

• 7:10Nathan Labenz?

  And, you know, some of the enthusiasm that you must have had to get into that position in the first place.

• 7:14Speaker 01

  Yeah, absolutely. I mean, so I joined the board in 2021, but I had been familiar with the company and with many of the folks working there since they were founded. So they were set up in, you know, San Francisco in kind of 2015, 2016 kind of timeframe. And at that point, I was working in San Francisco and that was right around when I was starting to work on AI issues. So it's really interesting reflecting at the time, it felt like kind of being behind the game. Cause this was, you know, 2012, you had Alex Nat, the deep learning revolution was really in full swing by 2015, 2016. And so by the time I kind of came around to the view that, okay, this is going to be a really big deal. There's a lot of work to be done here. I want to make this, you know, really focus on my work, AI and policy and national security.

• 7:58Speaker 01

  Yeah, at the time, it felt like that was sort of coming late to the party. And it's been fun to see since then, kind of a couple more waves, you know, I think around sort of. Twenty eighteen, twenty nineteen or so, I want to say, people started paying a little more attention and then obviously chat GPT in twenty twenty two, there was this huge new burst of of interest. So looking back, I no longer feel like I was as late to the party as I I kind of felt at the time. But yeah, I think I think it's also easy to underestimate how how how weird and how kind of against the grain it was for them to found a company to build AGI at the time. That was really not the kind of thing that you talked about in polite society, including in polite machine learning society. It was really Google DeepMind or sorry, just DeepMind at the time was sort of the only game in town of sort of

• 8:43Speaker 01

  serious researchers who were talking about AGI. And so I think that was also, when I look back on why I was invited to join the board, I think certainly part of it was being in the AI policy space, having a few years of experience at that when not many people did, having spent time in China, having, you know, that sort of China expertise and national security expertise, which I think was valuable for the board. I think it was also having taken the idea of AGI and taken their mission seriously for multiple years by the time I joined the board was really unusual. And it's sort of funny to look back on that now in twenty twenty five, because obviously AGI is on everyone's lips, OpenAI is such a famous company, that the situation looks kind of different. But I think for the community was really small, the set of people who had been thinking about these

• 9:23Speaker 01

  topics and who had actually informed perspectives was was really small. And so it was super interesting to get to be sort of familiar with the company from the very early stages.

• 9:33Nathan Labenz?

  Were you always a relatively short timelines person? One of the the blog posts that you shared the draft with me of just, I think rightly just kind of reminds everybody that like, even, you know, what people are now passing off as long timelines are actually quite short.

• 9:51Speaker 01

  Yeah.

• 9:51Nathan Labenz?

  But where were you, you know, five years ago in terms of your expectations?

• 9:54Speaker 01

  Yeah, no, I I wasn't. I still don't know if I am. The standards have changed so much. And yeah, hopefully the post you're talking about, that the current draft title is long timelines to advance AI have gotten crazy short. And hopefully by the time this comes out, it's published and hopefully the Substack is is launched and everyone who listens to the this episode will subscribe. But yeah, when I got into the space, I think so when I got into the space, I think I counted as short timelines for the time, which was As I described in the post, like this seems, it seems plausible. It seems likely enough to be worth preparing for, that we build very advanced systems in the next couple of decades, you know, in the next, in our lifetime. This seems like a potential development. And if it happens, it would need a ton of societal preparation, almost no one is thinking about it. So that would be, you know, worthwhile to spend time on. I think that did describe my view when I got into the space.

• 10:39Speaker 01

  Nowadays, I don't really identify as having short timelines because that means like expecting superintelligence before, you know, the 2020s are out or something like that. And I feel, I feel much more uncertain about that. I still tend to fall back to this view of, look, I think this is all likely enough that it warrants quite a lot of thought, quite a lot of preparation, which is different than, you know, I think it's very likely to happen or very likely to happen, you know, in the next five years, the next three years. So yeah, I guess the, the question is, you know, depending on what your standards are for short timelines, maybe yes, maybe no.

• 11:07Nathan Labenz?

  Yeah, I used to make a very similar argument to people when the whole notion of like powerful AI was fanciful and certainly any notion of like safety concerns related to AI was doubly fanciful. I used to just say, you know, we have a small number of people that like scan space to try to find asteroids so that we don't get taken out like the dinosaurs did. And that seems really good. And this, you know, kind of seems like another thing. And now it definitely feels like there is an asteroid and it's like coming at us and we don't know if it's good or bad, but it's definitely going to be both.

• 11:39Speaker 01

  Yeah, I mean, I never, I never, to me, the sort of underlying like internal motivation to kind of work on this space was, was related to the way that I think if you look across the scope of history, huge new technologies tend to really change what society looks like for better or for worse, often for better and for worse. And so it was really coming to believe in sort of the early, early to mid 2010s, okay, this looks like we're going to go through one of these transformations probably in my lifetime. And that's going to be a really huge deal. And if I'm interested in trying to, you know, leave the world a better place than, than when I found it, to the extent that I can, then maybe this is an area to go work in and shape. And so to me, it was never like, oh, this is definitely going to kill us. And so we have to get into the space to prevent it from killing us. It was much more this, this broader argument about seems like pretty likely we're going to go through this massive transformation.

• 12:23Speaker 01

  Can I get into a line of work that, that can help contribute to that going better?

• 12:28Nathan Labenz?

  That mindset point is really interesting and I want to ask you about the sort of mind, prevailing mindsets at OpenAI from your perspective. Before getting into that, I know you've, you know, given a couple of different interviews about this and spoken about different aspects of it and probably are tired of it understandably. What kind of governs what you can and can't say at this point? I mean, we've all seen also the like, you know, non-disparagement clauses that were then nullified and I don't think you've ever had any like equity in the company.

• 12:56Speaker 01

  That's right.

• 12:57Nathan Labenz?

  Maybe you did, but I don't think so, right? So, yeah, like, how much is sort of external constraint on what you can say and how much is sort of just, you know, you deciding like how much you really want to talk about this?

• 13:09Speaker 01

  Yeah, it's two big factors at this point, or maybe three, depending how you count. As a board member, I'm under ongoing confidentiality obligations that don't apply, for example, to former employees. So just in terms of like conversations on the board, topics that we're discussing, you know, I want to respect those obligations, I take them very seriously. And then there's also just ongoing legal processes where I could be, you know, my statements could be compared against each other for consistency and minor discrepancies could cause problems. I might need to, you know, testify under oath or otherwise, you know, say things. And then I think also there's, you know, there's just a range of other stuff. So confidentiality, conversations that I've had in confidence with people. A lot of this stuff as well, it's like so, there's just so many details and so much, you would need to go back and explain so much context and bring in like other people

• 13:54Speaker 01

  who really don't need to be dragged into this and like the payoff wouldn't even be that great because none of it, you know, I gave an interview, the best interview I've, I've been able to give on this, the most detail I've been able to go into was on the TED AI show last year. If folks are interested, they should go listen to that. And, you know, it's not that there's like hidden secrets that are more shocking than what was, what was there. There's just a lot more kind of, I want to say almost like boring detail, but that brings in a lot of stuff that's maybe confidential or maybe involves other people who don't need to be dragged in. It's sort, so it's sort of like, I don't, I don't think the payoff is there. And so certainly if people are like, oh man, there's still this big, deep, dark secret that Helen still just hasn't spat out, that's not the case. Yeah, there are still reasons that I, that, you know, that I just talked about, that I am not just sharing everything totally publicly, but I think even if I were able

• 14:39Speaker 01

  to, it wouldn't change the overall picture in any, in some kind of dramatic way.

• 14:43Nathan Labenz?

  Yeah, context is that which is scarce, as you say.

• 14:46Speaker 01

  And actually, maybe one, this is a good point to share one thing, you know, one example of something that I didn't comment on because I thought it was, I wanted to respect my obligations to the company around confidentiality was this rumor at the time about Q Star and Q Star contributing to the board's decision, which is totally false. So I didn't haven't really commented on it because I didn't want to get out ahead of OpenAI's sort of reasoning work and the, what has now been released as one or three, the board was aware that that research was underway, but we never, we never got some letter about a breakthrough. We didn't make our decision based on a letter from employees like that. That whole Reuters story was totally false. So that's one example of something that is now slightly easier to talk about because the underlying sort of confidential information around that line of research is now, you know, now out in the open.

• 15:32Nathan Labenz?

  Yeah, okay. So about the, the mindset or the sort of motivations. I mean, you said, you know, wanting to leave the world a better place, you know, was part of what motivated you to get involved and just kind of recognizing the stakes, feeling like this is sort of a high leverage activity. That seems to me to be a big part of how I understand what I think is kind of motivating people at OpenAI in general. And, you know, in a way that's like, good, right? I mean, everybody should want to make a positive difference. But I do sometimes worry that it can cross over into a sort of main character mindset or kind of a hero mentality. And especially as I like,

• 16:14Nathan Labenz?

  hear more and more things about sort of guru style, you know, coaching going on and, you know, practice of like detachment and, you know, sort of a elite performance mindset. I'm not sure if it's like maybe gone too far. And I sort of, you know, part of me is like, maybe we should want some amount of attachment among the people that are developing, developing the potential like superhuman AI systems. So, so much of this is like hearsay, you know, I don't even really know how pervasive some of these ideas are, but there are definitely like. Quite a few data points at this point. How would you, would you say that's like a prevailing sense at the company that there's this like heroic, uh, world-changing quest that they're on?

• 16:59Nathan Labenz?

  Or would you put that as a more like minority position that's just occasionally popping up?

• 17:05Speaker 01

  I mean, I think, I think board members are generally not in the best position to talk about the culture of the company because they're not kind of immersed in the employees the way that, that a team member would be. So I honestly don't know that I have perspective on that, that, that you wouldn't, you know, these perspective I do have comes from knowing people who work there. Um, and I know you know people who work there as well. So I don't really like, I don't feel like I have more to add necessarily.

• 17:25Nathan Labenz?

  Okay. Fair. How do I get at that? I mean, it seems, it does seem like a very important question. Like I, I do have this sense that sort of detachment in frontier AI development seems somehow wrong. Like it's, it feels like we're taking, you know, to put it in machine learning terms, feels like we're taking something that was learned like out of domain. You know, we, this sort of mindset that I think I'm seeing is kind of like what they tell NBA three-point shooters to, to adopt, right? Just like, keep shooting and, you know, don't worry, but, you know, trust the process over the outcome and, you know, case in a way, sort of.

• 18:07Nathan Labenz?

  And I, I don't, don't know that that generalizes, you know, super well.

• 18:11Speaker 01

  Yeah. I mean, then.

• 18:12Nathan Labenz?

  These high stakes technology environments.

• 18:14Speaker 01

  Yeah. I mean, I think, I think the other, the other version of it generalizing would be just this separation that we sort of implicitly have in society more broadly between kind of the people building new technologies and doing scientific work and the people who are figuring out how to apply them and how to regulate them. And, and I, I think it's generally pretty, you know, generally pretty reasonable for someone who's figuring out how to make, you know, an airplane wing be shaped slightly more efficiently to not be thinking about like, how should the FAI regulate this or how should airline seats be priced or, you know, things like that. I think often it does make sense to kind of decouple those more technical and more sort of societal questions. And so to me, that's more that out of distribution is, this might be a technology where the, if the technical progress outpaces society's ability to adapt, then the people who have just been doing that sort of decoupled technical work might end up having these huge societal

• 18:59Speaker 01

  consequences that sort of, you know, only they or almost only they were able to, to affect or to prevent. So to me, that, that's the way that I would think about kind of the OOD, um, uh, element here.

• 19:12Nathan Labenz?

  Yeah. That's interesting.

• 19:15Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 19:20Nathan Labenz?

  Another thing that I mean, you know, actually, I think last time we actually spoke, I was still participating in the GPT-4 red team. You were still on the board and you know, it's been quite a journey since then. I've definitely watched the company like super close. And I feel like I've kind of been on this roller coaster where I've been like many times sort of disappointed and kind of at times even like I was scared by what I'm seeing. And then at other times, I'm like, well, that's like dramatically reassuring. And I'd say there's probably 10 episodes, you know, that and it might be like five to five. The most recent two would be the publication of the obfuscated reward hacking paper, which I would put up there in the, you know, pantheon of like most important, you know,

• 20:01Nathan Labenz?

  and clearly stated warnings about like how AI can go wrong if it's not developed with like, you know, utmost care. And then at the same, like maybe the same week or like within like 72 hours of that, there was the response to the White House request for comment on like what AI policy should be. And there, you know, we got a rather, yeah, I would say escalatory vibe, you know, certainly with respect to China. We've had, you know, Altman has said things like it's our values or their values. There's no third way, which seems again, like ruling out a lot of possibility space, you know, quite prematurely. And then also asking for things like, you know, just cancel all, you know, property rights. So we can just train on everything. Cause again, if, if we don't do that, you

• 20:42Nathan Labenz?

  know, that China will. So it seems that there's this like almost schizophrenic nature to the company. And I wonder how you understand that?

• 20:54Speaker 01

  Yeah, I find it confusing as well. And I think it has gotten, seems like it's gotten more pronounced over the last year or two. One explanation could just be that there's, which I think is, as far as I know, is the case that there's relatively, you know, a decent amount of freedom given to employees to, you know, tweet as they choose to, you know, do some research directions and maybe write about those research directions. I think there's different processes that different things go through. But again, I, you know, I haven't been close enough to those sort of on the ground decisions about what gets published, when to have any kind of insider perspective on that. But I do think I agree with you that it's striking how, how different some of the voices from inside the company seem to be. And I wonder if, I don't know, I hope, I hope that the, the more technical folks there are paying attention to the kind of policy messages that are being sent out by the company, given how much they, they sort of contradict each other.

• 21:40Nathan Labenz?

  How would you advise people that are there today? I mean, if you are inside and you are, and this could generalize beyond OpenAI. I think, you know, I don't see any reason to think that like, you know, XAI won't have similar issues and potentially, you know, other companies that are generally held in higher esteem in terms of their, you know, safety practices very well could too.

• 22:03Speaker 01

  Yeah.

• 22:03Nathan Labenz?

  It's all coming at us pretty fast. So if you are somebody inside a company and you're like concerned about what you're seeing, what should those people be thinking about? And maybe like, you know, flip side of that or complimentary question is like, what should policymakers be thinking about in terms of protecting whistleblowers? Or, you know, facilitating? I think, you know, I also like, respect the idea that, you know, these companies should be able to keep some trade secrets.

• 22:35Speaker 01

  For sure.

• 22:36Nathan Labenz?

  But then it also seems like the level of secrecy that was requested, like of me at one time, was too much where it was like, the public sort of at some point does need to know what capabilities exist. So I don't really have a great sense for how to find that line, but maybe let's start with the policymakers. Like, what do you think the rules should be? And then we can go into like, if you're in a position where maybe the rules aren't there yet, or whatever, you know, how should one as an individual think about taking responsibility themselves?

• 23:05Speaker 01

  So the rules specifically around whistleblowing?

• 23:08Nathan Labenz?

  Yeah, I mean, you go broad of that if you want, but I'm, I'm definitely interested in, how do we get things that the public really needs to know?

• 23:15Speaker 01

  Yeah.

• 23:16Nathan Labenz?

  To come to light when, you know, company policy says it's a secret.

• 23:20Speaker 01

  Yeah. Yes. I mean, I think the whistleblowing piece does like, connect to other parts of the policy picture. I won't, I won't try and give like a comprehensive view on policy right now, but, but sort of starting from the whistleblow piece and expanding out. I mean, the way whistleblowing usually works is it's for illegal behavior. And so, you know, the SEC has very clear processes for if you're seeing financial misconduct, you can go talk to them and lots of other whistleblowing processes are similar. So I think for policymakers, a big challenge I would want them to have in mind here is, A lot of the concerns that we're talking about are potential concerns are behavior that is actually not illegal. And so where is the line for when can you whistleblow? What kind of behavior should be protected? So I think the best way to do whistleblower protections is to pair it with some kind of disclosure or some kind of, some kind of, you know, rules around what information

• 24:05Speaker 01

  needs to be shared, because then it creates a clear standard for if the company is either not sharing information it's supposed to share or being misleading or, or, you know, inaccurate in the information it's sharing. That's kind of like structurally a simpler way for the whistleblowing to work as opposed to trying to have this vague standard of like, if you're worried, you know, then call this hotline or something, because that's just so squishy and hard, both for employees and also for the company, as you say, trying to think about trade secrets or, or other reasons that they don't want their employees just kind of blabbing. It's much more helpful to have kind of clear standard to compare against. So that's, that's one thing I would say on the policy side is if you compare it with some kind of expectations, requirements around information sharing of some kind, or around processes that you have to carry out internally, even if you don't share the results.

• 24:49Speaker 01

  But then that lives, leaves employees able to say, oh, actually, we didn't carry out that process. So, you know, to be a little more concrete, a version of this that I think can work quite well is, is the kinds of ideas around creating a safety and security plan, potentially publishing that, that plan or sharing it with the government. And then that creates an opening for whistleblowing activity if you're not sticking to that plan, basically, which again is just a little crisper than, uh, you can whistleblow if you're, if you're worried, if you're concerned, if you think there's too much risk being taken. And the other thing for policymakers, I think that they need to, to keep in mind is that these are technical folks, they're not legally sophisticated, they might be scared, they might not have that much time, they might be working really hard. And so the simpler and clearer the process can be in terms of how do you know if you're eligible? How do you know what your next step is?

• 25:31Speaker 01

  I think that kind of, almost like, you know, UX set of questions matters a lot as well. If the, the user is a whistleblower, then, then what is their, what is their user experience? For the people in the companies, I mean, sort of very concretely, you know, there's, there's other, you know, formal whistleblowers who have gone on the record. And so those are definitely people you can reach out to if you're kind of looking for advice. I think more broadly and conceptually, a really important thing to keep in mind for the people who are contributing to AGI companies work, to these frontier companies, is I think they are in a very powerful position. You know, I think we've seen multiple times how powerful employees can be. And I heard this point recently, and I thought it was really smart, that they might be in the most powerful position they're going to be in, because they're actively working to replace

• 26:15Speaker 01

  themselves and actively working to sort of hand away their own power. So if you're in one of these companies, I think there might be a temptation to sort of sit tight and wait until things get more serious. And that might be right. But I think it's worth thinking about, you know, both the case that you might actually be less powerful in the future than you are now, if your work is more automated. It seems like in general, tech workers' power is going down right now in terms of the labor market and so on. And also, there, I think it's perfectly likely that there will not be some sort of clear crisis moment in the future, but instead it might really be kind of boiling frog style, this is worrying, this is worrying, I don't like this, this seems a little bit dishonest, this seems a little bit too risky. And so just being realistic with yourself about if you're only ever going to do something, if there is some big moment, then being realistic with yourself that that

• 26:59Speaker 01

  might mean that you never do something. Maybe that's what I call it, but, but not, not kind of kidding yourself about that, I guess.

• 27:06Nathan Labenz?

  If you were organizing a union at one of the frontier developers right now, like, do you have a sense of what your demands would be?

• 27:14Speaker 01

  I haven't thought about much. I think it's an interesting line of thought. I, I think, in general, I think we're starting to get to the point where, you know, there's a whole world around labor organizing, worker power, and I think it really hasn't connected much with either sort of the technical AI world so far, or the AI policy world so far. I, I think that's going to change. And I'm, I'm pretty interested to see kind of folks who have more of a background in that space and thinking about kind of how to use this kind of power and how to, how to, what kind of leverage is productive, how to represent a broad set of interests. I think we'll see more of that in the coming years. And I'm pretty interested to see where it goes.

• 27:44Nathan Labenz?

  You mentioned the sort of disclosure requirements. I mean, we had briefly a sort of 10 to the 26 threshold where at least you had to sort of say they were doing it. And I think a little bit more about sort of what tests you ran and kind of how they came out. My sense is that's now gone.

• 28:01Speaker 01

  I think it's unclear. Last I heard, it was unclear if it was gone because it had started to go through the official sort of notice and comment process in the government. So I think it might still be at the discretion of the agencies, so specifically Commerce. So I haven't heard that that is definitively dead. It certainly seems like it's on a wobbly footing right now.

• 28:20Nathan Labenz?

  Yeah, they've announced the intention to remove it at least. If there's really nothing else, right, that is sort of at an actual rule at this point?

• 28:29Speaker 01

  There's the EU AI Act, which is in the process of putting together their Code of Practice, which I think involves some transparency around, I think for them, it's models over 10 to the 25 and maybe some other criteria. And I think the details of what exactly is going to be, what exactly is going to be transparent about is still being hashed out. And there's also a lot of political pressure to water that down right now. So we'll see. But that is at least one other legal process that is underway.

• 28:53Nathan Labenz?

  Yeah. What do you think is, I mean, compute thresholds are one thing. What do you think is most important for the public to know? Like, I tend to focus on just observed behaviors, but I'm also really mindful that like all of these things sort of have at least potential for unintended consequences, you know, that with the observed behaviors, there's sort of the like, well, we don't look, we don't observe. And so, you know, that can fall down pretty fast.

• 29:21Speaker 01

  Yeah, I think there's, I think there's a lot of things that could be helpful to know more about, to share. Certainly trade offs in terms of what do you share with the public? What do you share with the government? How confident are you that the government will keep things you want to keep private, private? So I think lots of details to be worked out. I think, I tend to think in terms of test results, both for capabilities and risks, being pretty important to share. So what do we think these systems are capable of? I think also just being transparent about, you know, what kind of processes and protocols are you using to make sure that things are safe? Again, just disclosing, not having the government come in with a checklist and say, here's what you have to do, but just saying like, tell us, how are you thinking about this? And I liked, I liked ideas as well, from Daniel Cocotello and Dean Ball had a, you know, a sort of joint piece on transparency. I thought some of the things they pointed out there, like looking at, you

• 30:04Speaker 01

  know, the model spec, what is your model actually being trained to do? Also makes sense to me to have that, that kind of thing be shared publicly. Again, not because the government should be saying what it should be, but because this is a very fast-moving space. And the way I think about it is there's this huge information gap between the companies that are developing this stuff and everyone else. And so if you can narrow that information gap a little bit, I think that's decent.

• 30:30Nathan Labenz?

  Yeah, it seems important to me. And I mean, honestly, there's just a lot of work to be done, even in educating people about what is already fully public. You know, one of my kind of mantras is like, if people understood what is already out there today, better they would probably have a healthier fear of what might be coming down the pipe, you know, in the not-too-distant future. And so to some extent.

• 30:48Speaker 01

  Or even not a healthier fear, but a clearer understanding, clearer picture.

• 30:52Nathan Labenz?

  Yeah, I think a little fear is healthy, personally, honestly. But, you know, mileage may vary on how much that does for different people. I also sometimes describe myself as an adoption accelerationist and hyperscaling pauser, meaning like, I love the tools that I have today and I absolutely am trying to use them to the maximum. And I also am like, man, you know, there's a huge overhang for society broadly on what we already have. And we continue to see so much more pulled out of models of a certain kind of resource input that I do think we're, you know,

• 31:36Nathan Labenz?

  potentially starting to get close to the line of where like certain thresholds could be crossed in ways that we just like can't take back and might ultimately regret. You know, I mean, obviously, the canonical one is like, you open source a model that is later found, you know, to be able to help people make bioweapons or whatever. And like, you've just created a new sort of Damocles that hangs over everybody, you know, kind of indefinitely. Yeah, I don't know. It seems like we're close. Do you feel like we're not that close to that? It feels to me like we're fairly close.

• 32:04Speaker 01

  I don't know. Yeah, I feel like I'm, I also, I was, I had four and a half months of parental leave over the winter. And so I feel like I'm still, I just came back to work a few weeks ago. So I feel like I'm still kind of reorienting around O3, you know, DeepSeek R1, this new ChatGPT image release, Gemini 2.5. I haven't had a chance to try yet, but like, you know, there's just so much, I feel like my picture is changing all the time. I definitely think we're at the point, to me, the version of compute thresholds that makes sense is not saying, oh, models over 10 to the 26 are dangerous, so we have to restrict them more, but saying, okay.

• 32:37Speaker 01

  We need some way to target the models that are newest and best and most capable, because those are the models where the sort of potential risks of potential unknown unknown risks of like, what can they do are highest. And so those models should be subject to a little bit more scrutiny. And so I do think we're at a point where it makes sense to say, like, look, seems plausible that the next generation of models could be really concerning in a whole bunch of ways. Totally plausible, they won't be, but probably we should be like looking a little more closely than, you know, when, you know, GPT-3 came out and that was just, I think, really unlikely to be anything worrying. And I think it was right that there would kind of no rules in place for for releasing that kind of model. So yeah, that's that's the way that that I'm thinking about it right now.

• 33:15Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 33:21Nathan Labenz?

  This may be a good opportunity to talk about your concept of adaptation buffers, which is, I really like that phrase and notion. And I think it helps deconfuse, or hopefully, I think I think it will, I'm hopeful that it will help deconfuse people about the sort of apparent, you know, sort of hard to reconcile idea that like these AI advances that we see are really hard to achieve and like cost huge amounts of money. But then we also see that like, you know, deep seek and otherwise, like, things are becoming dramatically cheaper. So maybe set up that dynamic a little bit and talk about this adaptation buffer and how you see the window of time that we have to kind of adjust to capabilities advances.

• 34:00Speaker 01

  Yeah, I mean, I think an important underlying thing here is, I'm generally a believer that humanity, you know, society at large is very adaptable and has adapted to a lot of things in the past. So a lot of things, you know, it's sort of a cliche that often when new technologies come in, whether it's the printing press, or the television, or the telephone, or whatever, people cry that the sky is falling, and this is a terrible thing, and it's gonna ruin everything forever. And then it doesn't. And so I think the starting point here is thinking that for a lot of different kinds of technology, we actually have a pretty good track record of sort of digesting them, figuring out how to incorporate them into society in a good way, how to have a set of, you know, institutions, or barriers, or social norms around them that that make them sort of positive rather than negative, or on balance positive. And I think

• 34:43Speaker 01

  with AI, I think there are a set of questions around AI that seem less that way to me. So I think the whole thing around, are we building something that is, you know, a successor species, or, you know, more intelligent than than humanity by a long way, those set of questions, I think,

• 35:00Speaker 01

  Do seem potentially very different to me, but I get a little worried on whether this this post, which again, you know, you saw I draft of hopefully now on on Substack Live by the time this comes out. A place where I think people sometimes overstate how new this is, is when it comes to AI misuse. This idea you mentioned of like, are you going to have some open source model that can help anyone create a bioweapon or can really make it much easier to carry out really sophisticated cyber operations, hack critical infrastructure, so on. I sometimes see this impulse in kind of AI policy circles of, oh, that's too dangerous. That can't, we can't let that happen, can't let that out, can't have that technology, you know, be proliferated. It's just like, uh, sort of in an absolute sense, too dangerous, no go, big problem.

• 35:42Speaker 01

  And so what you see from that is people talking about wanting to ensure or wanting to work for, work toward nonproliferation of these systems, meaning can you prevent them from being distributed in any way? Can you prevent, you know, access from getting beyond a small number of, you know, very controlled people? And yeah, your question about kind of deep seek versus the frontier giant cluster training models gets at why I think that is really problematic, which is we have this weird dual dynamic at the moment in AI development where it is both true that developing the next best model that pushed frontier being at the cutting edge keeps getting more and more expensive in terms of compute power, also in terms of, you know, the number of the amount of expertise you need, that, you know, you need a really absolutely top team of researchers and engineers.

• 36:25Speaker 01

  That kind of amount of, you know, expense is going up and accessibility is less and less accessible. At the same time, every time we reach a certain point on that development curve, every time we reach a new set of capabilities, it's very expensive the first time we build it, but then it gets cheaper and cheaper and cheaper. So, you know, deep seek was an illustration of this dynamic where they didn't actually build, you know, a model that pushed the cutting edge and was better than anything we'd seen. What they did was they matched what some US companies had had, depending on how you count, you know, a month or two ago, if it was the reasoning model alone, more like six to nine months, if we're looking at the base model. And so, and they had done that at a lower price point. We can fight about what the actual price point was, but I think that's not the point for here, for this, you know, the situation. And the reason this matters is I think if you're taking

• 37:10Speaker 01

  If you're trying to have a policy approach that says we're going to prevent anyone from having access to a certain kind of model, but that model is getting cheaper and cheaper and easier and easier to get your hands on. That means that your kind of policy regime is going to have to get more and more invasive to prevent people from having access to that. So the comparison I give in the in the post is we have nuclear nonproliferation. It works pretty well. You know, the only about a dozen countries have nuclear weapons. That's pretty good, I think, compared to what a lot of people would have expected in, you know, the 50s. But imagine how that could have worked or how that would have not worked if nuclear technology were improving over time, getting much more efficient at the same kind of crazy rate that the AI technology is, such that you needed less and less uranium to build a nuke and you needed it to be less and less enriched.

• 37:52Speaker 01

  So right now you need quite a lot and you need very, very enriched uranium to for you to actually be able to to build a bomb. Imagine if that number is going down over time. At some point, you would need to have the IAEA coming and inspecting, you know, what you're doing in your farmhouse because you have a couple acres of land and, you know, across that couple acres of land, there's enough uranium in the soil that in theory you could build some like very efficient, you know, teeny tiny nuclear bomb. That, you know, that's a totally untenable regime. That the nuclear nonproliferation we have right now only works because there's a limited amount of physical material that people don't necessarily need for other purposes that needs to be enriched in these very highly specialized facilities. And that's really not the case for AI. Um, but I think if we're thinking, instead of thinking purely about how we prevent people

• 38:35Speaker 01

  from getting access to this, if we're thinking more about how do we make the most of the time that we have to adapt to build our societal resilience, to be doing things like scaling up our vaccine production infrastructure or scaling up our outbreak detection infrastructure. So, you know, more wastewater monitoring, how do we have more test kits available in more places around the world? So if someone does use a bioweapon, we can more quickly identify it, more quickly respond to it, similar things for on the hacking hacking side of things. I think that approach of how do we, how do we maximize the value that we get out of the time we have as opposed to how do we lock this down and prevent this sort of capital B bad technology from being spread is both going to be more more productive and also less invasive. I do think it might not be enough.

• 39:20Speaker 01

  It might be that we're just in a really bad situation if if AI progresses incredibly rapidly, but I think it's a much better approach and a much more sort of societally healthy approach.

• 39:30Nathan Labenz?

  Does that imply a certain pessimism about technical solutions? Like, another one of the blog posts is about the, you know, kind of fundamental challenge of just like getting AI to do what you want it to do at all. You know, I'm old enough to remember the Eliezer-led discourse from years past on how these things are going to just turn us all into paperclips. Of course, that's like, was always kind of a caricature, but you know, there was a, I think, a felt sense that we have no, kind of a genie problem where we have no idea how to communicate our real values and real intent to a system like this. And so they're just going to be like extremely, extremely unwieldy. I would say relative to that, I've been very pleasantly surprised on the upside that

• 40:13Nathan Labenz?

  like, today's models do seem to have a pretty good internalization of human values broadly and sort of a general respect for norms. Although, at the same time, like, one always has to be situationally aware, we are now seeing many of the problems that, you know, Eliezer at all sort of predicted back in the day of like, once they have values, they also sort of seem to be inclined to try to protect them by, you know, lying to users if that's what's needed or trying to subvert a training process that they understand themselves to be going through. So, you know, this is another one of these roller coaster rides where I feel like, man, it's gone way better than I thought. But also, some of the, you know, some of the doomsaying is like, starting to be proven correct.

• 40:58Nathan Labenz?

  But I guess I would be optimistic that like, if we had a sort of adaptation buffer that was maybe a little bit more required or, you know, imposed by authorities as opposed to just kind of trusting the, the sort of natural sawtooth motion. Because it does seem like DeepSeek might be about to challenge that, or at least those sawtooth time intervals might be getting really short. Like, yeah, I would, you know, definitely love to see all the great things of wastewater monitoring and, you know, I mean, the fact that we haven't done anything really about it, the last pandemic has not bode well. Indeed. But I also kind of feel like... We need that time to maybe sort of figure out like, how does one distribute a frontier model with a better sense of like what capability, you know, making unlearning work or making like mixture of experts work in such a way where you can like distribute all but two of the experts or something, you know,

• 41:44Nathan Labenz?

  so that certain capabilities are like redacted while the the core utility of the overall thing can be diffused. Long question, I guess the core of it is, do you think we'll see technical solutions that could allow us to sort of square the circle and have, you know, free distribution, but also, you know, a pretty confident sense that like what we're distributing isn't going to come back to bite us?

• 42:09Speaker 01

  I think the thing I want to challenge is to focus on only technical solutions because this does come up a lot. It's like people talk about the the offense, defense balance of like, you know, AI for cybersecurity, for example, does it help hackers more or defenders more? Same for bio, people talk about, does it help you, you know, design a new vaccine as much as it helps you design a bioweapon? And I think that's, I think that's just one small part of the picture. I think the thing that I'm trying to point to with the idea of an adaptation buffer is that a lot of the ways that we, specifically, we're talking about these misuse risks. So are there going to be terrorists who build a bioweapon? Are there going to be hackers in their basements who can suddenly bring down the US power grid? A lot of the things we can do about that don't actually relate to AI. They relate to what is going on in society. Or if they do relate to AI, it's not actually the frontier model. So it's like, can you, you know, can you use AI tools to look at, you know, large scale

• 42:53Speaker 01

  disease monitoring data, for example, and notice anomalies or something like that?

• 42:58Speaker 01

  For bioweapons, it's like, you know, if you talk to people who who work in biosecurity and bioterrorism, there's a lot of stuff that has nothing to do with AI, nothing to do even with biomaterials. It's things like, who is the FBI tracking? You know, what is, how good are they at detecting plots before they get very far? So, yeah, I think certainly as the technology advances, there will be sort of new defensive tools that become available to us and we should make use of those. But I sometimes think that the discussion here gets too focused on only those as opposed to looking at kind of broader parts of the picture. And, you know, if we are also, if we're talking about the AI tools, I think a huge part of the discussion needs to be about the application and dissemination of those tools. So for example, in cyber, I, I think it's sort of less a question of what can the absolute

• 43:41Speaker 01

  most advanced model do for cyber defense and more a question of how can you have well-designed, ready to ship, you know, defensive tools that you can get in the hands of? Operators who are not very sophisticated on AI. So you're people who are running your your water treatment plants and your power grid and your chemical plants and so on. How do you have those, even if they are AI related, it's not a kind of capabilities question, it's more of a dissemination application question for how you get those defenses kind of out into the real world.

• 44:11Nathan Labenz?

  Yeah, I just talked to somebody not long ago who is applying language models to the challenge of formal verification of software. And it sounds like there is a multiple order of magnitude speed up that is becoming possible in that domain. And it does feel like if we just have enough of an adaptation buffer, then a lot of the things that people are most worried about could really be brought dramatically down in terms of the, you know, the absolute magnitude of the risk.

• 44:42Speaker 01

  Yeah.

• 44:42Nathan Labenz?

  And so, yeah, it's just a question I think of like, are we investing enough in that? Almost really not. And, you know, do we have enough time for the sort of, you know, next disruptive thing hits?

• 44:53Speaker 01

  For sure. And I mean, not to sound too optimistic here. I mean, I do think sometimes I hear from from folks, you know, in the cyber domain that, well, it's fine because AI is going to help defenders more than attackers. So so it'll all be good. And I think that is also, you know, in my mind, a quite naive take if you're looking at the sort of dissemination and real world use case here, where I think even if that is ultimately the case long term, for example, if you can, you know, use AI to develop formally verified code, you're still likely going to go through this sort of dangerous transition period where it's just obviously going to be much quicker. The time lag between some model is released and some disaffected teenager in the basement can use it to carry out an attack is going to be much shorter than the time lag between the model release.

• 45:35Speaker 01

  And when, yeah, the, I don't know how many it is, thousands of critical infrastructure providers in the US can go through their very old code bases where they have this complicated division between their IT and their OT operational technology and what gets updated when, you know, it's it's they're not going to be nimble. They're not going to be agile. They're not going to have all these defenses kind of wrapped up or built in really quickly. So I don't think the fact that these advances seem promising and could help defenders means that we'll get away from that sort of dangerous transition period. But I do think that thinking in terms of a transition period, rather than like a permanent state of danger, helps us respond much better.

• 46:12Nathan Labenz?

  So have you seen any regulatory proposals that you'd like? I mean, you mentioned Dean Ball, friend of the show. He recently put out a post I thought was quite interesting, basically proposing a regulatory market type structure where the government would essentially accredit, you know, or sort of authorize private regulators to sort of, let's say, approve the practices of AI developers. And as long as the developers were able to keep the private regulator happy, then they would get some sort of liability shield. And, you know, then, of course, that could be like withdrawn if they didn't do it. And even, you know, the regulators, the private regulators, you know, authorization could

• 46:56Nathan Labenz?

  be withdrawn if, you know, if the state found them to be out of compliance. I understand now, I haven't read the text, but I understand there is now a California bill that is kind of moving in that general direction. Interesting your takes on that, or, you know, there's also like proposals to really embrace liability and go the other way and say like, you know, maybe you should even be liable for like close calls, you know, because the close calls could be like so big and bad that, you know, even probabilistically, if you were, you know, if it was a near miss, you know, maybe you should sort of face liability consequences for that. So I don't know, react to those or, you know, tell me any other policy proposals that you think are particularly promising.

• 47:35Speaker 01

  Yeah, so I haven't seen a policy, a sort of like proposed regulatory regime to really sort of comprehensively manage the risks that we're facing from frontier AI systems. I haven't seen one that I like. I think it's a really big problem. It's a complicated problem and it's especially difficult because we're not actually sure exactly what the problem is or when we'll face it. So to me, the policies that I have seen that that I'm interested in are more kind of building blocks that put us in a better position for the future as opposed to sort of solutions, which I think is, I think is in some ways appropriate, given that there's so much uncertainty about the technology, though also certainly scary, given that one way the future might go is we might have very little time, in which case maybe some initial building blocks right now are going to be far from sufficient.

• 48:19Speaker 01

  So I think the proposal by Dean is an example of a building block that seems potentially pretty useful, seems certainly better than nothing. I think it's, it's, you know, he's, he's written it deliberately to be something that could be implemented at the state level, which certainly seems far, far, far more feasible than any kind of federal legislation. I don't know that it does as much as we would need to target those kind of cutthroat financial incentives and also just like other many incentives for the frontier developers to do anything other than push ahead as fast as they can. But I, you know, I think it's a, an interesting idea and it does seem better than nothing. And then other, other kinds of building blocks, we talked about transparency. I do think that that is the kind of thing that can, doesn't in itself solve any problems,

• 49:01Speaker 01

  but does put many actors in a much better position to help solve problems as they rise down the road. Likewise, I think there's a lot of stuff that can be done that isn't like sort of regulatory, isn't obliging anyone to do anything, but things like funding, trying to, trying to really boost our science of measuring AI, I think could be a target of, of research funding, potentially something for like a, you know, focused research organization or things like that, similarly for interpretability, of course, and alignment research and lots of things like that. I think even just, even like building blocks as basic as trying to get more technical capacity into governments so that they're able to kind of handle things as they arise and, and make better decisions as the technology progresses. These are all kind of, again, individual components that don't add up to a comprehensive

• 49:45Speaker 01

  solution, but that I do, I think, put us on better footing for the future. So that's the terms that I'm thinking of right now.

• 49:52Nathan Labenz?

  Just as an aside, it was Gabe Wile, I had to look up and make sure I had his name right, who's arguing for this sort of embrace of liability. And I hope to do an episode with him about that. But in the meantime, he's written about it for Fox One, I go into that in more depth. You know, one thing that I thought OpenAI always had right was the idea of iterative employ, iterative deployment, you know, the idea that like, if we sort of develop superintelligence in secret and then, you know, drop it on the world one day, that's going to be like far more disruptive than if we sort of, you know, launch a bunch of products along the way and people can see what they're good at and get used to them and so on and so forth. That itself now seems to be sort of, uh, at risk, both in the sense that like, Ilya

• 50:35Nathan Labenz?

  has gone off and started a company that has a Explicit strategic statement that they are not going to release anything until they achieve superintelligence, which seems crazy, but also, you know, to borrow a term like strikingly plausible that they might actually achieve it. And even at OpenAI, I was really, you know, kind of taken aback by Sam's recent statement when they released 4.5, that, you know, let us know if you like this or not, because, you know, we've got a lot of other models to build. And, you know, this one is pretty compute intensive. And so if it's not like, really doing it for people, then we might like, take it offline and, you know, focus our resources on building

• 51:16Nathan Labenz?

  more models. This has me thinking like, you know, we, we might actually be at sort of risk of like, these companies kind of closing down what they put out into the public and just kind of going for broke totally internally. Miles Brendig is also, you know, formerly of OpenAI, has made some, I think, kind of cryptic comments about sort of the rising importance of internal deployment decisions. So I'm sure you'll find major flaws with this. But one idea that I've had is like, could we put a sort of speed limit in place, not an absolute speed limit necessarily, but a relative speed limit where we might say, you can only develop a model that is so many times bigger in terms of resource inputs than the biggest

• 52:00Nathan Labenz?

  one you currently have deployed. And if you want to go bigger than that in your development, you got to deploy something that's kind of, you know, helping us as the rest of society, like, understand where all this is going, so that we don't have these, you know, sort of, not exactly unilateral, because we know that we've got multiple voices inside the companies, but, you know, these like, very small, kind of concentrated decision makers, without much at all in the way of visibility, just going for something that they seem to believe could be like, a world takeover capable technology. So I guess, how big of a problem do you see that possible retreat from iterative deployment being? And, you know, do you like my relative speed limit solution, or have any others to address that?

• 52:43Speaker 01

  Yeah, I agree with you. I think iterative deployment in general seems like a good approach, with of course, the caveat of at some point, you should probably have some criteria in place for when you would not just iterate. Because you know, the idea of iterative deployment, I think, is to kind of put it out in the world, see what happens, adjust, try again. And I think that's great, as long as you're confident enough that what you're putting out in the world is not going to have any really severe irreversible consequences. So the question is, how do you know, when do you decide to do it differently? I'm not sure that I see a retreat from that. I think, I don't know that it's ever been, like, certainly was or has been OpenAI's MO, but I don't know that it's ever been kind of an across the industry approach. Your idea, I think it's interesting. I've heard kind of similar proposals.

• 53:26Speaker 01

  Could also be things that companies adopt internally in terms of, you know, how much scale up are you going for at a given time? It sort of feels to me like it runs into the same problem that so many of these run into, which is, one, how are you going to implement it? Like, to do that in the US, you would certainly need legislation, I think. I don't see how you would do it without legislation. We're not going to get legislation, so then how do you do it? And then also, doesn't it just mean that we lose to China? It's going to be the other, you know, the other big question. So I think conceptually, things like that could work, maybe, or could make sense if they were implementable. I don't really see how they're implementable. And then if they were, then I would want to come back to this question of like, okay, is this actually the right approach? For me, I tend to be pretty pessimistic about sort of solutions that involve slowing down at some, kind of the input level,

• 54:09Speaker 01

  meaning slowing down, pausing for a certain number of months, or slowing down how much, you know, how quickly you're advancing versus slowdowns that are sort of conditional, more of this kind of like if then approach of, you know, we're not going to keep progressing until we have hit this level of understanding of our system, or this level of risk mitigation, which I think a lot of the companies now have in place, or, you know, have made voluntary statements that they will think in that way. So that would tend to be my preferred approach. But I think this kind of thing, I think we're in a rough situation right now for anything that will involve kind of cross industry coordination, because there's so little political appetite at this moment. Maybe that'll change, probably that'll change, but uh-uh. For now, it seems hard to imagine.

• 54:51Nathan Labenz?

  Do you think it all, at the end of the day, is about the fact that the policymakers, the members of Congress, whatever, just like, just don't buy it? I mean, it seems like if they really believed what Ilya is saying, that he's gonna not release anything until he has superintelligence and he thinks that's gonna happen in the not-too-distant future, then they wouldn't just sit back and be like, well, let us know when you got the superintelligence, right? Like, it seems to me that they just fundamentally don't believe it, and that's like the biggest barrier to something, you know? And there's a lot of questions, obviously, I don't know what that something should be. But I find the notion of like, well, we're not gonna get legislation, to me, that still feels like a sort of education challenge.

• 55:33Nathan Labenz?

  Like, again, if, I think if people had a better sense of what already is deployed, you know, they might be like, yeah, I don't know that I'm comfortable with Ilya and like, how many people, how many people have even worked there? You know, it's like, are we talking like, a couple dozen, potentially, maybe up to, you know, low hundreds now? It can't be that big. And then we're just gonna, you know, sort of wait for them to pop up with superintelligence? Like, that seems so crazy.

• 55:56Speaker 01

  I agree with you. I think it is, in large part, a question of how seriously do people take the possibility that this, that AI will get as good as similarly Ilya thinks it will. But I also think that US Congress is really broken right now, not in an AI way, but just like incredibly dysfunctional. A friend of mine who really knows his way around Congress, has worked on the Hill for years and years, said that he thinks that the US House is, House of Representatives is less functional than it's been since after the Civil War. So like, really, like, really, really, really dysfunctional, separate from AI. So I agree with you, I think there will be most likely windows that open again if the technology keeps progressing, and I think people will, will change their views and change their level of urgency around it. And also, I don't know that that will be enough to actually get thoughtful, productive

• 56:41Speaker 01

  regulation through Congress at the federal level. It might be enough to get some kind of bill, and I certainly know people who are working on, you know, sort of like a Patriot Act for AI, where the Patriot Act was put in place after 9/11, but had really been developed in advance. And sort of setting aside the merits of that bill, I think that approach makes sense of saying, what, you're gonna get some window after some crisis. I think that is a reasonable way to be thinking about AI policy right now. I think it'll be a, a, a big lift, even in the wake of a crisis, to get the right kind of productive, useful legislation through, not just kind of fighting the last war, or... Doing a bunch of stuff that that people wanted to do for other reasons. I mean, the other model for this in a past era, when it seemed like we might actually get something through through Congress, the other model for this that that I have thought

• 57:25Speaker 01

  about is sort of the trying to avoid a Three Mile Island situation where Three Mile Island as a nuclear disaster seems to have basically killed the US nuclear industry because the safety regulation that was put in place afterwards was just too onerous and wasn't comparing to risk posed by other sources of energy and wasn't actually kind of commensurate to the level of risk. And instead, it just sort of shut down the whole industry. And I think that that story, in my mind, should be motivation for AI developers to want to have some more safety guardrails in place earlier to prevent that kind of accident or to mean that if that kind of accident happens, you have a better answer or, you know, legislators have a better answer for the public of here's all the stuff we did in advance. And this really was just a freak accident. I think right now we're on track for a massive knee jerk overreaction when something something happens. But we may have

• 58:10Speaker 01

  passed the point where we can prevent that at this at this point, given how unlikely the regulation looks. I don't know, maybe the states will exceed my expectations. Maybe they'll maybe there'll be kind of more useful stuff that comes up there.

• 58:19Nathan Labenz?

  Yeah, I say something similar to like AI developers and investors, not even so much at the frontier level, but even just like your rank and file app developers all the time. Like, we are right now, the voice AI world is totally taking off. And also, I think is if they're not, you know, pretty quick to sharpen up how they handle the technology, we're headed for a world where there are going to be some high profile things. And because the voices are getting really good, and you can still go to all these products and just drop in whatever voice you want, click the checkbox, and the next thing you know, you're calling, you know, as Trump or as Taylor Swift, I did it with Biden, you know, during the election as well, just call anyone, say anything, there are zero guardrails

• 59:03Nathan Labenz?

  on these products. And, you know,

• 59:07Nathan Labenz?

  That's not going to be good for the industry if it, you know, they're like definitely, again, playing with a certain kind of fire that I think self-interest alone would dictate better governance or stewardship of such powerful technology, but that's

• 59:20Speaker 01

  Yeah, I mean,

• 59:21Nathan Labenz?

  all on deaf ears.

• 59:22Speaker 01

  And one thing I think is maybe gets a little bit glossed over in sort of the kind of more detailed or like wonky discussions of this, people who are thinking really hard about the risks, thinking really hard about the policies, is I think there's, it's quite unpredictable or quite maybe unintuitive which kinds of things will catch the public imagination or will create sort of a perceived crisis. So, you know, I think, seems to me like in the, after the GPT-4 release, one of the things that really caught fire was this conversation that Kevin Roos had with Sydney and, you know, was trying to get him to leave his wife or whatever. And, you know, the AI people I know who read that transcript were like, look, he really led it there. Like, he was really kind of prompting the model in a way that got it to go there. It's not that surprising. It wasn't dangerous, didn't actually harm his marriage at all, probably was a huge boost to his career, you know?

• 1:00:05Speaker 01

  But that is what really kind of caught attention and got people worried. And likewise, you know, I don't personally feel that worried about the risks from voice synthesis. We could, you know, talk about that maybe. But I agree with you that it's the kind of thing that is very vivid and very easy for people to latch on to. And so it could be the kind of thing that produces a backlash, sort of disproportionate to the actual risk or harm of that specific use case.

• 1:00:27Nathan Labenz?

  Well, in the interest of time, let's keep moving. You alluded to maybe the one thing that can unite the Congress, and that is the threat of China. And, you know, certainly a growing number of my AI conversations sort of get backstopped or kind of run into this like final barrier of like, well, you know, but China, you know, we'll lose to China. One thing I think is very much under discussed, I'd love to hear your take on, is what is the threat from China? You know, like, I don't get great answers to this usually. And I sometimes joke like, am I supposed to expect that my grandkids are going to be speaking Chinese, you know, if we don't like develop AI as fast as we can? How do you understand the threat from China to the United States, the West, like my values, my way of life?

• 1:01:13Speaker 01

  Yeah, I've heard some of the conversations you've had about this. And something that jumped out to me there, coming from the world I come from, this sort of national security, foreign affairs, geopolitical kind of viewpoint, is you seem to be starting from a point of view of like, well, the US and China should be friends unless there's some strong reason otherwise. And I think for a lot of people with experience in international relations, defense, military history, the starting point is more, okay, we're an established power. China is a rising power. Who has power on the world stage matters a lot. And by default, if they're coming in to be rivaling us in terms of how much power they have and how much they can throw their weight around on the world stage, by default, it's going to be a more hostile relationship.

• 1:01:57Speaker 01

  And maybe you can have exceptions to that. So, you know, the classic exception that from last century was as the US was rising and kind of eclipsing Great Britain, as you know, the British Empire was crumbling right around when the US was really coming into the height of its power. But that was a relationship that was actually very close, actually very cooperative. And so there wasn't a huge amount of tension there. But that's really unusual. I don't know how to give a, I don't want to, you know, go off on a 20-minute tangent about sort of US-China history in the specifics. I think here, there was a real effort, real effort in the 80s, 90s, 2000s, to try and

• 1:02:37Speaker 01

  usher China into a position in what was seen as sort of the rules-based international order, which was, you know, I guess the background here is, usually, for much of history, in many places around the world, most things operated under a might-makes-right framework. Whoever has the most power, whoever has the most guns, gets to push around everyone else. The second half of the 20th century was a big exception to that, where, it's called the Pax Americana, or other things, where the US was sort of the leading power in the world. There was also the USSR for, you know, a good chunk of that. But the US was instrumental in setting up this set of institutions, this way of countries relating to each other.

• 1:03:18Speaker 01

  There's a, you know, UN Charter, which puts sovereignty central, it makes sovereignty of countries central, so that you can't just invade other countries and do whatever you want instead, you know, borders are sacrosanct and so on. This whole system that was put in place in the second half of the 20th century, with the US leading, which was seen as a big improvement on the might-makes-right sort of default.

• 1:03:40Speaker 01

  And I think we're in a weird place right now where if you look back at the last few years, few decades of US-China relations, I think a lot of the hostility now comes from the failed attempt to bring China into that order. So this is, you know, of course, there was, you know, warming throughout the 80s when Deng Xiaoping was doing his reform and opening strategy to try and, you know, make, make China more market focused and freer. Big hit to that in 1989 with Tiananmen Square, then sort of more optimism again in the 1990s, culminating in China entering the World Trade Association. The classic phrase, I forget who used it first, was trying to make China a responsible stakeholder in this system. And then that all kind of fell apart in, you can date it different ways. Certainly by the time Xi came into power in, in 2012, that was starting to crumble and Xi has accelerated

• 1:04:25Speaker 01

  that where China's become more illiberal again, has become more hostile in the South China Sea, more aggressive, really making it clear that they, they want no part in this sort of quote unquote rules-based international order. And so that is the context for, I think, why they are seen in a more hostile light. The challenge now is that the US itself is retreating from that rules-based international order and moving back into a might makes right kind of frame where the idea is, well, the US has all this power. We have, you know, the dollar is the world's reserve currency. We have the biggest, best military, so we should be able to get what we want. So sort of in that light, it comes, it becomes confusing again. Like, why would we have a hostile relationship with China? But I think that is very much a kind of a set of changes that are still in process and that the system

• 1:05:07Speaker 01

  hasn't quite figured out how to orient towards yet. Yeah, I mean, I, I think, I think it definitely comes down a lot to, I guess, there's sort of two big questions. One is China's position in international institutions and its relationship to sort of the whole world. And then there's also these sort of territorial military questions around, is China going to try and take Taiwan? Is China going to be aggressive around Japanese, Filipino, Korean assets? Sort of more of a like hard power military set of questions as well.

• 1:05:37Speaker 01

  Are they going to, you know, for example, come back to the sort of rules-based international order, are they going to damage the freedom of navigation norms that the US has been so instrumental in preserving, that are so good for international trade, for example? Are they going to prevent people from using their, what they claim to be their waters? Or, you know, there's a whole kind of set of questions around who has power, what are the rules, what are the norms, where China is very clearly not wanting to cooperate with the US on that, and so the US has shifted since 2016, 2017 into a more confrontational posture.

• 1:06:07Nathan Labenz?

  I have to say, though, that, you know, all of that stuff, like, I'm generally familiar with that history. Plenty of things we can, you know, complain about China doing. You didn't even mention stealing all of our intellectual property, you know, which is definitely a rightful point to which, you know, many American business leaders and others object. So, you know, there's wrongdoing inside the Chinese nation as well that we can point to and, I think, again, like, justifiably and rightfully criticize. But I still don't quite get the flip, and I'm not sure if I should be understanding what is happening now as just like strategic communication where everybody's like trying to influence, you know, the guy that I call he who must always be named. But like, both Altman and Dario have done a pretty dramatic flip, you know, where there's

• 1:06:52Nathan Labenz?

  video evidence of them not that long ago saying, everybody, you know, is too worried about China, like, erase China, be one of the worst things, you know, we should make our own decisions about what's right to do and not worry so much about them. And we can, you know, point to these clips. And now we've got both of them basically saying, you know, we gotta go as fast as we can or we're gonna lose. And Dario's even like saying, you know, we can't accept a multipolar world, we need to maintain a unipolar world. And I sort of am like, man, who's really being aggressive here? You know, I haven't heard China say they want to be at the center of a unipolar world. I've only heard Americans say, we want to be at the center of a unipolar world.

• 1:07:35Speaker 01

  Depends who you read and how you read it, the Chinese, I don't know. But I mean, I agree with you.

• 1:07:40Nathan Labenz?

  Their technology, right? Like they're not, we're the ones that are saying like, we need to box them out and have this unassailable lead. Meanwhile, they're just like open sourcing everything. It doesn't seem like they're like trying to dominate us as far as I can tell.

• 1:07:52Speaker 01

  I mean, open sourcing, I think, makes a lot of sense if you're the, in the, if you're in the following position. I think it makes a lot of sense to be trying to show off how good you are, trying to attract talent, trying to, so I don't know that that, that is like just coming from pure goodwill and lack of competitive spirit. I think that is more that that makes a lot of sense if you're not leading and it's, it's much less clear how to handle openness if you, if you are leading. I agree with you that the, the position change and the rhetoric change from a lot of the top CEOs has been pretty striking. And I think, honestly, I think it's just, it's just the path of least resistance at this point. There's so many different issues to handle here with AI and so many, so little agreement on what to do about it. And the one thing that people can't agree on is, well, we've got to be China. And so it doesn't surprise me that the companies are leaning into that message as a way to say, look, we should get funding, we should get government contracts, we should have

• 1:08:36Speaker 01

  no regulation, we should get shielded from liability. Yeah, I, that's, that's the way that, that's the explanation that makes most sense to me right now. And I'm sure it also depends on, you know, different individuals, how they're thinking about those specific statements.

• 1:08:52Nathan Labenz?

  You've got a paper coming out on decision support systems in the military. And I think this is really interesting as a, you know, okay, yeah, we got to beat China, whatever, but like, we also have to confront the fact that the systems that we have have, you know, for all the upside, which, you know, I'm well on the record embracing that and use it every day, yada, yada, yada, also have a lot of problems in terms of, you know, their reliability, their hallucinations, again, like now scheming against their human users in some cases. And, you know, I for one, would not want to go into combat with an AI buddy until I was like quite confident that all of these sort of scheming issues were like well and fully resolved. So.

• 1:09:34Speaker 01

  Not to mention, both a nation and reliability and.

• 1:09:37Nathan Labenz?

  Yeah, I mean, there's a lot of issues, right, that I like if I'm if I'm taking this thing and trying to really rely on it in a genuinely life and death situation. You know, and I say this as a, you know, top tier enthusiast, I would say, I wouldn't want to use it in that sort of context.

• 1:09:52Speaker 01

  Yeah.

• 1:09:52Nathan Labenz?

  So that also seems to be like a big disconnect to me in terms of how the AI debate, you know, sort of seems kind of, especially with respect to China, it seems like a little bit decoupled from the actual reality of systems that we have. So I'll shut up, give you the floor and just, you know, tell us about your work on decision support systems and, you know, what we can and probably shouldn't be relying on them for.

• 1:10:17Speaker 01

  Yeah, so this paper, this paper is led by a colleague of mine called Emmy Probasco, and she is a super interesting person to be working on this because she actually served in the Navy and her job in the Navy was operating AEGIS missile defense systems onboard US Navy ships. So this is actually, people really don't, a lot of people don't know kind of that we actually have essentially autonomous weapons systems already. So AEGIS is a system onboard a ship that looks at incoming missile fire and automatically identifies and then takes out incoming threats, not based on deep learning, developed in like 60s, 70s, employed in the 80s. So, yeah, super cool to get to work with Emmy on this paper because she's just bringing such a grounded, informed perspective. The paper is about, yeah, what gets called decision support systems.

• 1:11:02Speaker 01

  And what I think is really important here is to move that discussions about AI in the military don't just get stuck on this autonomous weapons question because there's so many things you can use AI for in the military. And so this category, decision support, is another, is a big, broad category of, that's one example of, you know, other uses. And it can mean a lot of different things. It basically, you know, it's what it sounds like. It's systems that are helping commanders or operators make decisions. And so there's a long history of different kinds of tools like this. And recently, there's been interest in how do you kind of add AI or how do you use AI to do, perform some of those decision support functions or upgrade existing decision support systems. Big range of different types of things we could be talking about here. So on the simple end, could be things as simple as you look at a photograph of an area

• 1:11:47Speaker 01

  and you use AI to determine the view sheds in that photograph. So what a view shed is, is basically if you're a sniper, where can you see? What you know, where is in your field of vision and where is not in your field of vision? For example, or if you're, you know, not necessarily a sniper, anyone. And so that could be a computer, you know, vision sort of segmentation image that is doing some kind of processing of an image to figure out what is visible from where. That that counts as, you know, in our definition, decision support. Likewise, you could have a system doing medical triage. So you have, you're on a battlefield, you have a bunch of wounded. How do you figure out who to treat first, where to take them, that kind of thing? Likewise, student support or looking for, looking at movements of ships and doing kind of anomaly detection. You could be using, you know, anomaly detection has gotten way better over the last 10, 15 years, could be using kind of upgraded AI systems to do that kind of thing.

• 1:12:32Speaker 01

  And in the paper, we basically sort of look at a whole range of different sort of AI-based decision support systems that are being either used or advertised. So on the, that's on the sort of simple end, the ones I talked about so far. On the more complex end, you have companies like Palantir, also Scale AI, advertising, large language model-based systems that are really trying to be more of what you described of this kind of all-purpose battle buddy. Or at least that's how some of the early marketing looked. They've changed their marketing since then to look a little bit more restricted. But this could be something that some of the early videos would have a huge range of potential functionalities kind of in these demo videos. They might include things that to me like make a good amount of sense, like you're using a natural language interface to access some like clear, clearly documented information elsewhere.

• 1:13:17Speaker 01

  So for example, you've identified and some, you know, enemy movement and you want to know where's your nearest unit, like geographically, where is it? So you could ask that and then it could refer to some database or, you know, some other system and show you, okay, here's the nearest enemy unit. And then you could say, okay, and how many, you know, missiles of such and such type do they have? You get access to that information. In theory, that all makes good sense to me. But these demos are mixing that in with things like, okay, now generate three courses of action that are non-escalatory. And having the AI, presumably the LLM, just like write out potential courses of action for how you could engage this this enemy with what kinds of, you know, tactics from what kind with what kinds of routes. With this caveat of non-escalatory of like, how the hell does the LLM know what's escalatory and non-escalatory? Like, what, who, who is making these decisions?

• 1:14:02Speaker 01

  How is that being evaluated? And these kind of different use cases, or, you know, another one was like, looking at Chinese writing, or you're looking at the writings of some, some country and figuring out, you know, what they think about some set of questions. And these use cases will all kind of be mixed together in these demo videos. And so part of why we wrote this paper was to say, look, this is a category of use for AI that makes a lot of sense. There's a lot of ways that it can make, help the military work better, help clarify, you know, get rid of the fog of war, reduce fog of war. But these systems are not perfect. And there are a lot of ways you could use them that could go badly for you. And so how to think about that? And so, you know, just briefly in the paper, we, we talk about kind of three types of considerations that we suggest be considered. If you're thinking about whether to use a system of these, so the first is scope.

• 1:14:46Speaker 01

  What is the, what is the scope of this system? How tightly bound is it? How well can it be tested for that particular set of activities versus is it more sprawling? Is it more general purpose? Is it more kind of whatever happens to come to mind that you might want to type to your battle buddy? Second is data. So what data has it been trained on? How confident are we that that data reflects the situation that you're in? A huge problem for, you know, military operations in general is both that you're likely to be in sort of situations that are novel. And also that you have an adversary trying to mess with you. So how do you think about whether the data that a system has been trained on are going to be really reflective of the, the real world situation you find yourself in? And then the, the third kind of factor that we talk about, which I think is really, really important and really often neglected is this kind of human machine interaction component.

• 1:15:28Speaker 01

  So how is a system designed to help the person operating it? Uh, understand what it can do, what it can't do, help them make decisions that are, uh, good decisions, help them not overtrust. And there's a long history of, uh, kind of user interface, I think, being undervalued in military circles and, and then contributing to, to unwanted outcomes as well. So, uh, yeah, essentially we're trying to lay out this sort of category of systems and describe both why militaries want to employ them and how they can employ them productively rather than counter productively.

• 1:16:05Nathan Labenz?

  Do you see any stable equilibrium in the future here? I mean, I was struck by, I'm sure you read Dan Hendricks and Alex from Scale was one of the co-authors, Eric Schmidt, the main thesis.

• 1:16:19Nathan Labenz?

  I, sad to say, like, didn't find that super compelling as a idea of like what a stable equilibrium could look like, although I really applaud the idea of trying to articulate something that could be a stable equilibrium. But it just feels like, you know, where we sort of inevitably end up, especially if we don't like get on the same page with, you know, the countries that we're currently most fearful of, is what I'm starting to call alpha go for the army, which is just like self-play, you know, simulation-driven, kind of go to superhuman performance by just like having these things battle it out amongst themselves and he'll climb to a level that human tacticians,

• 1:17:01Nathan Labenz?

  especially when you consider speed, just can't get to. And, you know, that's like one way we get to Skynet and that just seems like a pretty bad situation where we have these sort of inscrutable, but like hyper-lethal systems that we build in theory to defeat an adversary and maybe it even happens that way. But, boy, that seems like another way that we add a pretty scary sort of Damocles to, you know, all of future humanities life and that, you know, without giving them any chance to vote on it, obviously. Is there any way to avoid that, though? I mean, right now it just seems like we're sliding into that and I don't love it.

• 1:17:42Speaker 01

  Yeah, I mean, many thoughts here.

• 1:17:47Speaker 01

  The main paper was really interesting, basically for folks who didn't read it. They had a few different things in there, but the key idea, this main idea, I think was like mutually assured AI malfunction or something like that.

• 1:17:57Speaker 01

  The idea was, which I think is, I think there's a A correct core to this, which is if one country is going around saying, hey, we're going to develop superintelligence and then we're going to rule the world, the universe forever, that's going to create an incentive for other countries to react and respond. And certainly if, like, I think it could be quite stabilizing if it's true that it's much easier to sabotage a AI project than it is to build, to keep building it. That could be a stabilizing dynamic, both because, you know, maybe you have these projects getting sabotaged, but also maybe that affects what kind of projects you undertake in the first place. And in the paper, they get into like, how could you harden your project and make it harder to sabotage? And how many years of development with that? You know, if you have to build your data centers in a mountain, like how much, many more years

• 1:18:40Speaker 01

  does that take, et cetera, et cetera. So I think there's a core logic to that that is like helpful. And I thought was a useful contribution to the, to the discourse of saying like, hey, there's actually going to be, you know, multiple, uh, multiple parties in this decision-making system that you don't just get to say, hey, we're going to race ahead and, and win the race. And the other parties have options beyond just trying to develop their own AI systems. They can actually kind of engage with you in other ways. I thought that was helpful. I agree with you that I don't think it has the force that mutual assured destruction did in the Cold War. I think mostly because of the clarity. The mutually assured destruction was so clear. We knew what nukes did. We knew how they looked. No one wanted to use them. We understood how it would work. You know, once second strike was really guaranteed, we understood what it would look like. Here is just all so much unclear, like what, so much more unclear. What does superintelligence

• 1:19:24Speaker 01

  look like? How much does it matter strategically? So I think, I think of their sort of main idea as kind of a helpful contribution, directionally useful, but definitely not that kind of crystal clear strategic logic that, that they, I think presented it as. To your question about the sort of alpha go for war, I don't know. I don't think we're particularly close to that. I, that just seems like such an intractable simulation problem because you don't just need, you need the battlefield dynamics of a specific battle. You need the broader theater dynamics of, you know, what is where in the world? How are you bringing, how are you bringing your assets to different places? That needs to be connected into broader economic and political questions of what is going on

• 1:20:05Speaker 01

  with the whole world. That needs to be connected to the, you know, the public attitude. It just, I think that is definitely people are trying to build simulations like that. I think they can be useful in, in limited ways, but I don't. I don't know. To me, it brings up this idea that I got from David Chapman, who's a really interesting, I think of him as a philosopher, but he doesn't identify as a philosopher. And he's written a lot of interesting stuff on sort of rationality and what he contrasts it with as reasonableness, meaning the how you make practical decisions in real world situations. Like when you're cooking breakfast, you don't sort of sit down and make a 10 step plan to cook breakfast. You just start and then you see what happens and you adjust from there. And a point that he makes that I think is really correct is that

• 1:20:50Speaker 01

  people in technical domains sometimes think of the messy real world as kind of a rough approximation of some much cleaner, more abstracted system. So like warfare is kind of a messy approximation of chess or of Go or of Starcraft. But in reverse, in actuality, it's the reverse. It's that these clean games, these sort of simplified, abstractable systems are really special cases, very special cases, very unusual special cases of the actual world that we find ourselves in. And I think that, I don't know, this is maybe opening up a whole other can of worms that we sadly don't have time to get into. But I think it seems to me like a lot of the focus on AI development today is focused on assuming that you can build these clean, abstracted systems that are amenable to, for example, reinforcement learning.

• 1:21:33Speaker 01

  And in the meantime, the models continue to struggle with that kind of real world, practical troubleshooting, problem solving, readjusting along the way. All this is a long way to say that I think military simulation is going to be incredibly, incredibly messy. Way too many factors, way too many unknown unknowns, way too much ability for the adversary to deliberately throw a spanner in your assumptions and make your simulation sort of inaccurate. So I don't personally see that as AlphaGo for War as any kind of near term, near term possibility. I think the broader question of what does equilibrium look like? I have no idea. I honestly don't know what the nation state looks like in a world of superintelligence. I don't know what democracy looks like. I don't know what the Chinese Communist Party looks like. So I definitely don't have kind of a broader answer for you there, unfortunately.

• 1:22:19Nathan Labenz?

  Well, that's maybe a great place to leave it. We have more questions than answers. And that's, like it or not, that's the reality. That's the timeline that we're in. Short or long, we've got a lot of questions that remain pretty vexing. This has been great, though. I really appreciate you for humoring me on some of the OpenAI questions at the beginning and also really admire how you've stayed in the arena and to look forward to your continued contributions to try to make the AI future a positive one for us and for our kids.

• 1:22:51Speaker 01

  No, thanks very much. It was a great conversation.

• 1:22:53Nathan Labenz?

  Helen Toner, thank you for being part of the cognitive revolution.

• 1:22:57Speaker 01

  My pleasure.

• 1:23:00Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.