How AI safety took a backseat to military money

This is Hayden Field, senior AI reporter at The Verge — and your Thursday episode guest host. I have another couple of shows for you while Nilay is out on parental leave, and we’re going to be spending more time diving into some of the unforeseen consequences of the generative AI boom. Today, I’m talking with Heidy Khlaaf, who is chief AI scientist at the AI Now Institute, about the tech industry’s shift toward AI military applications. I wanted to know what’s motivated this shift, and why Heidy thinks leading AI firms are being far too cavalier about deploying generative AI in high-risk scenarios. Links: OpenAI is softening its stance on military use | The Verge OpenAI awarded $200 million US defense contract | The Verge OpenAI is partnering with defense tech company Anduril | The Verge Anthropic launches new Claude service for military and intelligence use | The Verge Anthropic, Palantir, Amazon team up on defense AI | Axios Google scraps promise not to develop AI weapons | The Verge Microsoft employees occupy headquarters in protest of Israel contracts | The Verge Microsoft’s employee protests have reached a boiling point | The Verge Credits: Decoder is a production of The Verge and part of the Vox Media Podcast Network. Our producers are Kate Cox and Nick Statt. Our editor is Ursa Wright. The Decoder music is by Breakmaster Cylinder. Learn more about your ad choices. Visit podcastchoices.com/adchoices

• 0:00Speaker 00

  You're busy. We know that, so we'll keep this quick. Introducing AMEX Corporate, now featuring the Corporate Cash Back Card. For one annual flat fee, get 1.5% cash back on eligible business purchases, plus intelligent expense management with the unmatched expertise and backing of American Express. Built to simplify, designed to scale. Get started at go.amex/ammexcorporate. Terms and cap apply.

• 0:31Speaker 01

  New and exclusive holiday decor just launched online at The Home Depot. Check out the viral Grand Duchess Collection with smart holiday lights you can control from your phone. Easily customize the twinkle across trees, wreaths, and garlands to match your holiday vibe. Create stunning light effects, then switch it up with the tap of your fingers. Even the man in red will be impressed. Shop the holiday decor you want, online only, at homedepot.com.

• 1:02Speaker 02

  This episode is brought to you by Michelob Ultra. Good times mean competitions between you and your friends, like seeing who can throw some pigskin the farthest, or rush the field the fastest. Winner gets an Ultra. Because it's not just a superior light beer, it's also giving fans superior access to their favorite sporting events. Superior is worth playing for. Visit michelobultra.com/superioraccess to enter. Enjoy responsibly. Copyright 2026 Anheuser Busch InBev Michelob Ultra Beer, St. Louis, Missouri.

• 1:32Speaker 03

  Hey there, and welcome to Decoder. I'm Hayden Field, senior AI reporter at The Verge, and your Thursday episode guest host. I have another couple of shows for you while Neelai is out on parental leave, and we're going to be spending more time diving into some of the unforeseen consequences of the generative AI boom. Today, I'm talking with Heidi Klaff, who is chief AI scientist at the AI Now Institute, and one of the industry's leading experts in the safety of AI within autonomous weapons systems. Heidi has actually worked with OpenAI in the past. From late 2020 to mid 2021, she was a senior systems safety engineer for the company during a critical time when it Was developing safety and risk assessment frameworks for the company's Codex coding tool.

• 2:15Speaker 03

  But now, the same companies that have in the past seemed to champion safety and ethics in their mission statements are now actively selling and developing new technology for military applications. In 2024, OpenAI removed a ban on military and warfare use cases from its Terms of Service. Since then, the company has signed a deal with autonomous weapons maker Anduril and, this past June, signed a $200 million Department of Defense contract. And OpenAI isn't alone. Anthropic, which has a reputation as one of the most safety-oriented AI labs, has partnered with Palantir to allow its models to be used for US defense and intelligence purposes, and it also landed its own $200 million DoD contract.

• 2:58Speaker 03

  And big tech players like Amazon, Google, and Microsoft, who have long worked with the government, are now pushing AI products for defense and intelligence despite growing outcry from critics and employee activist groups. So I wanted to have Heidi on the show to walk me through this major shift in the AI industry, what's motivating it, and why she thinks some of the leading AI companies are being far too cavalier about deploying generative AI in high-risk scenarios. I also wanted to know what this push to deploy military-grade AI means for bad actors who might want to use AI systems to develop chemical, biological, radiological, and nuclear weapons, a risk the AI companies themselves say they're increasingly worried about. Okay, here's Heidi Klaff on AI in the military. Here we go.

• 3:58Speaker 03

  Heidi Klaff, chief AI scientist at the AI Now Institute, welcome to Decoder. Thank you for having me. First up, I wanted to talk about how AI companies have moved their goalposts a lot with regard to what they're okay with and what their mission statements allow regarding work with the US military and other militaries. So do you remember the whole controversy over Google removing the phrase "don't be evil" from its code of conduct? Yeah, absolutely. It reminds me a bit of something more recent, which is how OpenAI and Anthropic both used to have certain bans on military use of their products, and then they relaxed them. OpenAI walked its ban back in January 2024, when it began to work with the DOD on AI tools. And the month before that, it partnered with Anduril.

• 4:42Speaker 03

  And for Anthropic, it partnered with Palantir in 2024 to offer Claude to intelligence and defense agencies with the US government. So I wanted to ask what you were thinking when you saw these announcements, like made month after month. What did you make of that kind of parade of changes that were happening over a couple of months?

• 4:59Speaker 04

  Well, to many, including myself, the timing didn't seem like it was a pure coincidence of when OpenAI removed the ban in January 2024. If you consider, for example, that Israel at that time was ramping up its mass targeting campaign in Gaza, that we now know is being supported by Microsoft's cloud services that offer AI, and in this case, OpenAI models, as extensions of their IT and cloud infrastructure. And so this rollback was really a signifier on where AI companies were heading, what they were interested in deploying their technologies, despite OpenAI being well aware of the risks that their models posed in defense and safety critical settings, which is something that

• 5:41Speaker 04

  I actually worked on with them in one of our papers together when we were looking at evaluation of models like Codex. And interestingly enough, with the announcement of these collaborations that you mentioned, and that includes Meta, Anthropic, and OpenAI, all announcing US national security work that aligns them with defense contractors like Palantir, Android, and Lockheed. These AI companies never address their previous statements on their stance that LLMs or foundation models are unsafe and insufficient for defense use. So it was almost like a clean slate was being created, where they behaved as if this was always aligned with their mission, right? For example, they started making claims that

• 6:26Speaker 04

  U.S. national security is synonymous with safety under this pretense of an AI arms race with China. And this push for this AI adoption seems quite convenient when you're considering the current unprofitable reality of AI and how expensive it is, where it seems like they're trying to sort of de-risk their portfolio through government subsidies and military contracts. So now we have this complete pivot from banning military uses, all this talking about their main mission being building systems that benefit all humanity, to now their reliance on this narrative of a U.S.-China AI arms race to drive policy initiatives that not only boosts the use of AI, but allows them to sort of avoid

• 7:11Speaker 04

  safety and security scrutiny within military applications.

• 7:15Speaker 03

  That makes total sense. And that reminds me of how, you know, I think last week Senator Warren sent a letter about XAI's own contract with the DOD and expressing concern that the company hadn't done the same level of safety audits as other companies before receiving that type of contract, that they weren't ready. She was worried about how the company would use data it had access to as part of, you know, its government partnerships. What did you make of that letter and kind of XAI's, I guess it seemed grandfathered in kind of approach to this DOD contract announcement?

• 7:55Speaker 04

  I think the way that I see it is it's part of the trend of the USUD not recognizing that there's a national security risk with the use of commercial foundation models in the military, because they do significantly expand the tac vectors of military systems and defense infrastructures that they interface with, because commercial models are unvetted. They don't have a supply chain that follows the typical military supply chain, and they can be compromised. In a lot of ways. So to me, sort of the contract with XAI is another risk that's added on to that, that sort of stems from the issues that all commercial models have, right?

• 8:38Speaker 04

  And obviously, depending on sort of the platform that these models are trained on and the personal data that these models are trained on, they come with a lot of risks and capabilities that allows them to promote or even deploy surveillance systems because they're able to use data that other companies may not have. For example, XAI has a huge amount of data, you know, could be from not just public posts, but also private messages of their users. And what does that mean for that to be used in military applications, right? There's a legitimate concern here, but there's also this larger concern that these systems

• 9:17Speaker 04

  are also unsafe and trained on data that can have been compromised by an adversary, including China, right? And that sways the way that the AI system behaves. So there's like risks on both sides here, right? From both the data that is meant to be protected, right? And also the type of data that could have been compromised and then thus change the behavior of an AI system that's being used in something like very sensitive military operations.

• 9:48Speaker 03

  So something that comes up a lot in my reporting and in my conversations with other people, even my pitching my editor, is that a lot of these companies are pre-profit, I mean, maybe all of them. And we're seeing a bunch of them unveil government products, enterprise products, and that seems to be where a lot of the money lies. So, you know, obviously, OpenAI, Anthropic and XAI have all unveiled government products designed for US defense and intelligence agencies to use. They also all received, you know, those government contracts from the DOD. So with companies that burn through cash at super high rates, do we think that the government play is about seeing cold, hard cash come in finally or staying above regulatory pressure or both? I'd love to hear your thoughts on that.

• 10:34Speaker 04

  It's definitely both, right? Because, as you mentioned, these companies are pre-profit and there's a really big pot of money in the military industrial complex. There simply is, and I think that's well-known. But also, there's the aspect that these companies would not traditionally pass any of the testing and evaluation required for military procurement. And here's the thing that a lot of people don't know, is that defense and military procurement is actually some of the most strict, you know, prior to this AI era, I mean. To sort of evaluate these systems, they have some of the most strict standards. And I think a lot of people assume that's not the case,

• 11:16Speaker 04

  but often our safety critical systems, if you're talking about like energy infrastructure, you know, and so on and so forth, is derived from those defense standards because of how robust they are. The thing with AI systems, and we're talking about generative AI systems, because AI systems have been used in the military for decades at this point. But these foundation models, or large language models, whatever it is that you want to call them, they do not meet the sort of very basic threshold that is typically expected for a military system, right? And so there is this kind of issue now that they want this pot of money. You know, as I mentioned before, they're trying to de-risk their portfolios through military contracts,

• 12:00Speaker 04

  but they have this issue where safety as defined by defense and safety critical system is too stringent for their systems to meet just by their nature, right? They're highly inaccurate systems. And when you're looking at, I can't really get into defense systems, but I'm going to talk a little bit about like safety critical system, if you're looking at like a nuclear power plant, for example, you're looking at safety of like 99%, right? That's like the minimum. And often, the accuracy of AI systems is like 60% if I'm being optimistic about specific types, right? So there's an enormous gap. Here to make AI systems as they exist for foundation models be able to satisfy the strict testing and evaluation

• 12:45Speaker 04

  measures often required by military procurement.

• 12:48Speaker 03

  For the listeners, let's just define military procurement really quick. What is that?

• 12:53Speaker 04

  Military procurement is, well, it really depends, right? There's a huge amount of processes that exist for different types. And the process is often strict depending on how critical the technology is going to be used, like, for example, if it's going to be used for lethal operation versus bureaucratic operations. Very different types of procurement. If you, if we are looking at a sort of more general idea, typically the government puts out a specific ask for the type of systems that they're looking for, and people submit to that, you know, procurement ask. Ultimately, these systems often have to go through what we call a testing and evaluation process for them to even be considered, you know,

• 13:37Speaker 04

  even before they sign the contract, right? This process is quite stringent in that it has their specific thresholds on how accurate these systems need to be and how secure they need to be. Often, the security thresholds is extremely, extremely high, right? They have to be air gap, the supply chain has to be completely traceable. They have to know who coded the system, who developed the system, if there's any sort of backdoors that can be compromised, so on and so forth. And once sort of a system goes through that procurement, that safety and security procurement process, the DOD often just takes complete control of that technology, right? Like, this is now something that they possess and are in complete

• 14:19Speaker 04

  control of using in whichever way that they see fit, right? And so this is often why procurement for the military takes, can even take many years, right? This is not a process that is meant to take. A couple of weeks or even several months, often this is quite a rigorous process. So this is very different from signing a commercial contract, right? Where you have traditional terms of service. The nation state, not just, you know, the US UUD, often are the ones that get to define the terms of how they want to use the technology. And typically, people abide by it because people want that pot of money, right? It's very lucrative to even be considered up for procurement because it means that you're sort of be on call for them for potential other technologies. But even to get your foot on the door can take years.

• 15:03Speaker 04

  So it's a very different type of assessments than I think what people expect for commercial contracts.

• 15:13Speaker 03

  We need to take a quick break. We'll be right back.

• 15:24Speaker 01

  New and exclusive holiday decor just launched online at The Home Depot. Check out the viral Grand Duchess collection with smart holiday lights you can control from your phone. Easily customize the twinkle across trees, wreaths, and garlands to match your holiday vibe. Create stunning light effects, then switch it up with the tap of your fingers. Even the man in red will be impressed. Shop the holiday decor you want, online only, at homedepot.com.

• 15:56Speaker 02

  This episode is brought to you by Michelob Ultra. Good times mean competitions between you and your friends. Like seeing who can throw some pigskin the farthest or rush the field the fastest. Winner gets an Ultra. Because it's not just a superior light beer, it's also giving fans superior access to their favorite sporting events. Superior is worth playing for. Visit michelobultra.com/superioraccess to enter. Enjoy responsibly. Copyright 2026 Anheuser Busch InBev Michelob Ultra Beer, St. Louis, Missouri.

• 16:25Speaker 05

  This episode is brought to you by Subaru. You know that place your mind always wanders to? The lake, the mountains, the homes of the people you love most? The completely redesigned 2026 Subaru Outback is built to help you get there. With legendary capabilities, standard symmetrical all wheel drive, and the confidence to go further. The 2026 Subaru Outback. Love never looked so good. Visit subaru.com/outback to learn more.

• 16:54Speaker 03

  We're back with Heidi Klaff of the AI Now Institute. Before the break, Heidi was breaking down the standard military procurement process and why it feels like AI systems don't meet the rigorous standards we might expect when it comes to being used for high-risk operations. Now, I want to ask Heidi about the specific AI products being sold to the US military and whether they're really much more secure than the commercial models on the market today.

• 17:19Speaker 03

  I also wanted to ask, the models these companies use for government products, like their government-designed products, like Claude Gov, OpenAI's government product, XAI's government product, by design have looser guardrails for government use and they're trained to better analyze classified information. And although these types of models allegedly underwent the same type of safety testing as these companies' other models, I'm using Anthropic here as an example, they have certain specifications for national security work, like they have a greater understanding of intelligence and defense documents and they refuse less when they are asked to engage with classified information that's being fed into them.

• 18:03Speaker 03

  So, in your eyes, how does the development work? How secure really are they and what are the implications here?

• 18:09Speaker 04

  So I wouldn't say they're much more secure. They may be more secure in that they're more air-gapped. So for example, you can take a commercial model, right? And you can fine-tune it on a sort of sensitive military data and then that model then becomes accessible to the military. But that still misses out on some of the biggest risks of that commercial model is that it was trained on data sets that were publicly available. And so, a lot of research has shown that not only can you poison web data that these models are trained on, but you can implement what's called like a sleeper agent, which is... Given a specific prompt or a command, it will then behave in a sort of a harmful way

• 18:53Speaker 04

  that sort of the operator of that system did not intend, based on something that was implemented in the training data or something that, you know, the model was trained on. And so we see this all the time with like prompt injections, but this can happen on a sort of deeper level with what we call sort of web poisoning attacks, which can then be used to implement these like sleeper agents, as we call them. And so this is in the commercial supply chain, right? The only way that these models are trained is to be trained on sort of mass amounts of data that are publicly available. So they're already compromised, right? These models are also fine tuned through methods like reinforcement, learning, human feedback, which unfortunately uses basically sweatshops

• 19:37Speaker 04

  of people in developing nations that are paid nothing to then make these models behave in a specific way. And you can imagine a military operation, right? Where some and a foreign adversary has is able to sort of have a covert operation in which they run one of these data labeling and data fine tuning shops essentially, and are sort of aware that they might eventually be used to be fine tuned for military application and implement backdoors or sleeper agents, which trigger a specific behavior based on a specific command. And because of that, that's what makes them so unsafe. So sure, you might be able to fine tune it on specific data that isn't then released publicly,

• 20:21Speaker 04

  which might remove some vectors of attack. But ultimately, at the end of the day, commercial models are already compromised. So when you're saying that they are more secure, I mean, they're more secure in a traditional security way in that you air gap the system. So you kind of limit the control of people who have access to it. And so thus, people who can probe it and get information out of it. But it doesn't remove the fact that commercial models are already compromised from the day that they're built because they're based on public data.

• 20:49Speaker 03

  I wanted to see if you agree with this take I'm about to tell you. So I once interviewed Meg Mitchell from Hugging Face. And she said that for these types of military contracts, even if you have in your mission statement, like, you know, Anthropic and OpenAI do, that your tech can't be used to directly harm others. The problem is that you don't have control in the end over how your tech is actually being used with the military. If you do have any control, you definitely don't have control in the longer term once you already shared that with the military organization, especially without having security clearance and knowing really how it's being used down the line. She also was talking about what's considered direct harm. You know, what if you're summarizing social media posts that then lead to making a list

• 21:34Speaker 03

  of enemy combatants or potential people of interest that have a certain view on a topic, on X, for example. I wanted to see if you agree with that in terms of, you know, these companies often have in their mission statements, oh, don't worry, even though we're working with the military on this, this, and this, we know for a fact that our tech isn't being used to directly harm people. But yeah, how can they really know that, can they?

• 21:59Speaker 04

  I completely agree with that statement. And I think something that often people miss out on is that militaries do not follow the terms of service. They might do that if they're buying like a Microsoft Office suite, right? For their bureaucratic purposes. But when it comes to military procurement, the companies do not have control and they know that over how these systems are being used. And they actually have no say in terms of the terms of service as well. These things get often determined by international law and also by the nation state itself, right? They have the power here. And so when people tell me, oh, but wouldn't that break the terms of service? And it's like, this is not how military procurement works, period, right?

• 22:43Speaker 04

  And we've also seen examples of, as I mentioned before, Microsoft working directly with militaries to implement some of these systems. So I would say that in a lot of cases, they are well aware of how their systems are being used to some extent, right? We don't know all the details, but governments put out procurement documents of the type of data they want, how they want to use it, and how they want to store it. Because then companies can offer services like what their AI can do with that, right? So I do think that it's not the case that they're just selling something commercially like they do to everyone else and then they hope the military abides by it. It's a much more involved process to do military procurement that often requires testing and evaluation.

• 23:26Speaker 04

  And the companies typically have to be in the know about the technical details to see if they can offer support for that. Again, as I use a Microsoft case as one of the most recent examples of that being the case with their work with IDF. So I think it's easy for them to point to terms of service, but as someone who has worked on procurement before, this is not a commercial contract that these companies are signing.

• 23:51Speaker 03

  Okay, so now I want to get into the CBRN side of things. Although, as we've talked about, you know, it may not be as big of a concern as AI companies are making it out to be, it is a big concern for the public, probably because of that marketing and just because it's a scary idea. So let's be real here. Obviously, smarter AI that can do anything for you isn't always good, especially when people want to use it to do bad things like creating chemical, biological, radiological, and nuclear weapons. So top AI companies say they're increasingly worried about the risk of that. Of course, they're not maybe worried enough to stop building, but I want to get into again how big of a risk this is. Let's just go into more detail there.

• 24:35Speaker 04

  We have not seen any proof of CBRN capabilities right now, but those capabilities could come to fruition if we start training very, very sensitive nuclear data, for example, nuclear technologies on these models. And I actually believe that the risk that comes with that is very different from what most people are thinking about. Most people are thinking about that the AI is somehow going to develop weapons by itself, right? Or it will give access to adversarial actors to do so. But even if it's just a military who has access to a model that has been trained on CBRN data, what that means is that they are likely going to use it for those purposes within the military. And that's extremely dangerous.

• 25:20Speaker 04

  Like if you're thinking about nuclear command and control, right? Who gets to essentially make the decisions about nuclear weapons deployment? And it certainly shouldn't be AI systems because regardless of the data distribution, these systems are highly flawed and they're always going to have inaccuracy. As I mentioned before, Often when we're looking at military systems that deploy AI, they can have as low of an accuracy rate as like 20%. And if you're being really optimistic, maybe 60 to 80%, right? These are the levels of accuracy that you're looking at with these types of system. And so then to train a model on CBRN data and then to then attempt to use it for those tasks is extremely dangerous when you're looking at those accuracy rates, right?

• 26:05Speaker 04

  For me, the concern that I have is that they will then think that these models are reliable because we trained them on that set of data. And thus we can then use them and the military and in defense operations to dictate decisions about those types of systems and where they should be used and when. And I think, you know, that's a very different type of risk than most people are thinking about. As before, this idea that like they're somehow going to gain CBRN capabilities by themselves, very hypothetical and not really like tied to the reality that we're in right now with AI systems. But if we're taking AI systems today and we train them on on sensitive military data, I have a concern of how those systems are going to be used.

• 26:48Speaker 03

  We need to take another quick break. We'll be right back.

• 26:59Speaker 01

  New and exclusive holiday decor just launched online at The Home Depot. Check out the viral Grand Duchess collection with smart holiday lights you can control from your phone. Easily customize the twinkle across trees, wreaths, and garlands to match your holiday vibe. Create stunning light effects, then switch it up with the tap of your fingers. Even the man in red will be impressed. Shop the holiday decor you want online only at homedepot.com.

• 27:30Speaker 02

  This episode is brought to you by Michelob Ultra. Good times mean competitions between you and your friends, like seeing who can throw some pigskin the farthest or rush the field the fastest. Winner gets an Ultra. Because it's not just a superior-like beer, it's also giving fans superior access to their favorite sporting events. Superior is worth playing for. Visit michelobultra.com/superioraccess to enter. Enjoy responsibly. Copyright 2026 Anheuser-Busch InBev Michelob Ultra Beer, St. Louis, Missouri.

• 28:00Speaker 05

  This episode is brought to you by Subaru. You know that place your mind always wanders to? The lake, the mountains, the homes of the people you love most? The completely redesigned 2026 Subaru Outback is built to help you get there. With legendary capabilities, standard symmetrical all wheel drive, and the confidence to go further. The 2026 Subaru Outback. Love never looked so good. Visit Subaru.com/Outback to learn more.

• 28:29Speaker 03

  We're back with chief AI scientist Heidi Klaff, discussing the ways in which AI companies are pushing into defense contracting. Before the break, we were talking about how real the risk is that AI systems might be used to develop nuclear or biological weapons. But now I want to zoom out and talk to Heidi about the broader field of AI safety and how she thinks it's changed since she worked with OpenAI years ago.

• 28:54Speaker 03

  Let's shift and talk about AI safety for a bit. So you helped establish and pioneer the field of AI safety engineering. What's the technical meaning of safety? Like with your background? And how has the AI safety world changed that meaning? Or how has it become more colloquial now?

• 29:12Speaker 04

  So if we take a step back and not think about what the AI companies have been telling us what safety means for the past four years, or even more than that at this point. Safety has historically meant, especially in the context of safety critical systems, ensuring no harm to humans or the environment. So if you're thinking about aviation or nuclear power plants, for example, you want to ensure that when your systems fail, and systems do fail, that humans are not harmed, that there's no death, and that there's no environmental catastrophe. It's quite a simple definition. Now, what is happening in terms of what safety now means is very different. And it has been redefined by AI companies as of late.

• 29:57Speaker 04

  So I believe AI labs engage in what I call safety revisionism, where they use the same safety terminology. That are often used for regulating and assurance, defense and safety critical systems, but instead redefine those safety techniques with washdown alternatives that actually accelerate the deployment of inaccurate AI in high-risk scenarios like defense or nuclear. So for example, AI companies often reduce the term safety to now mean alignment or existential risks. Now, this is pretty distinct, right, from what the definition of safety that I just gave you, because alignment focuses on human preference, right?

• 30:40Speaker 04

  And that makes us question, well, alignment with whom and which humans, right? And whose preferences? And existential risks that are also emphasized, like CBRN, are hypothetical, like I mentioned, and are often used as sort of a pretense for an AI arms race to ignore other risks and safety thresholds like surveillance systems, right? So in allowing AI companies to do this, right, which a lot of governments have, they've sort of ceded that control of defining what safety is to them, puts them in a position to define what a risk threshold is or what actually safe enough means. And entire idea of risk thresholds, because I imagine a lot of people might not know

• 31:19Speaker 04

  this, is to provide sort of a metric or a measure of the level of risk exposure that our society collectively agreed to take. And this often shapes how we determine the safety of technological systems, including nuclear plants. And typically, this is done through a democratic process that we have established over decades and in other high-stakes scenarios. Like all of our thresholds for other safety critical systems have come from sort of democratically determined idea of what society thinks safe, that safety is. So in allowing AI companies to sort of co-opt these traditional safety terms, we've sort of given them permission to not only decide what counts as safe enough, which again breaks

• 32:02Speaker 04

  these democratic norms that we've had, but it also lowers and undermines existing safety threshold that would have otherwise regulated AI use in things like defense. So ironically, you know?

• 32:16Speaker 04

  This is kind of their way of how they bypass some of the safety measures that I've talked about earlier in that they're looking for this pot of money, right, from the military, but the safety thresholds for defense are extremely high. So what do you do? Well, you redefine what safety means and you say it's different for AI. You say because our systems are so different, they're at a scale we've never seen before, we cannot abide by these safety rules, which is definitely not accurate. I think a lot of our existing safety critical standards hold for AI systems. And ironically, this hollowing out of safety, although being sold as crucial to win the AI arms race, can't be regulated, we have

• 32:57Speaker 04

  to beat China, is accelerating AI adoption at the cost of more unsafe and insecure systems, which may be exactly what disadvantages the US military and our technological capabilities against China. If we're sort of letting inaccurate and easily compromised systems be deployed in our front lines, because it's profitable for these AI companies.

• 33:21Speaker 03

  Let's talk a little bit about, earlier you mentioned, safety of 99%, for example, at a nuclear power plant. What does that mean in context? We talked just now about the meaning of safety in that regard, but what would a safety of 99% entail? It's that there's a 99% chance that it won't harm people or the environment? Or, you know, what does that mean in practice?

• 33:45Speaker 04

  Yeah, I mean, it's a lot more technical than that. But basically, right, we have these thresholds of these systems have to be accurate and be able to perform. So these are typically what we call reliability and availability measures of the system. So they can only fail. Often even, 99% is like one of, it's like the lowest threshold for a nuclear plant. It even goes up to 99.99%, right? And so, obviously, if we allow zero risk, we're never going to build anything, right? Like, I think that's very important to remember is that with every technological system, that there is some sort of risk. But you have to mitigate for when those systems fail. So this idea that our safety critical systems have this like 99.99% reliability means that

• 34:31Speaker 04

  They're meant to operate basically well ninety-nine to ninety-nine point nine nine nine percent of the time, depending on the kind of system that you're looking at, right? And the safety criticality. And then when that system fails, we then have to have mitigations in place, right? And there will be risks with that. And typically, these mitigations are based on, like I said, these thresholds of how many people could be harmed. So in the case of like airplanes, I think that's a very simple example. A catastrophic incident is considered if everyone on a commercial airplane dies. So typically, that number is like three hundred people get harmed or die. That's the threshold for aviation as being the most catastrophic thing that could happen. So safety is actually very specific to the use cases.

• 35:15Speaker 04

  What we mean by ninety-nine point nine nine percent reliability often relates to these systems failing. But if you're looking at how to actually mitigate for those risks and what the threshold is, that depends on every single field because kind of the impact of the system will vary. An airplane crashing is very different from a nuclear plant crashing. So the nuclear plant doesn't have this idea that three hundred people dying is the worst case scenario. In fact, it's much more than that and also has to do with environmental like nuclear disaster, right? And so this is why this idea of AI and the safety that they push forward is problematic because they want us to adopt this idea of universal or general safety that has to

• 36:00Speaker 04

  do with like, as they call it, alignment. And it's this misguided idea that there exists like a universal safety solution that would make all general functionality of all L L Ms safe, right? And this is also one of the ways that, you know, procurement and training in the military, when you're looking at companies like Scale AI, they are putting forward these types of general frameworks. But there is no standard safety approach to generic systems in any domain. In fact, this would contradict established safety practices that require sort of a well-defined use case to map risks against. And so. Often what we're seeing now happen is that companies like Scale AI, they say, we're going to build a risk assessment framework for AI systems because existing ones simply don't work.

• 36:46Speaker 04

  Like, I'm being sarcastic, that's not the case. And then the way that they define safety, again, is through the safety revisionism, right? They call it something else, it ends up being about something else, and completely disconnected from actually being accurate for military operation. It ends up being, again, these high-level ideas of safety that we're seeing them push, whether it's about like CBRN. It's like, right, but can it do the thing that we're asking the AI systems to do? You actually never see an assessment of that in a lot of these frameworks. And so this is sort of why this idea of safety becomes very confusing, because it has diverted so much from how we've traditionally used it

• 37:28Speaker 04

  to assess like nuclear plants or airplanes and so on.

• 37:33Speaker 03

  And you led the safety evaluation of Codex at OpenAI. So what was that like? And would it be a different process, do you think, if you were leading that work now?

• 37:42Speaker 04

  For Codex, the idea was to introduce something like a risk assessment for AI, which is not what people were doing before. Prior, there was like a lot of benchmarking, right? And these benchmarks didn't really consider the risks that the AI system imposes with having specific capabilities. So the idea was to really try to investigate that and use some techniques inspired from safety critical fields. It was not meant to be a replacement for assessments for safety critical systems, right? And I think that's a really, really big distinction. And in terms of like, what would I be doing now? It was my choice to not continue working with OpenAI,

• 38:26Speaker 04

  because it became very clear to me, again, this idea that they're pushing of general safety just does not align with. How safety actually should be assured in sort of the real world, right? And so this idea of existential risks, CBRN alignment, to me was like, no, but these are not the current harms that we're going to see if we're going to deploy AI systems in these safety critical situations. And if we are going to deploy them in safety critical situations like defense, we have to assess these systems as we always have for every other system. I thought introducing something like risk assessments would be helpful to the field because then people could understand the risks that come from using the systems. But

• 39:07Speaker 04

  what it ended up unfortunately evolving to in being used by many labs is that these risk assessments are now sort of being used like the end all be all of all assessments of AI being used in all systems everywhere. And that I regret very much, but that was never sort of the intention to begin with when we set out this work.

• 39:27Speaker 03

  Was there one thing or a couple of different examples of what made you kind of decide not to continue? Do you remember anything specific?

• 39:34Speaker 04

  Yeah, I think it is the existential risk, this concern that AI will somehow become self-aware or have these capabilities that lead to nuclear proliferation. And as someone who has worked on sort of risk assessments now for about a decade, you have to have real data to back up your claims. And so when you're then using risk assessment frameworks to try to substantiate hypothetical claims that there are no proof for, you're not doing science. And to me, I'm willing to be convinced that perhaps AI models could have CBRN capabilities in the future. I am not opposed to that idea, but they don't have them now. And so for all of us to put our safety and regulation efforts, right, and that includes by the US

• 40:18Speaker 04

  government and the UK governments, to be about hypothetical risk that can't be measured, right, that can't be quantified or qualified. And our entire regulatory system then becomes about risks that we have yet to see, you might as well not have regulation at all, right? So I think a lot of people talk to me about, well, what do you think of this framework? And what do you think of that framework? I'm like, to me, practically speaking, as someone who has done risk assessment, it is equivalent of having no regulation because we're actually not addressing the risks of the harms that AI is posing.

• 40:53Speaker 04

  And we're in fact focusing on hypothetical risks. And there's this idea that I've heard before, well, what if those risks come true and you're unprepared? And the way that I see it is that if you're not prepared for today's risks and you're not building the frameworks for that, you're not going to be prepared for future risks because these frameworks and risk assessments built on top of each other. So if you're not able to mitigate for the lack of safety and security of AI models today, then you have no chance of mitigating against these hypothetical risks that people like to bring up, right? Because that is kind of one of the core concepts of safety is the smallest catastrophe, not the smallest catastrophe, like the smallest hazard can cascade into a large catastrophe.

• 41:36Speaker 04

  So if you're not able to address the very, the things that are considered, you know, they consider this stuff inconsequential, then you're never going to be able to prepare for these, you know, much more large scale events that they talk about. And that's very much like a standard safety perspective to have.

• 41:51Speaker 03

  The snowball effect in practice. Well, thank you so much, Heidi. This is incredibly helpful and, you know, your perspective is so unique. So I'm really glad we were able to, you know, talk about this and have the audience kind of weigh in in comments and stuff. I think this is, you know, something that's not talked about enough. So I'm really glad we were able to talk and thanks for making you the time and moving your schedule around.

• 42:12Speaker 04

  Thank you for having me.

• 42:17Speaker 03

  I'd like to thank Heidi for taking the time to speak with me and thank you for tuning in. I hope you enjoyed this episode. If you'd like to let us know what you thought about this show or what else you'd like us to cover, drop us a line. You can email us at decoder@theverge. We really do read every email or hit me up directly on X, Blue Sky or Threads. I'm at Hayden Field on all platforms. Decoder also has a TikTok and Instagram and now also a YouTube channel. Check those out at DecoderPod. They're a blast. If you like Decoder, please share it with your friends and subscribe wherever you get your podcasts. Decoder is a production of The Verge and it's part of the Vox Media Podcast Network. Our producers are Kate Cox and Nick Statt. Our editor is Ursa Wright. The Decoder music is by Breakmaster Cylinder. See you next time.

• 43:04Speaker 01

  New and exclusive holiday decor just launched online at The Home Depot. Check out the viral Grand Duchess Collection with smart holiday lights you can control from your phone. Easily customize the twinkle across trees, wreaths, and garlands to match your holiday vibe. Create stunning light effects, then switch it up with the tap of your fingers. Even the man in red will be impressed. Shop the holiday decor you want, online only, at homedepot.com.

• 43:34Speaker 06

  At Zocdoc, we know you care about your health. You wouldn't do that early morning cardio if you didn't. So the fact that your annual physical is overdue just shows how hard it can be to get seen by a doctor.

• 43:45Speaker 07

  Sorry, that doctor's not in network.

• 43:47Speaker 08

  Press 1 for more options. You've reached us outside normal business hours.

• 43:51Speaker 07

  The next availability is in three months.

• 43:53Speaker 06

  But Zocdoc makes finding the right doctor easy. Download the app and search by specialty, insurance, and availability. Read real patient reviews, then book instantly. Zocdoc, ready to be seen.

• 44:04Speaker 09

  Mr. Monopoly here. Miss me? Monopoly is back at McDonald's with millions of prizes. A 2027 Jeep Cherokee, a Carnival Cruise vacation, gift cards from DoorDash, and Best Buy. You'll then play in the McDonald's app. Get your bag!

• 44:22Speaker 10

  No purchase necessary. 50 U.S. and D.C. age 18+. See playatmcd.com for rules and playatmcd.com/amoe for info on playing without purchase by mail by November 2nd. Chance card game ends November 8th. Game ends November 22nd, 2026. See rules for prize details. Monopoly is a registered trademark of Hasbro.