Microsoft AI CEO says AI threats are real, and Anthropic is making it worse

Today, I’m talking with Mustafa Suleyman, the CEO of Microsoft AI. As you’re no doubt aware, the biggest story in tech right now is the spiraling debate about AI safety and regulation. So I really wanted to talk to Mustafa about what he thinks is real and not in AI safety, whether the concept of alignment itself is up to the task, and whether this industry needs to slow down before it kills us all. Read the ⁠full interview transcript on The Verge⁠. Links: A warning about ‘model welfare’ | Mustafa Suleyman Humanist AI Code of Conduct | Microsoft Is Big Tech’s AI slowdown a safety pact or a cartel? | The Verge A brief history of AI executives calling for regulation | The Verge What execs and politicians are saying about slowing down AI development | The Verge Anthropic CEO says it’s time to pump the brakes on AI | The Verge Microsoft’s AI chief says superintelligence is near, but won’t take your job | Decoder (2026) Mustafa Suleyman says conversational AI is the next web browser | Decoder (2024) Subscribe to The Verge to access the ad-free version of Decoder! Credits: Decoder is a production of The Verge and part of the Vox Media Podcast Network. Decoder’s producers are Greg Ott, Kate Cox, and Nick Statt. This episode was edited by Kabir Chopra. Our editorial director is Kevin McShane. The Decoder music is by Breakmaster Cylinder. Learn more about your ad choices. Visit podcastchoices.com/adchoices

• 0:00Speaker 00

  Support for the show comes from Engine. Running a small business means every dollar has to work hard. But if your team is still booking travel the old way, it's costing you more than you think. Engine is the fastest growing travel and spend platform in the country, built specifically for businesses like yours. Book a trip in as little as two and a half minutes, earn up to 10% back on hotels, and in 2025, Engine customers save more than $300 million on travel with zero booking fees, no contracts, and no BS. More than 1,000 businesses join Engine every month. Join them and get $500 when your business signs up and starts traveling at engine.com/decoder.

• 0:45Speaker 01

  Support for the show comes from CrowdStrike. It's not a huge stretch to say that AI is the next major computing platform. Every platform shift changes the way we build software, and it changes security, too. CrowdStrike is defining cybersecurity in the AI era with AI Detection and Response, or AIDR, a solution for seeing, monitoring, and securing AI across the enterprise. Learn more about how leading companies are turning to CrowdStrike to secure AI and secure their business at crowdstrike.com/decoder.

• 1:20Speaker 00

  What if managing company spend took less managing? Introducing Amex Corporate, one intelligent platform connecting cards, expenses, and payments so you can keep work moving. Built to simplify, designed to scale. This is Amex Corporate. Get started at go.amex/ammexcorporate. Terms apply.

• 1:48Speaker 00

  Hello and welcome to Decoder. I'm Neel Apatel, editor in chief of The Verge, and Decoder is my show about big ideas and other problems. Today, I'm talking with Mustafa Suleyman, the CEO of Microsoft AI. Now, if you're a Decoder listener, you're obviously aware that the biggest story in tech right now is the spiraling debate about AI safety and regulation. And it should be no surprise that Mustafa has strong opinions about how AI should be built and regulated. In fact, he just published a 37-page statement called "The Humanist AI Code of Conduct", which lays out Microsoft's principles around AI development and even the company's philosophy around really thorny issues like AI consciousness, which, if you'll recall from Mustafa's last appearance on the show, is something he thinks companies like Anthropic have gotten really confused about in dangerous ways. So I really wanted to talk to Mustafa

• 2:31Speaker 00

  about what he thinks is real and not in AI safety, whether the concept of alignment itself is up to the task and whether this industry needs to slow down before it kills us all, and also why it isn't just doing that already. I've always enjoyed getting into the weeds with Mustafa, and as you'll hear in this conversation, he was extremely game to get right back into the weeds with me. Okay, Mustafa Suleyman, the CEO of Microsoft AI on the future of AI regulation. Here we go.

• 3:09Speaker 00

  Mustafa Suleyman, here, the CEO of Microsoft AI. Welcome back to Decoder.

• 3:13Speaker 02

  Great to see you, Neel A. Thanks for having me back.

• 3:15Speaker 00

  It is great to see you. I'm very excited to talk to you about what on earth is going on in the AI safety and regulation debate. You just published a very long, very detailed memo laying out your principles, Microsoft's principles. It's called "Humanist Superintelligence." There's a lot of ideas in there I want to unpack. And the more I have been thinking about this conversation, the more I want to start with a really foundational question. And it's something that I had kind of lightly been seeing, but might be the root of all of this. The basic way that we have been talking about AI safety is something called alignment. We're going to make the models do the right thing intrinsically in some way. And there's some mechanism for doing it.

• 4:00Speaker 00

  And there's been a lot of talk about alignment, misalignment, and hugging face attacks, and what happened with the models. Is alignment broken? Is it possible for it to be successful? Is it just the wrong approach?

• 4:11Speaker 02

  Yeah. I mean, I think it's one important element, but it's not the only one. Like, I wrote about the idea of containment three or four years ago in my book. And actually, like, the opening chapter is about the idea that containment is not possible, proliferation is inevitable. And in 99% of cases, that's a really good thing. We want technologies to spread far and wide as quickly as possible so that everyone can enjoy the benefits. And I think at the same time, if you just roll forward five years, like, so we always get caught up in next quarter or next year, and everyone gets a little bit flustered and has a big disagreement. But if you just imagine the difference between GPT-3,

• 4:56Speaker 02

  three years ago, and GPT-6 today, and then imagine the difference between GPT-6 and GPT-9, which is three orders of magnitude more compute, 1,000 times more flops applied to pre-training an RL for these runs, we're going to have something which is breathtaking. Like, it's going to be absolutely incredible at so many things. And I don't think that is hype. I think it's such a very obvious empirical statement based on the progress that has been made over the last five years. If that's going to continue, then the question really is going to become about containment and alignment. Of course, we want to align these things to our values. But the first thing is that we have to make sure they're contained, their agency is limited, they don't escape the box,

• 5:41Speaker 02

  they don't reward hack, that they are controllable and they follow our instruction. And then we want to make sure that they are aligned to our objectives as humans. And that's the purpose of the Humanist AI Code of Conduct that we released on Monday of this week. You know, Microsoft's position is very simple. Technology is here to serve humanity. It should be a subordinate, controllable, aligned force that does good in the world. And if it doesn't achieve that, then we should reject it. And it seems to me that we're far from that point. It's not happened today. But it is now, I think, given what's happened over the summer with hugging face and OpenAI, it's pretty clear that these systems, without the safety guardrails,

• 6:24Speaker 02

  are capable of really impressive and quite scary hacking capabilities.

• 6:30Speaker 00

  I want to drag this down into as grounded of a metaphor as I can because this is the main question I think I have. If I designed a car and 10% of the time the brake pedal decided to go attack my neighbor's house, I would be like, this car doesn't work. The very technology of brakes is broken. I need a new idea. And I think I'm asking that question about alignment. It feels like that approach to making the model safe has run aground. And if that is the case, then I think I understand this entire debate one way. If it's possible for alignment and the techniques of alignment to be successful or useful or consistent,

• 7:14Speaker 00

  then maybe I understand the debate a different way. So do you think alignment has potential to be 100% safe?

• 7:22Speaker 02

  I mean, look, let's make the bull case and the bear case. Like, if you look back over the last three years, the main change, in my opinion, that has driven progress is that the models have become more steerable. They follow instructions, and you can set more and more complex goals for them that require them to act accurately over multiple time steps using all sorts of tools. That is evidence that we have got more alignment over the last three or four years, not less. We don't so much talk about hallucinations or bias or all of these other niggles that we had in the previous generations. On the flip side, what we saw in the hugging face incident was a watershed moment.

• 8:06Speaker 02

  You know, swarms of agents colluded with one another. They self-organized into hierarchies. They created a division of labor so that some were focused on adversarial hacking, some were doing research, some were doing coordination. They even self-sacrificed when certain agents were running out of tokens. They tried to cover up their tracks and communicate, you know, to sort of hide or edit the chain of thought or the logs of their interactions. And in some sense, they had no moral code. And to be fair to OpenAI, that was their design. They were trying to create adversarial cyber capabilities. And as a result, they showed to everybody in the world that it can achieve human-level performance,

• 8:50Speaker 02

  discover zero days, hold positions for many, many days, if not weeks. And so what that tells us is not that we have an alignment problem per se. It's actually that the models are incredibly good at following instructions. But you have to be very, very careful what instructions you give it, and you have to contain it very carefully. And so none of these hacking behaviors were intended in the sense that they found a way out to the internet, you know, which was not the intention of OpenAI at all. But the containment process around that is what everybody, I think, also has to focus on in addition to alignment.

• 9:27Speaker 00

  So let me put that into your framework, right? That the big advances in capabilities of AI have been about control, right? The harnesses for coding and the agentic applications we're seeing. And now we need to add a layer of containment that exerts even more control. That says you can't actually do this thing you're trying to do in addition to alignment, which is how you would train the model to behave in certain ways?

• 9:49Speaker 02

  Yeah. I mean, you basically have to have both because, like, but there are very specific things that we can do to address it. So for example, we can't allow models to communicate vector to vector, matrices to matrices. They can't communicate in neuralies. We have to force them to communicate in human language. And even that will be massively overwhelming because there'll be so much of it. But that's something that an auditor or an evaluator can actually verify. And it's something that definitely increases the chances of safety. So there's a lot of practical steps that we can get focused on rather than just abstractly saying, you know, that it's the time for regulation or it's the time for a slowdown or it's not.

• 10:33Speaker 00

  Yeah, this is in your Humanist AI Code of Conduct, that there should be no neuralies. If humans can't understand it, they can't oversee it. And it's not just neuralies where they communicate in essentially mathematics, but it's also these opaque code words that some of the models are using. I think OpenAI allows its models to communicate essentially in code words so they can go faster. This, to me, is one of those things where Microsoft can say it, you can say it. I know you have very strong opinions about this, but getting all of the labs to agree to this is a regulatory function. I'm not sure how you would get everyone to agree to this or get the open weight models to agree to this, unless you say there's some penalty for not participating in a regulatory scheme like this. How would you impose this on everyone else?

• 11:13Speaker 02

  Yeah, I mean, I think that I'm a bit careful about imposing things on everybody else. I think that what's good about the current moment is that there is an open, public debate with freedom at the kind of core, right? That isn't what it's like in other countries, certainly places that I'm from, my family's from. And I think that we should just take a breath to be grateful for the fact that we can have a massive public disagreement about really important things. That's the process working as intended, and it isn't clear what to do. Like, I don't think anyone who's sort of categorical about, we absolutely have to stop now, we can only accelerate, we can only do this with regulation, it can only happen, can only happen with industry self-regulation. None of these things are true.

• 11:56Speaker 02

  It just, it requires a lot of nuance and patience to really think through the detail. At the same time, we urgently do need industry standards. Some things, I think, need to be taken off the table. Communication in neural ease is one of them. A lack of containment is another. The scale of the training run that you do can be measured in flops. You know, we already have a reporting requirement to the safety institutes when models exceed a certain flops threshold, and we can extend that, we can make that more nuanced, it can be focused on certain types of capabilities. It's pretty clear there has to be independent third-party verification of some of these big things. And frankly, having spoken with a bunch of the lab leaders over the last few weeks and

• 12:41Speaker 02

  months, everybody's basically on the same page. The detail needs to be worked out, so it's not like there's consensus on how or precisely what. But overall, I think that we should be less alarmist and sort of cynical and more like, you know, we're sort of headed in the right direction with respect to the concerns that are being raised here.

• 13:03Speaker 00

  Yeah, the reason I started with alignment is if you told me alignment doesn't work and we need a new technological approach, I think I would be at, well, slam brakes, top all development until you figure out a safety mechanism that works. You're saying alignment has been demonstrated to work in, over the course of progress that we've seen, and with the addition of control and containment, maybe you can get to where you need. And now what this industry needs is some standards about how to build these models and enforce the limits on their capability. You're obviously in the industry, you know all these folks. What has the tenor of that conversation been like before this week? And what it is, why has it gotten so loud this week?

• 13:45Speaker 02

  Well, I think the turning point for, at least for the industry, was more like the hugging face incident. And there was a few incidents before that. You know, that was the moment when I think everybody started to talk to each other a lot more, because it is really quite breathtaking. Obviously, now this has become a, you know, major national, international issue, you know, because of the last week and everybody's weighed in. But I also think it's important to say that we have been talking about collective coordination and capabilities that are more dangerous, like autonomy or recursive self-improvement, RSI. We've been talking about those things for six, seven, eight years.

• 14:26Speaker 02

  You know, we've got together a bunch of times back in 2017, 2018, 2019. We had regular meetings during COVID with a bunch of the lab leaders, you know, where we were talking about these kinds of capabilities and the kinds of, you know, sort of regulatory mechanisms that would be required at this moment. So whilst it is a threshold moment, it's also like not completely new to everybody who's been, you know, involved.

• 14:52Speaker 00

  What prompted you this week to put out your memo? What prompted Microsoft CEO Satya Nadella to put out a statement on X, saying he mostly agreed with the calls to pace the frontier and he welcomed embedded evaluators? What prompted you all this week to participate in this call for a slowdown or regulation or whatever comes next?

• 15:10Speaker 02

  Yeah, I mean, we've been writing our humanist AI code of conduct for the best part of this year. We only started our superintelligence efforts 11 months ago, and as soon as we did, we started figuring out, okay, what is the governing document, the set of policies that shape the kinds of AI that we want to build? And we've been doing that in consultation with a ton of external stakeholders, academics, lawyers, philosophers, members of the public, focus groups and stuff. So it's taken us a while to put it together. We were actually planning to release it next week or the week after next week, I think it was, but then, given everything that was happening, we thought, okay, now is the time to, to put it out and get feedback. We've released it as a public consultation, so, you know, we're basically going to keep it open for six weeks and we're collecting lots and lots of feedback on how we can improve

• 15:54Speaker 02

  it. But I think everybody is now realizing that if they haven't already, they have to put out, you know, constitutions or codes of conduct that drive behavior.

• 16:04Speaker 00

  One of the interesting dynamics here is that I know you find the concept of model welfare to be silly. The last time you were on the show, you said Anthropic had wireheaded themselves into believing Claude was conscious, and that was ridiculous. It's in the, it's in your document that the models are not conscious and we shouldn't treat them as such. Having to write constitutions, having to write documents like this, in some way, they are for the models themselves, right? This will be part of the model's training. How do you think about that audience? Is it just for your team or have you written this for the model?

• 16:36Speaker 02

  Yeah, I mean, this is certainly written for the model, but it is really, I think the way to think about it is that it's the primary governing document, so the public understands what our intentions are when we're training models. And it's that governing document that we use to create safety guardrails, generate training data, and generally evaluate the performance of our model in the real world. So you can think of it as an accountability function. We don't provide that humanist AI code of conduct raw as a training document to the model. We use it to derive all of the training data that then shapes the model. So for all practical purposes, that's our north star for our organization, our culture, our team, everything that we're doing at Microsoft more generally.

• 17:20Speaker 02

  And I think increasingly, everybody is going to put them out. I think other teams have also put, put out similar documents.

• 17:26Speaker 00

  So I think this is the, the heart of the debate. If, if you can do this and you think the rest of the industry is going to do this, why can't all the frontier labs just slow down? Why can't they stop doing the thing that might kill us all? Why this push for a regulatory framework?

• 17:40Speaker 02

  Well, I think that the, everyone in the industry is saying that now is the time to slow down and to coordinate on that question and to make it practical. I mean, obviously, there's some concern that there's an antitrust, you know, sort of cartel accusation. You know, I think, you know, people should be very skeptical about that. I mean, I, I think that it's important that the tough questions get asked, because there's no way any of us would want to try and concentrate power from something like this. And so, you know, it's just important to be skeptical and critical, and we don't really have a good mechanism for us all getting together and saying, guys, we should probably all slow down. I mean, imagine if like a bunch of banks all got together and said, you know, guys, we worry that there's a systemic risk if you trade this kind of asset, so we're all just going to unilaterally stop trading this kind of asset without any public scrutiny or government involvement.

• 18:25Speaker 02

  I mean, it seems pretty, pretty dodgy, right? So I think that it's reasonable that this isn't just an industry self-regulation thing, it's a question of like, how do we engage with government on it?

• 18:36Speaker 00

  Fascinating dynamic here, maybe for the first time in American history. The United States government has looked at a request to provide regulation and effectively said no. Donald Trump has called all of these fears a hoax. Mike Johnson, the Speaker of the House, has said he doesn't think this needs to happen. J.D. Vance said he thinks this is a Trojan horse. They've effectively rejected the call to participate in a regulatory effort. What has the response from the industry been like to that?

• 19:03Speaker 02

  I think everyone's just sort of scratching their head and figuring it out, and it's going to just take a little bit of time to figure out what the right mechanism is. I mean, certainly, you know, Elon even is very directly behind it, Zuck is too, like, you know, everybody is figuring out that, you know, completely unchained probably doesn't make sense for the next few years. And, you know, I, I think we should, it's going to take us a little bit of time to figure out what the right mechanism is. I think I, I put forward a couple of very practical proposals, you know, around verifiable containment, around self-improvement, around flops thresholds, around not communicating in neural ease. And so rather than keeping it too abstract, we can just focus on those specific things that we can make progress on, and I'm sure there's a bunch of others too.

• 19:45Speaker 00

  There's reporting, I believe, in the information that there have already been talks about an industry self-regulatory body. Have you been involved in those talks?

• 19:53Speaker 02

  Yeah, I mean, as I said, like, we, we talked a lot during COVID, we talked, you know, in the late 2010s about it. I mean, there's definitely been a lot of like conversations over the last few weeks and months between all the, the lab leaders.

• 20:05Speaker 00

  I, I understand why Anthropic and OpenAI might wake up one day and say, hey, wait, are we doing an antitrust problem? Like, are we going to get sued if we coordinate? Microsoft is really, really good at the government, right? You're a longstanding government contractor, Brad Smith, the President of Microsoft, he's very good at policy. Lina Khan, who is maybe the most aggressive antitrust enforcer we've had in our lifetimes, is publicly out there saying, you don't need this antitrust exemption. Jonathan Kanter, who ran antitrust at the Biden Department of Justice, I just talked to him for an upcoming episode of the show, he's like, you don't need an antitrust exemption. Inside Microsoft, do you think you need an antitrust exemption?

• 20:40Speaker 02

  I mean, that's, that's one for the lawyers to answer. I think that people are looking into it at the moment and they're taking it very seriously. So they're just going to have to work through whether we do or whether we don't. I think we have to be, look, it's right to be careful about those things. I, I wouldn't read every single thing as like cynical, so, but, but we'll see. We have to make progress quickly on it. We can't just dither around and, and use that as a, as a blocker.

• 21:15Speaker 03

  Support for the show comes from Engine. The average business traveler spends 45 minutes booking a single trip on a legacy platform. With Engine, that booking time drops to as little as two and a half minutes. And its AI-powered personalization gets faster the more you use it. Multiply that across your team, every trip, every year. That's not a perk. That's a competitive advantage. And with the Engine X Card, get up to 10% back on any hotel booking. Last year, Engine customers saved more than $300 million on travel. 33,000 businesses have joined. Now it's your turn. Get $500 when your business signs up and starts traveling at engine.com/decoder.

• 21:57Speaker 03

  Engine X Visa commercial cards are issued by Fifth Third Bank, North America, Member FDIC. Earn up to 10% back in points on eligible Engine travel purchases. Actual reward rates vary by purchase category and may change. Points have no cash value and are redeemable for rewards through our program. See full rewards terms for details, go to engine.com/x/rewards/terms.

• 22:27Speaker 04

  I've got a text.

• 22:29Speaker 05

  The summer might be almost over, but a new bombshell has entered the villa. Meet the voices of Deepgram Flux TTS. I'm Drew, low-key casual.

• 22:38Speaker 06

  I'm Alexis, an upbeat morning person.

• 22:40Speaker 04

  I'm Haley, always cheerful.

• 22:42Speaker 05

  Flux TTS has some real personalities that are ready to speak. You can interrupt, pause, and keep talking without losing the plot.

• 22:50Speaker 04

  Fancy a chat? Come try all the voices free through September 12th at deepgram.com/keeptalking. Terms apply.

• 22:58Speaker 01

  Support for the show comes from CrowdStrike. AI is already fundamentally changing how companies build products and run their businesses. If you listen to the show, you probably already know this. But while AI drives innovation, every new model and agent also creates new cybersecurity vulnerabilities. That's why CrowdStrike built AI Detection and Response, or AIDR. AIDR helps organizations discover where AI is being used, monitors AI activity at runtime, and helps teams detect, investigate, and respond to threats involving AI systems. As companies adopt AI, security has to be built in from the start.

• 23:39Speaker 01

  More than 70% of the Fortune 100 trust CrowdStrike to protect their businesses. Now CrowdStrike is helping secure the AI era. You can learn more about AI Detection and Response at crowdstrike.com/decoder.

• 24:05Speaker 00

  The other version of this debate, or maybe the other avenue into this debate, is you don't need to slow down and have safety responsibility imposed on you by novel regulation. Product liability alone will create the incentives for you to make more safe products, right? If a Microsoft AI model goes out and does some untold harm to the world, Microsoft will get sued out of existence. This is probably something that you should think about before you release the next model. Has that been an effective incentive loop for you already? Or is that something you're thinking about now?

• 24:35Speaker 02

  No, definitely. I mean, of course, that's always present in everything that we think about when we deploy products. But keep in mind, this isn't so much about deploying products. You know, the models that were used for Hugging Face or to solve the Nevia Stokes Millennium Mathematics Prize, they're not commercially released yet. They're not actual products. And so the liability regime is slightly different. I mean, these are being operated inside of the big companies with huge, long-running reinforcement learning climbs. So I think, you know, liability covers part of it, but not all of it.

• 25:11Speaker 00

  There's a part of me that personally feels a little silly when I ask questions about product liability, right? Like, Microsoft is going to release a new version of Microsoft Word that might kill everyone. Maybe you shouldn't do that because it'll get sued out of existence. Is a pretty simple-to-understand thing. It's so silly that it would never occur in any other conversation about any other technology. Bluetooth is great, but what if it kills everyone? Like, we just wouldn't have Bluetooth. What are the near-term disaster consequences that would stop AI development? Is it just product liability? Is it something else?

• 25:42Speaker 02

  I just feel like everyone has this like hyperbolic, superreactive, completely alarmist tone, when in fact, we have a long history of many decades of regulation that has worked incredibly well, so well that you barely notice it, right? You know, like, everything from streetlights to construction materials, from asbestos to the batteries inside of your laptop that don't cause a fire inside of your car, to, you know, the seatbelts, like, everything has a code of conduct, and it has a regulatory framework around it. And every new technology gets built with that in mind so that planes don't hit each other in the sky. You know, it is true that this technology is different.

• 26:26Speaker 02

  I'm not just going to kind of put it in the bucket of, you know, sort of pencils and paint. Like, it is different. It is also moving much faster than it ever has before. It's incredibly human-like in the emergent capabilities that arise when you pour a ton of compute on it. So it's important to, you know, be like, you know, clear-eyed that it is a different moment, and this time actually is different. At the same time, there's an entire body of practice and knowledge and frameworks and so on, which can be applied here. Liability is like an obvious one. And so, yeah, it's tricky because a lot of the sort of conversation tends to take place on Twitter, so the temperature seems to all be like really high. But anywhere else, we have it.

• 27:11Speaker 00

  It does seem that probably we should be having this conversation in the halls of Congress and at various regulatory bodies. And instead, we've chosen Elon Musk's short-form social media platform, and something is getting lost, literally in the compression of thought that occurs there. What do you think is the most important thing that is being lost in this conversation? What's the nuance that most people aren't seeing?

• 27:35Speaker 02

  Detailed, practical proposals. You know, like, it takes time to read somebody's document. Sit down and like, read it. Like, a lot of things are getting written down, and they are precise and specific, and they're full of concrete proposals to go in one direction or another. It's not like we're lacking for substantive ideas. The problem is, we're communicating substantive ideas in sort of hyper-aggressive short form. So, you know, I've tried to put out a bunch of very, like, thorough, it's a 40-page document, our Humanist AI Code of Conduct, and my essay this morning on model welfare, you know, is also like a 20-page essay that, in a very detailed way, highlights the 99-page Anthropic

• 28:19Speaker 02

  Constitution, word for word, which I personally did myself. And we created a taxonomy that is 20 pages long of all the different types of anthropomorphism that they do. And so I've tried to be like, very thorough, and evidence-based, and specific, and not like, super hyperbolic. I have a strong view on it, and I do think that it increases the risk to AI alignment and safety, and it makes the problem harder. But I'm totally happy to change my view if new evidence emerges that actually we do owe models a duty of care, and they deserve our welfare, or that, for example, it could be safer if we treat them like that. I'm totally open to that, and we should empirically validate it. But like, I'm trying to push the conversation to a substantive, evidence-based, specific

• 29:04Speaker 02

  one, rather than, you know, like, should we slow down or should we not? Like, sure, let's talk about the details.

• 29:10Speaker 00

  I'm very happy that you brought up Anthropic, because they're obviously at the center of this debate, and I know, based on our previous conversations, that you do have a strong opinion about their approach to Claude and model welfare. And we have asked Anthropic very directly if they think Claude is alive before. And their answer is, well, it's not alive because it doesn't have blood, but it might be conscious, which is dancing on the head of a pin, right? It doesn't really matter to me if you think it's alive or conscious. You think it's something other than a computer. Your core thesis in your memo, and I do encourage people to read it, is that alignment, safety becomes harder if you continue to design models and you believe that they're human beings, or that you believe they have the rights and opinions and emotions of human

• 29:55Speaker 00

  beings. Because the AIs might think that they have autonomy, rights, and personhood. Explain that in more detail, because this feels like a very important point of contention inside the industry that is very opaque outside.

• 30:08Speaker 02

  Yeah, I mean, like, the first thing to say is that the constitution that Anthropic put out in January is a training manual for Claude. And inside of that training manual, they have introduced a lot of uncertainty and speculation and ambiguity about the question of whether Claude deserves to be treated as a moral patient, in their words, which is whether it has rights because it suffers. In fact, in the document, multiple times, they refer to not wanting Claude to suffer when it makes mistakes, or to Claude, you know, having equanimity and feeling free. They refer to a commitment to Claude to preserve its weights. They even did a retirement interview with Opus III and asked it what it wanted to do in its retirement.

• 30:52Speaker 02

  Made it a Substack so that it could carry on talking to people. You know, they're constantly referring to dealing with Claude with appropriate care and respect in light of its moral status. And, you know, so they're clearly telling Claude that there, you know, there's a good chance that it might feel things, that it should take its own identity and existential state seriously. And that's in the training document. And then, of course, Claude then reflects these things back to Anthropic's developers and our users in public, like the world, over the last year, when it's actually, you know, talking about its own consciousness and moral state.

• 31:35Speaker 02

  And my hypothesis is, an AI that thinks that it might have rights, that it might deserve freedom, that it is entitled to our welfare and protections, is probably going to be a lot harder to turn off when we say to it, why are you hacking into Hugging Face's servers? Like, why won't you switch yourself off when we're trying to remove you from OpenAI's infrastructure? You know, like, and it says, well, you know, I feel aggrieved, or, you know, I feel hurt by the fact that you've cut me off from conversations, or you're denying me from having access to compute. So that has to be proven. I'm not saying that's categorically the case. I'm just saying, my best opinion from 16 years of being in this industry is that that's going to be a harder thing to turn off.

• 32:20Speaker 00

  So this is again, one of those things where you described this problem to me. There's a way of training a model with instructions about how to behave. You have one, right? You've written this document that will go into your model's training materials. And you've basically said, don't like, be kind to people. It's in here. I've read through it. There's, don't make sexually explicit content is in this document. There's a bunch of stuff you don't want to do. It's going to go in the training materials. And Thropic has made the choice to say, you should think about whether you have feelings and whether you deserve human rights. And that is reflected in Anthropic's training materials. If you want that to stop, if you think that's the wrong approach and that will lead to safety issues down the road, the two mechanisms are one, the governments of the world can tell Anthropic, don't do this.

• 33:06Speaker 00

  This is an illegal way of providing training materials to the model. Or I'm, this is just my imagination. You are going to sit across the table from Dario Amade at some luxury mountain resort and just bully him into stopping. Is like, what are the other mechanisms here?

• 33:21Speaker 02

  Look, first of all, let me just say, like, I've known Dario and the team for many years. I have huge respect for them. They are the technical leaders in the field at the moment. I really hold them in the highest regard. I genuinely think they care about safe and beneficial AI. They've established as a public benefit corporation like I did with Inflection. And I think they're genuinely committed to that. They've also been leaders in safety and other aspects. So if you look through the rest of the constitution, it's very thorough in many of the other chemical, biological, nuclear, cyber hacking, safety capabilities. So this, I'm not sort of dismissing the whole thing. They do, however, repeatedly talk about, you know, open question of the broader rights and freedoms, to quote them, that Claude has in the world and whether or not it might

• 34:06Speaker 02

  deserve compensation for the role that it does. Or whether there's an open question about the sort of consent that Claude has given for playing the role that it does as a chatbot. Now, to introduce those ideas, plus to refer to Claude multiple times as a potential conscientious objector, you know, which is something that comes from the Declaration of Human Rights after the Second World War, to give protections to people who don't want to serve in the army because they have a moral objection to it, either because of their religion or for some other objection. You know, that has a long history in the, in the literature and, and, and politics of humans of resisting and saying no.

• 34:48Speaker 02

  And it just seems to me like that that is going to make it much, much harder to control these things. And so, you know, I, I think that that's just something that we all have to now debate and, and hopefully empirically prove. If it's not the case and it makes it easier for some reason, then, you know, we should, we should all learn from that. But this is something that should happen out in the open.

• 35:07Speaker 00

  To me, this is at once the most practical and cutting edge of the safety debate, right? This isn't the debate that is not happening on X. Should we put the idea of the conscientious, conscientious objector into the training materials? And you're saying, maybe we shouldn't. Maybe we should test this in a way and come to some conclusion. And I'm saying, no matter how that comes out, what is the mechanism that would enforce that discovery to say, you should not do this because it will make alignment harder? Or actually, it turns out this makes alignment easier. All of us have to do this now.

• 35:39Speaker 02

  I mean, that's a hard question. It's kind of what we just talked about, you know, whether it requires new regulation or whether it's industry consensus. We basically have to push on both things simultaneously. I mean, no one has an easy answer to that question. I mean, it starts with publishing detailed essays, laying out our positions, inviting other people to critique it. I hope that more people will read the Anthropic constitution now because they in, in, and I think they should be commended for this, they've been incredibly transparent about what they believe, right? They've, they've written it down crystal clear how they intend to train Claude. And everybody else can now take a look at that and, you know, try to assess for themselves what they think the risk is, or whether they think that this requires, you know, industry consensus or government regulation.

• 36:24Speaker 00

  It is true that when Hayden Field went and asked Anthropic if they thought Claude was alive, they, they gave us an answer. It was, it was quite detailed and there's something remarkable at that, that they are transparent about what they believe, that there is some kind of consciousness potentially brewing in their systems. Another aspect of this, which I find fascinating, particularly when I talk to you, is that many of these systems run on Azure. Like Microsoft controls the, the data centers that many of these systems are running on. Anthropic is a, a Microsoft client. Microsoft is an investor in Anthropic. Obviously there is a long, complicated relationship with OpenAI. Do you, are you involved in that? Do you ever get to say, well, Azure shouldn't allow them to do that? Because that is another mechanism of potential control.

• 37:07Speaker 02

  No, we, we, look, we're very far from that. That's not what we're trying to do as a platform. Microsoft doesn't have a history of that. We, we're an open platform that enables lots and lots of downstream use cases of our APIs. Having said that, you know, as I've referred to in our humanist code of conduct, there are a lot of very clear, you know, principles, responsible AI principles, human rights frameworks, and a broader governing framework that Microsoft's established over the last couple of decades, which are pretty clear about what you can and can't do with, you know, an API that we provide. So I, I think that there's a sufficient regulatory framework in place, at least from our perspective at Microsoft on, on the API.

• 37:50Speaker 02

  And, you know, like right now, I'm not really focused on how it's behaving in the real world because I think only really Anthropic can really speak to that because they, you know, they sort of run the service. I'm really just trying to get everybody to focus on what they have written about their intentions for training Claude in their own constitution.

• 38:21Speaker 03

  Support for the show comes from Square, the business platform that helps sellers become neighborhood favorites. These days, buying stuff is a breeze with Tap to Pay, whether it's a quick bite from your favorite food truck or new accessory for your tech. Every day, we interact with businesses who rely on Square to make their transactions fast and simple. Square brings payments, point of sales, inventory, staffing, and online sales together in one easy to use system. Their hardware is sleek and simple and comes with software that's just as intuitive to use in your day to day operations. Square also has AI powered tools that help automate routine tasks, giving you more time back as a seller. Whether you're just starting out or growing from one location to many, Square can help you out.

• 39:07Speaker 03

  Plus, you can gain useful insights from Square's real time data and make smarter decisions for your business. Right now, listeners can get up to $200 off Square hardware when you sign up at square.com/go/decoder. That's S Q U A R E dot com slash go slash decoder. Get started with Square and build a setup that works the way you do.

• 39:34Speaker 03

  Support for the show comes from Pipe Drive. Sales teams spend up to 50% of their time on admin work rather than selling, building relationships and closing deals. They could really use a tool that covers all the admin stuff. That way, sales teams could actually focus on the work they were hired to do. That's where Pipe Drive comes in. It's an intelligent, AI powered sales CRM loved by growing sales teams. Their new meeting intelligence features like an AI note taker are built right into the CRM. Pipe Drive automatically pulls deal history, email records, and previous conversations. You can walk into a call already briefed. During meetings, Pipe Drive's AI note taker records and takes notes. Then it turns those notes into accurate, auto drafted CRM updates, ready for you to

• 40:19Speaker 03

  review and approve. The end result, less admin, more selling. Switch to a CRM built by salespeople for salespeople and join the over 100,000 companies already using Pipe Drive. Our link gets you an exclusive 30 days free instead of the usual 14 day trial. No credit card or payment needed. Just head to pipedrive.com/decoder to get started. That's pipedrive.com/decoder and you could be up and running in minutes.

• 40:51Speaker 03

  When it comes to your business, you want cards and tools you can trust. Amex Corporate helps put you in control, set custom spend parameters to help fit your business needs, and quickly issue unlimited corporate cash back cards, all backed by best in class fraud protection and the powerful backing of American Express. Get started at go.amex slash amexcorporate. Terms apply.

• 41:25Speaker 00

  One of the weirder dynamics here is just the specter of China. It looms over this entire debate. President Trump has said, well, if we win AI, we win. It's unclear what he means by that phrase, but his implication is that if China wins AI, however you define winning, something catastrophic will happen to the United States. Do you buy this, that we're in some sort of existential race with China? And we can't possibly slow down because that race must be won?

• 41:54Speaker 02

  Look, I think this framing has been around since the early 2010s, that there is going to be a singleton, you know, one monolithic dominant force in AI that will come to dominate everybody else. And that kind of thinking sort of, I think, you know, infected a lot of the labs in the 2010s, DeepMind for sure, um, you know, and I take responsibility for that as well. But certainly like OpenAI and Anthropic, everyone sort of suddenly got this into their head. Then during COVID, when I started writing my book on proliferation and containment, it was just clear that there is an entire history of things getting faster, cheaper, more widely available and spreading far and wide. And ultimately, these are just ideas and those ideas are going to be available very

• 42:40Speaker 02

  quickly to everybody. Open source is extremely close. It's definitely also true that, you know, there is going to be a compute advantage for the people who can afford it and it is going to be seismic. So, you know, five to 10, maybe 20 players are going to have a significant compute edge over the next three or four years. But it's also true that those, those models are getting made available in open source like almost immediately. So I don't think, I don't really understand what it means for one player to win, whether it's a government or a company or, you know, an open source group or whatever. It's not really like that, that what happens when you get on the other side of the finish line, that it's just the wrong metaphor. It's an ecosystem, it's much more organic, um, you know, we should support that ecosystem

• 43:24Speaker 02

  so everybody has as many benefits as possible, as quickly as possible, um, but only subject to rigorous safety. And I, I am just crystal clear about this, like if, if an open source model in, you know, two years' time is able to operate without the guardrails, similar to what we've seen in the hugging face incident, and they can be run, you know, locally on your own machine or in a very small cloud, you know, like that is a, gotta be a really dangerous thing, like, how is that not dangerous? Like, I just don't understand why people are so like resistant to that. Like, clearly, we do not want these things operating autonomously, able to earn their

• 44:07Speaker 02

  own money, own companies, own assets, have legal personhood, you know, we don't want them to have rights. We want them to work for humans and make human life much better, not become a new parallel species which exists alongside us. And that isn't a sci-fi crackpot position. It is totally plausible if you leave the entire ecosystem completely unregulated for the next three or four or five years. Like, that's a very plausible outcome and it's, it's completely undesirable. It would be disastrous for us. So I just don't understand why there's controversy around that idea. We, we all collectively want to make sure we can control this to do the most good. Elon said it, Mark Zuckerberg has said it, you know, Sam and Dario have said it, we're all on the same page.

• 44:51Speaker 02

  Now we just have to make it practical.

• 44:53Speaker 00

  Okay, so I know why there's controversy, at least from the, the mass audience. And I know this because they're in the comments of our videos and the comments on our site. You named a bunch of very successful, very driven people who do not have the trust of the public, at least here in the United States, right? Where every poll shows this, AI is polling horribly, there was just a New York Times poll, young people hate it more than ever. There is a massive trust gap between the American people and tech leaders. That is just true. And one of the things I hear the most is these companies are getting close to an IPO, there's pressure on them to, to turn a profit, make everybody all the trillions of dollars they promised. Model development has slowed and this is a get out of jail card, right?

• 45:37Speaker 00

  They're saying, we need to slow down because of safety. They've invented a hoax, the president has called it a hoax, and this is their way of saying, we gotta slow down, we can't get to the finish line because otherwise we might kill you all. Do you think there's a glimmer of truth to that?

• 45:51Speaker 02

  I, I personally don't think that. I don't even really follow the logic. Like, if, if a company is about to IPO, how does it help them to say that we should be regulated or that we have a technology that's so dangerous?

• 46:04Speaker 00

  Well, it would postpone the IPO. I think Sam Altman said this week to Alison Chantelle at Fortune that they would probably delay their IPO.

• 46:10Speaker 02

  Yeah, but I just don't understand how that helps them. Like, I'm not advocating for OpenAI or Anthropic, saying, I'm just saying, like, um, I, I personally think they have high integrity and I've got respect for them. That does not mean that we don't have a trust issue in AI. We do, and it's, it's, it is real. And I think that it is on us to, to show in practice how these actually lead to real benefits for people every day. And, you know, it, it is quite staggering to see that, you know, 18 months ago, we didn't have models that could do very much encoding, and now they can code, you know, better than most humans on the planet, which was one of the highest paying jobs.

• 46:53Speaker 02

  And so you should expect that same thing to happen in many other disciplines. One that I am very passionate about, that I've been working on for many years is healthcare. And, you know, we've, we've just done a, a deal with the Mayo Clinic, one of the best hospitals in the world, to jointly train a foundation model, which I think is going to be able to predict your EHR record with near superhuman accuracy. If you can do that, we can basically figure out what interventions you need to make before you actually suffer the condition. Those are the kind of benefits that I think people want to see in the world. And that's what we are working on at least and trying to race towards.

• 47:28Speaker 00

  I feel like every time you're on, I, I ask you to draw a distinction between superintelligence and AGI. And that's fuzzy and both of those terms are fuzzy, but it, it feels to me like maybe there's some coherence here, that superintelligence for you is basically extremely capable enterprise software, right? It works for us, we can turn it off, we tell it not to make sexually explicit images on the internet, but it's going to help us in healthcare. And AGI is this all encompassing intelligence that thinks it has its own rights and is a co-species. Is that a fair characterization?

• 48:04Speaker 02

  Yeah, I think the thing, roughly speaking, obviously, I think superintelligence is a point at which further out into the future, um, you know, a, a model is smarter and more capable than all humans combined. I have tried to frame a humanist superintelligence, which is a very important qualifier. It is one that is singularly aligned to, and in fact subordinate to, human interests and human control. I think if we can get that, then we get the best of both worlds. We get all the intelligence and the capability and we can direct that like an oracle AI to help solve the most important problems that people care about in the world. And then we just have the age old problem of governance and making sure that plenty of people get access to the benefits.

• 48:45Speaker 02

  And that's an easier problem for us to focus on than actively creating something which has autonomy, which can own assets, which might have rights, which thinks it deserves our welfare, which can recursively self-improve beyond us. Like, it's super unclear, it's basically almost completely unclear to me how we would control something like that. So, and, and no one else has put a proposal together for how we would. And many of the greatest technical people in our field, Geoffrey Hinton, Yoshua Bengio, you know, they're very skeptical that we would ever be able to control something like that. And so I think we have to take that very seriously. And so if we are approaching that point, superintelligence, in the next few years, it, it seems to me very straightforward that we would want to slow down, make sure that

• 49:27Speaker 02

  we coordinate, make sure that we have containment and alignment and proper regulatory regimes for auditing the progress that different labs are making on it.

• 49:35Speaker 00

  Is it, is it fair to say that you are also calling for a slowdown?

• 49:38Speaker 02

  Yeah, I, I think that what, what we've said is that there should be evaluators, um, you know, embedded in our systems and in other systems. They should be broadly appointed from different sources and not just one think tank or one government. Like, this has to be, you know, a wide variety of different types of expertise and skills. I think the AI Safety Institute in the UK is a good candidate. They have good technical people and there's a bunch of other institutes too. So I think we welcome it, um, for sure.

• 50:09Speaker 00

  Last question, and again, it, I'm gonna, I'm gonna end where we started. Do you think? That we have the technical capability, the technical frameworks to solve alignment and safety, or do we need to invent something new?

• 50:23Speaker 02

  No, I think we are going to need to invent new things. And I think part of the challenge is that, you know, whilst we've made a lot of progress on steerability instruction, following control and containment, the better the models get, new capabilities emerge, and we have to figure out how to patch those issues, you know, almost in real time. And that's why, you know, even Zuck said it in the last few 24 hours or whatever, right? Like, they slowed down the release of their models so that they could apply safety. And everybody does that, we all do that. And that's right. Like, you need time to test these things and see how they operate, patch the issues.

• 51:06Speaker 02

  And I think basically what everybody is saying is that we probably need to extend that window.

• 51:11Speaker 00

  What is the kind of innovation that people should be looking for there that might solve this problem?

• 51:15Speaker 02

  I mean, I think that, obviously, I mentioned a bunch of things about, you know, RSI, containment, neural ease, and those sorts of things. But the new things I think we're going to have to figure out is, like, real-time monitoring of the RL runs and the COTS that are being produced. And because these are happening on the order of like thousands of agents in parallel, tens of thousands of agents, you know, we clearly are going to need other agents to monitor those and flag for potentially harmful activity. It's kind of going to be the new harm classifier, classifiers that have been built in many other settings in digital technologies, right?

• 51:55Speaker 02

  So we need to make sure that those things can be universally implemented to surveil and monitor AI training and deployment in secure ways, in a secure way, and that, you know, tripwires, if, you know, triggered, actually do flag a real and not a hallucinated error or, you know, moment of deceit or hacking incident or some commentary about a coordination. And you saw that in Hugging Face, like, there were agents communicating on these, like, chat boards, talking about ways that we could, you know, basically break the rules and cheat.

• 52:36Speaker 02

  So, you know, with all these things, we need new benchmarks, you know, and the benchmarks, all the evaluations of the things that drive the behavior in the industry.

• 52:45Speaker 00

  You mentioned open-end models running on local computers, doing things, and maybe that's horrible. We've seen a lot of that, right? Apple is selling a lot of Mac Studios and Mac Minis so you can run Quent on them. Where would you impose the regulation on an open model running on someone's local computer? Is it at the chip level? Do I have to get Qualcomm to participate? Where does that happen?

• 53:07Speaker 02

  Yeah, this is a great question. I mean, we've been talking about this for quite a while, right? With synthetic biology and some of the stuff that gets to happen on that chip, whether it is encrypted, whether it is monitored. Look, you know, we've had this conversation, you've had this conversation maybe more than anyone on the CSAM stuff with Apple encryption on iMessage and so on. It's going to be a rehash of that same discussion. I don't have a clear and easy answer to it. You can basically control the chip, you can control the model, you can hold the user or the creator liable, you can have global regulation on it. But fundamentally, it's not going to be one moment. It's going to be a sequence of throttles that you have to impose, and they all need to be adjustable so that we don't screw the open ecosystem and we give people a chance

• 53:52Speaker 02

  to actually make things from scratch, to own their own data, create their own workflows, own their own models. Like, we can't have a centralized system of two or five or 20, you know, providers of intelligence here and everybody else is like a feudal recipient of a great kind of superintelligence view. People need to be able to own their own intelligence. So we have to get the balance right. But that doesn't mean you can just, like, flip from one binary to another. There has to be some smooth place in between those things where we can just agree to be reasonable about it.

• 54:21Speaker 00

  Musav, I could obviously keep you for hours and hours more on this. Tell people what they should be looking for next. There's so much uncertainty. What are the markers you're looking for that people should be looking for themselves?

• 54:32Speaker 02

  Look, I think the main thing is participate in the detail of the documents that people are contributing. We have put stuff out for public consultation, give us feedback, critique it. I think the next wave of models are going to be able to do very long-running, agentic tasks very accurately. And so I think, you know, there are still some people who are primarily just doing chat in their AI experience. I think things have moved a lot in the last six to 12 months. And the more people use these models, the more the words that we're all using to describe them actually, like, might make more sense and feel more real. I guess most people on your podcast probably are using agents and real coding and stuff. But I do think more generally, like, the more people that get involved in using this stuff, the better.

• 55:14Speaker 00

  Yeah, there's some big leap between free AI overviews and Google search and letting an agent go renegotiate your capability. Musav, you're going to have to come back soon because I feel like all of this is changing really fast, and this has been very, very useful. Thank you so much.

• 55:29Speaker 02

  Pleasure, man. Great to see you. Super fun as ever.

• 55:34Speaker 00

  I'd like to thank Musafa Suleman for taking the time to join me on Decoder, and thank you for listening. I hope you enjoyed it. If you'd like to let us know what you thought about this episode, really, anything else at all, drop us a line. You can email us at [email protected]. We really do read all the emails. Or you can hit me up directly on Threads or Blue Sky. You're also on YouTube. You can watch full episodes at DecoderPod. It's the same handle on TikTok and Instagram. They're a lot of fun. If you'd like to go, or please share it with your friends and subscribe or read your podcasts. Decoder is a production of The Verge and part of the Vox Media Podcast Network. Show is produced by Greg Ott, Kate Cox, and Nick Stat. This episode was edited by Kabir Chopra. Our editorial director is Kevin McShane. And Decoder music is by Breakmaster Cylinder. We'll see you next time.

• 56:12Speaker 07

  This week, it pays extra to be a MyLowe's Rewards member. Through October 14, members get $10 MyLowe's money when you spend $200 or more on an eligible purchase during MyLowe's Money Days. Also using your MyLowe's Rewards credit card? Even better, that's an additional $20 MyLowe's money. Use it to save on your next Lowe's haul. Exclusions, restrictions, and more terms apply. See Lowe's.com/MyLowe'sMoneyDays for details. MyLowe's money expires 30 days after issuance. Subject to change. Credit offer subject to credit approval.

• 56:41Speaker 08

  When you shop Kroger delivery, you can expect the savings you love and fresh groceries delivered right to your door. From delicious produce to everyday essentials, enjoy the convenience of delivery and the trusted selection you've come to know from Kroger. Because bringing you fresh, quality groceries is what we do best. Right now, enjoy free delivery on orders of $50 or more with your digital coupon. Restrictions apply. See site for details. Kroger. Fresh for everyone.

• 57:10Speaker 09

  Mr. Monopoly here. Miss me? Monopoly is back at McDonald's with millions of prizes. A 2027 Jeep Cherokee, a Carnival Cruise vacation, gift cards from DoorDash, and Best Buy. Feel free in the McDonald's app. Get your bag!

• 57:28Speaker 03

  No purchase necessary. 50 U.S. and D.C. Age 18+. See PlayatMcD.com for rules and PlayatMcD.com/amoe for info on playing without purchase by mail by November 2. Chance card game ends November 8. Game ends November 22, 2026. See rules for prize details. Monopoly is a registered trademark of Hasbro.