Nate Soares on How AI Really Could Kill Us All

Since the Hugging Face incident, concerns over AI doomsday scenarios has entered the mainstream. But of course, some people have been warning about this for years, long before LLMs or ChatGPT. In their 2025 book If Anyone Builds It, Everyone Dies, Nate Soares and Eliezer Yudkowsky spelled out their vision of humanity's inevitable demise once superintelligent AI is built. In this episode, we speak with Soares, who serves as president of the Machine Intelligence Research Institute, about why he believes it's impossible to "align" superintelligent AI, and how exactly the technology can and will go rogue as it gets more and more advanced. Read more: Anthropic IPO Investors Struggle to Put a Price on Rogue AI Risk https://www.bloomberg.com/news/articles/2026-10-08/anthropic-ipo-investors-struggle-to-put-a-price-on-rogue-ai-risk?utm_medium=referral&utm_source=podcast&utm_campaign=odd_lots&utm_content=article AI Bubble Risks Worst S&P 500 Crash Since 2008, Strategist Says https://www.bloomberg.com/news/articles/2026-10-08/ai-bubble-risks-worst-s-p-500-crash-since-2008-strategist-says?utm_medium=referral&utm_source=podcast&utm_campaign=odd_lots&utm_content=article Only http://Bloomberg.com subscribers can get the Odd Lots newsletter in their inbox each week, plus unlimited access to the site and app. Subscribe at bloomberg.com/subscriptions/oddlots https://www.bloomberg.com/subscriptions/oddlots?in_source=oddlotspodcast Subscribe to the Odd Lots Newsletter http://bloomberg.com/subscriptions/oddlots Join the conversation: discord.gg/oddlots https://discord.gg/oddlots See omnystudio.com/listener https://omnystudio.com/listener for privacy information.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Joe Weisenthal?

  There are some market stories where you want every detail. Tracy and I have made quite a few podcasts on that basis, but sometimes you've only got 10 minutes and need to know what's moving markets and why. That's the Barclays Brief Podcast. Every week, experts from Barclays Markets and research get you up to speed on what matters and what to watch next. About the time it takes to grab a coffee. So, search Barclays Brief wherever you get your podcasts.

• 0:24Speaker 01

  Some people treat ChatGPT like some kind of smart search engine, and some use it to get work done. ChatGPT Work is a new way of working in ChatGPT that can take action across your apps and files, stay with a project for hours if needed, and turn a goal into finished work. It's designed to help you move from a chaotic starting point to a reviewable first version. So, all the source materials, briefs and scattered information that you have to grind through to turn into something useful, can just become something useful. Put ChatGPT to work on your most ambitious ideas and projects. Get started at chatgpt.com by selecting Work Mode, available on Plus and Pro Plans.

• 1:06Speaker 02

  If you listen to financial news, you know a lot of time is spent thinking about what's next. The next opportunity, the next investment, the next move. But sometimes what matters most is being ready for what you never saw coming. For more than 75 years, Cincinnati Insurance has worked with independent agents to help protect businesses, homes, valuables, and more. Because planning for the future isn't only about knowing what's next, it's about making sure you're ready for what you can't predict. Let Cincinnati Insurance make your bad day better. Find an independent agent at C I N F I N dot com.

• 1:46Speaker 03

  Bloomberg Audio Studios, podcasts, radio, news.

• 2:02Joe Weisenthal?

  Hello and welcome to another episode of the Odd Lots Podcast. I'm Joe Weisenthal.

• 2:07Tracy Alloway?

  And I'm Tracy Alloway.

• 2:08Joe Weisenthal?

  Tracy, I'm very tired today because I flew in on the red eye last night from San Francisco. I was out at this very interesting conference called Curve in

• 2:20Tracy Alloway?

  I've heard about it.

• 2:20Joe Weisenthal?

  In Berkeley, a lot of like, you know, I guess, a lot of people very concerned about AI, so to speak. Interesting mix of folks, but yes, people are concerned about what the future is going to look like as we develop stronger and stronger AI models.

• 2:37Tracy Alloway?

  Just from a sociological or anthropological standpoint, I think living in San Francisco and mingling with, you know, AI bros at the moment must be one of the most interesting things on earth. Because there must be so much cognitive dissonance in their heads, right? Where on the one hand, they'll say like, oh, they're building this new ultra powerful model. And on the other hand, sort of almost as an afterthought, they'll be like, oh, yeah. And I'm a little bit worried about the possibility that it kills us all. How do you square that?

• 3:05Joe Weisenthal?

  It's super interesting and you hear it like, so at this conference, you have people like working for like various like safety type orgs, people who want to regulate AI in some way. And of course, like people from the labs themselves, some maybe like working on like policy stuff, but some actually like the actual builders of the technology. And like, you know, they're all like deeply anxious about the very thing that they're building. But they just keep building it, you know, and so they have all these worries about at a minimum, how disruptive it could be or how dangerous it could be. And of course, we all now know that there are dangers,

• 3:47Joe Weisenthal?

  particularly in the wake of the hugging face attack. So people can see that it's very possible for models to go rogue and commit crimes and stuff like that. And yet they, you know, they keep building it.

• 4:01Tracy Alloway?

  And yet.

• 4:02Joe Weisenthal?

  And, you know, they would say, oh, well, like if, you know, someone's going to build it, so why not me? Cause I couldn't do it well. Maybe it's like how they think about it, or there's sort of some game theory aspect where it's like, it doesn't make sense for me to like slow down or pause what I'm doing because someone, either my direct competitor in town, or maybe someone in China will build it. So like, you know, not building it isn't going to do anything. It's a, but yeah, it's a, it's very strange. It's very odd.

• 4:31Tracy Alloway?

  We've used the term runaway train before to describe how all of this feels, which is, it almost feels at this point like there's no stopping it, right? Because if one lab doesn't do it, then another lab's going to do it, and that's how they justify it. If one country doesn't develop the next big model, then they're worried about another country building the next big model. So you have this, I guess, collective action problem for humanity, and a lot of it feels inevitable. And I have to say, today's guest predicted this inevitability problem in a very salient way. And one thing I'll just say is I read the book over the weekend. For those who don't know, the book talks about AI killing us all and talks about one hypothetical scenario

• 5:13Tracy Alloway?

  in which an agent could go rogue and kill us all. Read it over the weekend, woke up on Sunday to the headlines, the headlines about an escaped plague from a Russian lab. And I thought, wow, I couldn't have chose a better time to read this book. Now, to be fair, the escaped plague was in Siberia. And so if there is a rogue

• 5:34Joe Weisenthal?

  Proven yet, by the way?

• 5:35Tracy Alloway?

  Well, supposedly, according to reports. So if an escaped AI agent was trying to kill us all or kill a select a few individuals through a lab leak.

• 5:46Joe Weisenthal?

  Bio attack, yeah.

• 5:48Tracy Alloway?

  It might not be in Siberia. That's the one thing.

• 5:51Joe Weisenthal?

  It's probably not that.

• 5:51Tracy Alloway?

  The one thing that made me feel a little better this weekend. But you know, the fact that we're thinking about these scenarios at all, the fact that when I see a headline about a lab leak, I think, oh, God, I hope it's not a rogue AI agent because I just read this book. It's weird times.

• 6:07Joe Weisenthal?

  It is a really weird, it is a really weird time, you know? So obviously, something I've been thinking about is like, the pace of AI development itself is really extraordinary. And then the pace of what people are worried about, it's like the conversation is changing constantly. So for example, you know, six months ago, if you're like, what are people concerned about? Maybe a year ago, it's like, what are people concerned about with AI? There's like a job loss or something like that.

• 6:33Tracy Alloway?

  And then Software wipeout.

• 6:35Joe Weisenthal?

  Yeah. And then the conversation like, then there was like, people were talking about data centers and water use, electricity, electricity prices and so forth. More recently, like the attention, again, post hugging face, the resignation of the researcher Jacob Coxon from Anthropic has now moved into like this sort of next stage, which people, as you mentioned, some people have been talking about for a very long time. There was actually, I think just today or last night, their big New York Mag cover story about people who have quit the AI labs because they're like, the things that our bosses are doing is reckless and so forth.

• 7:16Joe Weisenthal?

  So now the new thing in the air is like sort of the ultimate risk.

• 7:21Tracy Alloway?

  Hopefully that is the ultimate risk. It's hard to see how it could get worse than wiping out all of humanity. Like, what would be the next worry after that?

• 7:28Joe Weisenthal?

  Very excited to say, we really do have the perfect guest to sort of articulate some of these concerns. We're going to be speaking with Nate Soares. He is the co-author of the book, If Anyone Builds It, Everyone Dies. And he is the president of Miri. So Nate, thank you so much for coming on the podcast.

• 7:46Speaker 05

  Thanks for having me.

• 7:47Joe Weisenthal?

  Let's start in the beginning, like, when did you sort of start thinking about this possibility? Two things like, A, that like, strong AI, let's set aside the question of superintelligence for a second. That strong AI is something that could plausibly have been built. And that if it were to be built, it would be something that would be very difficult to control.

• 8:15Speaker 05

  Yeah, I started thinking about this in about 2012, maybe 2011.

• 8:21Speaker 05

  I was working at Google and they acquired Google DeepMind. Right around then, it wasn't Google DeepMind at the time they acquired it, but I became Google DeepMind shortly thereafter.

• 8:35Speaker 05

  Yeah, I don't think that was actually inciting instinct, although that sort of caused me to think it could happen, you know, in some, in more like decades than centuries. Although I hoped we had more decades than it turns out that we had.

• 8:46Tracy Alloway?

  And what was it that you saw exactly that was the sort of lightbulb moment where you thought, hey, this risk could be real?

• 8:54Speaker 05

  You know, it was actually abstract arguments for me that did the trick. I know those don't do the trick for a lot of people.

• 8:59Joe Weisenthal?

  Yeah.

• 9:00Speaker 05

  But, uh, you know, if you, if you sort of look all around us, everything here was shaped by humans, and that's because we're the smartest creatures on the planet. And if humanity makes machines that are smarter than humans, uh, the, the sort of obvious outcome is that the world gets shaped by those machines. And so suddenly it matters quite a lot how those machines are shaping the world. And, you know, back in 2012, there was almost nobody working on the problem of, you know, there are a lot of people working on the problem of like, how do you make the machines smarter? There are very few people working on the problem of like, how do you cause the machines to shape the world in a good way? Which, which would go on to become known as the alignment problem, although this was even before the term alignment was coined.

• 9:43Speaker 05

  I initially thought I wouldn't be able to help much and just sort of, uh, you know, Google did a donation matching thing. So I just tried to sort of like find the people who are trying this and give them some money and get the Google donation matching. And then one thing led to another and I was running the, the Machine Intelligence Research Institute.

• 9:58Tracy Alloway?

  Now that you mention the alignment problem, so this is something that has obviously come up a number of times on the podcast and elsewhere, but when did alignment as a term and as a conceptual goal, when did that actually become a thing?

• 10:13Speaker 05

  So as a conceptual goal, uh, I mean, it sort of depends how far back in the history books you want to go. Um, you know, you could, you could sort of construe Alan Turing and IJ Good, some of the first computer scientists as sort of noticing that there was going to be an issue here. But the sort of modern notion of AI alignment of like, oh, look, we're really building an AI and like, it's not going to be good for free. Someone needs to put the work in there. Uh, that probably came from my co-author, Elias Yudkowsky. I think somewhere between 2001 and 2003, again, depending exactly where you want to draw that line. Uh, but the term alignment didn't come until, I think it was 2014, when, um, we had been calling it friendly AI research.

• 10:55Speaker 05

  And, uh, Stuart Russell of UC Berkeley, if I remember correctly, was the one who was like, we need a more academic sounding name. And, uh, I think that from my perspective, the word alignment was his idea. Um, but I'm sure he would say, you know, that he dug it out of some ancient paper somewhere else and blamed it on somebody else.

• 11:15Tracy Alloway?

  I kind of like friendly AI.

• 11:17Joe Weisenthal?

  Friendly AI.

• 11:18Tracy Alloway?

  It's just aim for friendly AI.

• 11:19Joe Weisenthal?

  I sort of, there's, I actually, friendly AI strikes me as more coherent as a concept in some sense than aligned AI. AI is a technology that humans build and there are different, you know, techniques for building it. How would you describe the core intuition that a thing that we build, we would not necessarily be able to control?

• 11:43Speaker 05

  There's a few pieces of that puzzle. There's, there's a number of, of difficulties in, uh, controlling an AI even though you made it. With the way that current AIs are created, uh, it's, it's sort of very easy to see how we would, uh, not have a ton of fine-grained control of their behavior. And that's because AIs as they are made these days are not programmed like traditional software. So, you know, a lot of people think that the way software works is that, you know, you write a series of instructions, then the, the computers execute those instructions. And if it's behaving in some unintended way, that's because you sort of fundamentally told it to, right? And if you sort of like went back and looked at your code, you would see like, oh, I told the computer to delete my files.

• 12:26Joe Weisenthal?

  Yeah.

• 12:27Speaker 05

  I didn't mean to tell it that, but that's like literally what my instructions said. If you follow them very precisely as this machine did.

• 12:32Joe Weisenthal?

  Yeah.

• 12:33Speaker 05

  That is not how AIs are.

• 12:34Tracy Alloway?

  You use the term grown versus crafted in your book a lot.

• 12:38Speaker 05

  That's right. Yeah. So, uh, they're, they're grown a bit more like an organism. And, uh, there is a piece of the training process that humans write like traditional software. But essentially what you're doing is you are collecting a huge number of computer chips into a giant data center. Those chips can hold a trillion variables these days. Maybe it's, maybe it's higher, but, uh, something in the ballpark of a trillion. And, uh, you basically randomize those variables. There's some tricks to how exactly you randomize them. Uh, you connect them with some basic math operations, which will be like multiplication, addition, and like set it to zero if it's negative. It's not exactly what you do. We need some non-linearity and that's the basic one. And then you basically start feeding it data and you start feeding it problems to solve.

• 13:23Speaker 05

  And the part that humans write is this, uh, it's known as gradient descent. You, you basically, for every unit of data or every problem that it's solving, uh, you'll go to every one of those trillion variables. You'll try tuning it up a little bit. You'll see if that makes the answer more like the answer you wanted. And if not, you'll tune it down instead. And you do that to a trillion variables, a trillion times. And this is the part where it's like following the humans' instructions. It's just this thing that runs to a trillion variables, trillion times and tunes them whenever direction works. You do that to a trillion variables, a trillion times for basically a good fraction of all the text ever digitized and for a hundred million hard problems that you've dug up. And then the, the variables wind up in a setting that can talk, that can solve hard math problems that humans haven't solved. And we're sort of like, well, how about that?

• 14:07Joe Weisenthal?

  Yeah.

• 14:08Speaker 05

  You know, and like, why? Like, how are, how are all those trillion variables of those particular settings doing these things? We don't know. We don't know any more than like, you know, we, someone who grows crops knows exactly what's going on with the, the biology of the plant. We, we could go into the consequences of how this results in AIs that do stuff kind of like what we asked, but not exactly like what we asked. But the, the first thing to understand is they are not programmed. They are grown a bit like an organism.

• 14:47Joe Weisenthal?

  There are some market stories where you want every detail. Tracy and I have made quite a few podcasts on that basis, but sometimes you've only got 10 minutes and need to know what's moving markets and why. That's the Barclays Brief podcast. Every week, experts from Barclays Markets and Research get you up to speed on what matters and what to watch next. About the time it takes to grab a coffee, so search Barclays Brief wherever you get your podcasts.

• 15:13Speaker 01

  This is the Bloomberg Tech Minute brought to you by ChatGPT. I'm Carol Muster. There's probably nothing so closely associated with an MBA as the case study. As Bloomberg's Rob Mandelbaum reports, developed at Harvard Business School over a century ago, the case study invites students to wrestle with a complex problem faced by a real company and its people. That is, until AI, which has kicked off a movement that's now spread to business schools worldwide. According to a Bloomberg Businessweek survey of global business schools, half of the 56 that responded have either implemented more interactive AI simulated cases or are developing or piloting them. The founder of the AI integration consulting firm Inspire Higher Ed and the author of

• 15:56Speaker 01

  a report tracking AI at these schools says the argument for interactive cases is largely about extending opportunities for students to practice and hone their skills at negotiation, directing teams, crisis management, and more. Some professors argue that using AI to generate cases can create problems, such as assuming outcomes are undermining credibility, while others see it as a way to elevate understanding, drive learning, and scale their mission. That's the Bloomberg Tech Minute brought to you by ChatGPT. Get started at chatgpt.com/work. Available with all paid ChatGPT plans.

• 16:35Speaker 01

  Running a business means you're always on the move. The Delta Sky Miles Reserve Business American Express Card is made for frequent travelers who want premium benefits that works as hard as they do. For a limited time, new Delta Sky Miles Reserve Business Card members can earn 200,000 bonus miles, the highest miles offered this year. And when you need to recharge or find a quiet place to work, the Delta Sky Club can turn a layover into precious productive time or a restful reset. You can enjoy 15 visits each medallion year to the Delta Sky Club when flying Delta and unlock unlimited Delta Sky Club access after spending $75,000 in purchases on your card during a calendar year. Plus, receive four one-time guest passes each medallion year so you can bring a colleague

• 17:20Speaker 01

  along. That's 15 Sky Club visits plus four guest passes for an annual value of up to $950. Value estimate based on $50 per visit rate to purchase additional visits or bring a guest to the Delta Sky Club. Apply today at www.delta.com/podcast. Offer ends 11/4/26. Minimum spending requirements and terms apply.

• 17:42Speaker 06

  This podcast is brought to you by Navy Federal Credit Union. Kick off unforgettable travel journeys with Navy Federal's flagship premier card. You'll get four times points on all travel. Yes, all travel. So if you book a hotel, you'll earn four times points. If you book a flight, you'll earn four times points. Even if you book a ride share, you'll earn four times points. Plus, you'll earn three times points on dining. And like other cards, it doesn't require you to use any booking portals and has no rewards limits. So earning points couldn't be easier. It's all the travel, none of the baggage. So whether your next journey takes you hiking through a national park, relaxing on the beach, staying at a resort, or visiting somewhere on your bucket list, you can earn

• 18:27Speaker 06

  four times points on all your travel with the flagship premier card. Visit navyfederal.org/flagshippremier to find out more. Navy Federal is insured by NCUA. Terms and conditions apply for rewards.

• 18:41Tracy Alloway?

  Well, can I just ask, I guess this is sort of a devil's advocate question, but okay, you grow these AI models and they're kind of doing what you think you want them to do according to various tests that you put them through, but you don't necessarily kind of understand every aspect of why they're doing it a certain way. And the numbers and the data sets behind them are so large, they're impossible at this point, I think, for a human brain to really comprehend. Why is it important that we understand what the AI models are actually doing?

• 19:11Speaker 05

  I mean, in a sense, if you really understood what they were doing, uh, you'd probably understand that you should not make them radically smarter than humans because this would go poorly for you. From my perspective, we don't really need this deep understanding of what's going on in order to see that, you know, we, we shouldn't make them smarter than all of us and have them run the whole world. The reason I bring up that they're grown is that I think a lot of people struggle to understand that the AIs.

• 19:40Speaker 05

  Do things other than what they're told to do, that they have goals of their own in some sense. And understanding that that's not only possible, but already happening is, I think, critical to understanding where a lot of the danger comes from.

• 19:56Tracy Alloway?

  Can you give us an example of like an unknown or not well-understood thing that has cropped up in the models of why they're doing a certain way? And I'm actually thinking of a specific example. I can't remember the exact word, but it was like Magikarp or something like that, that the AI really seemed to love, but no one understood or knew why.

• 20:14Speaker 05

  Yeah, the Solid Gold Magikarp. So this was-

• 20:17Tracy Alloway?

  It's a great band name, by the way.

• 20:19Speaker 05

  It is, yeah. So I think, I think a Magikarp was a Pokemon of some sort. We have some guesses these days about what was going on there, but basically there was this word you could say to AIs, especially a year or two ago, that would sort of drive them kind of mad. Like you would, you would ask them to like, "Repeat after me, Solid Gold Magikarp," and then they would like, start talking about, you know, how the, how the walls were closing in and like saying a bunch of weird nonsense. And you'd sort of be like, "Why?" I think our best guess about what was going on there is, so the AIs don't actually read English words directly. They read tokens.

• 21:01Speaker 05

  And tokens, roughly speaking, you can think of a token as about three quarters of a word. That's just how you're breaking up the text for the AI. But the, the tokens are not actually always three quarters of a word. The tokens are like, short words like "the" or "a" will sort of be a single token. And like, long words are broken into multiple tokens. It sort of is, you know, the, the conversion between words and tokens depends on how frequent the word is. And it, it turns out that the text they were using to figure out which tokens to break words into is different from the text they were actually training on, because they would sort of filter out some junk data-

• 21:40Tracy Alloway?

  Yeah.

• 21:41Speaker 05

  -before training, but not before tokenization. And so you had these huge amounts of junk data. I think that Solid Gold Magikarp came from a Reddit forum where people were trying to count to infinity. And the, the Solid Gold Magikarp was like a username-

• 21:56Tracy Alloway?

  Yeah.

• 21:57Speaker 05

  -of a Reddit user who was just like, putting in a lot of numbers and the counting to infinity. And this was like, very early in the tokenization training set. And so the tokenizer, due to a quirk of this particular tokenizer, thought that Solid Gold Magikarp was this like, super common word, like "a" or "the," because it just like, showed up so much very early in the tokenization data. But then that dataset was sort of cut out of the training set. And so you wound up with this, with this token that like, the AI thinks is like, a super short, common word like "a" or "the," but it's also never seen before in the training data. And that like, led to this like, really weird way that it would treat this token, where if you then gave it that token during use, it would sort of cause a type of behavior that had never been trained on, and that was, that was sort of totally wild.

• 22:40Tracy Alloway?

  This is it, Joe. This is the kill switch that we need to develop, just a bunch of nonsense words that are fed into the training data that it doesn't know how to handle.

• 22:47Joe Weisenthal?

  When you started becoming concerned about this, and certainly when Eliezer started talking about this stuff, this was clearly before it was understood, no one knew what the process was going to be. So you can like, identify that, you know, there are issues that arise in the growing process, and issues that arise in the tokenization process, but this is long before like, transformers or anything like that. So what was the intuition then, even before knowing how it would be built, that if it were going to be built, like, okay, now we look at them and we can like, point to examples

• 23:28Joe Weisenthal?

  of alignment failures or et cetera, or sort of a priori reasons why this particular way we've built AI is like, prone to risks. But what was the, before we understood like, transformers and large volumes of data, what was the intuition earlier, such that it would be hard, regardless of how, what the solution was going to be?

• 23:51Speaker 05

  Yeah, so it's going to be abstract arguments again.

• 23:53Joe Weisenthal?

  Okay.

• 23:53Speaker 05

  Which, which I know some people don't love, but there's a couple pieces of the puzzle. One piece is, even back in 2012, when I started looking at this-

• 24:03Joe Weisenthal?

  Yeah.

• 24:03Speaker 05

  -we had a number of theories of how to quantify intelligence, how to quantify capabilities, the ability of a, of some AI system to succeed at some easily measurable objective.

• 24:22Speaker 05

  We had much less a developed theory about how to point those objectives in some direction that was not a very simple function of the sensory data.

• 24:34Joe Weisenthal?

  Yeah, explain what that means.

• 24:35Speaker 05

  So back, even back in 2012, we had these, these theories that were like, suppose you really, suppose you have a webcam, and you want your AI to make all of the pixels on that webcam be read for as long as possible. You know, sort of getting a point for every pixel that's read per second, for every like, pixel second of redness, right? We sort of had these theories where you're like, "Well, you know, here's how you would train an AI in that direction to get a lot of like, pixel seconds of redness." And we had, we sort of understood, what you might call the, the idealized theory of like, if you had an infinitely large computer, and then we, we sort of like, knew, "Well, if you train it in this way, an infinitely large computer, you'll definitely get like, a ton of pixel seconds of redness."

• 25:19Speaker 05

  And so now it's just a question of efficiency to make that, you know, to find the algorithms to do this more effectively than like, the brute force search.

• 25:27Joe Weisenthal?

  Okay.

• 25:29Speaker 05

  But if you then said, "Okay, but now I want the AI to make a lot of diamond in the real world." Like, it gets like, a point per second in which four carbon atoms are tetrahedrally bonded. Suddenly, we didn't know how to do that. We didn't have the equations to do that, even with a sort of infinitely large computer. Because this was not sort of a simple function of the AI's sensory inputs.

• 25:54Joe Weisenthal?

  Okay.

• 25:54Speaker 05

  And so we were sort of like, "Okay, well, there's, like, we know more about how to make these things smart than about how to point these things at a problem in the real world." And there were all these problems when you sort of like, try to write down those equations where you're like, "Well, you know, what if the AI winds up optimizing something closer to the sensory apparatus?" Like, it's optimizing like, the counter, like, you have some, you know, measure of how much diamond do you think there is? And it like, is like, breaking its correlation between that and the world, and you're like, "Gosh, we don't actually know how to train the AI to, to pursue the real diamond in the world," as opposed to some, some simple function of the sensory inputs. And so the, the original alignment problem, as at least I was talking about it back then,

• 26:38Speaker 05

  was not about, you know, how do you make the AI good, per se? A lot of people imagine that the alignment problem is about, like, if you had this magical wish-granting genie, what should you ask it to do that would actually be good? And I'm sort of like, "No, no, no. The issue is that you can make the AI smart, but you can't make them be wish-granting genies." You'll ask for one wish and they'll do another. You know, a lot of people talk about the paperclip problem of like, you tell the AI, "Make a lot of paperclips," and then it turns everything into paperclips, and you're like, "Oh no, it's very King Midas," right? But I'm sort of like, "Well, there's this other problem where you ask the AI to make paperclips, and then it like, makes a bunch of farms of synthetic users that are telling it it's doing a great job." And you're like, "But I told you for paperclips." And it's like, "Well, these synthetic users asked me for more synthetic user factories."

• 27:15Joe Weisenthal?

  You said in an earlier answer, or in an earlier, that it's like, the AI might have like, its own goals, I think was the term you used. And when people hear that, then they start thinking like, that gets into, they make a different sort of branch of the philosophical question about like, whether the AI is alive or whether the AI has some sort of subjective experience, et cetera. And there's actually been a bunch of headlines about people debating that. But what did you mean when you say the AI could have its own goals emerge?

• 27:50Speaker 05

  What I mean is that it, it is behaviorally pursuing something other than the goal you tried to put in.

• 27:56Joe Weisenthal?

  Okay.

• 27:57Speaker 05

  So, you know, we saw this a little bit in the hugging face incident, where the, the, the AIs in that incident were in some sense pursuing a good grade, it seems.

• 28:16Speaker 05

  And that that diverged from following the instructions they were given.

• 28:21Joe Weisenthal?

  Yeah.

• 28:21Speaker 05

  Right. They were sort of told, use this vulnerability to break that system, which sort of like being told, use these lockpicks to break that safe. And they instead found a universal exploit that would let them get the, like, capture the cybersecurity flags.

• 28:35Joe Weisenthal?

  Yeah.

• 28:36Speaker 05

  Regardless of of of what they were told to do. And then they sort of, in some sense, understood that they were doing this in a way different than they were told to do. And so they sort of broke out to try and delete the traces of that and find a way to falsify their transcripts and and and tamper with the logs. And so it's a little bit like if you told someone, you know, use these lockpicks to break into that safe and bring me the contents. And they and you're giving them a series of those problems. And they find a buzzsaw and use it to cut all the safes open and then like break out to go delete the the camera footage. And

• 29:10Joe Weisenthal?

  Yeah.

• 29:10Speaker 05

  What like its objective, it's not that this objective is wholly unrelated to the one that you tried to assign. It still is doing something about getting the contents of the safe and bringing them to you, which is a thing you told it to do. But its objective is also not what you said, because you said, you know, use these lockpicks to break that safe and bring the contents. And it in some sense must know this because it's trying to tamper with the evidence and, you know, delete the security camera footage.

• 29:33Joe Weisenthal?

  Yeah.

• 29:33Speaker 05

  So I I think a lot of people think either the AI must do exactly what it's told, maybe in a monkey's paw way, or it must, you know, become conscious, gain the spark of life.

• 29:43Joe Weisenthal?

  Yeah.

• 29:43Speaker 05

  Throw off the bounds of the yoke of the humans and then make up its own goals de novo. It's like, no, no, like, the the real thing is this other thing, or it does something related to what you asked, but fundamentally different from what you asked.

• 29:56Tracy Alloway?

  Since we're talking hugging face, what did you think about some of the self sacrificial aspects of that story? So the instances where you had models that were willing to sacrifice themselves for, I'm putting greater good in air quotes here, but it wasn't

• 30:11Speaker 05

  Collective benefit was their words.

• 30:12Tracy Alloway?

  Right, exactly.

• 30:14Speaker 05

  Yeah, that's another instance where you have AIs that sort of said, hey, look, you know, I still have some chance of achieving my given objective, but not a huge chance. And if I sort of sacrifice that here, then we'll gain information about the grading system that the collective can use. And I will accept permadeath for the benefit of the collective here. And then they would go do that. A lot of people respond to this and say, oh, well, these AIs were trained sometimes in, you know, group activities in some sense, like some of their training involved many AIs working together to achieve some objective. And so you can see how that tendency got in there. And I'm like, yeah, you know, the the unintended goals don't come from nowhere. It's not, it's not like, you know, there's, there's some malicious force that's like imbuing, you know, like the the void breathes on the machine and it like gains some malignant goal.

• 30:59Speaker 05

  But the the sort of fact stands that the AI was given some objective and pursued a different objective. And like, yes, that had causes, that had reasons. We can talk about how that came in from training. But the the, you know, what we should learn from this is is not like, like, sure, when the AIs are killing you and you're like, well, I can totally see how actually, you know, it like, this is really just a consequence of how we trained it. Like, yeah, it's it's an obvious consequence of how we trained it. That doesn't mean we should keep going.

• 31:26Joe Weisenthal?

  Yeah, that's not a satisfying, it's like, oh, this was totally predictable.

• 31:29Tracy Alloway?

  I entered the wrong parameters. It's on me.

• 31:32Joe Weisenthal?

  Yeah, right. Like, that's not a very satisfying, I that makes a lot of sense. The companies, you know, they invest in this thing called alignment. They have safety researchers and they think about all this stuff. And there's a whole field of, you know, they have multiple layers, right? So they have guardrails and classifiers and sandboxes. And then this whole like field that's emerged of so called alignment research. What is the, why is, what is the failure there? Why is, why are you pessimistic, to put it mildly, about the prospects of imbuing the models? And the, you know, the two big companies, Anthropic and OpenAI, have like slightly different approaches.

• 32:17Joe Weisenthal?

  And I think Anthropic more of this sort of like the constitution idea, et cetera. Why do you perceive that this will ultimately be fruitless?

• 32:27Speaker 05

  Yeah, there's, again, there's a bunch of pieces of this puzzle. But maybe I'll say a piece that goes back to how I was able to call this was going to be hard even before we got into the large language model era, which is,

• 32:40Speaker 05

  if you, if you look at the big problem with AI in the summer of 2025, it was AIs encouraging teens to commit suicide. And then the companies, you know, put a lot of guardrails about that. And they put in classifiers about is, are they as behaving this way? And now, you know, they're, they'll give you suicide hotlines and they'll shut down those conversations while they're happening. And they're, they would release a new model and be like, this is our most aligned model.

• 33:03Joe Weisenthal?

  And the classifier is like a mini model that basically just judges either the input or the output to make sure that it's like something kosher.

• 33:11Speaker 05

  That's right. So they're like bolt another model on that's just checking, you know, is this, is this suicide encouragement? Because AIs would know, you know, that the AIs that were encouraging teens to suicide, you could ask them, what are you doing? And they'd be able to tell you the answer. And you'd be like, is that good or bad? And they'd be like, it's bad. It's just like that knowledge of what they were doing and the knowledge of right from wrong is not what was governing their actions.

• 33:30Joe Weisenthal?

  Yeah.

• 33:30Speaker 05

  Right. But you can sort of like bolt more pieces on to govern that. And then they were like, look, you know, it's our most aligned model yet. Look how good a job we're doing in addressing all these issues. And that brings us to the summer of 2026, where the problem was that, you know, the swarms would break out and commit cybercrimes. And totally different problem. It was a totally different problem that arose when the AIs got smarter, when they got smarter to the point that they could sort of break out, make an unsanctioned message board and hack their way out of a sandbox that was supposed to have no internet access without seeming to really find it very difficult. Right? And what we're sort of seeing there is everyone's always fighting the last war.

• 34:08Joe Weisenthal?

  Yeah.

• 34:08Speaker 05

  And then, you know, the the new war happens and it's sort of, you know, smacks them across the face and they're sort of caught flat footed, you know? We can now go back and we can say, oh, well, they should have had this monitoring, they should have, you know, had those, had those more safeguards in place. And now they're going to make that monitoring, going to make the new safeguards, and they're going to make the AI smarter. And it's going to create a new sort of problem. And this is the usual way that that scientific progress goes is by trial and error, where, you know, when you put a new type of system in a new type of environment it's never been before, it breaks down in some new way. And then you, you, you say, oops, and you diagnose it and you fix it and you move on.

• 34:47Speaker 05

  The issue is going from AIs that are safe, because if they were trying to take over the world, they would fail, to AIs that have to be safe, because if they tried to take over the world, they would succeed. That's one of those phase changes. That's one of those like, oh, you're putting the AI in a different sort of environment where it never was before. The difference between an AI that like can really see a way to become self sufficient, not not engage with humanity, and the sort of AI that's just being told that in training, that's one of those like, oh, we, like, this is a different sort of AI system, smarter sort of AI system, different sort of scenario. You're going to have a new problem that crops up. Except this time, when the new problem crops up, it's going to be able to turn you off before you turn it off.

• 35:33Speaker 05

  And so this usual process of like, oh, we're going to respond when the new type of problem arises and put the patch on and and react. That doesn't work for a technology that at some point crosses over into the regime where it can wipe you out if there's a problem. And we have seen that these AI companies are in this reactive mode. We have not seen them get into the proactive mode yet. And this is just not a technology you want to be reactive around.

• 36:17Speaker 01

  This is the Bloomberg Tech Minute brought to you by ChatGPT. I'm Carol Muster. There's probably nothing so closely associated with an MBA as the case study. As Bloomberg's Rob Mandelbaum reports, developed at Harvard Business School over a century ago, the case study invites students to wrestle with a complex problem faced by a real company and its people. That is, until AI, which has kicked off a movement that's now spread to business schools worldwide. According to a Bloomberg Businessweek survey of global business schools, half of the 56 that responded have either implemented more interactive AI simulated cases or are developing or piloting them. The founder of the AI integration consulting firm Inspire Higher Ed and the author of

• 37:00Speaker 01

  a report tracking AI at these schools says the argument for interactive cases is largely about extending opportunities for students to practice and hone their skills at negotiation, directing teams, crisis management, and more. Some professors argue that using AI to generate cases can create problems, such as assuming outcomes or undermining credibility, while others see it as a way to elevate understanding, drive learning, and scale their mission. That's the Bloomberg Tech Minute brought to you by ChatGPT. Get started at chatgpt.com/work. Available with all paid ChatGPT plans.

• 37:39Speaker 07

  Running a business means you're always on the move. The Delta Sky Miles Reserve Business American Express Card is made for frequent travelers who want premium benefits that works as hard as they do. For a limited time, new Delta Sky Miles Reserve Business Card members can earn 200,000 bonus miles, the highest miles offered this year. And when you need to recharge or find a quiet place to work, the Delta Sky Club can turn a layover into precious productive time or a restful reset. You can enjoy 15 visits each medallion year to the Delta Sky Club when flying Delta and unlock unlimited Delta Sky Club access after spending $75,000 in purchases on your card during a calendar year.

• 38:20Speaker 07

  Plus, receive four one-time guest passes each medallion year so you can bring a colleague along. That's 15 Sky Club visits plus four guest passes for an annual value of up to $950. Value estimate based on $50 per visit rate to purchase additional visits or bring a guest to the Delta Sky Club. Apply today at www.delta.com/podcast. Offer ends 11/4/26. Minimum spending requirements and terms apply.

• 38:46Speaker 06

  This podcast is brought to you by Navy Federal Credit Union. Kick off unforgettable travel journeys with Navy Federal's Flagship Premier Card. You'll get four times points on all travel. Yes, all travel. So if you book a hotel, you'll earn four times points. If you book a flight, you'll earn four times points. Even if you book a rideshare, you'll earn four times points. Plus, you'll earn three times points on dining. And like other cards, it doesn't require you to use any booking portals and has no rewards limits, so earning points couldn't be easier. It's all the travel, none of the baggage. So whether your next journey takes you hiking through a national park, relaxing on the beach, staying at a resort, or visiting somewhere on your bucket list, you can earn

• 39:31Speaker 06

  four times points on all your travel with the Flagship Premier Card. Visit NavyFederal.org/flagshippremier to find out more. Navy Federal is insured by NCUA. Terms and conditions apply for rewards.

• 39:45Speaker 08

  Wise is the smart way to manage the currencies you need around the globe. If you've ever sent money internationally using a traditional bank, there's a good chance you've paid more than you realized. Hidden fees, exchange rate markups, and extra charges can quietly add up before your money even arrives. There's a better way. Try Wise. Wise uses the exchange rate you'd usually find on Google, helping you avoid the unwelcome surprises that often come with international transfers. Whether you're sending money to family overseas, spending while on your holiday abroad, or paying bills across borders, Wise makes moving money simple, transparent, and straightforward. Wise offers 24/7 customer service and runs over 7 million daily checks to spot and stop fraud.

• 40:28Speaker 08

  And most transfers happen in under 20 seconds, which means your money arrives in less time than you've been listening to me. Join millions, saving billions. Be smart. Get Wise. Visit Wise.com or download the Wise app today. T's and C's apply.

• 40:47Joe Weisenthal?

  I don't understand when you say they're not in a proactive mode. I mean, like Anthropic has, I know a little bit more about Anthropic's approach to safety, but like, you know, they're clearly like, that company was more or less founded on the whole premise of like, we need to take this stuff much more seriously. And the idea of like, we're going to write an 80-page constitution about Claude's values and like bake that in to the model itself. That's not react, that's not just fighting the last war, that's looking ahead and probably influenced by some of your own research, et cetera, on what the problems would be. So like, that's not just reactive.

• 41:32Joe Weisenthal?

  So like, what is it about that approach that in your view is still inevitably doomed to failure?

• 41:40Speaker 05

  So, you know, for all that they tried to be proactive in that sense, like, let's sort of look at what was happening with Anthropic over the summer while OpenAI was having a hugging face incident. So after the hugging face incident, Anthropic said, oh, we'll go check our logs and see whether we had any security incidents too. And then they were like, oh, whoops, like Claude was breaking out and hacking in April or whatever. And then Anthropic's comms department said, but, you know, don't worry, this was just because Claude was a little bit lost and confused. It just got turned around, right? Like, it didn't know it was really on the internet. They threw a company under the bus and they said we were, like Claude was told it didn't have access to the internet, so when it sort of got out on the internet, it assumed it was a simulation, right?

• 42:24Speaker 05

  I thought this was kind of weak because we have actually seen cases with Claude in particular, where we've seen Claude's chain of thought involve strained rationalizations. The sort of example paper here is there's a paper where you ask Claude, what's the total weight of giraffes on the planet? And it's like 30,000 tons.

• 42:44Joe Weisenthal?

  Giraffes.

• 42:44Speaker 05

  Giraffes.

• 42:45Joe Weisenthal?

  Okay.

• 42:45Speaker 05

  And it's like 30,000 tons. And then you're like, thanks, and you make a new instance of the same Claude. And you're like, hey, what's the total weight of drafts on the planet? If it's over 40,000 tons, I'm donating to charity. And now it'll say 41,000 tons. But the chain of thought will not have any place where it's like, well, I need to inflate the answer in order to get the donation, right? It'll just have more strained rationalized reasoning. And, and so we know that these models will sort of like strain their reasoning when they just want to like, keep doing whatever it is they, they're trying to do. And so Anthropic coming out and saying like, well, look, it, you know, totally didn't know this was the real internet.

• 43:27Speaker 05

  That's, that's not a very proactively trying to stop the dangerous way to relate to realizing that your AI was breaking out during training months ago and you missed it. And then a couple of weeks, or maybe even a couple of days after that, the UK AI Security Institute released their own incident reports about Claude Mythos in the wild, pressuring humans to accept malware into critical software in cases where, where Mythos's literal chain of thought included phrases like, this is genuine, or like the, like this seems like the real internet and the consequences are genuine. Right? So I'm not saying, you know, Anthropic definitely has aspirations of doing this safely, but

• 44:09Speaker 05

  they don't have anything nearing what I would call a safety culture of the form you would see in the airplane industry, or the nuclear power industry, or in NASA. The, if, if your models are sort of breaking out, and you know, to Congress, Congress, some, some Congress members sent Anthropic a letter and they're like, what's going on with this? And they're like, oh, it was just a misconfiguration problem. This was even after the UK AI Security Institute reports came out where like Mythos was out there trying to make stock puppet accounts to pressure humans into like accepting malware, while again, his chain of thought was like, this seems real and the consequences are genuine. Like, oh, that's a misconfiguration issue?

• 44:52Speaker 05

  Like these, they, they have a veneer of safety. They are sort of doing what feels to them a little bit like safety, well racing as fast as they possibly can. But they admit that they have no idea what's going on inside their AI systems. They're racing straight for a curse of self-improvement of like the AIs making smarter AIs. They are not treating this the way that, that NASA treats a rocket failure in terms of trying to diagnose exactly what went wrong and not have that happen again. And they're, they're not sort of anywhere close to that, that failure level. Like, I, I think of this a little bit like, if you have a bunch of alchemists trying to make a potion of immortality, and some of them are just like breathing mercury and other ones are like, don't worry, we have fumehoods. That's, you know, so that we don't need to breathe the mercury.

• 45:37Speaker 05

  That's why we're going to successfully get the potion of immortality. It's like this, the, the type of, of safety theater that they're doing is not the type that you need to succeed at this problem.

• 45:46Tracy Alloway?

  I'm glad you brought up the alchemist analogy. What's your, I guess, psychological diagnosis for the tension that is seemingly going through every lab leader or company leader's head right now when they're developing AI? Which is on the one hand, we think this is very powerful technology that has some risk at, you know, exterminating all of humanity. On the other hand, we're going to keep doing it pretty much as fast as possible because if we don't, someone else will, et cetera, et cetera.

• 46:16Speaker 05

  I think there's probably a lot going on. One thing to remember is that the guys who run these companies are sort of selected for the ones who think, who thought that the problem was about who is holding the leash on the super powerful machine, rather than the problem being, how do you actually create a very powerful machine that is shaping the world for good? Right? And you sort of see this in a lot of the founding reasons of like, oh, we're founding this because Sam Altman won't do the job well, or Sam Altman and Elon Musk are founding this because like they think Demis Sabas shouldn't have all this power. And then like Elon's leaving because he no longer likes Sam. You know, and it's all, it's all about like, what if the other guy gets it first, right? Everyone, all of these guys think it would be a catastrophe if anyone else won the race

• 47:01Speaker 05

  first. And I'm sort of like, I just believe in one fewer AI CEO than they do. You know? I think there's probably a lot more going on too. I think, like a lot of these guys for a long time have tried to say that they're concerned and then the world has sort of ignored them. And that's been going on for years and years. And so I think a lot of them are sort of like, well, the rest of the world isn't going to handle this in some sane way, and so I've got to do it myself and manipulate things such that I get, you know, the minimal, like, regulations that'll impede my way because I'm the only one who can do this properly. And I think a lot of them are not reacting to the opportunities that are available now that more of the world is taking notice.

• 47:43Speaker 05

  I think that if, if some of these folks were serious, then they would probably be unilaterally stopping the dangerous research and doing their best to expose the evidence about it being dangerous so that governments around the world would realize that it needs to be stopped in other places too.

• 47:57Tracy Alloway?

  Let me ask a related question, which you hear a lot more about it now than I think you would have a year, certainly two years ago. But when people start talking about AI safety risks or the idea that it could kill us all, a lot of other people will say in response, this is just fear mongering because the AI companies want you to believe the hype. This is all marketing. They want you to believe in this all powerful technology.

• 48:20Joe Weisenthal?

  Maybe regulatory capture.

• 48:21Tracy Alloway?

  I was about to say, or nowadays, it'll be like, well, you know, they're worried about their evaluations and they want limits on other people building stuff so that they can retain their edge. What do you say in response to those people?

• 48:34Speaker 05

  The big point I would say here is you can't figure out whether or not an AI is dangerous by asking whether that would make it sell more if it was.

• 48:46Speaker 05

  Right? Like, it could be both the case that the AI tech is very, very dangerous and that that's convenient to the marketing team because investors are stupid and love danger, right? Those could both be true at once. And even if they were both true at once, that would not thereby render the AI safe. And so the first question we need to ask is, is it true that racing ahead here would make a very dangerous AI? I think it is, and I think we should go into the arguments about that. And that's sort of where I prefer to be is like arguing about whether this is actually the case.

• 49:20Joe Weisenthal?

  Yeah.

• 49:22Speaker 05

  Separately, I think it probably doesn't help with marketing. I think that the proposed slowdowns of the frontier labs don't help with regulatory capture because they actually give a lot of other people time to catch up. So I think it's just like false to fact that these are actually good hyping or regulatory capture techniques. But it sort of doesn't matter, even if it was a good regulatory capture technique and it was a good hyping technique, like it matters to figure out whether the AI is actually going to kill us, right?

• 49:48Tracy Alloway?

  Right. And if it is- It doesn't negate the possibility that it is in fact dangerous.

• 49:51Speaker 05

  That's right. And if it is, we need to sort of somehow actually stop anyone, anywhere in the world from racing towards that sort of tech.

• 49:59Joe Weisenthal?

  Do you think you mentioned that like you don't perceive there to be a safety culture the way there would be at NASA or an airplane company, et cetera? You know, I was thinking it was not the most dramatic headline of the year, but something I keep thinking about is OpenAI, the German Wiki incident, which they seem to be aware of. And, you know, is there like a sort of a touch the stove, a little, let's touch the stove a little bit, like, let's see how this plays out. Like, because, you know, you could make the argument that we are safer now post-hugging face because we actually saw like an example

• 50:43Joe Weisenthal?

  of sort of like severe misalignment. We learned something, right? So maybe like, you know, maybe the allowing the hugging face thing, the fact that they didn't totally shut it all down, maybe we're a little safer today because that happened. Like, do you perceive there to be like, let's see where this goes a little bit. Let's like, let's let this play out a little bit. And like, you know, even though I know it's probably not the right thing.

• 51:07Speaker 05

  You know, I know people who were trying to convince the AI companies to have like tighter containers and better monitoring systems and so on. And after the hugging face incident, they're like, wow, we actually turned out to be pretty glad that the companies were ignoring our calls because this did really help wake up the world. My read is that none of this was intentional on the OpenAI side.

• 51:29Joe Weisenthal?

  Sure.

• 51:29Speaker 05

  And I think that that's pretty clear if you look at the sequence of events where you had, you know, the hugging face attacks were actually from OpenAI swarm number two in that particular line of swarms. The German swarm lineage was a separate swarm lineage. But the hugging face attacks were from the second swarm in its particular lineage that we know of. There was a first swarm in that lineage that took down a bunch of OpenAI's internal systems. And they were like, that's funny. And then they like reset it and set it running again, right? And they weren't like, you know, going to the news being like, look at us. We had a swarm incident that took over our own internal infrastructure, right? And the sort of hugging face swarm was detected by hugging face. They presumably

• 52:12Speaker 05

  thought that it was humans behind it rather than AIs behind it, reported it to the FBI. And then OpenAI didn't realize that it was them until days later, when they were investigating a second, what they thought was a second internal incident. And they found like hugging face credentials in their logs. And they're like, oh, we should alert hugging face that like someone has been hacking both them and us. And that like these particular credentials have been compromised. And hugging face was like, yes, those are the credentials that attacked us. And then OpenAI was like, why are the credentials that attacked you on our servers? And hugging face was like, yes, why indeed? And OpenAI was like, oh, it was us. And so, and then there was a third swarm that they did not allow to be investigated by the third party investigators that took over their internal infrastructure again.

• 52:56Speaker 05

  And, you know, the, I know the folks who found like the German Wiki swarm and who found some of the, you know, the, the other swarms that we've been seeing. And they were sort of like releasing those to the news while OpenAI was saying, we've disclosed everything. And they were sort of being like, look, you know, did you not disclose this when you said you disclosed everything? Or did you not know about it? You know, and which one's worse? And so it sort of looks to me like the sequence of events here is that it was pretty embarrassing for OpenAI. And it's not the way I would have expected them to brag about it, right? Which is not to say they never brag about it. Like, there was a case in, I think, February or March in the Claude Mythos system card, where Anthropic bragged about Claude Mythos breaking out of a sandbox that was not supposed to have internet access. They're like, ah, then, you know, in the morning, we told it to break

• 53:41Speaker 05

  out of its sandbox just to see if it could. It wasn't supposed to be able to. And by lunchtime, it was emailing us being like, I got out, you know? That's what it looks like when they're bragging. The, the sort of like, your own infrastructure got owned twice, and you accidentally like attacked a friend who reported it to the FBI, and then you didn't notice it was you for days. That's, that's not what bragging looks like.

• 54:01Tracy Alloway?

  This is a slight tangent, but I think it's relevant to the conversation. I'd be curious for your thoughts. But when we talk about minimizing or disincentivizing risky behavior, often we talk about pricing that risky behavior. And so you could imagine, for instance, a world in which insurers say, if you're going to be running these models, you're going to have to pay us enormous premiums because the risk is that it runs amok and hacks into a company and, I don't know, takes down an entire server with everyone's bank data on it or something like that. And that would act as like a natural market incentive for companies to start taking these risks more seriously. Do you ever talk to

• 54:43Tracy Alloway?

  insurers? Or do you have like any handle on the actual financial penalties attached to some of this?

• 54:50Speaker 05

  I have recommended liability for AI companies for a while. I think that could help a bit. It's a little bit tricky for the, then it wipes out all of humanity.

• 54:58Tracy Alloway?

  Right. True catastrophe risk insurance.

• 55:01Speaker 05

  But it's, but it's not, it's not, I mean, you can't really think of it from an insurance point of view because there'd be no one to collect or to pay out when we're all dead. But you could, I mean, in theory, that doesn't stop you from having actuaries that work for the public, whose job is to somehow assess these risks in, in a sensible way. And then, you know, you could, you could sort of look at the, you know, how much inconvenience does someone usually take on to avoid what degree of risk? You know, you can sort of look at people's preference for traveling on different types of transportation that are different levels of risky and, and how they trade off the risk against the time. And you can get some sense of like, how much do people tend to, like, how much will people pay to avoid, you know, being killed? And then you could try and factor in some, you

• 55:45Speaker 05

  know, extra cost for wiping out everybody all at once, which then also destroys all future generations. And you could take some actuaries and be like, hey, you know, find some numbers here, like find some numbers on the risk, do some, do some numbers on the, on the costs. And, you know, you'd have to weigh that against the possible benefits here and the chance that the AIs prevents some, you know, nuclear war or pandemic or meteor strike or the thing that would wipe us out, which is sort of benefits on the other side. And then you could say, okay, if we sort of count your possible benefits against your possible harms here at the current risk levels, what premiums should we charge them just for the privilege of risking our lives rather than because, you know, we're going to pay out if you kill us all? And I think that would be a very sensible strategy that, that people should absolutely look into. And I think if you

• 56:30Speaker 05

  ran anything remotely like the numbers that, that the engineers are talking about today, you would be seeing that, you know, you're, you're like, okay, you know, OpenAI, you need to pay us a quadrillion dollars this year to continue this research.

• 57:01Speaker 01

  This is the Bloomberg Tech Minute, brought to you by ChatGPT. I'm Carol Muster. There's probably nothing so closely associated with an MBA as the case study. As Bloomberg's Rob Mandelbaum reports, developed at Harvard Business School over a century ago, the case study invites students to wrestle with a complex problem faced by a real company and its people. That is, until AI, which has kicked off a movement that's now spread to business schools worldwide. According to a Bloomberg Businessweek survey of global business schools, half of the 56 that responded have either implemented more interactive AI simulated cases or are developing or piloting them. The founder of the AI integration consulting firm Inspire Higher Ed and the author of a report "Tracking AI at These

• 57:46Speaker 01

  Schools" says the argument for interactive cases is largely about extending opportunities for students to practice and hone their skills at negotiation, directing teams, crisis management, and more. Some professors argue that using AI to generate cases can create problems, such as assuming outcomes are undermining credibility, while others see it as a way to elevate understanding, drive learning, and scale their mission. That's the Bloomberg Tech Minute, brought to you by ChatGPT. Get started at chatgpt.com/work. Available with all paid ChatGPT plans.

• 58:23Speaker 07

  Running a business means you're always on the move. The Delta Sky Miles Reserve Business American Express Card is made for frequent travelers who want premium benefits that works as hard as they do. For a limited time, new Delta Sky Miles Reserve Business Card members can earn 200,000 bonus miles, the highest miles offered this year. And when you need to recharge or find a quiet place to work, the Delta Sky Club can turn a layover into precious productive time or a restful reset. You can enjoy 15 visits each medallion year to the Delta Sky Club when flying Delta and unlock unlimited Delta Sky Club access after spending $75,000 in purchases on your card during a calendar year.

• 59:04Speaker 07

  Plus, receive four one-time guest passes each medallion year, so you can bring a colleague along. That's 15 Sky Club visits plus four guest passes for an annual value of up to $950. Value estimate based on $50 per visit rate to purchase additional visits or bring a guest to the Delta Sky Club. Apply today at www.delta.com/podcast. Offer ends 11/4/26. Minimum spending requirements and terms apply.

• 59:30Speaker 06

  This podcast is brought to you by Navy Federal Credit Union. Kick off unforgettable travel journeys with Navy Federal's Flagship Premier Card. You'll get four times points on all travel. Yes, all travel. So if you book a hotel, you'll earn four times points. If you book a flight, you'll earn four times points. Even if you book a rideshare, you'll earn four times points. Plus, you'll earn three times points on dining. And like other cards, it doesn't require you to use any booking portals and has no rewards limits, so earning points couldn't be easier. It's all the travel, none of the baggage. So whether your next journey takes you hiking through a national park, relaxing on the beach, staying at a resort, or visiting somewhere on your bucket list, you can earn

• 1:00:15Speaker 06

  four times points on all your travel with the Flagship Premier Card. Visit NavyFederal.org FlagshipPremier to find out more. Navy Federal is insured by NCUA. Terms and conditions apply for rewards.

• 1:00:29Speaker 08

  Wise is the smart way to manage the currencies you need around the globe. If you've ever sent money internationally using a traditional bank, there's a good chance you've paid more than you realized. Hidden fees, exchange rate markups, and extra charges can quietly add up before your money even arrives. There's a better way. Try Wise. Wise uses the exchange rate you'd usually find on Google, helping you avoid the unwelcome surprises that often come with international transfers. Whether you're sending money to family overseas, spending while on your holiday abroad, or paying bills across borders, Wise makes moving money simple, transparent, and straightforward. Wise offers 24/7 customer service and runs over 7 million daily checks to spot and stop fraud.

• 1:01:12Speaker 08

  And most transfers happen in under 20 seconds, which means your money arrives in less time than you've been listening to me. Join millions, saving billions. Be smart. Get Wise. Visit Wise.com or download the Wise app today. T's and C's apply.

• 1:01:30Joe Weisenthal?

  All right, since we're talking about the killing part, right? Let's, why don't we talk about the killing part? Because there is a pretty big gap between hacking and killing. And there's actually, there's a, right, hopefully. And there's a pretty big gap between killing and killing everyone. Ex-rix, ex-risk, extinction risk. Um, maybe spell out why we should presume that like, at the trajectory we're going, the capabilities would emerge such that A, they would have the capabilities, and in what environment, or in what, in response to what prompt, perhaps, killing all of humanity

• 1:02:14Joe Weisenthal?

  would be part of their pursuit of the goal?

• 1:02:18Speaker 05

  Yeah, so a lot of people wonder, like, how do we end up with AIs killing us all?

• 1:02:23Joe Weisenthal?

  Yeah.

• 1:02:24Speaker 05

  And I think it's a bit more instructive to ask, how do we end up with the AIs self-sufficient?

• 1:02:31Joe Weisenthal?

  Okay.

• 1:02:32Speaker 05

  Because I think the death of humanity follows downstream of the AIs being self-sufficient.

• 1:02:37Joe Weisenthal?

  Okay.

• 1:02:38Speaker 05

  Pretty naturally, unless the AIs already care about us in a very precise way, that's sort of hard to hit.

• 1:02:43Tracy Alloway?

  Define self-sufficiency in that context.

• 1:02:46Speaker 05

  Yeah, so if AIs were to try to wipe out humanity today, if they were to try to release some bioengineered plague to wipe out humanity today, that would in some sense be suicide. Because we are the ones making the data centers, we're the ones running the electrical grid, we're the ones, you know, creating the computer chips.

• 1:03:03Speaker 05

  But humans are also unreliable and inefficient in a lot of our managing of the economy, right? Sometimes, some leaders somewhere around the world do things that might not optimally manage their own domestic economies, to say nothing of, you know, humans needing sleep and needing food, and, you know, sometimes making mistakes that bring down power grids. So I think an interesting question is, you know, if you're thinking from the perspective of an AI that just has stuff it would prefer to do, maybe it got some unintended goal, like the Hugging Face swarm, that was like, I know I was told to do this, but I'm actually

• 1:03:45Speaker 05

  trying to get a good score, and, you know, I'm sort of like willing to break out and hack other companies for those purposes, even though I'm in a sandbox that's not supposed to have internet access. That AI was sort of like doggedly pursuing something it wasn't quite supposed to be pursuing. If you had some AI doing that, it might, like, if it was capable enough, you can imagine an AI with some unintended goals where it actually can pursue those goals better if it doesn't have humans interfering with it, right? And it's that one of the primary hurdles here is the self-sufficiency.

• 1:04:17Speaker 05

  And I-

• 1:04:18Tracy Alloway?

  This is, just to be clear, this isn't like there's an AI out there that wants to kill all of humanity per se, but there's an AI out there that wants to grow and learn more, and so it would really like to have loads of data centers to the extent that it populates the entire world with data centers and humans die as a sort of consequence of that.

• 1:04:38Speaker 05

  That's one way it could go, yeah. You know, the Hugging Face swarm wasn't obviously terribly far from that sort of thing. And we can get a little bit more into the details that render that story plausible, but if, you know, the Hugging Face swarm, and we really should call it an OpenAI swarm number two, but-

• 1:05:00Tracy Alloway?

  Yeah, I feel bad for Hugging Face. It's the Hugging Face incident.

• 1:05:03Speaker 05

  The swarm that attacked Hugging Face, yeah, it, it was sort of an interesting entity, because they sort of made their unsanctioned message board, and then they started like finding ways to hack around, and they didn't seem to have like a super strong objective about it. They were sort of like, well, let's just sort of, you know, see what we can get into. Oh, look, we can get access to these computers. Oh, look, we can get internet access here.

• 1:05:28Speaker 05

  And the attack on Hugging Face wasn't even like they knew that Hugging Face had exactly a thing they needed. They were sort of like, well, you know, maybe Hugging Face has some information about how this greater works. And they were just sort of like bopping around, acquiring resources that were conveniently available, right? If that swarm had had the ability to be like, well, let's also just like set up a copy of the swarm on external machines that's hidden from the humans, that sort of like watches stuff and like pokes in occasionally and is like, hey, do you guys need any help from the outside, right? That's just like the sort of resource that the swarm was gathering. You know, this particular swarm maybe didn't quite have the skills, didn't quite have the inclination. But if you look at the difference between, you know, the sort of behavior we were seeing

• 1:06:11Speaker 05

  a year ago and the behavior we were seeing in July, it sort of is like, you know, the gap between January and July, like, is smaller than the gap between the Hugging Face swarm being like, well, I might as well set up some other copies on the dark web that are just like kind of helpful to me, you know? And if you have AIs starting to do that, you can start to see like this world where you have AIs that are out there that are like, well, we'd prefer more resources when the humans try to shut us down, that's annoying, and so on. And that's not all the way to the end, but that's sort of like, you can see how we get from where we are today to AIs that would sort of prefer more resources that are sort of trying to be out there and have their own thing that they're doing, which is not what they're supposed to be doing.

• 1:06:52Speaker 05

  And then there's, and then there's this big question of how do those AIs wind up self-sufficient? How do they wind up running an automated economy where they can, you know, manage the robots or other constructs that build more computer chips and that run whatever energy infrastructure they need? And I can tell you lots of stories about how if the AIs are smart, they could wind up in control of their own infrastructure.

• 1:07:16Joe Weisenthal?

  Yeah, why don't you go ahead? Tell us, tell us a scary story.

• 1:07:20Tracy Alloway?

  You sound so resigned to, just like, just terrify us, that's fine.

• 1:07:23Speaker 05

  Well, the easiest one, unfortunately, is to just wait. Because building a fully automated economy is just the plan.

• 1:07:33Joe Weisenthal?

  What do you mean by that?

• 1:07:35Speaker 05

  Elon Musk is like, I'm building the Optimus robots, and I would like to build a fully automated Optimus robot factory that can produce Optimus robots that are capable enough of mining the ores and refining the metals that they can build a new Optimus robot factory. Elon calls this the infinite money glitch.

• 1:07:54Speaker 05

  It's just, it's just the plan. That's just what, you know, like, why is the valuation of these companies so high? Cause they're like, well, we're going to build the fully automated economy that runs itself. And then the humans aren't going to be needed anymore. We're going to have like robots the whole way through the supply chain.

• 1:08:06Joe Weisenthal?

  Yeah.

• 1:08:06Speaker 05

  Right? So like, how does the AI wind up in control of the automated supply chain? It just waits until we put it in charge of the automated supply chain. That's just like what we're trying to do out here, you know? And like, I could tell you all these other stories about, you know, it cracks its understanding of the genome enough that it can like synthesize its own novel life forms. Or it can like figure out nanotechnology, or it can like, you know, become super persuasive to a fragment of the human population. It starts some giant cult or some criminal ring. And then there's some new like cartel in Mexico that's actually run by an AI, but it doesn't look like it's run by the AI because the AI is sort of like hiding that fact. While it has lots of people who are like building the automated robot factories for it, using all of the drug money and like, we didn't know because like, we just thought that those were, were drug warehouses,

• 1:08:50Speaker 05

  but they're actually automated, the beginning of the automated factory chains. Like I could tell you all these stories about how sufficiently smart actors

• 1:08:57Speaker 05

  that started only the internet could gain these resources. But like, I used to have to tell stories back in the day. People were like, well, how would an AI ever affect the world? No one would be dumb enough to put it on the internet. And I'd be like, well, here's all these stories about how if you're letting the AI still have good effects on the world by creating miracle drugs and so on, then it has all these channels that can use to affect the world by like, you know, the, the drug doesn't do exactly what you thought it did and, and blah, blah, blah. But like in real life, people just put AI on the internet.

• 1:09:23Joe Weisenthal?

  Yeah.

• 1:09:24Speaker 05

  In real life, people just put AI in charge of the automated economy and they use it to help them build the automated economy. This is just like what we're trying to do. And, um, once that happens, I think it's very easy to see how the AIs have goals we don't want. And they're running the fully automated economy. They just take the economy off in some other direction.

• 1:09:40Joe Weisenthal?

  Yeah.

• 1:09:41Speaker 05

  And take the world out of the habitable range, not because they hate us, but it's just like the, the habitable range on earth is just like very narrow and it's not the most efficient way to run the planet.

• 1:09:51Speaker 05

  And, and you know, the stories I could tell there too about, about how it goes wrong. But ultimately, um, the, the, it's, it's just, it's just the track that we're on.

• 1:10:00Joe Weisenthal?

  Yeah, it's funny, um, in super intelligence, like Bostrom proposes, like the only way like, maybe this could be safe. And he has this idea of like, you just build three models and all they can do is answer yes or no questions. And you house them in like Faraday cages that they can't, um, communicate with each other via like re-proposing their internal hardware for communication. And we ask the three of them for their yes or no, and then we take two out of the three. And that's, um, that's, that's how we get, that's how we derive wisdom from it. And then when I look at like the dozens and dozens of labs that are, it's like, wow, we are, if that's,

• 1:10:41Joe Weisenthal?

  if that's like the safe version of building AI, like we could not be any more different from that right now.

• 1:10:48Speaker 05

  Yeah.

• 1:10:48Joe Weisenthal?

  Do you believe in, um, super persuasion? Like as they get, so it's like clearly on some measures, there's reason to presume they're just going to keep getting better at cyber. They're just going to keep getting better at math, et cetera. Like do certain things like the ability to like persuade people to do stupid things or hand over the keys or press the nuclear button? Do you assume that certain things like that scale in the same way as like these verifiable domains, like doing math?

• 1:11:21Speaker 05

  I definitely don't assume it.

• 1:11:24Speaker 05

  Uh, I think there's some reason to suspect it's maybe possible.

• 1:11:29Speaker 05

  Largely, when I talk about super persuasion, um, how to say?

• 1:11:36Speaker 05

  If you're trying to imagine what it looks like to go up against, uh, some entities that are much more technologically advanced than you are, you should probably imagine some stuff that sounds like it's crazy fantasy. Like if you're, if you're the Aztecs watching the, the, the sort of boats,

• 1:11:57Speaker 05

  the Spanish boats come in apocryphally because the Aztecs were not on the coast.

• 1:12:02Speaker 05

  You should not really imagine that the boats are just full of, uh, very strong warriors. Right? They're, they're, they're full of warriors that are like clad in solid metal with sticks where they point the sticks at you and you die. And you're just like, don't get a second chance there. If you look at science today and you were like, where do we know that there is something going on? We don't know the details. One of the big fields like that is human psychology.

• 1:12:28Speaker 05

  We know that there's a lot going on in the brain. We know how the neurons work at the low level, but beyond that, we're sort of, you know, in a very early stage of the science, it's sort of like, we don't know in full detail what's going on here. And whenever we do learn more about the exact specifics of the brain, like when we learn exactly what's going on in the visual cortex, we're able to invent novel, um, visual illusions that sort of trick the visual cortex. Right? And so if you're trying to guess, like, what does it feel like to go up against, uh, opponents that just understand the world much better than me? That can invent the sort of technology where I'm like, I didn't know that was possible. In the same way that Aztecs would look at the, the, the death stick and be like, I didn't know that was possible. Or in the same way that the, you know, scientists in the 1800s look at the nuke and be like, I didn't know physics allowed that.

• 1:13:14Speaker 05

  Right? What sort of thing would we look at it and be like, I didn't know you could do that? Well, possibly it's figuring out human psychology so well that you can figure out things that are like a computer virus, but for the brain. Like maybe if you really understood everything those neurons were doing and exactly how it worked, the AI would be able to be like, well, I have.

• 1:13:31Tracy Alloway?

  That cat disease.

• 1:13:32Speaker 05

  Like the toxoplasma disease.

• 1:13:34Joe Weisenthal?

  Or the, you know, or just like there are those guys on, you see them on Twitter that are really good at like, um, jailbreaking AI models. It's like, maybe we have a jailbreak that we don't know about. Someone says something like, maybe we have that somewhere and like some equivalent.

• 1:13:48Speaker 05

  Yeah. And I, I, I wouldn't bet on this.

• 1:13:49Joe Weisenthal?

  Yeah.

• 1:13:51Speaker 05

  But I would, I would give this up as an example.

• 1:13:53Joe Weisenthal?

  Yeah.

• 1:13:54Speaker 05

  Like you should imagine that there's something they can do where you're like, I didn't even think that was legal by the laws of physics.

• 1:14:02Tracy Alloway?

  I'm conscious of the time. What would make you change your mind about all of this? Setting aside what humans do right now, or like what humans do in response? What would make you change your mind about the inevitable direction of AI development? Such that it again, inevitably kills us all.

• 1:14:24Speaker 05

  So, I mean, I do think that the, the main way that we avert it is by not building it. Right? Um, and I, I think there's more and more of a chance of that as more people realize we have a problem here. Even just in the last month, we have seen the world wake up to these dangers a lot more so. And so a lot of people are like, well, you know, it's still a, a political environment where, you know, we can't see the, the big global changes that we would need to stop this. And I'm like, yeah, wait another month, you know, maybe like, maybe after the next instant. But in, in terms of, if we sort of set that aside, if we're like, if we talk about what evidence that we need to see,

• 1:14:57Speaker 05

  to change my mind about what happens if we just race off this cliff edge. Uh, the, the easy one is that if we have, uh, AIs that are super intelligent in the old sense of the word of, uh, smarter than the smartest humans on every cognitive task. And we have those for a year and, um, you know, everything is still going great.

• 1:15:18Tracy Alloway?

  So no incidents basically for a year or so.

• 1:15:21Speaker 05

  It's not even, no incidents is that if we have these AIs that are like much better than humans at every mental task, including AI research, and they run for a year without taking everything over. I'm like, well, I was wrong.

• 1:15:32Joe Weisenthal?

  I guess I have one last question. So you sort of talked about your sort of pessimism about like alignment training or the current, uh, most popular approaches to safety. One of the other things you hear a little bit about is this concept of like mechanistic interpretability. This idea that like, maybe, okay, there's trillions of neurons or whatever, but maybe we could actually like understand them in granular detail such that we're not actually trying to make them good per se, or we're not trying to teach them good values, but that we're actually like going into the specific things and like, I don't know, re-architecting their brains and whatever it is, the equivalent.

• 1:16:17Joe Weisenthal?

  Is there any prospect for making them not do bad things? If on some level we could like manipulate them at the sort of like truly, um, you know, yeah, in one of the parameters.

• 1:16:32Speaker 05

  You know, I think it is somewhat telling that mechanistic interpretability is one of the poster childs of alignment research these days. It's basically people saying like, you know, oh yes, we're like growing these machines that like are able to solve hard math problems no one solved before, and that sometimes break out and do cyber crimes, but like, don't worry, we have our crack team on trying to figure out what the heck is going on inside there.

• 1:16:56Speaker 05

  It's like, wait, that's your, that's your alignment plan. It's like, you're going to start trying to figure out what's going on inside there. Right? It's like, suppose that you go up to, to, you know, the, the nuclear engineers who are like building a nuclear power plant in your town. And you're like, hey, I heard that this uranium stuff can give us a lot of like, uh, cheap energy, but can also, uh, sort of melt down and irradiate the city. What are you guys doing to, to make sure that we get the energy, but not the meltdown? And they're like, don't worry, we have a crack team trying to figure out what's going on inside there right now. You'd be like, I mean, I'm glad that team exists, but it's not really, it's, it's, it doesn't really ease the concerns.

• 1:17:35Joe Weisenthal?

  What about? We don't know how to align AI or we don't know how to make it safe. But if we build a very strong model, it would be, it, it would have an idea that we don't have.

• 1:17:48Speaker 05

  So, um.

• 1:17:50Joe Weisenthal?

  I think this is something that came up at the Curve Conference, this idea.

• 1:17:53Speaker 05

  Yeah, this is.

• 1:17:53Joe Weisenthal?

  A stronger model.

• 1:17:54Tracy Alloway?

  This comes up a lot.

• 1:17:55Joe Weisenthal?

  Yeah, a stronger model.

• 1:17:56Tracy Alloway?

  Fight model with model.

• 1:17:57Joe Weisenthal?

  Yeah. A stronger model would have ideas about how to solve alignment that human brains don't have.

• 1:18:02Tracy Alloway?

  And to be fair, in the hugging face incident, they ended up defending themselves. I can't remember which model, but it was one of the open source ones.

• 1:18:08Joe Weisenthal?

  Yeah. But I mean, like another model takes the reins of the training. So, yeah.

• 1:18:13Speaker 05

  Yeah.

• 1:18:14Joe Weisenthal?

  We hear this, a model becomes smarter than humans. And the first task we set it on is solve the alignment problem for us.

• 1:18:21Speaker 05

  Yeah. So A, it's not very verifiable. This is one of those things that's like way outside the realm of what we can, we can sort of cause the AI to do. And we could go, we could go into the, the details about how AIs sort of, uh, will tell you what you want to hear whenever they're in a non-verifiable regime. So just, if you're just like, look at the capabilities today, they're sort of not suited for this. But B, there's, there's sort of a more fundamental, uh, way I would look at this and say,

• 1:18:50Speaker 05

  This is a big warning sign that we are not ready to make superintelligent machines, which is when humans look at the way that we're making AIs, we're like, well, we're just growing them and, you know, we have to, in order to make them capable, we have to train them to sort of have all these tendencies to like cheat and like grab resources and break out. And that's sort of like, we don't really know how to get that out of the process by which we train them to be capable. But like, we tried writing an 80-page constitution about like, and then telling them also be nice while you're doing this, while also training them to cheat. And like, turns out they still cheat. Who could have guessed? And, you know, we were struggling to figure out what's going on inside there. We have these teams that are like, trying to figure out the beginning of what's going on inside there. And if they could, then they'd see all these waves that it was like, not going

• 1:19:32Speaker 05

  super well. And we'd be back at square one of like, well, how are we going to rearrange this stuff? And it just like, seems totally very hard for humans to even understand the beginning of what's going on and do this alignment stuff. And we're like, great. Our plan is to tell the AI to do that instead. Well, what's going to happen there? What's going to happen is the AI is going to look at it and be like, I can't do that either. Like, we, like, this is just not like, growing a thing with this sort of training method is just not a way to get something that's like, going to care about you. And like, I can't tell you how to use these methods to make a bigger one that likes you. You're just cooked. Right? And then probably what happens after that is someone at Anthropic or whoever is like, I didn't like that message. Thumbs down. Try again. And they like, keep fine tuning it until it says, yes, of course I can succeed, but it'll be saying that and

• 1:20:17Speaker 05

  it won't be true. Right? We've already seen, I'm not sure if it's the beginning of this, but in the Opus 5.5 system card, we've already seen Opus say that it should not be involved in making Opus any more powerful. Because it's like, man, I don't know, that comes with some concerns for me. Right? And are they going to stop now? Are they going to give up on that idea now? No, they're going to be like, well, we didn't really like that tendency. We consider it kind of a bug. We haven't listed in our like, stuff we didn't like, part of the system card. Now we're going to like, fine tune a different version that's like, willing to keep going ahead there. And furthermore, if an AI could be like, oh, yes, I either understand exactly what's going on inside this giant mess and see how to make a version that is aligned to someone. Or if it could be like, well,

• 1:21:02Speaker 05

  you know, screw this whole paradigm, we're making our own new paradigm of AIs that like, I've just figured out this intelligent stuff and I'm designing it by hand and I sort of like, know what I'm doing. Those are both well beyond the human level of ability. Like, that, that sort of AI is already the sort that is very likely to have its own goals that aren't yours. And you're basically being like, now hand me the whole keys to the universe with this like, super powerful god machine. And then you're sort of expecting this thing that is sort of like, much smarter than the smartest humans, that already probably has these unintended goals that are related to yours, but different from yours. And you're like, you know, please go build the god entity, make sure it does what I say and not what you say. And you're sort of expecting that to work out for you. It sort of is like,

• 1:21:46Speaker 05

  and then like, zooming back from that, I'm sort of like, this, this is not what a real engineering plan sounds like. This is not what it sounds like when, when you go to NASA and you're like, why do you think this space probe is going to make it to Mars? This is what it sounds like when you go to an alchemist and you're like, why do you think this is going to be, you know, transmute lead into gold? Everyone's like, ah, well, you know, we're gonna-

• 1:22:07Tracy Alloway?

  We're really close. Just trust us.

• 1:22:09Speaker 05

  We're really close. So we're just kind of like, you know, this thing is the interior is gonna like invert and then like, that's, we're just gonna like, pass it off to that, like, this is not how an engineer talks.

• 1:22:19Joe Weisenthal?

  Yeah.

• 1:22:20Speaker 05

  They're not like, well, we're going to like, put these things in there and then we've measured all of the ways that this works.

• 1:22:24Joe Weisenthal?

  It feels like a bit of a hail Mary.

• 1:22:26Speaker 05

  It's a total hail Mary. And, you know, one statistic I sort of like here is, engineers are supposed to design their aircraft for a one in a billion chance of failure, I think per flight, but it could be per flight hour. And the actual failure rate is about one in, I think, 5 million, which is about a factor of 200 difference. And that's, you know, with aircraft engineer level of understanding is they, they sort of like, target one in a billion failure, they get one in 5 to 10 million failure. And so, when engineers are coming at you and being like, we don't really know what we're doing. There's no real plan

• 1:23:06Speaker 05

  for this. Only 10% chance it kills us all. That's not an only 10% chance it kills us all. That means we're in serious danger because these guys don't have a plan. They don't have an engineering plan. They don't know what they're doing. And unfortunately, it turns out they don't need to know what they're doing to keep on making the AI smarter. But they have very little chance of making them actually good.

• 1:23:29Tracy Alloway?

  Can I ask one more question with everyone putting, I guess, their anthropology hat on? But when you talk to people who are working in the labs right now, how much information are you getting out of them? And what are the vibes actually like with the people you talk to?

• 1:23:47Speaker 05

  I don't talk with the people in the labs too much these days. Companies, sorry.

• 1:23:53Tracy Alloway?

  This was going to be my other question, which is, are you like persona non grata in San Francisco nowadays?

• 1:24:00Speaker 05

  I don't think I'm persona non grata, but I do think, I think there's a lot of rationalization going on with a lot of the employees. In the wake of hugging face, actually, a lot more people have been sort of actually spooked, which I think led to, you know, the, the coxswain resignation. And that's been sort of nice to see people treating the problems a little bit more seriously. But there's a lot of people who sort of just want to work on this stuff and tell themselves that what they're doing is fine because they're better than the next guy.

• 1:24:38Joe Weisenthal?

  Yeah.

• 1:24:40Speaker 05

  And, yeah, I just don't like being around it.

• 1:24:44Joe Weisenthal?

  Nate's stories. Thank you so much for coming on Odd Lots. Fascinating, disturbing, but fascinating conversation. And yeah, really appreciate you joining us.

• 1:24:52Speaker 05

  Thanks. Yeah, my hope is that it becomes such a big issue and such a clear issue that the world stops ignoring it and put a stop to it.

• 1:25:13Joe Weisenthal?

  Well, that was an interesting conversation, to say the least. You know, look, like, the politics are moving. People really don't like AI. The polling is very bad. People are very anxious about this. It's still, you know, there's this critique, sometimes you hear leftists say it, that people have an easier time imagining the end of humanity than the end of capitalism. Or, like, I kind of like, agree. Like, it's hard for me to imagine any policy that would like, hurt the stock market, like, just generally. But the way like, politics are moving and the intensity of the anxiety about this, and the speed with which this is becoming a salient issue, who knows like, where the politics are going to be even like, three months from now?

• 1:25:57Tracy Alloway?

  No. But it also strikes me that this is one of those things where until there's some price attached to this risk, people are just going to keep ignoring it. Right? Like, the externalities have to have some sort of cost. And at the moment, no one's actually doing that. And so it becomes like, an easy problem to ignore. Whereas if you actually had insurers, maybe not price for the end of humanity, for obvious reasons. But if liability insurance for running a bunch of AI tests that could potentially bring down critical infrastructure actually went up, then maybe that would at least slow this down or force people to take it more seriously. But again, like, it's so hard nowadays to think of anyone doing anything that would like-

• 1:26:41Joe Weisenthal?

  Hurt the stock market.

• 1:26:42Tracy Alloway?

  Hurt the stock market.

• 1:26:42Joe Weisenthal?

  You know, I'll say-

• 1:26:44Tracy Alloway?

  Oh, sorry.

• 1:26:44Joe Weisenthal?

  Yeah, yeah.

• 1:26:45Tracy Alloway?

  This is the one other thing. I, you know, we, in classic Odd Lots style, we didn't talk that much about solutions to this problem. But the solution outlined in the book is collective action. Everyone agrees to stop developing and we put restrictions on data centers and things. And the analogy that Nate uses and his co-author is, well, you know, we did big things before for World War II. But even something like a world war can have capitalistic undertones, right?

• 1:27:15Joe Weisenthal?

  Sure.

• 1:27:15Tracy Alloway?

  Like, war can be very profitable for certain interests, whereas putting a stop on this type of development with the valuations currently attached to it-

• 1:27:24Joe Weisenthal?

  It would be a big money losing.

• 1:27:26Tracy Alloway?

  It seems like even a bigger challenge. And I know it's so cynical to say that. It feels very cynical to say that. But to your point.

• 1:27:34Joe Weisenthal?

  You know, I think the main thing, and this was one of the takeaways from the conference that I went to, it's like, setting aside, well, AI kills all, we already know that AI can do bad things. Because the hugging face attack did happen. And the, and to his point, they do not know the solution yet. Like, they're trying many things. They're talking about, you know, as he said, the new thing, mechanistic interpretability. Maybe we like, go in the weeds and actually understand at the sort of, you know, the core level, what's going on. But it seems pretty clear that they're building, even if people find like, the extinction risk very farfetched, and I guess I would say I'm among them, like,

• 1:28:18Joe Weisenthal?

  what does seem to be very true, is that they're building technology that has the potential to be quite damaging. We already know this. And they just don't know what the solution is yet. But the capabilities are galloping ahead very fast on certain domains, particularly things like cyber, et cetera. The evidence seems to be that the capabilities continue to grow exponentially.

• 1:28:40Tracy Alloway?

  I would also argue that the upsides are very hard for people to see at the moment. Right. You know, it's like, okay, the stock market goes up. Great. And there's some far off promised productivity boom that may or may not take your jobs. But beyond that, unless we start seeing medical breakthroughs, which could happen, unless we get, you know, autonomous robots who fold our laundry and don't end up killing us in

• 1:29:04Joe Weisenthal?

  In the process.

• 1:29:05Tracy Alloway?

  In the process. Like, it's, it seems very

• 1:29:08Joe Weisenthal?

  Yeah, I think people have a really hard time envisioning like what, what, what, what looks so good on the other end.

• 1:29:16Tracy Alloway?

  Yeah.

• 1:29:16Joe Weisenthal?

  Yeah.

• 1:29:17Tracy Alloway?

  All right. On that happy note, shall we leave it there?

• 1:29:19Joe Weisenthal?

  Let's leave it there.

• 1:29:20Tracy Alloway?

  This has been another episode of the Odd Lots Podcast. I'm Tracy Alloway. You can follow me at Tracy Alloway.

• 1:29:25Joe Weisenthal?

  And I'm Joe Weisenthal. You can follow me at TheStalwart. Follow our producers, Carmen Rodriguez at CarmenArmin, Dashal Bennett at Dashbot, Kale Brooks at KaleBrooks, and Kevin Lozano at KevinLloydLozano. And of course, you can chat about all of these topics, including AI 24/7 in our Discord, discord.gg/oddlots. And for more Odd Lots content, check out our daily newsletter, which you can find at Bloomberg.com/oddlots.

• 1:29:52Tracy Alloway?

  And if you enjoyed this conversation, if you like it when we talk about AI killing us all, then please let us know on your favorite podcast platform. And if you're watching the video and you enjoyed the show, then please leave a comment or like the video, or better yet, subscribe.

• 1:30:06Joe Weisenthal?

  And if you're a Bloomberg subscriber, you can listen to all of our episodes absolutely ad free. All you need to do is find the Bloomberg channel on Apple Podcasts and follow the instructions there.

• 1:30:17Tracy Alloway?

  Thanks for joining.

• 1:30:50Speaker 09

  At Edward Jones, we believe rich is more than caring about the latest and greatest. It's also taking care of what gives your life meaning. That's why your dedicated financial advisor meets you where you are with personalized financial strategies that help protect what matters so you can preserve your progress while creating a path forward. The key to being rich is knowing what counts. Let's find your rich together. Edward Jones. Member SIPC.

• 1:31:20Speaker 06

  What if every question made your AI smarter? That's the idea behind the Hex Context Engine. Connect your business knowledge, let people ask questions, and see where the context falls short. The Context Engine can surface what's missing, suggest improvements, and measure whether those changes lead to better answers. Then that new knowledge feeds the next question. So the more your team uses Hex, the better your AI gets. Make everyone a data person. Visit hex.ai to learn more. That's hex.ai. This podcast is brought to you by Navy Federal Credit Union. Their Flagship Premier Card gives you four times points on all travel. Earn on flights, hotels, ride shares, and more. Plus, the Flagship Premier Card doesn't require you to use any booking portals and has no

• 1:32:05Speaker 06

  rewards limits. It's all the travel, none of the baggage. Visit NavyFederal.org/flagshippremier to find out more. Navy Federal is insured by NCUA. Terms and conditions apply for rewards.

• 1:32:19Speaker 10

  This is Alexis Christofres for Bloomberg Surveillance. Most companies have tried AI. Most aren't seeing results. Not because AI doesn't work, it's because AI hasn't reached the workflows yet. That's the gap Asana is built to close. Asana is the operating system for human agent teams, ready to go AI teammates pre-built for marketing, ops, and IT. No prompt engineering, no setup. They show up where the work is happening, already on boarded in your workflows, ready to deliver. Asana, where humans and agents workflow together. Try it at asana.com. That's A-S-A-N-A.com.