Stopping The AI Safety Cult ft. Adam Becker & Cal Newport

Ed Zitron is joined by computer science professor and writer Cal Newport and journalist and astrophysicist Adam Becker to get the word out that the narrative about existential risk from AI is driven by a religious cult of effective altruists and rationalists. AI Is Already In Dangerous Hands newsletter: https://www.wheresyoured.at/ai-is-already-in-dangerous-hands/ Adam https://bsky.app/profile/adambecker.bsky.social https://www.youtube.com/@DreamingAgainstTheMachine https://www.hachettebookgroup.com/titles/adam-becker/more-everything-forever/9781541619593/ Cal Podcast & Videos: https://www.youtube.com/@CalNewportMedia/ Newsletter: https://calnewport.com Save $10 off a year of my premium newsletter: https://edzitronswheresyouredatghostio.outpost.pub/public/promo-subscription/gzqwkv54e1 YOU CAN NOW BUY BETTER OFFLINE MERCH! Go to https://cottonbureau.com/people/better-offline and use code FREE99 for free shipping on orders of $99 or more. --- LINKS: https://www.tinyurl.com/betterofflinelinks Newsletter: https://www.wheresyoured.at/ Reddit: https://www.reddit.com/r/BetterOffline/ Discord: chat.wheresyoured.at https://discord.com/invite/QUUQUP9szv Ed's Socials: https://twitter.com/edzitron https://www.instagram.com/edzitron https://bsky.app/profile/edzitron.com https://www.threads.net/@edzitron Email Me: [email protected] mailto:[email protected] See omnystudio.com/listener https://omnystudio.com/listener for privacy information.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  This is an iHeart Podcast. Guaranteed Human.

• 0:04Speaker 01

  Snoring, gasping for air during sleep, daytime sleepiness? I'm Shaquille O'Neal, and this shouldn't be anybody's experience. Ask your doctor about Zepbound Trisepotide, the first and only FDA-approved prescription medicine for moderate to severe obstructive sleep apnea in adults with obesity.

• 0:20Speaker 02

  Zepbound is a prescription medicine used with a reduced calorie diet and increased physical activity to help adults with moderate to severe obstructive sleep apnea, OSA, and obesity to improve their OSA. Zepbound is approved as a 2.5, 5, 7.5, 10, 12.5, or 15-milligram injection. Zepbound contains trisepotide and should not be used with other trisepotide-containing products or any GLP-1 receptor agonist medicines. It is not known if Zepbound is safe and effective for use in children. Don't share needles or pens, or reuse needles. Don't take if allergic to it, or if you or someone in your family had medullary thyroid cancer, or if you've had multiple endocrine neoplasia syndrome type 2. Tell your doctor if you get a lump or swelling in your neck. Stop Zepbound and call your doctor if you have severe stomach pain or a serious allergic

• 1:05Speaker 02

  reaction. Severe side effects may include inflamed pancreas or gallbladder problems. Tell your doctor if you experience vision changes before scheduled procedures with anesthesia, if you're nursing, pregnant, plant-to-be, or taking birth control pills. Taking Zepbound with a sulfonylurea or insulin may cause low blood sugar. Side effects include nausea, diarrhea, and vomiting, which can cause dehydration and worsen kidney problems.

• 1:27Speaker 01

  Talk to your doctor. Call 1-800-545-5979, or visit zepbound.lily.com.

• 1:34Speaker 03

  Small businesses are the pulse of every community. They bring people together, create opportunities, and drive growth. Chase for Business helps business owners like you with personalized guidance and convenient digital tools all in one place. With that guidance and your determination, you can take your business further and help build a brighter future for your community. Learn more at chase.com/business. Chase for Business: Make More of What's Yours. The Chase mobile app is available for select mobile devices. Message and data rates may apply. JPMorgan Chase Bank, N.A., member FDIC. Copyright 2026, JPMorgan Chase and Company.

• 2:16Speaker 04

  Even on my antidepressant, I still struggled with depression symptoms.

• 2:20Speaker 05

  You okay, honey?

• 2:21Speaker 04

  I'm okay, ish. My doctor said adding Rexulti Brexpitrazole could help.

• 2:26Speaker 05

  When added to an antidepressant, Rexulti was proven to significantly reduce depression symptoms, more than an antidepressant alone. Elderly people with dementia-related psychosis have increased risk of death or stroke. Antidepressants may increase suicidal thoughts and actions and worsen depression in children and young adults. Call your healthcare provider right away to report new or sudden changes in mood, behavior, thoughts, or feelings, or if you develop suicidal thoughts or actions. Report fever, stiff muscles, and confusion, which can be life-threatening, or uncontrolled muscle movements which may be permanent. High blood sugar can lead to coma or death. Weight gain, increased cholesterol, unusual urges, dizziness on standing, falls, seizures, trouble swallowing, or sleepiness may occur. Learn about these and other side effects at rexulti.com. Tap ad for PI. Hey, how you doing, honey?

• 3:08Speaker 04

  Okay.

• 3:08Speaker 05

  Okay-ish?

• 3:10Speaker 04

  No-ish. I'm okay, really.

• 3:12Speaker 05

  Ask your doctor about adding Rexulti.

• 3:17Speaker 06

  Your Saturday's packed, and overthinking skincare isn't on the agenda. Simple can still be effective, because going overboard with active ingredients can weaken the skin's moisture barrier. New Aquaphor Hydrating Daily Lotion helps repair your skin's moisture barrier. It's lightweight and fast-absorbing on skin. Even though your Saturday's packed, your skin's moisture barrier is supported. Grab Aquaphor Hydrating Daily Lotion at your nearest retailer. There's before, and then there's Aquaphor.

• 3:47Speaker 00

  Coca-Cola. For the big, for the small, the short, and the tall. Peacemakers, risk-takers. For the optimists, pessimists. For long-distance love. For introverts and extroverts. The thinkers and the doers. For old friends and new. Coca-Cola. For everyone. Pick up some Coca-Cola at a store near you.

• 4:17Speaker 07

  Jardiance has a big story to tell. Discover Jardiance: Empigliflozin 10 or 25-milligram tablets. Visit jardiance.com, call 1-888-968-6648, or talk to your doctor to see if Jardiance is right for you.

• 4:33Speaker 08

  Your social media feed delivers plenty of advice, but it doesn't know you. It doesn't ask questions. It doesn't give physical exams or order tests. Doctors do. At the American Medical Association, we believe the best care starts with a real conversation with someone who understands the science and your unique health. So stay curious, ask questions, but when it's time to make decisions, make them with a doctor. Learn more at amahhealthvshype.org. That's amahhealthvshype.org.

• 5:04Speaker 09

  Call Zone Media.

• 5:07Ed Zitron?

  My name's Ed Zitron, and this is Better Offline.

• 5:22Ed Zitron?

  I am back, and I'm announcing that we will be speeding up the process of researching powerful and dangerous podcasting technology. My researchers are spending over $200 million a day on GPU compute trying to create and generate a recursive self-improving zitron, and no amount of pressure from world governments will stop me. But before my greatest work is complete, I must continue to record. And today, we've got an absolute barn burner. I'm joined by the claviculars of computer science, by which I mean Professor Cal Newport, an astrophysicist and author of more, Everything Forever. Adam Becker, fellas, welcome back to the show.

• 5:54Speaker 11

  Thanks for having us.

• 5:55Speaker 12

  Yeah, I'm terrified of the idea of recursively self-improved. That means your newsletters will be 200,000 words long now? Is that what we'll expect?

• 6:04Ed Zitron?

  Yeah, they'll either be 200,000 words or 100 words long.

• 6:08Speaker 12

  Oh, maybe it's not the worst thing then. All right, keep going.

• 6:10Ed Zitron?

  When the perfect matrix of zitron is complete. Well, today, though, this was inspired by your wonderful piece, Cal, in New York Times about the rationalists. And we are going to start very simple. Who are these people, and why are they suddenly notable?

• 6:27Speaker 12

  Well, I'm probably going to actually throw the baton to Adam here in a second to give the history of this group, because he and his book was one of my primary sources in trying to understand them. But let me just set the stage for why I was writing about them. I don't think people realize just how weird some of the lead figures are who are at, in particular, OpenAI, Anthropic, and DeepMind. And I don't think people realize that until last week, when you had a whole group of 20-somethings casually debating in their lunch hour whether it's 10% or 50% chance that we're all going to die within the next couple of years. And that, I think, caught a lot of people's attention of like, wait a second. What is going on over there in those labs?

• 7:10Speaker 12

  Why is this type of rhetoric only coming out of those particular labs? Why is Jinson Wong think that this is ridiculous? Why is China laughing at this narrative? There's something going on here that's not right. I think that I think people are starting to wake up to this. And I think it really is this family of futurist philosophies that come out of the Bay Area that have mutated and split in various ways that are really defining our current moment, and no one really realizes it. But Adam spent years, God help him, immersed in this world, has read everything they've read, and he knows the history well. So actually, Adam, I'll throw the baton to you. Maybe you can give us a story of rationalism and effective altruism, and where they came from, and how they've split. To me, it's a fascinating, odd story.

• 7:54Speaker 11

  Oh, God. I mean, my, my standard response when people ask me this is, you know, well, I could write a whole book about it, and I did. And, and, you know, I'm not going to get into the same level of detail I do in the book, but like, basically, this is something that was spawned out of transhumanist message boards and email lists in the 1990s.

• 8:16Ed Zitron?

  And what is transhumanist?

• 8:18Speaker 11

  Yeah, that's a good question. Transhumanists, I guess the shortest good definition I could give you is, transhumanists are people who don't want to be human anymore. They want to use machines, and especially computers, to basically escape human bodies and solve the problem of death.

• 8:44Ed Zitron?

  Okay.

• 8:45Speaker 11

  Yeah.

• 8:45Ed Zitron?

  So those guys.

• 8:47Speaker 11

  Yeah, nor normal shit, normal shit, normal.

• 8:50Ed Zitron?

  Normal way to do.

• 8:51Speaker 11

  Yeah, exactly. Yeah. And, um, and so, you know, these are people who are really excited about the prospect of AI, the prospect of, you know, uh, nano technology that could, you know, basically serve as a kind of magic to do whatever they wanted. Um, this was the nineties. Nano tech was like the hot thing. Uh, and it was a bunch of ideas that basically come from science fiction.

• 9:21Ed Zitron?

  Right.

• 9:22Speaker 11

  You know, they, they read too much sci fi and confused it with reality and thought, you know, oh, well, because all of the sci fi I read takes place in the future and all of the sci fi I read has super intelligent machines in it and nano tech and, you know, people uploading their brains into computers, that must be what's going to happen in the future. Uh, if we just, you know, work hard and wish hard enough for it.

• 9:48Ed Zitron?

  And so just so I'm clear, this is, this is a certainty for them. They're 100% sure this will happen.

• 9:56Speaker 11

  They're 100% sure that they could make it happen.

• 10:00Ed Zitron?

  Oh, so it's in their hands.

• 10:02Speaker 11

  Yeah, yeah, yeah, yeah. Like, you know, they, they think, oh yeah, this is going to happen unless something, you know, ends civilization first, or, you know, nobody listens to us or something like that. They think that they're working on the most important stuff in the history of humanity. Basically, they think that they are going to, uh, you know, make themselves immortal.

• 10:23Ed Zitron?

  And when you say working on, that seems like a fairly broad thing though, because a lot of these rationalist types don't actually work on the stuff. They work next to it.

• 10:34Speaker 11

  Yeah.

• 10:35Ed Zitron?

  Philosophy people.

• 10:36Speaker 11

  Yeah. I mean, it's not even so much that they're working next to it. It's like they talk about it. They like to talk about the prospect of nano technology. They like to talk about, uh, you know, super intelligent AI. They like to talk about these things, but, but yeah. So these people spent a lot of time doing that on message boards in the nineties. And there was this one guy who showed up in the mid to late nineties on, uh, one of the most sort of influential of these message boards, the extropians, uh, which, you know, don't get me started on that name. It comes out of a misunderstanding of physics.

• 11:10Ed Zitron?

  That's a annoying name.

• 11:12Speaker 12

  No, that is, that's, that's how you know, this is a serious group of scientists.

• 11:15Speaker 11

  That's right.

• 11:16Speaker 12

  The extropians. Yeah.

• 11:17Speaker 11

  That's, that's correct. Yeah, that's exactly right. And they're like, yeah, we're going to fight the second law of thermodynamics, to which my response as a physicist is, you know, yeah, good luck with that. But, um, basically, uh, this guy, this, this kid shows up, uh, named Eliezer Yudkowsky, who is a teenager at that point.

• 11:39Speaker 12

  Dropped out, dropped out of middle, after middle school, dropped out of schooling. This is like right after he dropped out.

• 11:43Ed Zitron?

  That is wild.

• 11:44Speaker 11

  Yep. Dropped out after eighth grade. That's correct.

• 11:47Ed Zitron?

  That's crazy, man.

• 11:48Speaker 11

  Yep. Um, and, uh, and yeah, Yudkowsky then goes on to, uh, write an enormous amount of, you know, his own deep thoughts on what he thinks the future of AI is going to be. Uh, and relatively quickly comes to the conclusion that, uh, AI is going to, you know, he, he takes an article of faith among these people, which is that AI is going to become self-improving and lead to a super intelligent AI at some point in the relatively near future and lead to a singularity where

• 12:31Speaker 11

  everything changes and, you know, all of the stuff that the transhumanists believe is inevitable is going to happen. Uh, he comes to, Yudkowsky then comes to the conclusion that, uh, such an AI might not be friendly to humanity and probably wouldn't be unless it's done properly.

• 12:50Ed Zitron?

  And the singularity is when the computer is smarter than us, just so that I understand.

• 12:54Speaker 11

  It's when, uh, it's really ill-defined. It's when the computer is smarter than us, or maybe it's when the computer can upload us into more computers. So we can all be computer and everything is computer. Uh, the singularity is when, uh, is when a lot of tech, but all the time and everything, like it's not a well-defined thing. Um, yeah.

• 13:18Ed Zitron?

  And this is the thing they obsess over, the thing that's ill-defined.

• 13:21Speaker 11

  Oh, yeah, yeah, yeah. They, they think that that is pretty much inevitable unless, you know, we have like a nuclear apocalypse first or something like that.

• 13:31Speaker 12

  This is worth, let's just pause on this for a second, right? I think it's really important because these ideas that I was talking about sound very familiar to what people are talking about today. They were talking about this. This is about the early 2000s.

• 13:43Speaker 11

  Yep.

• 13:43Speaker 12

  Is roughly when Yudkowsky kind of had this change of heart of, um, the super intelligence, of course, is inevitable, but now he was worried about it instead of being excited about it. This was all thought experiment. Okay. I mean, this is, there's no LLMs there, but this was during an AI winter from the computer science perspective. Um, it was entirely inductive thinking directed by ideas from sci fi. So they just sat there and thought about this, right? So these are not, this is not an engineer looking at a particular technical trajectory and saying, oh, I'm worried where this is going to go. This is people just thinking through the thought experiments of, well, if AI could recursively self-improve, it would get super intelligent. What would it happen if you had a super intelligent AI? And then just sort of exhaustively working out in their own

• 14:28Speaker 12

  minds, various scenarios about, well, maybe it would be friendly, but maybe it would trick us. And what if it did this or that? And, and so I think that's important. So you have all these ideas are established well before there's any particular technology that people are worried about, established entirely based off of, at this point, thought experiments.

• 14:45Speaker 11

  Yeah, I think that's.

• 14:46Ed Zitron?

  So this is, so this isn't through proximity to actual scientists?

• 14:50Speaker 11

  Nope.

• 14:50Ed Zitron?

  Or so no, nothing about witnessing the actual technology following, other than, uh, arms reach?

• 14:58Speaker 11

  Yeah, no, I mean, it is exactly as Cal just said, and another way of phrasing that is, it's, it's essentially a, a kind of extremely dry fanfiction that they're engaging in. They are writing, uh, you know, anxiety fueled scenarios based on these ideas that they got from science fiction, thinking that that's what the future is going to be.

• 15:23Ed Zitron?

  Do they believe it?

• 15:24Speaker 11

  Oh, yeah.

• 15:25Ed Zitron?

  So they're sincere, they're just odd.

• 15:28Speaker 11

  Yes. Yeah. And at the time you can like dismiss them as sincere oddballs. What happens is, uh, Yudkowsky just keeps writing and writing and gets invited to contribute to some group blogs and then spins off his own sort of blog, uh, called, uh, Less Wrong, which becomes a community hub for people. Yeah, that's him. But the other thing that happens is it's around this time, like the late 2000s, early 2010s, somewhere in there, that, uh, Yudkowsky meets Peter Thiel.

• 16:03Ed Zitron?

  Oh.

• 16:04Speaker 11

  And Peter Thiel gives Yudkowsky a bunch of money.

• 16:07Ed Zitron?

  How old is he at the time?

• 16:09Speaker 11

  Oh.

• 16:10Speaker 12

  We did this math. We did this math at him with the fact checkers.

• 16:13Speaker 11

  Yeah.

• 16:14Speaker 12

  This is like 2000, the first decade of the 2000s, he was like mid to upper twenties.

• 16:20Speaker 11

  Yeah, something like that. Yeah.

• 16:21Speaker 12

  Yeah, that's right.

• 16:23Ed Zitron?

  Huh.

• 16:23Speaker 12

  It's like a decade of him, a decade after.

• 16:25Ed Zitron?

  Just giving a child money.

• 16:28Speaker 11

  I mean, that's not out of character for Peter Thiel.

• 16:32Speaker 12

  But he literally pays people not to go to college, right?

• 16:34Speaker 11

  Exactly.

• 16:35Ed Zitron?

  Yeah. The thing is though, there's like the Thiel Fellowship where it's like, I'll pay you not to go to college to build something. This is just like, I liked your blog kiddo.

• 16:44Speaker 11

  Pretty much.

• 16:45Ed Zitron?

  You think the world's going to explode? You better blog about that some more.

• 16:49Speaker 12

  They met at, is, uh, tough to get this right, Adam, but they were at the, an event called the Singularity Summit. Um, and, and that's when.

• 16:56Speaker 11

  That's where they met. Yeah.

• 16:58Speaker 12

  Or maybe that's where it gets complicated.

• 17:01Speaker 11

  They definitely.

• 17:02Speaker 12

  It gets complicated.

• 17:03Speaker 11

  Yeah, it does. Yeah.

• 17:04Speaker 12

  But a lot of money enters the scene, right? Like this is where like a lot of money starts entering this world.

• 17:09Speaker 11

  Yes. Yeah. Yeah. And, uh, and Yudkowsky, along with some other, you know, transhumanist futurist types like Ray Kurzweil, uh, starts this, you know, set of, uh, conferences, the Singularity Summit and, um, you know, just gets people together every year to talk about this stuff. And the community of rationalists on Less Wrong just, you know, start, you know, gets bigger and bigger and they start meeting up in the Bay Area. They start living together in group houses and.

• 17:45Ed Zitron?

  What does rationalist mean?

• 17:47Speaker 11

  Rationalist basically means you spend a lot of your spare time reading Yudkowsky's writing. Uh, Yudkowsky thinks at this point that if you don't believe that the singularity is coming and that AI is an existential threat to humanity, unless you figure out how to do it right. He thinks if you don't believe those things, you're not thinking rationally. And so he decides to post a lot about how to think rationally.

• 18:15Ed Zitron?

  That's it?

• 18:16Speaker 11

  Yes, that's it.

• 18:17Ed Zitron?

  That's it? That's the entire rationality part?

• 18:21Speaker 11

  Yeah, that's why he writes.

• 18:22Ed Zitron?

  It's not about living rationally. It's just about the ration, their so-called rational belief that the computer might wake up. Are you fucking kidding?

• 18:30Speaker 12

  Well, they apply it to their whole lives.

• 18:32Speaker 11

  Yeah, exactly.

• 18:33Speaker 12

  That's why the two big blogs are called overcoming bias and less wrong. It's all about this sort of like really sort of tech obsessive way of I'm going to be super accurate in the way I think, right? And this is, I'm gonna, I'm, I am not going to be overcome by cognitive biases. I'm not going to be tricked. And if we can just be like hyper Sherlock Holmes is like, that's how we're going to unlock the potential of the world. But Yudkowsky thinks like the ultimate application of this is preventing the extinction of humanity by preventing super intelligent AI. And it like, that is why we have to raise an army of like hyper rationalist people. It's our only chance to survive the, the coming AI apocalypse.

• 19:15Speaker 12

  So like, that kind of becomes his obsession is that like this idea of like hyper rationalism that he's the leader of is going to be, that's the John Connor of the Skynet scenario.

• 19:26Ed Zitron?

  Right, but they don't seem to be doing anything other than talking.

• 19:31Speaker 11

  Yeah, no, at this point they're not.

• 19:33Speaker 12

  They're doing some things. I mean, they're in the polyamory. I mean, there's some other things going on.

• 19:38Speaker 11

  But like, to, to be very, very clear, Yudkowsky is anointing himself as an expert on, you know, the, the art of rationality and cognitive biases and how to think. And, you know, this is, this is an entirely self-trained, self-taught guy who, whose experience of the world is pretty limited. And he is posting hundreds of thousands of words about this on Last Wrong and then starts writing as, uh, you know, a combination of artistic outlet and recruitment technique. Uh, he starts writing literal fanfiction, a Harry Potter fanfiction called, uh, Harry Potter and the Methods of Rationality.

• 20:24Ed Zitron?

  Oh.

• 20:26Speaker 11

  Yeah.

• 20:26Speaker 12

  It's like 600,000 words long or something.

• 20:28Speaker 11

  Yeah, it's longer than Warren.

• 20:29Ed Zitron?

  I'm sorry, even I would never write something that long.

• 20:35Ed Zitron?

  Even I would, 200,000 words on anything is terrifying.

• 20:41Speaker 11

  I mean, this is 600,000 words.

• 20:43Ed Zitron?

  But what the fuck was it even about? Was it like?

• 20:45Speaker 11

  Oh, it was about, it was about Harry, uh, who is, you know, I, I think fairly clearly an, an author self-insert of Yudkowsky. Harry being the specialist boy, uh, and finding ways to, you know, change and save the wizarding world of Harry Potter using his, his special logic and science powers.

• 21:09Ed Zitron?

  Through logic and reason.

• 21:10Speaker 11

  Yeah, exactly. Yes.

• 21:13Ed Zitron?

  Right. Well, why?

• 21:16Speaker 11

  I guess, I guess.

• 21:18Ed Zitron?

  I guess it's just creative?

• 21:20Speaker 11

  Yeah. I mean, I interviewed him for my book and that was what he claimed. And I, I believe that it started out as a creative thing, but you know, it was also very clearly him playing with these ideas that he has about, you know, this is the only way to think clearly. And, uh, and it did end up becoming, you know, quite a recruitment tool for the rationalists.

• 21:45Speaker 12

  And just to put a little, just make sure we have like the timeline beats clear and then Adam fact check me here. There's sort of this shift that happens where in the early 2000s, uh, actually, I think his first money was, it was Pierre Thiel, but it was also that other, that other internet entrepreneurship couple. They fund something called like the Institute for AI Singularity or the, what, what is

• 22:04Speaker 11

  Oh, yeah, the Singularity Institute for AI. Originally, yeah, originally the funding comes from this couple, the name of which I don't remember, but eventually like a few years after that, in like the mid to late 2000s, he meets Thiel and Thiel gives them like real money.

• 22:18Speaker 12

  Yeah. And this is where, so as the 2000s move on to that Singularity Institute was, super intelligence is going to be a beneficial godhead that's going to redeem history.

• 22:28Speaker 11

  Well.

• 22:30Speaker 12

  Sorry. He shifts pretty, my understanding is like, once that's a full-time job and he's thinking about it a lot, he shifts pretty quickly. That he does that first big report, he shifts pretty quickly, but like, actually, wait, I don't think we'll control AI. We should not build it at all. It's going to kill us. Like there was this shift that happened like kind of quickly in that area and he shifted Valence and then eventually they renamed it to Mira, Miri, the machine intelligent research institute.

• 22:53Speaker 11

  Yeah, that's correct. Yeah. So, so just, just to get the timeline straight here. Yudkowsky makes that shift, uh, from, you know, super intelligence is going to solve everything and we should be aiming for it as fast as possible to, oh my God, it's going to kill us all. He makes that shift around the same time that Singularity Institute for AI is founded in the early 2000s.

• 23:17Ed Zitron?

  Yeah, right. Is there a, is there an antecedent or does he just one day wake up?

• 23:21Speaker 11

  He's just, I mean, he's just been on these message boards for a long time. And he also saw that like, you know, the, the nanotech guys who wanted a lot of the same things and made a lot of the same promises back in the eighties and nineties, people like Eric Drexler. They had their own sort of institute and Yudkowsky, you know, at this point he's, you know, unemployed and, and about 20. And I think he's, he's like, I want to work on this full-time. Can I turn this into a job and finds a way of doing that by, you know, co-founding this institute. But, um, I, I also, I wanna, I wanna just push back on one thing that you said, Cal.

• 23:59Ed Zitron?

  Okay.

• 24:00Speaker 11

  It's not quite that, you know, Yudkowsky wants to stop AI at all costs. He, he wants to stop AI at all costs until he and others figure out how to do it safely. At which point he wants to go full speed ahead. He still believes in the, you know, the AI utopia, if you can get a super intelligent AI to not kill everybody.

• 24:24Speaker 12

  So he just thinks that, that problem is harder than he originally imagined. So he's like, this might take longer.

• 24:29Ed Zitron?

  There's an impedance towards him being the thinking king.

• 24:34Speaker 11

  Yeah.

• 24:35Ed Zitron?

  Like the thought king that can tell the computer how to be best, that loves us.

• 24:40Speaker 12

  That's basically, yeah. He will, he will be the, the right-hand man to the digital god. But, but Adam, is this right? There was no technical antecedent for this shift in his thinking from actually this might be harder. The best I could tell from your book and looking at other sources is that the shift in thinking just came from as soon as he got the funding for the Singularity Institute that he was helping to run, he had this big writing project he took on where he's like, okay, I'm going to produce this big report on super intelligence. And it was in just the, the internal, you know, cognition of thinking through this report that he just convinced himself, oh, wait, I don't think we're going to be able to control this as easily as we think. Because I think it's really important to emphasize, again, this is not someone who has seen something in a technical lab that is scaring him.

• 25:24Speaker 12

  All of these developments and his thinking at this stage, it's really just, it's Yudkowsky in his own mind.

• 25:36Ed Zitron?

  All right, everyone, I'm back talking about Quince, one of my favorite clothing brands. And as we speak, I'm currently wearing one of their excellent cotton knit overshirts. I wear like a jacket, nice prefall piece to wear as things cool off a bit. I've also been eyeballing one of their heritage water-resistant Harrington jackets too for later in the year to pair with their Mongolian cashmere sweaters I bought. And I have several of them and I really can't recommend Quince's stuff enough. I love Quince because they focus on high-quality wardrobe staples made with premium materials, jackets, pants, fitted tees and polos, active wear and even travel gear. Everything at Quince is priced 50 to 80% less than similar brands. And because they work directly with ethical factories and cut out the middlemen, you're getting more for less. Find the four pieces you'll reach for most at Quince.

• 26:19Ed Zitron?

  Download the Quince app for app exclusive offers or go to quince.com. Get free shipping on your order and 365 day returns. Now available in Canada and the UK too. And when Quince asks where you heard about them, let them know it was on Better Offline. It's the best way to support the show.

• 26:36Speaker 01

  Snoring, gasping for air during sleep, daytime sleepiness. I'm Shaquille O'Neal and this shouldn't be anybody's experience. Ask your doctor about Zepbound Trisepotide, the first and only FDA-approved prescription medicine for moderate to severe obstructive sleep apnea in adults with obesity.

• 26:52Speaker 02

  Zepbound is a prescription medicine used with a reduced calorie diet and increased physical activity to help adults with moderate to severe obstructive sleep apnea, OSA, and obesity to improve their OSA. Zepbound is approved as a 2.5, 5, 7.5, 10, 12.5, or 15-milligram injection. Zepbound contains trisepotide and should not be used with other trisepotide-containing products or any GLP-1 receptor agonist medicines. It is not known if Zepbound is safe and effective for use in children. Don't share needles or pens or reuse needles. Don't take if allergic to it, or if you or someone in your family had medullary thyroid cancer, or if you've had multiple endocrine neoplasia syndrome type two. Tell your doctor if you get a lump or swelling in your neck. Stop Zepbound and call your doctor if you have severe stomach pain or a serious allergic reaction.

• 27:37Speaker 02

  Severe side effects may include inflamed pancreas or gallbladder problems. Tell your doctor if you experience vision changes before scheduled procedures with anesthesia. If you're nursing, pregnant, plant-to-be, or taking birth control pills. Taking Zepbound with a sulphonal urea or insulin may cause low blood sugar. Side effects include nausea, diarrhea, and vomiting, which can cause dehydration and worsen kidney problems.

• 27:59Speaker 01

  Talk to your doctor. Call 1-800-545-5979 or visit zepbound.lily.com.

• 28:08Speaker 13

  The world is transforming faster than ever and standing still isn't an option. At Oppenheimer, we're working at the forefront of the innovation economy to invest where progress begins, finding opportunities that build and protect wealth for individuals and institutions that want a seat at the edge of tomorrow.

• 28:28Speaker 13

  Put the power of Oppenheimer thinking to work for you. Wealth management, capital markets, investment banking.

• 28:38Speaker 14

  What if you could make that stop? With LPL Financial, we remove the things holding you back and provide the services to help push you forward. If you're a financial advisor, what if you could have more freedom, but also more support? Ready to invest? What if you could have an advisor that really understood you? When it comes to your finances, your business, your future, at LPL Financial, we believe the only question should be, what if you could?

• 29:01Speaker 15

  Paid advertisement. Anna Kendrick is not a client of LPL Financial LLC and receives compensation to promote LPL. Investing involves risk, including potential loss of principal LPL Financial LLC member FINRA SIPC.

• 29:09Speaker 16

  Summer's still going strong. Make a stop at Total Wine & More for white wines, beer, and ready-to-drink cocktails. Just know, you're getting the lowest prices. Keep the summer celebrations going at Total Wine & More. Spirits are not sold in Virginia and North Carolina. Drink responsibly. B21.

• 29:24Speaker 17

  This is Danielle Fishel.

• 29:26Speaker 12

  Ryder Strong.

• 29:27Speaker 18

  And Will Friedle from Pod Meets World.

• 29:29Speaker 17

  Halloween has a funny way of making you buy things you'd never normally buy. A 10-foot skeleton? Sure.

• 29:35Speaker 12

  Full-size fog machine? Why not?

• 29:38Speaker 18

  Three bags of candy that somehow become six bags of candy? It happens.

• 29:42Speaker 17

  Which brings us to the Chase Freedom Flex card, the savvy decision that will earn you more cash back on the everyday purchases, plus the extraordinary Halloween tricks and treats.

• 29:52Speaker 12

  With Freedom Flex, you earn even more treats on your Halloween and everyday needs. You get 5% cash back on popular rotating quarterly categories, 3% on dining and drugstores, and 1% on all other purchases.

• 30:06Speaker 18

  More ways to earn, more ways to get rewarded. That's no tricks, all treats.

• 30:10Speaker 17

  More ways to earn, more ways to get rewarded. So sign up for the Chase Freedom Flex card. Learn more at chase.com/getflex.

• 30:18Speaker 12

  This is a paid collaboration with Chase Freedom Flex. Cards issued by JP Morgan Chase Bank, NA, member FDIC. Subject to credit approval. Offer subject to change. Terms apply.

• 30:28Speaker 04

  Even on my antidepressant, I still struggled with depression symptoms.

• 30:32Speaker 05

  You OK, honey?

• 30:33Speaker 04

  I'm OK, ish. My doctor said adding Rexulti Brexpirizol could help.

• 30:38Speaker 05

  When added to an antidepressant, Rexulti was proven to significantly reduce depression symptoms, more than an antidepressant alone. Elderly people with dementia-related psychosis have increased risk of death or stroke. Antidepressants may increase suicidal thoughts and actions, and worsen depression in children and young adults. Call your healthcare provider right away to report new or sudden changes in mood, behavior, thoughts, or feelings, or if you develop suicidal thoughts or actions. Report fever, stiff muscles, and confusion, which can be life-threatening, or uncontrolled muscle movements which may be permanent. High blood sugar can lead to coma or death. Weight gain, increased cholesterol, unusual urges, dizziness on standing, falls, seizures, trouble swallowing, or sleepiness may occur. Learn about these and other side effects at rexulti.com. Tap ad for PI. Hey, how you doing, honey?

• 31:20Speaker 04

  OK.

• 31:21Speaker 05

  OK-ish?

• 31:22Speaker 04

  No-ish. I'm OK, really.

• 31:24Speaker 05

  Ask your doctor about adding Rexulti.

• 31:34Ed Zitron?

  I think that that's a really important point. Is that consistent through time, though?

• 31:39Speaker 11

  Yeah, I mean, he's, from that point on, he's convinced of this and has been saying this.

• 31:46Ed Zitron?

  But he's not seen anything, and these people have not seen anything.

• 31:48Speaker 11

  No, I mean, you know, he, later on, he starts pointing to various advances that are happening in machine learning as evidence that the amount of time that we have to solve this problem that he came up with is, you know, running out. But no, you know, that original idea and his, his faith in it is based purely on thought experiments and conversations he has with other people in this community.

• 32:15Ed Zitron?

  Because the way these fucking people frame themselves when they talk about this, they talk, I'm speaking from, I don't know, a podcast episode I just did while I'm, they don't talk about, and maybe this is how they varnish it, they don't talk about that they would be the authority in some cases. They just like, we should have one, which sounds, that's how they always get you, these people, people like this. It's always, they sound reasonable, and then behind closed doors, trousers come off. OK, that's, that's not true, but, well, maybe, but, but it's about, it's about how, oh, actually, oh, we are, we are like Boris Balkan in The Ninth Gate. I alone am worthy to deal with this. That's terrifying. I don't like that one bit.

• 32:55Speaker 11

  I mean, oh.

• 32:57Speaker 12

  OK, let me, yeah, because we're gonna make this worse now.

• 33:01Speaker 11

  Yeah, great.

• 33:02Speaker 12

  All right, so this, this then becomes the key, this is one of the key arguments in my New York Times op-ed, like, why is this relevant to right now? Well, the issue is, so this world is going on, Yudkowsky has his followers, he's convinced of this, it's mainly thought experiments, right? There's not a particular AI technology of looking at. That world severely overlaps with OpenAI, Anthropic, and DeepMind. And let me give you like the very condensed version, Adam can add the pieces that I'm missing here, right? But let's go back to the founding of DeepMind, right? There's three founders of, co-founders of DeepMind, two of them, Hasabi's and Shane Legg, they're at a Singularity Summit, they are talking with Yudkowsky. Yudkowsky says, let me introduce you to Peter Thiel, and Peter Thiel gives the initial

• 33:47Speaker 12

  funding for DeepMind. Shane Legg has been deeply involved in sort of the Yudkowsky rationalist world for a long time. He was motivated by a talk that Yudkowsky came to give about superintelligence at the place where, where Legg worked. He went on to write a doctoral dissertation on superintelligence, a frequent poster on Less Wrong. So DeepMind has these deep connections to that community. When they get bought by Google, you know, that's when Sam Altman and Elon Musk are like, well, that's bad, right? Because this is, in a true rationalist sense, this is a technology that's going to either redeem humanity or kill us all. We can't have a company like Google control it. That's when Musk puts up the money for OpenAI. You have Sam Altman, you know, tweeting favorably about Yudkowsky at this point, saying he's done more for AGI than anyone else.

• 34:31Speaker 12

  Yudkowsky should get the Nobel Prize. The current president of OpenAI, Greg Brockman, was running a Less Wrong reading group at his, the company where he worked before.

• 34:39Ed Zitron?

  Wait, even Brockman? Brockman was the normal one.

• 34:43Speaker 12

  No, no, no. All right, so you have these deep connections, the board of OpenAI, half of that board comes strongly out of the sort of rationalist EA communities. When they tried to oust Sam Altman in 2023, as Cade Metz reported at the time, it was the faction of the board with rationalist EA connections that felt like he was being insufficiently aligned with rationalist-type ideas. The temporary CEO that the board put in place to try to replace Sam Altman is someone who is so entrenched in rationalist circles that there's a character named after him in the Harry Potter fanfiction that Yudkowsky wrote. Dario Ambede and his sister and the other employees who left OpenAI to start Anthropic

• 35:24Speaker 12

  did so because they felt like, again, they were being insufficiently aligned in part to rationalist ideals about how do we usher in superintelligence safely? You, you know, you have Ambede, a couple of years ago, parroting common rationalist talking points, like, yeah, we have about 25% chance that AI is going to kill all of humanity. So these worlds, those three companies, they come out of this world of rationalism, which had a really big foothold in the Bay Area of the time, which means our leading AI labs, the way they think and talk about AI is heavily influenced by everything we just talked about.

• 36:03Speaker 11

  Yep. No, that's all correct.

• 36:06Ed Zitron?

  So when they say alignment, because alignment as an idea is not the worst idea. It's making sure the models give good outputs and the outputs aren't inherently evil.

• 36:17Speaker 11

  That's not really what they mean when they say alignment.

• 36:19Ed Zitron?

  No, but that's exactly what I'm getting at, though. Because there is the term alignment, which means the thing I just said to most people I talk to, and then what they mean by alignment, which appears to be something else.

• 36:30Speaker 11

  Yeah, no, what they mean by alignment, and I mean, this is, this is basically word for word what Yudkowsky says is, an AI that, when it becomes superintelligent, will not kill 100% of, of humanity.

• 36:44Ed Zitron?

  Wow.

• 36:45Speaker 11

  Yeah.

• 36:46Ed Zitron?

  That's fucking annoying.

• 36:48Speaker 11

  Yes.

• 36:49Ed Zitron?

  That's, I know I meant to sound more eloquent than that, but that's, that makes me very frustrated.

• 36:55Speaker 11

  Well, to make you more frustrated and to make it even more annoying and infuriating, because these guys were so deeply entrenched and involved with the AI industry, and because they'd been writing about it for so long, it means that when the sort of AI moment arrived for the public with, you know, the release of ChatGPT in late 2022, the rationalists and effective altruists were able to convincingly portray themselves as experts who had been working on AI for decades.

• 37:37Speaker 11

  And this is how you end up with things like, you know, Bernie having, you know, a sit-down with Eliezer Yudkowsky.

• 37:47Ed Zitron?

  Right. This makes more sense now because the way they frame themselves and they'll say like, oh, AI 2027 has been accurate in some ways even. And as you'll know in a podcast coming up, I pushed them on that and they refused to actually open the blog in front of me. But it's, it's how they have scary, scary stuff framed in the terms of academia without actually being academic. Right?

• 38:12Speaker 11

  Yeah.

• 38:12Speaker 12

  Yeah. Now, there's a two important sidebars here. So sidebar one, we've been kind of casually referencing effective altruism.

• 38:20Speaker 11

  Yeah.

• 38:21Speaker 12

  This is important, right? Because so effective altruism, this is where it gets complicated. So this is a another movement that has like similar sensibilities that became intertwined and heavily influenced. They sort of mutually influenced each other, right? But you have effective altruism. You know, it comes out of a bunch of utilitarian philosophers at Oxford. And they have this idea of, oh, we can apply, this should sound familiar, but it's like, if we're very like rational and empirical about altruism, we can do altruism better, which is, I think, is like a, it's like modern utilitarianism. That's a fine idea. This is where they're going around and doing research, like, what are the actual best charities to give money to? How do we measure that? They have these

• 39:02Speaker 12

  ideas, like, actually, if you are, if you're at Oxford, like an elite university, the best thing you could do for charity is to go get a high paying banking job and then give up half your salary at charity. So they're trying to think through rationally, how do we be, do altruism as effectively as possible? Well, they overlap and are influenced by rationalism. And pretty quickly, you have a large segment of effective altruists. You say, actually, the most altruistic thing we can do is prevent an AI apocalypse. Because why bother? Why care about, you know, saving 20 million people now from malaria, when we could instead be solving, you know, saving 100 trillion future

• 39:42Speaker 12

  lives that will never be born if AI kills us all? So effective altruism, not all of it, it's a complicated field, but big segments of effective altruism. And this is where Nick Bostrom comes out of at Oxford, talking about existential risk and superintelligence. A big segment of effective altruism really begins to care about super intelligent AI is like one of the core things to care about. And a huge amount of money pours into effective altruism. We're talking hundreds of millions of dollars starts going to these institutes and groups.

• 40:11Ed Zitron?

  And nothing's been done yet, though. Just want to be clear, like, no actual actions have been taken to stop superintelligence. No safety standards, no actual science has taken place either.

• 40:24Speaker 12

  Well, they wrote the Harry Potter fan.

• 40:26Ed Zitron?

  Okay, so no, I'm not even being sarcastic. Like, they wrote some posts.

• 40:31Speaker 11

  They wrote some posts and, and kept talking about how it was really hard because, you know, what does it mean to align something with human values? And like, well, maybe, maybe your question is ill posed.

• 40:41Ed Zitron?

  Maybe go work on a fucking large language model. Go and work on AI instead of this.

• 40:46Speaker 11

  Some of them, I mean, some of them did, right?

• 40:49Ed Zitron?

  And that's how we got anthropic.

• 40:51Speaker 11

  Yeah, yeah, yeah. And DeepMind and OpenAI. But, but like, just to, everything Cal said is right, except, except one thing, and I think it's important. Nick Bostrom is definitely like, part of the whole effective altruism community. But Nick Bostrom, he wrote, he's probably best known for a book that he wrote called Superintelligence that came out, I don't know, a little over 10 years ago, that made a big splash, basically, you know, expounding on all of these ideas and saying, yeah, you know, superintelligence is really scary and we should, we should, we should figure out, you know, how to control it or stop it before it kills us all. And again, this is before,

• 41:32Speaker 11

  you know, transformer models and the current AI moment. And, and it's not from any particular thing he's looking at. But the thing is, Bostrom, who at that time is a philosophy professor at Oxford, came out of the same, you know, place, basically, as Yudkowsky. He was on those same 1990s listservs. And he's the one who sort of is most responsible for the connection between rationalism and effective altruism, because it was when his colleagues at Oxford were sort of working on effective altruism, that he went over to them and said, oh, if you're worried about that, you really need to be worried about the AI.

• 42:14Ed Zitron?

  But these are two different worlds.

• 42:16Speaker 11

  Yeah, but at this point, they're kind of conjoined. Yeah, they're, at this point, they're, they're like, they're really like, you can't disentangle the two communities.

• 42:26Speaker 12

  Yeah, there's a talk you can find. So William McCaskill is one of the two big utilitarians at Oxford that helped found effective altruism. And I'd crossed paths with him sort of earlier on in the EA world, when they were really caring about, especially like the idea of donate most of your salary to charity. There's a talk you can find, and I, it's on YouTube, and I forget exactly when this happens, but there's, there's a famous talk, you can find him at Caskill, where he basically comes out and says, this is the new thing we should care about is existential, the existential threat of super intelligent AI. This is the, this sort of long termerism. So they use this, there's another idea that comes out of Silicon Valley, but this long term view of what's going

• 43:07Speaker 12

  to happen over the next thousand years is actually, if we're just going to do the math, 100 trillion future lives matters more than, you know, 50 million lives now. So he kind of has this moment where he gives this big speech that where he's sort of announcing this is the new direction for EA. And so now they're like roughly interchangeable. This is why, like, if you read, like, Cade Metz's reporting on like the OpenAI board, or reporting about Amade and his philosophical connections, they'll sort of use the terms rationalism and EA interchangeably. They'll use one or the other, say both. And that's why these circles have become very, very similar.

• 43:43Ed Zitron?

  But they, and they seem to have had different goals that have now kind of conjoined.

• 43:48Speaker 11

  Yeah, I think that's correct. Yeah. I should also point out, you know, the most famous example of someone who, who took McCaskill's advice and, you know, went off to make a bunch of money in the finance industry and then tried to donate large amounts of it, is Sam Bankman Fried.

• 44:06Ed Zitron?

  Oh.

• 44:06Speaker 11

  Yeah.

• 44:07Speaker 12

  Big EA.

• 44:08Speaker 11

  Yep.

• 44:09Ed Zitron?

  Yep. And actually, early investor in Anthropic.

• 44:12Speaker 11

  Oh, yeah. Yeah. Early investor in Anthropic, gave a bunch of money to rationalist and effective altruist community organizations. And yeah, no, I mean, McCaskill met SBF when he was a college student at MIT.

• 44:29Ed Zitron?

  And get around with a lot of kids.

• 44:31Speaker 11

  Yeah, well.

• 44:32Ed Zitron?

  Kind of the Charlie Kirk style thing where they go to college campuses almost.

• 44:37Speaker 11

  Yeah, well, I mean.

• 44:38Ed Zitron?

  For thinking online dudes.

• 44:40Speaker 11

  That's the thing. They, they do do this. There's a, God, there's a piece, I want to say it was in the Washington Post a few years ago by Natasha Tiku about campus recruitment strategies that the effective altruists were into, because they have, you know, just enormous piles of tech money that they were using to, you know, get them while they're young.

• 45:02Speaker 12

  And we're talking like, the effective altruists will come in and someone will just sign a check for like $60 million to start an institute. Like, we're talking huge, huge money.

• 45:11Ed Zitron?

  Money coming from?

• 45:12Speaker 12

  It's tech billionaires.

• 45:13Speaker 11

  Yeah.

• 45:15Speaker 12

  Because think about it, who is the hero? Who are the heroes in the rationalist? You are Neo from The Matrix, you're John Connor from The Terminator, you are the people who are going to save the world. And if you're a, a tech, you know, 100 million, a centimillionaire, billionaire, you're like, yeah, that tracks. That kind of matches my understanding of myself.

• 45:35Ed Zitron?

  And I, I assume the EA rationalist types don't think, hey, wait, are we just playing into the hands of the richest people alive?

• 45:44Speaker 11

  I mean, they kind of either don't think about it or don't really care. They don't think, they think that if you just think really, really rationally, then things like conflict of interests and corrupting your motives. Well, you know, those are just cognitive biases, and you can overcome those by, you know, the, the power of pure logic.

• 46:03Ed Zitron?

  That reminds me of this guy, Alex Heath, who is ostensibly a journalist, who's also become a venture capitalist this week. I saw someone comment on his post saying, now you've got skin in the game, you can be truly objective. And they were serious.

• 46:16Speaker 11

  Oh, my God.

• 46:17Ed Zitron?

  That, but that is that, that it sounds like the rationalist people. It's like, now you know the stakes. You will be bereft of, of any sorts of human biases you may have and work towards this vague goal we all have that we've, none of us can describe.

• 46:33Speaker 11

  I mean, they, they believe that, you know, making bets on things is, is a way of thinking clearly about stuff.

• 46:43Ed Zitron?

  When you say making bets, what do you mean?

• 46:45Speaker 11

  Money. I mean, I mean, like, you know.

• 46:47Ed Zitron?

  Like what they betting on?

• 46:49Speaker 11

  Well.

• 46:51Speaker 11

  I mean, this is, this is all their like, you know, decision theory stuff, but they have, um, there's this, uh, um, forecast gambling website called, uh, uh, manifest, uh, sorry, manifold, um, uh, where they, you know, like, bet on, you know, it's, it's a prediction market. Um, and they, they love prediction markets.

• 47:17Speaker 12

  Because economic theory, right? You're going to be, you know, the, the rational consumer. When your money is on the line, that's going to help you like overcome biases and get towards, you know, there, there, all this theory behind it is you're going to get a clear rational solution.

• 47:30Ed Zitron?

  Yep. So no, this reminds me of just like the most annoying early people online.

• 47:34Speaker 12

  Oh, yeah.

• 47:35Ed Zitron?

  The people who are doing brain, like mind master shit being like, yeah, we're so smart. Oh, well, do you want to make a gentleman's bet about who is the smartest? And then they set very strange, vague goals that they can move the goalposts on. Yeah, yeah, I've seen this before. Except they control so much money now.

• 47:53Speaker 12

  Well, okay. So, but now we got to connect this to the current moment because it, it gives you, this is the argument I was trying to make my time's piece. It gives you a different filter to understand what's going on because it's a confusing moment because we have two things happening simultaneously at the, the small number of frontier labs that are heavily connected to this world. A, you're developing a, a, a complicated, powerful technology, spending giant amount of capex on this. Right? So transformer based L L M's are a legitimate breakthrough in AI. Um, they can, if you push them, put enough money into them, you can, it's, they're very jagged, but you can, there's all sorts of interesting things that happen. It's a, it's a live wire technology. And so you have a real technology that's developing, but then it's being run by people

• 48:35Speaker 12

  who have these, these sort of futurist, almost apocalyptic sort of eschatological views. We're, we're basically, I feel like that community split where the hardcore original rationalists like Yukowski or, or, you know, Nate Soares, et cetera, would argue like, you know, we need to bomb the data centers because we don't know how to control this yet. We're all going to die. And then you have like the Altman's and the Amadeus who are like, uh, I think what's more important is that we control the digital god we get there first. Right? So it's either going to redeem history or destroy us all. So you have people running these labs and they're hiring people who think this way. Um, Andrew Ho online was talking about, there's probably a lot of pressure within these places to think that way. If you're hired there, they're thinking in these almost like apocalyptic terms,

• 49:17Speaker 12

  like we need to build the digital god before someone less trustworthy does, right? Like the end of history is coming and we need to be the ones to get there. And you have this actual live technology that's powerful and a lot of money is going into. And so these two things are now mixed together. And I don't think our sort of public or media landscape was ready for that. Like this mix of weirdness with a real company and real technology. And it's like, these two things together is blowing everyone's mind. They're like, well, I don't understand. Like there's these people saying we're all going to die. Like, why would someone say that? Why are they so calm about it? And I think it's almost like the Rosetta Stone to understand what's really happening in this moment is that we have a mixture of a lot of money and a powerful technology with a weird group of people.

• 50:00Ed Zitron?

  And what does that, what does that manifest as? What is the specific thing that people need to know?

• 50:06Speaker 12

  My concern, all right, there's a lot, I mean, a lot of different things are going on here. But my concern is if you look at what specifically is causing trouble, what are we worried about coming out of the frontier labs? Like what, in particular, the sort of rogue AI narrative is what seems to have, um, caught people's attention. The experiments that led to something like the hugging face attack were extremely negligent. Right? It, it boggles the mind. Uh, you know, what we, I've talked about on your show before, Ed, and people point to that episode. So I won't go through the whole thing, but you know, they're taking, uh, an agent is basically a prompt loop. It's a, it's a program that tells a L L M with a prompt, hey, I'm an agent that has access to these tools. Here's my goal. What should I do next?

• 50:48Speaker 12

  And then it gets a response from the L L M and it executes that, updates its state. And then says, okay, here's my new state. Here's my tools. What should I do next? So it just blindly executes what an L L M says, uh, which could be okay. Millions of programmers every day use agents. There's no problems with them. But they ran these, uh, over the summer, OpenAI was running these negligent experiments where they gave them very powerful tools, uh, that you could use for computer hacking. They hooked them up to an L L M that had no guardrails and what information they would give them. They made the programs persistent. So they said, we want you to just keep demanding an answer. Don't accept, you know, don't accept an answer of like, I don't know what to do next. And then they let them run for days without supervision and they bred thousands of them at the same time. And why did they run thousands of them at the same time?

• 51:32Speaker 12

  It's because what we now know is what they wanted to do is see if any of these versions happened to like stumble onto a particularly good hacking strategy. They would then use that trace to go back and reinforcement train the model to push towards that type of solution in the future. So it's like putting a thousand self-driving cars onto the interstate and saying, yeah, most are going to crash and maybe hurt people. But those that actually make it to the destination, we can use their data to help update our system. So it's this very negligent experiment they were running. And I think my claim is-

• 52:03Ed Zitron?

  And they had no idea that their compute was running as well. They just was like letting these little bastards run.

• 52:08Speaker 12

  They, I, I don't think they cared. And the reason why I don't think they cared is because they have this bigger eschatological belief of we got to get to the digital god before anyone else.

• 52:17Ed Zitron?

  And you think that that's what's driving OpenAI though?

• 52:20Speaker 12

  Well, this is what I'm worried about is that when that's your mindset, instead of focusing on useful products, things that can give you return on investment, common sense, safety standards, you're going to take big risks because you're racing toward, when you're racing towards super intelligence, you know, there's going to be some casualties along the way. That's acceptable. But that's my concern. I don't know, Adam, if you, if you agree with that, but that's my concern is because they are so convinced this is going to happen and it's so important. And it trumps, this is what their writing says, it trumps any measly concerns we have about like the current moment, that they will be reckless. It's also why now Amade and Altman are talking about, well, we need a global slowdown, whatever that means, before we can stop doing dangerous things.

• 53:01Speaker 12

  Cause if you believe that's the only way to get digital god, you're like, we can't slow down unless everyone else does, because if someone else gets the digital god, it's going to enslave us.

• 53:21Speaker 01

  Snoring, gasping for air during sleep, daytime sleepiness? I'm Shaquille O'Neal and this shouldn't be anybody's experience. Ask your doctor about Zepbound trisepotide, the first and only FDA approved prescription medicine for moderate to severe obstructive sleep apnea in adults with obesity.

• 53:37Speaker 02

  Zepbound is a prescription medicine used with a reduced calorie diet and increased physical activity to help adults with moderate to severe obstructive sleep apnea, OSA and obesity to improve their OSA. Zepbound is approved as a 2.5, 5, 7.5, 10, 12.5 or 15 milligram injection. Zepbound contains trisepotide and should not be used with other trisepotide containing products or any GLP-1 receptor agonist medicines. It is not known if Zepbound is safe and effective for use in children. Don't share needles or pens or reuse needles. Don't take if allergic to it, or if you or someone in your family had medullary thyroid cancer, or if you've had multiple endocrine neoplasia syndrome type two. Tell your doctor if you get a lump or swelling in your neck. Stop Zepbound and call your doctor if you have severe stomach pain or a serious allergic reaction.

• 54:22Speaker 02

  Severe side effects may include inflamed pancreas or gallbladder problems. Tell your doctor if you experience vision changes before scheduled procedures with anesthesia. If you're nursing, pregnant, plant to be, or taking birth control pills. Taking Zepbound with a sulphonal urea or insulin may cause low blood sugar. Side effects include nausea, diarrhea and vomiting, which can cause dehydration and worsen kidney problems.

• 54:44Speaker 01

  Talk to your doctor, call 1-800-545-5979 or visit zepbound.lily.com.

• 54:52Speaker 13

  At Oppenheimer, we're proven because we're grounded in discipline. For 145 years, we've been building and protecting wealth through every market cycle with precision, clarity, and the courage to think boldly beyond the moment. This is what market tested legacy looks like for this generation and the next. Put the power of Oppenheimer thinking to work for you. Wealth management, capital markets, investment banking.

• 55:23Speaker 14

  What if you could make that stop? With LPL Financial, we remove the things holding you back and provide the services to help push you forward. If you're a financial advisor, what if you could have more freedom, but also more support? Ready to invest? What if you could have an advisor that really understood you? When it comes to your finances, your business, your future, at LPL Financial, we believe the only question should be, what if you could?

• 55:46Speaker 15

  Paid advertisement. Anna Kendrick is not a client of LPL Financial LLC and receives compensation to promote LPL. Investing involves risk, including potential loss of principal LPL Financial LLC member, FINRA SIPC.

• 55:53Speaker 19

  Football season is back. Stop by Total Wine & More before kickoff for cold beer, craft bourbon, and ready to drink cocktails. Get the lowest prices and get your game day lineup ready at Total Wine & More. Spirits are not sold in Virginia and North Carolina. Drink responsibly. Be 21.

• 56:08Speaker 17

  This is Danielle Fishel.

• 56:10Speaker 20

  Ryder Strong.

• 56:10Speaker 18

  And Will Friedle from Pod Meets World.

• 56:13Speaker 17

  Halloween has a funny way of making you buy things you'd never normally buy. A 10-foot skeleton? Sure.

• 56:19Speaker 20

  Full-size fog machine? Why not?

• 56:21Speaker 18

  Three bags of candy that somehow become six bags of candy? It happens.

• 56:26Speaker 17

  Which brings us to the Chase Freedom Flex card, the savvy decision that will earn you more cash back on the everyday purchases, plus the extraordinary Halloween tricks and treats.

• 56:36Speaker 20

  With Freedom Flex, you earn even more treats on your Halloween and everyday needs. You get 5% cash back on popular rotating quarterly categories, 3% on dining and drugstores, and 1% on all other purchases.

• 56:50Speaker 18

  More ways to earn, more ways to get rewarded. That's no tricks, all treats.

• 56:54Speaker 17

  More ways to earn, more ways to get rewarded. So sign up for the Chase Freedom Flex card. Learn more at chase.com/getflex.

• 57:02Speaker 20

  This is a paid collaboration with Chase Freedom Flex. Cards issued by JP Morgan Chase Bank, N.A., member FDIC. Subject to credit approval. Offer subject to change. Terms apply.

• 57:12Speaker 04

  Even on my antidepressant, I still struggled with depression symptoms.

• 57:16Speaker 05

  You okay, honey?

• 57:17Speaker 04

  I'm okay, ish. My doctor said adding Rexulti Brexpirizole could help.

• 57:22Speaker 05

  When added to an antidepressant, Rexulti was proven to significantly reduce depression symptoms, more than an antidepressant alone. Elderly people with dementia-related psychosis have increased risk of death or stroke. Antidepressants may increase suicidal thoughts and actions, and worsen depression in children and young adults. Call your healthcare provider right away to report new or sudden changes in mood, behavior, thoughts, or feelings, or if you develop suicidal thoughts or actions. Report fever, stiff muscles, and confusion, which can be life-threatening, or uncontrolled muscle movements which may be permanent. High blood sugar can lead to coma or death. Weight gain, increased cholesterol, unusual urges, dizziness on standing, falls, seizures, trouble swallowing, or sleepiness may occur. Learn about these and other side effects at rexulti.com. Tap ad for PI. Hey, how you doing, honey?

• 58:03Speaker 04

  Okay.

• 58:04Speaker 05

  Okay-ish?

• 58:06Speaker 04

  No-ish. I'm okay, really.

• 58:08Speaker 05

  Ask your doctor about adding Rexulti.

• 58:13Speaker 21

  Right now, you can save 60% on SimpliSafe home security. When an intruder approaches a home, they expect a camera. What they don't expect...

• 58:21Speaker 22

  Stop. This is SimpliSafe. You're being monitored and recorded. I'm requesting police dispatch.

• 58:26Speaker 21

  Unlike traditional alarm systems, SimpliSafe uses smart AI cameras that spot potential threats and alert live US-based agents to help stop a break-in before it starts. Our agents intervene in under 30 seconds to deter intruders while they're still outside, preventing them from getting close to your family. Plus, you control your privacy and recordings. And with a 60-day money-back guarantee, you can try us risk-free. Monitoring and deterrence plans start around $1 a day. Order today to get 60% off a new system with professional monitoring. Save now at simplisafe.com/iheartradio. SimpliSafe: Ahead of Crime in Real Time.

• 59:06Speaker 19

  30 seconds response time requires monitoring plan with outdoor protection starting at $49.99 a month. Visit simplisafe.com/licenses for alarm license information.

• 59:18Speaker 12

  Yeah, no, I agree basically with all of that. I mean, this is why they're doing what they're doing. And it's just awfully convenient, isn't it? Right? When you have this belief that this is inevitably what's coming, that this is what you are building, and you know that you're right about it, well, it makes it really easy to ignore any other concerns, and especially any concerns about, you know, something concrete that's definitely happening, as opposed to this... I mean, the nicest way that I can describe their belief in this is that it's highly speculative. I think that that's too kind.

• 1:00:02Speaker 12

  And that it's a more accurate way to describe it is, it's a religious delusion.

• 1:00:10Ed Zitron?

  So, but could there not be? Because I don't know if I buy that this is everyone working at these companies, but I think what you're seeing right now with this slowdown talk might be a result of the schism between there are a lot of rationalists, and then there are other people who are working at a computer company.

• 1:00:30Speaker 12

  I don't think you need that.

• 1:00:31Ed Zitron?

  And so, that's why you've got weirdos like Jacob Coxon going out being like, "I'm so scared of a computer," without really saying what they're scared of. And also people being like, "Shit, no, no, I love AI, of course, but this is cloud software." Because if it was just rationalists, I don't think they'd be so apologetic. But it kind of explains why Amade is like, "I think it's time we finally tell the truth about this," while also not telling the truth, or saying what they're telling the truth about.

• 1:01:01Speaker 12

  I don't think you need that, Ed. I mean, I think if you are, you know, in this sort of rationalist philosophy, the slowdown is like, no, they're afraid of the digital god. But they're like, "We can't stop if other people are going." Like, I mean, I've... The more I understand about the belief system here, the more I discount, for example, the arguments about marketing, or the arguments about this is a good way to make your technology seem powerful. I increasingly believe, like, no, this is just what they believe. Like, they started these companies for this purpose. And I think it's the controlling atmosphere in those companies. Not everyone at the... So how do we know that? Right? I think it's the controlling atmosphere. Yeah, not everyone else feels that way. But what do we notice with OpenAI in the last four or five months?

• 1:01:44Speaker 12

  They've had a steady stream of high-profile hires from other segments of the tech industry that within like a week or a month leave. Right? So we've been seeing this revolving door. I have to imagine, you know, they're hiring you because they're at a size now where you have to hire executives that aren't from your own, you know, world. They're coming in, and perhaps what's happening is they're hearing the rational rhetoric and are like, "Uh-oh, you know, danger Will Robinson." And they're out the door again, like, "I don't want to be involved."

• 1:02:11Ed Zitron?

  Oh, so you think that Jacob Coxon might be genuine?

• 1:02:14Speaker 12

  Oh, I think Jacob Coxon is definitely genuine. Yeah. Look, he gave an interview to Wired a few days ago, okay? And in that interview, this is what they used as the headline at Wired. They said, you know, that he was calling this "crunch time for humanity." What they didn't say, what Coxon didn't say in the interview, or at least wasn't published in the interview, is that is a direct quote from Eliezer Yudkowsky from like, you know, 15 years ago.

• 1:02:41Ed Zitron?

  What was the exact quote?

• 1:02:42Speaker 12

  That this is crunch time for humanity.

• 1:02:45Ed Zitron?

  So the reason, the only pushback I'll give you there is he uses the phrase "crunch time," and he says, and I'm going to pull the quote now, these are actually just literal quotes from my colleagues at Anthropic. The reason I push back is because all of this started with him doing an exclusive with The Wall Street Journal, which is not traditionally how that works. I used to run a PR firm. You don't get an exclusive with The Wall Street Journal because you left a company and you're scared without the assistance of a PR firm, or at least an organization like EA. So I'm like, it's... It feels like this is a co-opted thing, almost.

• 1:03:28Speaker 12

  Well, I mean, that might be true.

• 1:03:29Ed Zitron?

  I don't get the sense it's like, he can't even explain what he's worried about.

• 1:03:34Speaker 12

  Yeah. Well, no, I mean, I think it's true that, right, you have this schism in the EA rationalist world, where you have the EA rationalists who are at the actual labs themselves, who are like, "We have to be the ones to create the digital god." And then you have the giant infrastructure of EA and rationalist institutes that are incredibly well-funded, that are saying, "Don't try to build the god at all, because it's going to be really bad." So I think that's completely plausible, that one of these, you know, obviously has connections to some of these EA institutes from before he took the job at Anthropic. So sure, that's... But I think that just matches what we're saying, right? So he is more from the camp of don't...

• 1:04:12Ed Zitron?

  Yeah, but I think that there is a big difference between a young man taking an opportunity, which is genuinely what I think, I really think that that's what he's doing. We can disagree, that's fine. And the EA corps taking advantage of that, because like Daniel Kocilachiv, whatever his name is, Kukuracha, from AI 2027, quote posted it within like seven minutes. Like, it was very clearly coordinated on some level.

• 1:04:42Speaker 12

  But what's... To what end, Doug? I mean, like, I've seen that, and it might be true. Maybe he's the Rosa Parks of like the EA's or something, you know?

• 1:04:47Ed Zitron?

  No, I don't think he's... I don't think... I think he knew that if he did this, it would get a bunch of attention. Because my big thing with this is, I think, like, I think this is a burn-after reading situation. I think that the vast majority of... The vast majority of the problems caused by this situation are, there are people who genuinely believe this stuff. There are people who cynically believe it, and then there are the rest of them who don't know who to trust, who are just going like, "Yeah, right? Like, sure, I agree."

• 1:05:21Speaker 11

  Like, uh, this is really bad. Like Sam Altman, I don't think he's a rationalist. I think he's a guy. I think it's actually quite funny watching Sam Altman adjust to this. Because he's just like, yeah, um, slow down safety council, I guess. What do you think? I don't really care. Dario Amadei, uh, now that you've said this, I, I will buy that he actually believes this stuff, which only makes the whole thing weirder. But I think that the mi- the mishmash of different cynical people, honest people, everyone's kind of weird is just ca- it's causing chaos in the AI industry right now. Like this is not, people are saying, oh, it's an op. Oh, this is all a plan. No, I don't think they have a plan at all.

• 1:06:03Speaker 12

  Yeah. No, I mean, I think that's a fair way of putting it. Um, I mean, like to me, what was more important than Jacob's tweet was the multiple responses from current Anthropic employees who are like, yeah, he's right. Yeah.

• 1:06:14Speaker 11

  Yeah, it's so strange.

• 1:06:16Speaker 12

  And it was so calm. And I think that blew a lot of people's minds, but I'll tell you who was not confused by that was probably Adam.

• 1:06:22Speaker 11

  Right? Yeah, no, that's exactly what I expected. Yeah.

• 1:06:24Speaker 12

  Yeah.

• 1:06:25Speaker 11

  Yeah, exactly. No, because, you know, there are a lot of people who believe in this stuff who work at these AI labs because they think that that's the best way to keep a handle on this is to be the ones who control the coming godhead. And, you know, I understand, Ed, the, the impulse to, to think that, you know, that this is something that people are using cynically, and there probably are some people in the, in the industry who are using it cynically. But there are an awful lot of true believers. There are so, so many of them. And, and these beliefs about what's coming have been so, so influential in the industry.

• 1:07:05Speaker 11

  It is the conventional wisdom about AI in the tech industry at this point is that AGI is coming, superintelligence is coming, and it is going to radically and permanently transform the world. And it's going to come out of these AI companies working with these kinds of AI systems. And there's just no evidence for any of that and a lot of evidence against it.

• 1:07:30Speaker 12

  And, and that's, but the, the thing, just to be clear about what I'm saying.

• 1:07:34Speaker 11

  Yeah.

• 1:07:35Speaker 12

  Actually, this is a point, I actually need to make this point very clearly. I think that there are a ton of EA rationalist types in this industry. I think Anthropic is full of them. I also think the overall industry led by Sam Altman played into their fear tactics and said, oh, we're scared of what this might do, insincerely. Some of them were sincere, some of them were not. The outcome is they've created a very real panic about nothing. Like, they won't talk about the real harms of LLMs. They'll talk about, talk about the big scary computer that might wake up one day. Well, and I would, I would push back on Sam Altman being insincere about it. That's just the way people talk in the world he came out of.

• 1:08:16Speaker 11

  Yeah, but Sam.

• 1:08:16Speaker 12

  That's how they talk. That's how Elon talked. That's how they all, Elon was running all these safety conferences, you know? That's how they all talk about this.

• 1:08:23Speaker 11

  I'm serious. That is the thing, though. Using Elon Musk as an example of someone with a sincere belief. I've read those emails. The, these are just like, these are just regular rich guys being rich. Like, I will buy that Yudkowsky believes this or what have you. The rationalist types believe this, what have you. But it's like, the idea that Sam Altman, i- it's just laughable. I just, I don't, that one, I don't really, I can't get my head around it. Same with Musk.

• 1:08:49Speaker 12

  Well, and your defense, right? The defense of your, your claim there would be the fact that the, the EA rationalist faction of the OpenAI board tried to get rid of Altman. Um, and which was at the time widely reported as this must just be because he's kind of a slimy guy. But, you know, it's also true that specifically the people who voted to do this are strongly aligned with Yudkowsky's world and thought that he wasn't serious enough about it. So that would actually.

• 1:09:15Speaker 11

  Yeah, and Ellen Toner, who was a big EA person, she got kicked out by Altman. Yeah, they cleaned it up. All the EA people on the board got kicked out, including, uh, what's, uh, what's her name? Uh, God, I can't remember.

• 1:09:28Speaker 12

  Joseph Gordon-Levitt's wife got kicked out.

• 1:09:29Speaker 11

  Yeah, that's, that's the person whose name I can't remember. Yeah. Yeah.

• 1:09:33Speaker 12

  Jesus. Um, but then, okay, so putting aside that, that sort of nuance, what's, how else can we empirically test, you know, the effect of this philosophy on these particular companies? Well, I laid out a test in my piece. I said, here's two things we can do. Look at other companies in the space that aren't as strongly connected to rationalism, but are just as cutting edge and really understand technology and see how they talk about it. And I pointed the Nvidia in particular because, you know, Jensen Huang is, he started that company while Yudkowsky was still a teenager. He, he doesn't have a lot of strong connections to that, you know, Silicon Valley, uh, futurism. And I said, also you have to keep an eye on China. They don't, they have no connection to that sort of Silicon Valley philosophy. And for the most part, they really don't talk about AI

• 1:10:16Speaker 12

  with that same sort of fear narratives that we do. Well, what happened this week? Jacob Croxen came out, they became a big discussion. What did Jensen Huang immediately said? He said, you know, uh, that's dangerous and outlandish. So immediately he's like, that claims makes no sense, right? Uh, today, China came out, like, what, what are you talking about? A global slowdown? What is this like psy ops, Cold War nonsense? Why would we start, why would we slow down our industry? Because they don't see it that way. They, they don't see their industry. They don't see AI as like a nuclear weapon or a godhead they're building. They see it as like an economic tool that they're trying to get trade advantage. So I feel like this week validated the claim that this particular philosophy is playing a big role. It's like that Sherlock Holmes story, the dog that didn't bark.

• 1:11:00Speaker 12

  Right? Like, the thing that matters here is not the things being said by the two labs that we know are deeply connected to these philosophies. It's all the other people who are doing cutting edge research on this, who know it well, who were not coming out and saying, yeah, we also think we're about to kill all of humanity. It's Nvidia, which is, you know, they supply all the chips to all these companies. They're, uh, intimately involved in these models and their evaluation. And I mean, they, they, they customize the, the, the API interfaces to their chips to specifically match with these, they know what's going on. And he's like, this is outlandish. What the hell are you talking about?

• 1:11:34Speaker 11

  But what's funny is Jensen Huang is barely a week divorced from declaring AGI is here.

• 1:11:40Speaker 12

  Yeah, but this was a bridge too far. This was a bridge too far this week.

• 1:11:43Speaker 11

  I, so I'm kind of yes ending, because what I'm saying is, he has, as I think some have cynically, engaged in this, oh, AGI, superintelligence wank, this conversation about it. And now suddenly it's like, oh, wait, maybe not that much. Maybe we don't superintelligence that hard. Maybe we, maybe we, actually it's not superintelligence. It's cloud software. Stop freaking out. And I think that this is the thing. It's regular greedy assholes locking horns with a cult, a religious cult of posters.

• 1:12:21Speaker 11

  Yeah, I think that's definitely part of what's going on here. Um, what am I missing? Well, no, I mean, I, I, I think,

• 1:12:31Speaker 11

  I mean, I just think that, that it's totally, totally possible that even some of the rich assholes genuinely believe this stuff. It's just that genuine belief for these billionaires, uh, you know, manifests in a different way because they believe that they have, you know, uh, like the power to warp reality with their money, which, you know, to a point is true. Like, uh, you know, uh, uh, Mark Andreessen is someone who I would never ascribe sincere beliefs to. Um, you know, like, I don't, I don't, I'm sure he has some sincere beliefs and I just don't think that we have any evidence of what they are because I think he just says the stuff that he says because he thinks it'll make him money.

• 1:13:17Speaker 11

  But, you know, he put out a manifesto a few years ago where he said that slowing down AI means that you are guilty of murder. Because.

• 1:13:29Speaker 12

  Cool.

• 1:13:29Speaker 11

  Uh, yeah, cool. Yeah. Because, uh, you know, that means you're slowing down the advent of a super intelligent AI that will save lives.

• 1:13:38Speaker 12

  This is EA in reverse.

• 1:13:39Speaker 11

  Exactly. Yes. And.

• 1:13:42Speaker 12

  Jesus fucking God.

• 1:13:43Speaker 11

  You know, that's a great excuse to say, well, but no, I, I have to make the investments that I make and you have to let me do it without any regulation or slowdown of any kind, because otherwise you're a murderer.

• 1:13:56Speaker 12

  I.

• 1:13:58Ed Zitron?

  Okay. As we wrap up though, I was about to rant about these people. They sicken me. What should we do about this? As in, okay, maybe not the first thought in your head, but what, what is the way that people should look at this differently? How can they pierce the kind of rationalist veil and at least have a more sensible way of looking at how everything's happening?

• 1:14:20Speaker 12

  I mean, I have a few suggestions right off the bat because these are the calls I've been taking all day, every day, it feels like for the last week. First of all, is clarity and terminology, right? Just as I wrote that op ed, Michelle Goldberg has a similar op ed that came out the next weekend. I think it's really important that people actually understand there is a particular philosophy that's heavily influencing these companies. At the very least, we should be factoring that into trying to understand what's going on. There's a good quote in Adam's book from Timnit Gariboo, who basically said, and I'm paraphrasing a little bit, but it was referring to this issue. If it turned out that the major AI companies were run by Scientologists, we'd want to know what's the influence of Scientology on the decisions

• 1:15:04Speaker 12

  they're making or how they run their companies. That's what's happening. It's just rationalism EA instead of Scientology, but it's a very eccentric, sort of almost cultish internally looking self-policed type of belief system that's playing a big role. We, we should at least know that's the case. So we can factor that in. Two, we cannot allow the people in these circles to get away with co-opting the general term AI to refer to specific things they're doing that are causing problems. I see this terminology everywhere in the media this week where it's all about AI is getting out of our control. We need to slow down AI. AI is getting smarter and what's going to happen. We have to be very specific, right?

• 1:15:45Ed Zitron?

  And I made this point in my newsletter as well as it's

• 1:15:48Speaker 12

  Good.

• 1:15:49Ed Zitron?

  The terminology of pacing the frontier is in their terms. It's taking their philosophy about this. It's, we need to slow down. Everything we're doing is fine. We're just doing it too fast.

• 1:16:00Speaker 12

  Well, in fact, they want it all connected. This is, this is a Max Tegmark mental model from his book, Intelligence 3.0. They have this idea of AI.

• 1:16:09Ed Zitron?

  Future of life guy?

• 1:16:10Speaker 12

  The future.

• 1:16:10Speaker 11

  Yeah, yeah. Which is, which is also deeply, deeply connected.

• 1:16:14Ed Zitron?

  I've met him. He is, he's nothing special.

• 1:16:18Speaker 12

  But he had this idea, he had this idea of thinking about AI capability, like a water level that rises, right? So that if anything is getting better, it leads you to think all of AI capabilities are getting better, which is partially why OpenAI is spending so much money on obscure math problems, right? There's not a financial, why, why aren't you building tools that like businesses will buy? It's because of the mindset of, well, if we can solve a math problem this hard, we can do everything comparably as hard. So they, they want to connect it all together. But if we get more specific, now the issue is a very specific thing. First of all, it's in the world of LLMs. Most AI is not LLMs. Within LLMs, specifically in the world of agents, but not all agents. We use, you know, millions of people use coding agents every day. They never go rogue. There's no issue

• 1:17:03Speaker 12

  with them. Like we don't, we're not worried about that technology. So it's a particular type of agent, which I call over-equipped, under-constrained, where they give them very powerful tools. They run it with no constraints. It's a very specific type of system that caused the hacking attacks and all of their concerns about recursive self-improvement is because they want to next take these over-equipped, under-constrained, LLM-powered agents and allow them to update their own code. So now they're completely unconstrained and unmonotable. It's that that we're worried about. So I think it's very important that we talk about specifically the thing we're worried about because it's very narrow. And then third, if I'm Congress, don't pass a bill about, we need to, in some vague sense, slow down

• 1:17:43Speaker 12

  AI, bring these people in to testify and say, tell me about these over-equipped, under-constrained, LLM-powered agents. Why are you running those experiments? Was there no way to run those safer? Is there, is, is there, is, is there other things you could, why are you focused on that? Is this what you mean when you're saying you're worried? Like, okay, what do you believe? Do you believe, and going through like Yakowski, like we really need transparency, I think, on this situation. That's what I would do because I think it would really change the dynamic. If we could pick apart what's true about the technology, what is being hyped and what is being driven by these beliefs and is causing more danger than, because you could shut down those over-equipped, under-constrained, LLM-powered agent experiments

• 1:18:27Speaker 12

  tomorrow, it would have almost no impact on your ability, all the products they're working on that are actually money-making. It should have no, no effect on their ARR and their revenue or anything, but they're not because they think that's the way to get to super intelligence. So I, that's what I would do. It's like, I would be super clear about what's really happening here. I'd be super clear about what rationalism EA philosophy is, and I would start pouring transparency on specifically what they're doing and why, what they believe, what experiments are dangerous, why they're running them so dangerously. That's the hearings we need right now.

• 1:19:00Ed Zitron?

  Adam, any, any additions?

• 1:19:02Speaker 11

  Yeah, yeah. I, I think that those are all good ideas, but I, I think the thing that I would add is we also really need to try to shift the public conversation on this stuff, right? The, the response to, you know, Coxon, the whistleblower at, I don't even want to use the word whistleblower, the guy who left Anthropic and made a big splash recently. The response to that should have been, yeah, of course you believe that because, you know, you, you're part of this group that has these weird beliefs. Like it's a nothing burger. Like Cal said, I wasn't surprised by this. You know, I wasn't surprised by what he said and what his colleagues and former colleagues said in response. I think

• 1:19:44Speaker 11

  that we need to get the message out there that this narrative about existential risk caused by super intelligent AI going out of control and that that is the most important and most likely thing that we need to worry about with AI. We have to get the word out that that is just a belief that comes out of a almost religious, cult-like object, the, the effect of altruists and, and rationalists. Like this is not something that is based in reality. And the real harms imposed by this technology are very concrete and very well-known.

• 1:20:28Ed Zitron?

  Awesome. That is great. I'm so glad you guys were here for this because I feel like at this particular time in history, as both of you have said, there is no clarity here. There are a lot of assumptions being made. And I'm really, thank you so much for joining me because I think this will really help people clear this up.

• 1:20:46Speaker 12

  Well, thanks, Adam. I think it's, the more we get this message out, the better. Oh, by the way, I would add one little extra caveat. Don't look at X.

• 1:20:54Ed Zitron?

  Yes.

• 1:20:55Speaker 12

  It's, it's, yeah, it's completely caught up in this engagement farming off of this, this rationalist EA belief system. If you're a reporter out there who's listening to this, do not look at the X conversations. Call real sources, talk to them in person because X is off the rails right now.

• 1:21:15Ed Zitron?

  Indeed. All right. I'm going to end it there. This has been Better Offline. I'm Ed Zitron, joined by Adam Becker and of course, Cal Newport. I'll be back with a monologue next. Well, I think it's going to be Friday. Anyway, this Friday. Cheers, everyone.

• 1:21:36Ed Zitron?

  Thank you for listening to Better Offline. The editor and composer of the Better Offline theme song is Matt Osowski. You can check out more of his music and audio projects at mattosowski.com. M A T T O S O W S K I dot com. You can email me at [email protected] or visit betteroffline.com to find more podcast links and of course, my newsletter. I also really recommend you go to chat dot where's your ed dot at to visit the Discord and go to r slash better offline to check out our Reddit. Thank you so much for listening.

• 1:22:08Speaker 09

  Better Offline is a production of Cool Zone Media. For more from Cool Zone Media, visit our website, coolzonemedia.com, or check us out on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

• 1:22:40Speaker 23

  Snoring, gasping during sleep, feeling fatigued? Ask your doctor about Zepbound terzepatide, the first and only FDA-approved prescription medicine for moderate to severe obstructive sleep apnea, OSA, in adults with obesity. Zepbound is a prescription medicine used with a reduced calorie diet and increased physical activity to help adults with moderate to severe obstructive sleep apnea, OSA, and obesity to improve their OSA. Zepbound is approved as a 2.5, 5, 7.5, 10, 12.5, or 15-milligram injection. Zepbound contains terzepatide and should not be used with other terzepatide-containing products or any GLP-1 receptor agonist medicines. It is not known if Zepbound is safe and effective for use in children. Don't

• 1:23:24Speaker 23

  share needles or pins or reuse needles. Don't take if allergic to it, or if you or someone in your family had medullary thyroid cancer, or if you've had multiple endocrine neoplasia syndrome type two. Tell your doctor if you get a lump or swelling in your neck. Stop Zepbound and call your doctor if you have severe stomach pain or a serious allergic reaction. Severe side effects may include inflamed pancreas or gallbladder problems. Tell your doctor if you experience vision changes before scheduled procedures with anesthesia. If you're nursing, pregnant, plan to be, or taking birth control pills. Taking Zepbound with a sulphonal urea or insulin may cause low blood sugar. Side effects include nausea, diarrhea, and vomiting, which can cause dehydration and worsen kidney problems. Talk

• 1:24:03Speaker 23

  to your doctor. Call 1-800-545-5979, or visit zepbound.lily.com.

• 1:24:12Speaker 14

  With LPL Financial, we provide the services to help push you forward. When it comes to your finances, your business, your future, the only question should be, what if you could?

• 1:24:20Speaker 15

  Paid advertisement. Anna Kendrick is not a client of LPL Financial LLC and receives compensation to promote LPL. Investing involves risk, including potential loss to principal LPL Financial LLC member FINRA, SIPC.

• 1:24:27Speaker 16

  Summer's still going strong. Make a stop at Total Wine & More for white wines, beer, and ready-to-drink cocktails. Just know you're getting the lowest prices. Keep the summer celebrations going at Total Wine & More. Spirits are not sold in Virginia and North Carolina. Drink responsibly. B21.

• 1:24:43Speaker 24

  Small businesses are the pulse of every community. They bring people together, create opportunities, and drive growth. That's why Chase for Business helps small business owners like you by offering personalized guidance and digital tools that can help your business grow. With our guidance and your determination, you can take your business farther and help build a brighter future for your community. Learn more at chase.com/business. Chase for Business. Make more of what's yours. The Chase mobile app is available for select mobile devices. Message and data rates may apply. JP Morgan Chase Bank, N.A., member FDIC. Copyright 2026 JP Morgan Chase and Company.

• 1:25:18Speaker 25

  If you want something done right, you do it yourself. That's why you change your own oil. You're thinking about the long haul. Pennzoil Ultra Platinum Full Synthetic Motor Oil offers engine protection for the lifetime of your vehicle. And if you're doing it right, that's a long time. Plus, it's on rollback at Walmart for a short time, so you can save while protecting your engine. Pennzoil. Long may we drive. Limited lubrication warranty for lifetime engine protection. Other conditions apply, including enrollment and receipt requirements. See pennzoil.com/warranty for full details and terms.

• 1:25:48Speaker 26

  This is Jacob Goldstein from What's Your Problem? Running a business is hard enough. Don't make it harder with a dozen apps that don't talk to each other. One for sales, another for inventory, a separate one for accounting. That's software overload. Odoo is the all-in-one platform that replaces them all. CRM, accounting, inventory, e-commerce, HR, fully integrated, easy to use, and built to grow with your business. Thousands have already made the switch. Why not you? Try Odoo for free at O D O O dot com. That's Odoo dot com.

• 1:26:25Speaker 27

  The path to retirement is different for everyone. And as life changes, so do priorities. Fidelity can help you get where you want to go. With a free personalized plan, goal tracking, and timely insights, you'll be set to take on retirement. Whether you're saving for it or already living in it. Get started at fidelity.com/takeon. Expenses charged by your investments and other costs and fees associated with trading or transacting in your account apply. Fidelity Brokerage Services, member NYSE, SIPC. This is an iHeart Podcast. Guaranteed human.