Good Robot #1: The magic intelligence in the sky

Before AI became a mainstream obsession, one thinker sounded the alarm about its catastrophic potential. So why are so many billionaires and tech leaders worried about… paper clips? This is the first episode of our new four-part series about the stories shaping the future of AI. Good Robot was made in partnership with Vox’s Future Perfect team. Episodes will be released on Wednesdays and Saturdays over the next two weeks. For show transcripts, go to vox.com/unxtranscripts For more, go to vox.com/unexplainable And please email us! [email protected] We read every email. Support Unexplainable by becoming a Vox Member today: vox.com/members Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Speaker 00

  Support for the show comes from TikTok. If you listen to this show, you probably enjoy discovering new things. It is built into the DNA of the show that curiosity is its own reward and that there's always a deeper dive to do. And one place to explore some of the topics we've covered here is TikTok. Whether it's how to forecast volcanic eruptions, strange phenomena that the James Webb Space Telescope picked up near the beginning of time, or a literal deep dive on the ocean's deepest reefs, TikTok has a wealth of fascinating videos just waiting to be discovered. Your next discovery is waiting on TikTok. Learn more at tiktok.com/discovermore.

• 0:49Speaker 01

  Support for the show comes from Engine. Running a small business means every dollar has to work hard. But if your team is still booking travel the old way, it's costing you more than you think. Engine is the fastest growing travel and spend platform in the country, built specifically for businesses like yours. Book a trip in as little as two and a half minutes, earn up to 10% back on hotels. And in 2025, Engine customers save more than $300 million on travel with zero booking fees, no contracts, and no BS. More than 1,000 businesses join Engine every month. Join them and get $500 when your business signs up and starts traveling at engine.com/box.

• 1:35Noam Hassenfeld?

  It's unexplainable. I'm Noam Hassanfeld. Over the next couple of weeks, we're going to be bringing you a special series from the newest member of our team, Julia Longoria. She's diving deep into AI, which is a topic we've definitely talked about before. But she's taking a new kind of expansive perspective. It's not just about AI. It's about the people behind it, what they believe in, the stories they tell, and how those stories are shaping the future of AI itself. I really can't wait for you to hear it.

• 2:13Speaker 03

  Suppose in the future, there's an artificial intelligence.

• 2:22Speaker 03

  We've created an AI so vastly powerful, so unfathomably intelligent, that we might call it "super-intelligent."

• 2:31Speaker 03

  Let's give this super-intelligent AI a simple goal:

• 2:36Speaker 03

  Produce...

• 2:39Speaker 03

  paperclips.

• 2:44Speaker 03

  Because the AI is super-intelligent, it quickly learns how to make paperclips out of anything in the world. It can anticipate and foil any attempt to stop it, and will do so because its one directive is to make more paperclips. Should we attempt to turn the AI off, it will fight back because it can't make more paperclips if it is turned off. And it will beat us because it is super-intelligent and we are not.

• 3:13Speaker 03

  The final result? The entire galaxy, including you, me, and everyone we know, has either been destroyed or been transformed...

• 3:30Speaker 03

  into paperclips.

• 3:50Speaker 04

  Welcome. Thank you.

• 3:53Speaker 04

  Are you lost? This past summer, I found myself at a very niche event in the Bay Area. Cool. And what brought you to town? Because you don't live here, right? I came here for this festival conference thing.

• 4:09Speaker 04

  How much context on this will I say again? Please, dude. Yeah, so... It's so fun to watch people try to describe. The crowd is mostly dudes, a mix of people in their 20s, 30s, and 40s. It feels kind of like a college reunion meets costume party. I spot some masquerade masks and tie-dye jumpsuits.

• 4:33Speaker 00

  Around blogging.

• 4:35Speaker 04

  This festival conference thing is the first official gathering IRL of a blogging community founded about 15 years ago.

• 4:44Speaker 00

  I am the old school fucking rat. I am the oldest of schools.

• 4:49Speaker 04

  Amazing. And rats refers to?

• 4:52Speaker 00

  Rationalists.

• 4:53Speaker 04

  They call themselves the rationalists. Rats strive to be rational in an irrational world by thinking things through, often with quirky hypotheticals. They try to be rational about monetary policy, rational about evolution, rational even about dating.

• 5:12Speaker 05

  It got kind of mocked for trying to solve romance by writing long blog posts about it.

• 5:17Speaker 04

  But their most influential idea, their most viral meme, you might say, is one that influenced Elon Musk and created an entire industry. It's about the possibility of an AI apocalypse.

• 5:37Speaker 04

  As a bit of a normie myself. I was a normie once myself too. I just was drawn to the way that the community talks in these thought experiments, right? The paperclip maximizer in particular caught my attention.

• 5:50Speaker 06

  That was the one I had in mind.

• 5:51Speaker 04

  Yeah.

• 5:51Speaker 06

  Paperclip maximizer.

• 5:52Speaker 00

  Paperclip maximizer is a clear example of the thing people have classically been scared of.

• 6:01Speaker 04

  The paperclip maximizer is a thought experiment, an intentionally absurd story that tries to describe what rationalists foresee as a real problem in building AI systems. How do you kind of shape, control a artificial mind that is more capable than you, potentially as general or more general? They imagine a future where we've built an artificial general intelligence beyond our wildest dreams. Generally intelligent, not just at some narrow task like spell checking, and super intelligent. I'm told that means it's smarter, faster, and more creative than us.

• 6:45Speaker 04

  And then we hand this AI a simple task.

• 6:49Speaker 06

  Give it the job of something like, can you make a lot of paperclips, please? We need paperclips. Can you make there be a lot of paperclips?

• 6:57Speaker 04

  The task here, I'm told, is ridiculous by design. To show that if you are this future AI, you're going to follow the instructions you're given to a T. Even if you're super intelligent and you understand all the intricacies of the universe, paperclips are now your one priority.

• 7:17Speaker 05

  You totally understand that humans care about other stuff like art and children and love and happiness. You understand love. You just don't care about it because the thing that you care about is making as many paperclips as possible. And if you have the resources, maybe you'll turn the entire galaxy into paperclips.

• 7:39Speaker 04

  A lot of rationalists I spoke to told me they've thought this thing through.

• 7:43Speaker 00

  It was clear to me when I first heard the arguments that they weren't obviously silly.

• 7:49Speaker 04

  Was that thought experiment part of convincing you that this was something that we needed to worry about?

• 7:54Speaker 06

  Yes, definitely.

• 7:55Speaker 04

  And they are very, very worried. Not about a paperclip apocalypse in particular, but about how as we build more powerful AI systems, we might lose control of them. They might do something catastrophic. I think it in a way makes it hard, hard to like plan your life out or like feel like you stand somewhere solid, I think.

• 8:27Speaker 04

  The reason I, a mere normie, find myself at this festival conference thing is that I've been plunging my head deep into the sand about AI. I've had a general sense that the vibes are kind of bad over there. Will this tech destroy our livelihoods or save our lives?

• 8:47Speaker 01

  Use of artificial intelligence could lead to the annihilation of humanity.

• 8:53Speaker 04

  We never talked about a cellphone apocalypse or an internet apocalypse. I guess maybe if you count Y2K, but even that wasn't going to wipe out humanity. But the threat of an AI apocalypse, it feels like it's everywhere.

• 9:08Speaker 03

  Mark my words, AI is far more dangerous than nukes.

• 9:13Speaker 04

  From billionaire Elon Musk to the United Nations.

• 9:18Speaker 01

  Today, all 193 members of the United Nations General Assembly have spoken in one voice: AI is existential.

• 9:27Speaker 04

  But then, it feels like scientists in the know can't even agree on what exactly we should be worried about.

• 9:34Speaker 01

  These existential risks that they call it. It makes no sense at all, and on top of that, it's an enormous distraction from the actual harms that are already being done in the name of AI.

• 9:44Speaker 04

  It all feels way above my pay grade, overwhelming and unknowable. I'm not an AI scientist. I couldn't tell you the first thing about how to build a good robot. It feels like I'm just along for the ride of whatever technologists decide to make, good or bad. So better to just plug my ears and say, la la la la. But I recently took a job working with Vox, a site that's been covering this technology basically since it started. On top of that, last year, Vox Media, Vox's parent company, announced they're partnering

• 10:24Speaker 04

  with OpenAI, meaning I'm not totally sure what it means. But if I was ever going to have to grapple with AI and its place in my life, it's here, now, at Vox. So I'll start with a simple question: How did some people come to believe that we should fear an AI apocalypse? Should I be afraid? This is Good Robot, a series about AI from Unexplainable, in collaboration with Future Perfect. I'm Julia Longoria.

• 11:21Speaker 00

  Support for the show comes from Shopify. Starting a business requires you to be resourceful. And the more you flex that muscle, the more you start to see opportunities for growth. For example, launching your business with Black Friday approaching means you can take advantage of the extra online traffic. And Shopify is there to help you make it all happen. Shopify says they are the platform for a record-breaking, reliable holiday-selling season for businesses at every stage. They have pre-built website themes and an AI site builder, which should make it easy to launch a beautiful, professional-looking store. And you don't ever need to touch a line of code to do it. They also have a mobile app that lets you run your store from anywhere.

• 12:06Speaker 00

  And they say their AI e-commerce expert sidekick can help you manage almost any business task with a single prompt, including holiday discounts. With Shopify, you can easily sell everywhere people shop: online, in person, across social media, even in AI chats. If you are ready to launch your business before Black Friday, go to Shopify.com/unexplainable to start your free trial. That is Shopify.com/unexplainable. Go to Shopify.com/unexplainable today to hear your first.

• 12:46Speaker 00

  Support for the show comes from TikTok. Being curious is endlessly rewarding. There are few better feelings, in my opinion, than asking a question and then having that question lead you down a rabbit hole. It might even be the beginning of a new set of interests that was not on your radar beforehand. Maybe you're looking up at the night sky one night, and you start to wonder what the most distant thing you can see with the naked eye is. One place to start searching for answers to a question like that is TikTok. First, you might discover a video about the Andromeda Galaxy. And then, next thing you know, you're learning how to find different stars, or how to identify

• 13:29Speaker 00

  constellations, or how different astronomical discoveries got made, or how constellations got their names. You don't need serious equipment to make serious discoveries. Start with a simple question, and suddenly, you're somewhere you never expected to be.

• 13:46Speaker 04

  Your next discovery is waiting on TikTok. Learn more at tiktok.com/discovermore. At tiktok.com/discovermore.

• 14:01Speaker 04

  Support for the show comes from Engine. Running a small business means every dollar has to work hard. But if your team is still booking travel the old way, it's costing you more than you think. Engine is the fastest growing travel and spend platform in the country, built specifically for businesses like yours. Book a trip in as little as two and a half minutes, earn up to 10% back on hotels. And in 2025, Engine customers save more than $300 million on travel with zero booking fees, no contracts, and no BS. More than 1,000 businesses join Engine every month. Join them and get $500 when your business signs up and starts traveling at engine.com/vox.

• 14:48Speaker 01

  I'm an android. Lieutenant Commander Data.

• 14:51Speaker 04

  Take and brief counseling will be available at the conclusion of the test.

• 15:02Speaker 04

  Here we go.

• 15:05Speaker 04

  When I first started reporting on the idea of an AI apocalypse, and if we should be worried about it, my first stop was the Bay Area for the Rationalist Conference. But I also stopped by the house of a colleague nearby. Hi, Kelsey. How you doing?

• 15:21Speaker 05

  Good. How was your flight?

• 15:22Speaker 04

  It was actually seamless. Vox is largely a remote workplace. So it was one of those body dysmorphic experiences to meet Kelsey Piper in 3D, taller than she looks on Google Meets.

• 15:35Speaker 05

  I am a writer for Vox's Future Perfect, which is the Vox section that's about undercover issues that might be a really big deal in the world.

• 15:44Speaker 04

  We were joined by her seven-month-old. As she was saying Vox's Future Perfect is about...

• 15:49Speaker 05

  undercover issues that might be a really big deal in the world.

• 15:52Speaker 04

  Kelsey's thought that AI technology would be a really big deal in the world long before this AI moment we're all living. She's been thinking about AI since she was a kid, when she first found the rationalist community online.

• 16:07Speaker 05

  Oh, I was in high school. I was 15.

• 16:11Speaker 05

  Bored academic overperformer with a very long list of extracurriculars that would look good to colleges down the road. And in my free time, I read a lot of Harry Potter fanfiction, as you know, 15 year olds back in 2010 did. One of the most popular Harry Potter fanfictions was called Harry Potter and the Methods of Rationality.

• 16:32Speaker 05

  Harry Potter and the Methods of Rationality by Eliezer Yudkowsky. Eliezer was influenced by a lot of the early sci fi authors.

• 16:41Speaker 04

  Eliezer, as he's known to the rats, is the founding father of rationalism, king of thought experiments. Back in 2010, he started publishing a serialized Harry Potter fanfic over the course of years. It's since inspired several audiobook versions.

• 16:59Speaker 03

  Harry Potter and the Methods of Rationality, written by Eliezer Yudkowsky.

• 17:03Speaker 04

  And a version acted out by The Sims.

• 17:06Speaker 03

  Mum, if you want to win this argument with dad, look in chapter two of the first book of the Feynman Lectures on Physics.

• 17:13Speaker 04

  It too was a thought experiment. What if Harry Potter were parented differently?

• 17:19Speaker 05

  The initial promise is just that Harry Potter, instead of having abusive parents, has nerdy parents who teach him about science.

• 17:27Speaker 04

  So his aunt and uncle are like actually...

• 17:29Speaker 05

  Are nice people, yeah.

• 17:31Speaker 01

  Harry, I do love you. Always remember that.

• 17:35Speaker 04

  And in this version, Harry Potter's superpowers turn out not to be courage and magic, but math and logic, what Eliezer calls the Methods of Rationality.

• 17:50Speaker 04

  So Harry Potter has a quest to do what exactly?

• 17:55Speaker 05

  You know, fix all of the bad things in the world. And the combination of incredibly naive and also in some sense incredibly respectable, I think, as a teenager, that's super appealing and fun. Where you're like, why would I limit myself to only solving one of the problems? While there are any problems, I'm not done. We've got to fix everything.

• 18:17Speaker 04

  The idea that every problem should be thought about, every problem could be fixed, that was appealing to his readers, including 15-year-old Kelsey. She wanted to read more, so she found her way to Ali Azar's blog.

• 18:31Speaker 05

  Ali Azar was pretty openly like, "I wrote this to see if it would get people into my blog, Less Wrong, where I write about other issues."

• 18:40Noam Hassenfeld?

  So the question is, please tell us a little about your brain. What's your IQ?

• 18:47Speaker 04

  On his blog, called Less Wrong, he applies the methods of rationality, math and logic, to all kinds of topics.

• 18:55Noam Hassenfeld?

  So the question is, how to start training young children as rationalists?

• 18:59Speaker 04

  Like child rearing.

• 19:00Noam Hassenfeld?

  Training children to be self-aware, trying to get them more interested in being fair to both sides of an argument.

• 19:07Speaker 04

  Religion.

• 19:08Noam Hassenfeld?

  My parents, they're modern Orthodox Jews, always avoiding their real weak points of their beliefs.

• 19:14Speaker 05

  It had stuff about atheism, a lot of stuff about psychology biases, experiments that showed that, depending how you ask the question, you get very different answers from people. Because the idea is that you're supposed to, by, you know, reading the blog and participating, learn how to be less wrong.

• 19:32Noam Hassenfeld?

  I do it by stories and parables that illustrate it.

• 19:35Speaker 05

  Like, the default state is that we're all very confused about many things, and you're trying to do a little bit better.

• 19:40Speaker 04

  Interesting. So it's kind of like trying to, sort of, I don't know, like, work out the bugs in the human brain system to optimize prediction?

• 19:54Speaker 05

  Yeah, and a ton of the people involved are computer programmers, and I think that's very much how they saw it. Like, the human brain has all these bugs. You go in and you learn about all of these. You learn to correct for them. And then, once you've corrected for them, you'll be a better thinker and better at doing whatever it is you set out to do.

• 20:11Speaker 04

  The biggest human brain bug Ali Azar wanted to address was how people thought about AI, how he himself used to think about AI.

• 20:23Speaker 04

  His very first blog post, as far as I can tell, was in 1996, when he was just 17. And in a very 17 kind of way, he writes about his frustrations. "I have had it. I've had it with crack houses, dictatorships, and world hunger. I've had it with a planetary death rate of 150,000 sentient beings per day. None of this is necessary. We repeat the mantra, 'I can't solve all the problems of the world.' We can. We can end this." And the way to end this, he thought back then, was to build a superintelligent AI, a good robot that could save the world.

• 21:08Speaker 04

  But at around 20 years old, while researching how to build it, he became convinced building superintelligent robots would almost certainly go badly. It would be really hard to stop them once they were on a bad path.

• 21:23Speaker 06

  I mean, ultimately, if you push these things far enough without knowing what you're doing, sooner or later, you're going to open up the black box that contains the black swan surprised from hell.

• 21:33Speaker 04

  And at first, he was sending these warnings into the void of the vast internet.

• 21:39Speaker 06

  So the question is, do I feel lonely often?

• 21:44Speaker 06

  That's... I often feel isolated to some degree, but writing less wrong has, I think, helped a good deal.

• 21:57Speaker 05

  The way I tend to think about Eliezer Yudkowsky as a writer is that he has a certain angle on the world, which can be like a real breath of fresh air. Like, oh, there's someone else who cares about this. You know, you can feel very seen for the first time. And if...

• 22:13Speaker 04

  Is that how you felt?

• 22:14Speaker 05

  Oh, yeah, yeah. You have a good heart, and you are certainly trying to do the right thing, but it's very difficult sometimes to figure out what that is.

• 22:25Speaker 04

  That pursuit of being less wrong, of doing the right thing in the right way, brought many kindred spirits together on the blog.

• 22:33Speaker 05

  Actually, several of my housemates posted on Less Wrong back in the day. This is how I met a bunch of the people I live with. They were people whose blogs I read back when I was a high school student.

• 22:45Speaker 04

  Wow, that's kind of wild.

• 22:47Speaker 05

  Yeah.

• 22:48Speaker 04

  Many Less Wrong bloggers and readers like Kelsey were inspired to move to the Bay Area to join a pretty unusual community, IRL. And the weekend I visited, hundreds of rationalists from around the world gathered in the Bay to reason things out together for a festival conference thing called Less Online. Many rationalists I met there found the community the way Kelsey did.

• 23:15Speaker 07

  A friend of mine at math camp introduced me to Harry Potter and the methods of rationality. The post, it was written in all caps, saying, "Oh, my God, I've just read the most amazing book in my life.

• 23:25Speaker 04

  You have to read it right now," linking to fanfiction.net. Others found Eliezer on his blog.

• 23:31Speaker 06

  I mean, this, this event exists in very large part because of that series of blog posts.

• 23:37Speaker 04

  That series of blog posts has become known by the community as The Sequences. It includes the paperclip maximizer thought experiment. Eliezer Yudkowsky helped come up with the idea, intending to warn people of the danger of an AI apocalypse. And at least here, it seems to have worked.

• 23:56Speaker 06

  Like, I definitely think AI is the largest kind of existential risk that humanity faces right now.

• 24:01Speaker 04

  Yeah. I, the normie, wanted to try to take this threat beyond quirky hypotheticals to something more concrete. And can you walk me through, like, how could that happen? Like, how could an AI?

• 24:16Speaker 06

  It's really hard to say how it will happen if it does.

• 24:19Speaker 04

  Yeah.

• 24:19Speaker 06

  It's a little easier to say ways that it might happen and to kind of provide very-

• 24:23Speaker 04

  Examples to like just generate intuitions for why this might be. But anytime I pressed a rationalist on it, they gave me yet another series of thought experiments. Kind of the way it might happen is analogous to how a 21st century army might defeat an 11th century army. Which, I guess, might be the only way to try and describe a threat from a technology that's really still in its infancy. For rationalists first introduced into this world, like 15-year-old Kelsey, these thought experiments were convincing. AI to her was a really big deal.

• 25:01Speaker 05

  It was just like, whoa, all this is like really cool and exciting and interesting. And I tried to convince my friends that it was cool and exciting and interesting.

• 25:12Speaker 04

  I asked 30-year-old Kelsey to break it down for me without thought experiments.

• 25:17Speaker 05

  So I think ELIZER sort of had two big claims, the zooming out a lot.

• 25:24Speaker 04

  Claim number one, we will build an AI that's smarter than humans, and it will change the world.

• 25:32Speaker 05

  AI is a really big deal. Building something that is smarter than humans is possible, is probably achievable, is potentially achievable soon in our lifetimes.

• 25:43Speaker 04

  And then claim number two.

• 25:44Speaker 05

  Getting this right is extraordinarily difficult.

• 25:47Speaker 04

  Things are likely to go wrong.

• 25:51Speaker 06

  What is my advice to less wrong readers who want to save the human race? Well, if you're familiar with all the issues of AI and all the issues of rationality, and you're willing to work for a not overwhelmingly high salary.

• 26:07Speaker 04

  ELIZER helped inspire a new career path, and a new field was born, trying to make sure we develop superintelligence safely. One way to make sure it went safely was to try and actually build it. And as investment in that field began to grow, the community of believers in a someday's superintelligent AI experienced a schism.

• 26:37Speaker 05

  I think a lot of the people who were persuaded by ELIZER's first claim that AI is a really big deal were not necessarily so persuaded by his second claim that you have to be very, very careful or you're going to do something catastrophically bad.

• 26:54Speaker 04

  What the beginning of a so-called catastrophe looks like after the break.

• 27:18Speaker 04

  At Edward Jones, we believe rich is more than caring about the latest and greatest. It's also taking care of what gives your life meaning. That's why your dedicated financial advisor meets you where you are with personalized financial strategies that help protect what matters, so you can preserve your progress while creating a path forward. The key to being rich is knowing what counts. Let's find your rich together. Edward Jones. Member SIPC.

• 27:48Speaker 04

  New and exclusive holiday decor just launched online at The Home Depot. Check out the viral Grand Duchess Collection with smart holiday lights you can control

• 27:57Speaker 06

  from your phone. Easily customize the twinkle across trees, wreaths, and garlands to match your holiday vibe. Create stunning light effects, then switch it up with the tap of your fingers. Even the man in red will be impressed. Shop the holiday decor you want, online only, at homedepot.com.

• 28:19Speaker 06

  Somebody asked me how I define myself, and I didn't know what to say. Because how can someone be defined when we're always evolving? I've been so many things. I'll be so many others. Because identity's not a performance, it's an exploration. Never the same, but always myself. That's the paradox. Paradox Intense and the new Sweet Chemistry by Prada.

• 28:55Speaker 06

  That presently do not show. I'm sorry, Dave. I'm afraid I can't do that.

• 29:03Speaker 04

  When the paperclip maximizer meme first started circulating in the 2000s, our best example of a paperclip AI was Clippy, the animated little guy on Word with the eyeballs, Microsoft's AI office assistant. Back in the day, I remember it couldn't even tell you if you should use their, their, or their in a sentence. People weren't so much afraid of Clippy as they were annoyed with him. There are a remarkable number of think pieces from those years slamming Clippy. The consensus was, no one asked for this. This is dumb. So when Eliezer Yudkowsky warned about the dangers of a super intelligent AI

• 29:49Speaker 04

  that could someday destroy humanity, it was hard for a lot of people to take him seriously.

• 29:55Speaker 05

  The state of thought in 2010 was something like, yeah, AI may as well be a century away.

• 30:01Speaker 04

  Future perfect writer Kelsey Piper again.

• 30:03Speaker 05

  So if you are Eliezer Yudkowsky, you have a bit of a dilemma, right? You want to make two arguments. One is, super intelligent AI is possible.

• 30:12Speaker 04

  Building a robot that's smarter, faster, and more creative than humans at most things is possible. Clippy, be damned. And he needed to make that first argument before he could make his next one.

• 30:25Speaker 05

  But the second argument you want to make is, we need to not do it until we have solved the challenge of how to do it right.

• 30:33Speaker 04

  For a long time, both arguments, super AI is possible, but let's not for now, were dead in the water. Because AI tech was just not that impressive. But by 2014, Eliezer noticed that people outside his corner of the blogosphere had started to pay attention.

• 30:54Speaker 03

  AI is probably the single biggest item in the near term that's likely to affect humanity.

• 30:59Speaker 04

  Tesla chief executive and billionaire Elon Musk, who started this year sitting prominently in President Trump's White House, had tweeted, quote, "We need to be super careful with AI, potentially more dangerous than nukes."

• 31:14Speaker 03

  It's about minimizing the risk.

• 31:17Speaker 04

  Of existential harm.

• 31:20Speaker 04

  It seems like Elon Musk is a reader of Eliyahu's blog. He famously met his ex, the musician Grimes, when they joked on then Twitter about a very obscure thought experiment from the blog. I will spare you the details. The point is Elon Musk read the paperclip maximizer thought experiment, and he seemed convinced AI was a threat. It's very important that we have the advent of AI in a good way. And that's, you know, the reason that we created OpenAI. Elon Musk co-created OpenAI. You might have heard he left and then tried to buy it back. But if you haven't heard of OpenAI,

• 32:02Speaker 04

  you've probably come across its most popular product, ChatGPT.

• 32:09Speaker 04

  I was surprised to learn that Eliyahu Yudkowsky was in fact the original inspiration for the ChatGPT company, according to its co-founder, Sam Altman.

• 32:19Speaker 05

  Sam Altman has in fact said this on Twitter, that he said that he credits Eliyahu for the fact that he started OpenAI.

• 32:25Speaker 04

  Co-founder Sam Altman specifically tweeted that Yudkowsky might win a Nobel Peace Prize for his writings on AI, that he's done more to accelerate progress on building an artificial general intelligence than anyone else.

• 32:38Speaker 05

  Now, in saying this, he was kind of being a little cruel, right? Because Eliyahu thinks that OpenAI is on track to cause enormous catastrophe.

• 32:47Speaker 04

  Co-founders Sam Altman and Elon Musk bought Eliyahu's first claim, that superintelligence is possible, and it's possible in our lifetimes. But they missed the part about how you're not supposed to build it yet.

• 33:00Noam Hassenfeld?

  For this sort of most important technological milestone in human history, I view that as right around the corner.

• 33:05Speaker 04

  That's Sam Altman talking about superintelligence.

• 33:08Noam Hassenfeld?

  Like, it's coming soon enough, and it's a big enough deal that I think we need to think right now about how we want this deployed, how everyone gets it.

• 33:16Speaker 04

  From it, how we're going to govern it, how we're going to make it safe and sort of good for humanity. Human values, which are difficult to encode. It's still not clear to me what superintelligence actually is. I won't be the first one to observe that it has some religious vibes to it. The name makes it sound like it's an all-knowing entity. The CEO of OpenAI's competitor, Anthropic, said he wanted to build, quote, "machines of loving grace."

• 33:48Speaker 04

  Sam Altman was asked on Joe Rogan's podcast about whether he's attempting to build God. I guess it comes down to maybe a definitional disagreement about what you mean by it. It becomes a god. I think whatever we create will still be subject to the laws of physics in this universe. Sam Altman has called this superintelligence, quote, "the magic intelligence in the sky," which, I don't know, sounds a lot like how some people talk about God to me. How exactly this supposed superintelligence will be smarter, faster, and more intelligent than us, on what scale, is unclear.

• 34:30Speaker 04

  But for all the hype around ChatGPT, I only recently learned what the heck it is. It's what they call a large language model.

• 34:39Speaker 05

  At its most fundamental level, a language model is an AI system that is trained to predict what comes next in a sentence.

• 34:50Speaker 04

  I'm oversimplifying here, but the very basic idea of a language model is to generate language based on probabilities. So if I have a word or a set of words, what's the most likely next word?

• 35:04Speaker 05

  So if a sentence starts with, "On Monday, I went to the grocery," the next word is probably "store."

• 35:11Speaker 04

  The way the model guesses that "store" is probably next is based on how you train the language model.

• 35:19Speaker 04

  Training involves feeding the model a large body of text, so it can detect patterns in that text and then go generate language based on those patterns. Early versions of spell check, like Clippy, were language models trained on the dictionary, useful but only for a very specific task, like to tell you if you put the E in the word weird in the wrong place, or the Ages in the word rhythm. Clippy couldn't tell you if you should use there, there, or there in a sentence because it wasn't trained on enough text to be able to guess the right word in context. The dictionary can't tell you that. But OpenAI's products were very different from Clippy.

• 36:04Speaker 04

  A revolution was happening in AI tech that made language models look less like a simple spell check and more like the human brain, detecting patterns and storing them in a network of neurons. Technologists trained those neural networks through a process they called "deep learning." They trained the AI on a lot of data, close to the entire internet. Thanks to Vox Media's partnership with OpenAI, we know they're likely training the language model on this podcast, the words I'm saying right now. No one had ever trained an AI on the entire internet before, at least in part because

• 36:48Speaker 04

  of how expensive it is. It takes a ton of energy and compute power. But OpenAI, founded by a billionaire, raised the funds to make an attempt at the biggest, baddest, largest language model the world had ever seen.

• 37:04Speaker 05

  They started going, "Okay, what if the secret to trying to build super-intelligent God AI or whatever is just to spend more money and have more neurons and to have more connections, feed it more data? What if that's all there is? What if you can build something that is more intelligent than any human who's ever lived just by doing that?"

• 37:29Speaker 04

  One of their earlier attempts before ChatGPT was GPT-2 in 2019. You could similarly give it a specific task, like, "Design a luxury men's perfume ad for the London Underground. Make it witty and concise."

• 37:47Speaker 07

  "The London Underground is a great place to advertise. It's a great place to get your message across. It's a great place to get your product noticed."

• 37:56Speaker 04

  Look out, mad men.

• 37:59Speaker 04

  GPT-2 was not exactly coming for copywriter jobs. But for people like Kelsey, who were watching the technology closely...

• 38:08Speaker 05

  I was like, wow, this is like miles beyond what AI chatbots were capable of last week. This is huge.

• 38:15Speaker 04

  GPT-2, the language prediction machine, was showing some real promise. She wasn't alone in that feeling. Investors like Microsoft poured millions more dollars into the next few models, which were bigger and bigger.

• 38:31Speaker 07

  Be the scent that turns heads.

• 38:33Speaker 04

  And a couple of years later, OpenAI released ChatGPT.

• 38:39Speaker 07

  Visual: A captivating image of the perfume bottle, surrounded by vibrant city lights, symbolizing the urban lifestyle. Embrace the city. Embrace your scent.

• 38:50Speaker 05

  Most people weren't paying any attention to AI. And so for them, it was like a huge change in what they understood AI to do.

• 38:58Speaker 04

  ChatGPT was the first time that normies like me even thought about AI in any real way.

• 39:05Noam Hassenfeld?

  All I wanted to do was fix my email. I did not expect to have a minor existential crisis about how much the world is about to change. And this is only proving that one day, AI will take over human intelligence. I spent about two hours just typing back and forth with this AI chatbot, and it got pretty weird.

• 39:23Speaker 00

  The AI confessed to loving Kevin and tried to convince him to leave his wife.

• 39:31Speaker 05

  People at OpenAI or competitors were saying like, yeah, the plan is to build superintelligence. We think we're going to do it by 2027. People were like, okay, startup hype. For some reason, everybody who runs a startup feels the need to say that they're going to build God and the human race. And then after ChatGPT was genuinely impressive, people started taking them a bit more seriously. And a lot of those people were nervous.

• 39:58Speaker 04

  People weren't so nervous about ChatGPT, but what ChatGPT represented, the way they got the language model to sound so much smarter so quickly, wasn't through intricate code. They just made the model bigger, which suggested to some people that the path to building God, or whatever, was through brute force, spending more and more money to build a bigger and bigger machine. So big, we didn't really understand why it did what it did. We can't point to a line of code to say, this is why the robot got so much better at writing a perfume ad. And if we someday do build something that's smarter than us,

• 40:43Speaker 04

  whatever that means, we won't be able to understand why it's smarter than us. The trouble with this, it seems to me, is that AI will come for copywriter jobs. It could come for all our jobs. But rationalists I spoke to say, that's nothing compared to the bigger trouble ahead, a potential apocalypse.

• 41:06Speaker 06

  But I do also kind of think that it is a very important priority for me to have the best possible time in the next five to 10 years. And just to do the very best I can to squeeze the joy out of life while it is here. Do you have an example of that? One I can talk about on a podcast.

• 41:31Speaker 06

  I mean, yes, I joke, but I'm pretty involved in the King community, and that's very important to me. And it's important to me.

• 41:39Speaker 04

  Many rationalists I spoke to live in polyamorous communities because they believe monogamy is irrational. Some aren't sure if it's rational to have children, given the high probability of things going very, very wrong because of AI.

• 41:56Speaker 06

  What's my P doom, as our community says?

• 41:59Speaker 04

  P doom. It's a shorthand I heard at the conference, meaning probability of doom.

• 42:05Speaker 06

  It's a phrase that gets thrown around at this conference. People will literally go up to go, so what's your P doom? And it's a shorthand for what is the probability that humanity doesn't make it in the long term?

• 42:15Speaker 04

  And this is a mathy bunch, so they get specific.

• 42:19Speaker 06

  I guess the answer I usually give is something like over 50%. I mean, I think it's like somewhere around 80, 90.

• 42:25Speaker 04

  Eliezer Yudkowsky's P doom is very high. I've read it's over 95% these days. But then I've seen him tweet that P doom is beside the point.

• 42:39Speaker 04

  I spotted Eliezer Yudkowsky pretty much the moment I stepped into the conference. He was hard to miss. He was the one wearing a gold sparkly top hat all weekend. I was the one who was clearly lost, carrying a big furry microphone for three days, trying to get people to talk to me. It wasn't until day three of the conference that I mustered the determination to approach Eliezer for an interview. Determination was necessary because he was always surrounded by a cluster of people, a cluster of mostly dudes, listening to him speak. I asked him if it would be OK if I pulled out my microphone.

• 43:21Speaker 04

  Everyone has been looking at this like it's a weapon.

• 43:24Noam Hassenfeld?

  It is.

• 43:24Speaker 04

  It is, I know. Over the last few years, Eliyser and the rationalists have gotten some bad press. Some rationalists express their frustration at journalists who focus on the polyamory that happened in the community. Some critics of rationalism, to put it crudely, call them a sex cult. And then there's the unsavory things people associated with the community have said. One philosopher who helped popularize the paperclip maximizer, Nick Bostrom, once wrote that he thought black people were less intelligent than white people. He has since apologized. But critics highlight this comment and the mostly white demographics of the rationalist

• 44:07Speaker 04

  community to question their beliefs. I never really know why anyone agrees to talk to me, but can you introduce yourself?

• 44:15Noam Hassenfeld?

  I'm Eliyser Yudkowski. This event is probably more my fault than the fault of anyone else around.

• 44:23Speaker 04

  And can you describe your outfit right now?

• 44:25Noam Hassenfeld?

  Well, I'm currently wearing a sparkly, multi-colored shirt and a sparkly golden hat.

• 44:31Speaker 04

  You can probably hear it in my voice. I was nervous to talk to him. He's known for being a bit argumentative, very annoyed with journalists and with the world more generally, for not being smart enough to understand him, for not heeding his warnings.

• 44:48Speaker 04

  I don't know. How would you summarize what you want the world to know in terms of AI?

• 44:56Noam Hassenfeld?

  The world is completely botching the job of entering into the issue of machine superintelligence. There is not a simple fix to it. If anyone, anywhere builds it at the, you know, and anything, under anything remotely like the current regime, everyone will die. This is bad. We should not do it.

• 45:11Speaker 04

  Do you feel like gratified at all to see that, like, your ideas entered the mainstream conversation? Do you feel like they have?

• 45:18Noam Hassenfeld?

  The circumstances under which they have entered the mainstream conversation are catastrophic. And I didn't, if I was the sort of person who was like, you know, like deeply attached to the validation of seeing other people agree with me, I would have picked a much less disagreeable topic. I was here to try to like not have things go. Though, I was here to not have things go terribly. They're currently going terribly. I did not get the thing I wanted.

• 45:48Speaker 04

  Eliezer has been on a bit of a press tour, giving interviews and TED Talks, saying OpenAI is on track to cause catastrophe.

• 45:58Speaker 05

  So it's a funny thing because I have one position of deep sympathy with Eliezer. If you become convinced that this is a huge problem, it makes perfect sense to go on a writing tour trying to explain this to people. And also, I think it's kind of predictable that a lot of people heard this and went, "Oh, AI is going to be really powerful? I don't think you're right about the thing where that's a problem. I want the powerful, important thing."

• 46:23Speaker 05

  And some people seized on it and were like, "Because this is powerful and important, we should like invest now." And I feel kind of sad about this.

• 46:36Speaker 04

  I can understand why Eliezer was hesitant to talk to me. His message to the world has been totally lost in translation. In his mind, it's backfired. Even at his own conference, there were attendees who worked for places like OpenAI, the company's building the supposed death machine he was afraid of.

• 46:57Speaker 05

  He thought that our best chance of building a superintelligent AI that did what we wanted and didn't like, you know, seize power from humans, was to build one that was very well understood, one that sort of from the ground up, we knew why it made all the decisions that it made. Larger language models are just the exact opposite of that.

• 47:24Speaker 04

  I will say, even after talking to Eliezer and Kelsey and a bunch of rationalists, it's still hard to imagine how something like ChatGPT or Google's AI, which once told someone to add glue to stick cheese on pizza, is going to become the invention of all inventions and possibly catastrophic. But I can understand how building something big that you don't understand is a scary idea.

• 47:58Speaker 04

  The best AI metaphor I came across for my brain was not about paperclips. It was by a non-rationalist writer. A guy named Brian Christian describes that training AI is something that could go wrong in the way parenting a kid can go wrong. Like, there's a little kid playing with a broom. She cleans up a dirty floor, and her dad, looking at what she's done on her own, says, "Great job. You swept that really well." This little girl, without skipping a beat, might dump the dirt back on the floor and sweep it up again, waiting for that same praise. That's not what her dad meant for her to do. It's hard to get the goals right in teaching a kid to be good.

• 48:43Speaker 04

  It's even harder to teach good goals to a non-human robot. It strikes me as like, almost like a parenting problem. I ran this parenting metaphor by Kelsey with her seven-month-old on her lap.

• 48:57Speaker 05

  I think there's some serious similarities. And I do, with my kids, struggle with trying to steer something that you don't have perfect control over and that you wouldn't even want to have perfect control over, but where it could go extremely badly to like, just let the dice fall where they may.

• 49:24Speaker 04

  If we just let the dice fall where they may, rationalists say we could have an apocalypse on our hands. And they say it won't be one we saw coming. It won't be a Hollywood-style Terminator situation. It probably won't have paperclips either. They don't pretend to know exactly how apocalypse could befall us, just that it'll probably be something we haven't even imagined yet. But I have trouble getting caught in what could happen when it feels like, haven't bad things already started to happen thanks to AI? AI is not hypothetical anymore. It's arrived in our lives.

• 50:07Speaker 04

  I'm not kept up at night about a hypothetical apocalypse. I find myself asking now questions. Questions like, what is OpenAI doing with my voice right now?

• 50:24Speaker 04

  Is there anything to do about problems with AI short of the annihilation of humanity?

• 50:30Speaker 01

  It sounds very exciting. You know, like, if I were a big science fiction geek, I would be so into that.

• 50:38Speaker 04

  Not all technologists seize on Eliezer Yudkowsky's claims.

• 50:42Speaker 01

  What is he even talking about? This is like word salad. Like, this doesn't even make sense.

• 50:47Speaker 04

  One group of technologists didn't actually seize on any of his claims.

• 50:51Speaker 01

  There's one thing to have that conversation as a thought experiment. It's another thing when that kind of thought experimentation sucks up all of the money and the resources.

• 51:05Speaker 04

  The more I dig into the AI world, the more I see disagreement between technologists.

• 51:11Speaker 01

  I do worry about the ways in which AI can kill us, but I think about the ways in which AI can kill us slowly.

• 51:20Speaker 04

  They've been called the AI ethicists, and they say we've been paying attention to all of the wrong things.

• 51:32Speaker 04

  That's next time.

• 51:44Speaker 00

  Good Robot was hosted by Julia Longoria and produced by me, Gabrielle Burbay. Sound design, mixing, and original score by David Herman. Fact checking by Caitlin Penzimook. Editing by Diane Hodson and Catherine Wells. Special thanks to Future Perfect founder Dylan Matthews, to Vox's executive editor Albert Ventura, and to Tom Chivers, whose book, The Rationalist's Guide to the Galaxy, was an early inspiration for this episode. If you want to dig deeper into what you've heard, head to vox.com/goodrobot to read more Future Perfect stories trying to make sense of artificial intelligence. Thanks for listening.

• 52:40Speaker 04

  Support for the show comes from Engine. Running a small business means every dollar has to work hard. But if your team is still booking travel the old way, it's costing you more than you think. Engine is the fastest growing travel and spend platform in the country, built specifically for businesses like yours. Book a trip in as little as two and a half minutes, earn up to 10% back on hotels. And in 2025, Engine customers save more than $300 million on travel with zero booking fees, no contracts, and no BS. More than 1,000 businesses join Engine every month. Join them and get $500 when your business signs up and starts traveling at engine.com/box.

• 53:25Speaker 06

  At Edward Jones, we believe rich is more than caring about the latest and greatest. It's also taking care of what gives your life meaning. That's why your dedicated financial advisor meets you where you are with personalized financial strategies that help protect what matters, so you can preserve your progress while creating a path forward. The key to being rich is knowing what counts. Let's find your rich together. Edward Jones. Member SIPC.