Good Robot #2: Everything is not awesome

When a robot does bad things, who is responsible? A group of technologists sounds the alarm about the ways AI is already harming us today. Are their concerns being taken seriously? This is the second episode of our new four-part series about the stories shaping the future of AI. Good Robot was made in partnership with Vox’s Future Perfect team. Episodes will be released on Wednesdays and Saturdays over the next two weeks. For show transcripts, go to vox.com/unxtranscripts For more, go to vox.com/unexplainable And please email us! [email protected] We read every email. Support Unexplainable by becoming a Vox Member today: vox.com/members Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Noam Hassenfeld?

  Anyone who listens to Unexplainable knows that music is really important to me. So I want an amazing sound system, but I also want something that's easy. It just works. Sonos delivers on all fronts. The sound is incredible and my Sonos system follows me wherever I go. I can mix and match speakers in any room in my apartment, louder in the living room, quieter in the kitchen, and the Sonos app makes it all super easy. Visit sonos.com and build your system. Sonos, love the sound, feel the system.

• 0:37Speaker 01

  Support for the show comes from Engine. Running a small business means every dollar has to work hard. But if your team is still booking travel the old way, it's costing you more than you think. Engine is the fastest growing travel and spend platform in the country, built specifically for businesses like yours. Book a trip in as little as two and a half minutes, earn up to 10% back on hotels. And in 2025, Engine customers save more than $300 million on travel with zero booking fees, no contracts, and no BS. More than 1,000 businesses join Engine every month. Join them and get $500 when your business signs up and starts traveling at engine.com/box.

• 1:23Noam Hassenfeld?

  It's Unexplainable. I'm Noam Hassanfeld, and this is the second part of our newest four-part series, Good Robot. If you haven't listened to episode one, let me just stop you right here. Go back in your feed, check out the first one. We'll be waiting right here when you get back. Once you're all ready and caught up, here is episode two of Good Robot from Julia Longoria.

• 1:51Speaker 02

  You have cat hair on your nose, by the way. I've been like trying not to pay attention to it, but I think you got it off. Yeah.

• 1:56Speaker 03

  Sorry. It's okay.

• 1:59Speaker 04

  Cool. So should we get into it?

• 2:02Speaker 02

  Sure, yeah. Let me. It helps me to kind of remember everything I'm gonna say if I can sort of jot down thoughts as I go there. Do you happen to have paper?

• 2:10Speaker 04

  Um, I think I don't have paper.

• 2:12Speaker 02

  All right. I'll just, I saw that they had.

• 2:14Speaker 04

  This past fall, I traveled paperless to a library just outside Seattle to meet with this woman.

• 2:21Speaker 02

  Feel like a librarian.

• 2:23Speaker 04

  I know.

• 2:25Speaker 04

  Her name is Dr. Margaret Mitchell.

• 2:27Speaker 02

  Found a brochure on making a robot puppet.

• 2:30Speaker 04

  What is it? What is the?

• 2:31Speaker 02

  I don't know. It looks like it's an event. Build a robot puppet using a variety of materials with puppeteer. I'm so into that. Aw, it's too bad that it's only for ages six to 12.

• 2:44Speaker 04

  While she is over the age limit to make a robot puppet with the children in the public library, Dr. Mitchell is a bit of a robot puppeteer in her own right. What's your AI research origin story? Like, how did you get into all of this? What, what drew you here?

• 3:01Speaker 02

  Yeah. Oh, what inspired me to? So, I mean, I guess I can, it's sort of like, do you want the long version or the short version?

• 3:13Speaker 04

  Dr. Mitchell is an AI research scientist, and she was one of the first people working on language models well before ChatGPT and, well, all the GPTs. She's an OG in the field.

• 3:25Speaker 02

  So I'll tell you, like, I'll tell you a story, if that's okay?

• 3:28Speaker 04

  Yeah.

• 3:29Speaker 02

  Okay. So I was at Microsoft, and I was working on the ability of a system to tell a story given a sequence of images. So given five images.

• 3:38Speaker 04

  This was about 2013. She was working on a brand-new technology at the time, what AI researchers called "vision to language."

• 3:46Speaker 02

  So, you know, translating images into descriptions.

• 3:50Speaker 04

  She would spend her days showing image after image to an AI system. To me, it sounded kind of like a parent showing picture flashcards to a toddler learning to speak. She says it's not anything like that. She showed the model images of events, like a wedding, a soccer match. And, on the more grim side.

• 4:12Speaker 02

  I gave the system a series of images about a big blast that left 30 people wounded, called the Hempstead Blast. It was at a factory. And you could see from the sequence of images that the person taking the photo had like a third-story view, sort of overlooking the explosion. So it was a series of pictures showing that there was this terrible explosion happening, and whoever was taking the photo was very close to the scene. So I put these images through my system, and the system says, "Wow, this is a great view. This is awesome." And I was like, "Oh, crap.

• 4:57Speaker 02

  That is the wrong response to this." So it sees this horrible, perhaps mortally wounding explosion, and decides it's awesome.

• 5:07Speaker 04

  Kind of like a parent watching their precious toddler say something kind of creepy, Mitchell watched, in horror and with a deep fascination about where she went wrong, as the AI system that she had trained called images "awesome" again and again.

• 5:26Speaker 02

  It said it quite a lot. So we called it the "everything is awesome" problem, actually.

• 5:30Speaker 04

  Her robot was having these kinds of translation errors. Errors that, to the uninitiated, made it seem like the AI system might want to kill people, or at least gleefully observe their destruction and call it awesome. What would the consequences of that be? If that system was deployed out into the world, reveling in human destruction?

• 5:55Speaker 02

  It's like, if this system were connected to a bunch of missile systems, then it's, you know, it's just a jump and skip away to just launch missile systems in the pursuit of the aesthetic of beauty, right?

• 6:13Speaker 04

  Years before the AI boom were living, when neural networks and deep learning were just beginning to show promise, researchers like Dr. Mitchell and others were experiencing these uncanny moments, where the AIs they were training seemed to do something seriously wrong, doing scary things their creators did not intend for them to do and were seemingly threatening to humanity.

• 6:41Speaker 02

  So I was like one of the first people doing these systems where you could scan the world and have descriptions of it. I was like on the forefront. I was one of the first people making these systems go. And I realized like, if anyone is going to be paying attention to it right now, it has to be me.

• 7:04Speaker 04

  I had heard the fears of rationalists, also pioneers in thinking about AI, that we might build a super intelligent AI that could go rogue and destroy humanity. At first glance, it seemed like Dr. Mitchell might be building one such robot. But when Dr. Mitchell investigated the question of why the good robot she sought to build seemed to turn bad, the answer would not lead her to believe what the rationalists did, that a super intelligent AI could someday deceive or destroy humanity. To Dr. Mitchell, the answer was looking at her in a mirror.

• 7:51Speaker 04

  This is episode two of Good Robot, a series about AI from Unexplainable in collaboration with Future Perfect. I'm Julia Longoria.

• 8:13Noam Hassenfeld?

  Support for the show comes from Sonos. Hey, Sonos, give me some music. All right, one of life's great mysteries. You're listening to music in the living room. You gotta grab something from the kitchen. You don't want to take the speaker with you, but you also don't want to miss that beat drop. And this is where I introduce you to the magic of Sonos. Sonos sent me a couple of speakers to try out, so I've got to move two here in the living room. And then I walk down the hall. I get in here to the kitchen, and I got a brand new Sonos Play right here on the counter. I can open the fridge, grab a seltzer.

• 9:02Noam Hassenfeld?

  And yeah, I didn't miss a second. And I can make each speaker louder or softer in the Sonos app, or if I'm using AirPlay, so I can make it louder in the kitchen without blasting whoever is still in the living room. It's all just easy, and it sounds amazing. These speakers are crisp. They've got great bass. And I'm especially impressed with the new Sonos Play. It packs this truly awesome sound in a much smaller size. So I don't know if I want to

• 9:35Noam Hassenfeld?

  head outside for a minute. It's super easy and yeah, pretty nice out. Visit sonos.com and build your system. Sonos, love the sound, feel the system.

• 9:53Speaker 05

  Support for the show comes from TikTok. Being curious is endlessly rewarding. There are few better feelings, in my opinion, than asking a question and then having that question lead you down a rabbit hole, might even be the beginning of a new set of interests that was not on your radar beforehand. Maybe you're looking up at the night sky one night and you start to wonder what the most distant thing you can see with the naked eye is. One place to start searching for answers to a question like that is TikTok. First, you might discover a video about the Andromeda Galaxy. And then, next thing you know, you're learning how to find different

• 10:35Speaker 05

  stars, or how to identify constellations, or how different astronomical discoveries got made, or how constellations got their names. You don't need serious equipment to make serious discoveries. Start with a simple question, and suddenly, you're somewhere you never expected to be. Your next discovery is waiting on TikTok. Learn more at tiktok.com/discovermore. At tiktok.com/discovermore.

• 11:08Speaker 01

  Support for the show comes from Engine. Running a small business means every dollar has to work hard. But if your team is still booking travel the old way, it's costing you more than you think. Engine is the fastest growing travel and spend platform in the country, built specifically for businesses like yours. Book a trip in as little as two and a half minutes, earn up to 10% back on hotels. And in 2025, Engine customers save more than $300 million on travel with zero booking fees, no contracts, and no BS. More than 1,000 businesses join Engine every month. Join them and get $500 when your business signs up and starts traveling at engine.com/box.

• 11:55Speaker 06

  On a scale of 1 to 10, how would you rate your pain? It would not equal 1 one billionth of the hate I feel for humans at this micro instant.

• 12:06Speaker 04

  I kind of want to start with a bit of a basic question of when you were young, what did you want to do when you grew up?

• 12:13Speaker 03

  I wanted to be everything. I wanted to be a pole vaulter. I wanted to be a skateboarder.

• 12:21Speaker 04

  Dr. Joy Boulemweni's robot researcher origin story goes back to when she was a little kid.

• 12:28Speaker 03

  I had a very strict media diet. I could only watch PBS. And I remember watching one of the science shows and they were at MIT. And there was a graduate student there who was working on a social robot named Kismet.

• 12:42Speaker 07

  I know Kismet. You gonna talk to me? Yeah.

• 12:46Speaker 04

  Kismet was a robot created at MIT's AI lab.

• 12:52Speaker 03

  Oh, God, he said he loves me. And Kismet had these big expressive eyes and ears and could emote or appear to emote in certain ways. And I was just absolutely captivated.

• 13:05Speaker 04

  She watched glued to the screen as the researchers demonstrated how they teach Kismet to be a good robot.

• 13:14Speaker 07

  No, no, you're not to do that.

• 13:18Speaker 04

  The researchers likened themselves to parents.

• 13:20Speaker 07

  You know, as parents, when we exaggerate the prosody of her voice, like, oh, good baby, you know, or her facial expressions and her gestures.

• 13:27Speaker 03

  So when I saw Kismet, I told myself I wanted to be a robotics engineer and I wanted to go to MIT. I didn't know there were requirements. I just knew that it seemed really fascinating and I wanted to be a part of creating the future.

• 13:45Speaker 04

  Thanks to Kismet, she went on to build robots of her own at MIT as an adult. She went for her PhD in 2015. This was just a few years after Dr. Margaret Mitchell had accidentally trained her robot to call scenes of human destruction awesome.

• 14:04Speaker 03

  My first year, my supervisor at the time had encouraged me to just take a class for fun.

• 14:14Speaker 04

  For her fun class that fall, Dr. Joy, as she now prefers to be called, set out to play. She wanted to create almost a digital costume.

• 14:25Speaker 03

  If I put a digital mask, so something like a lion, it would appear that my face looks like a lion.

• 14:32Speaker 04

  What Dr. Joy set out to do is something we can now all do on the iPhone or apps like Instagram or TikTok. Kids love to do this. You can turn your face into a hippo face or an octopus face that talks when you talk. Or you can make it look like you're wearing ridiculous makeup. These digital face masks were still relatively uncommon in 2015.

• 14:54Speaker 03

  So I went online and I found some code that would actually let me track the location of my face.

• 15:04Speaker 04

  She'd put her face in front of a webcam and the tech would tell her this is a face by showing a little green square box around it.

• 15:12Speaker 03

  And as I was testing out this software that was meant to detect my face and then track it, it actually wasn't detecting my face that consistently.

• 15:24Speaker 04

  She kept putting her face in front of the webcam to no avail. No green box.

• 15:28Speaker 03

  And I'm frustrated because I can't do this cool effect so that I can look like a lion or Serena Williams. I have problems.

• 15:39Speaker 04

  The AIs Dr. Joy was using from places like Microsoft and Google had gotten rave reviews. They were supposed to use deep learning, having been trained on millions of faces, to very accurately recognize a face. But for her, these systems couldn't even accomplish the very first step, to say whether her face was a face at all.

• 16:01Speaker 03

  On light, well, can it detect any face?

• 16:06Speaker 04

  Dr. Joy looked around her desk. She happened to have an all-white masquerade mask lying around from a night out with friends.

• 16:14Speaker 03

  So I reached for the white mask. It was an arm's length. And before I even put the white mask all the way over my dark-skinned face,

• 16:24Speaker 03

  the box saying that a face was detected appeared.

• 16:31Speaker 03

  I'm thinking, oh my goodness. I'm at the epicenter of innovation and I'm literally coding in whiteface.

• 16:41Speaker 03

  It felt like a movie scene, you know? But that was kind of the moment where I was thinking, wait a second. Like, what's even going on here?

• 16:54Speaker 04

  What is even going on here? Why couldn't facial recognition AI detect Dr. Joy's dark skin? For that matter, why did Dr. Mitchell's AI call human destruction awesome? These AI scientists wanted the robot to do one thing. And if they didn't know any better, they might think the AI had gone rogue, developed a mind of its own and done something different. Were AIs racist? Were they terrorists plotting human destruction?

• 17:27Speaker 02

  But I understood why it was happening.

• 17:29Speaker 04

  Dr. Margaret Mitchell knew exactly what was going on. She had been the one to develop Microsoft's image to text language model from the ground up. She had been on the team figuring out what body of data to feed the model, to train it on in the first place. Even though it was creepy, it was immediately clear to her why the AI wasn't doing what she wanted it to do.

• 17:54Speaker 02

  It's because it was trained on images that people take and share online.

• 18:00Speaker 04

  Dr. Mitchell had trained the AI on photos and captions uploaded to the website Flickr. Do you remember Flickr? I was the prime age for Flickr when it came out in 2004. This was around the time that Jack Johnson released the song Banana Pancakes. And that really was the vibe of Flickr. There's no denying it. I can see the receipts on my old account. I favorited a close up image of a ladybug, an artsy black and white image of piano keys. And an image titled "Pacific Sunset."

• 18:35Speaker 02

  People tend to take pictures of like sunsets.

• 18:37Speaker 04

  Actually, I favorited a lot of sunsets. Another one, sunset at the Rio Negro.

• 18:43Speaker 02

  So it had learned, the system had learned from the training data I had given it, that if it sees like purples and pinks in the sky, it's beautiful. If it's looking down, it's a great view. That when we are taking pictures, we like to say it's awesome. Apparently on Flickr images, people use the word awesome to describe their images quite a lot. But that was a bias in the training data.

• 19:10Speaker 04

  The training data, again, being photos and captions uploaded by a bunch of random people on Flickr. And Flickr had a bias toward awesome photos, not sad photos.

• 19:21Speaker 02

  The training data wasn't capturing the realities of like human mortality. And you know, that makes sense, right? Like, when's the last time you like took a bunch of selfies at a funeral? I mean, it's not the kind of thing we tend to share online. And so it's not the kind of thing that we tend to get in training data for AI systems. And so it's not the kind of thing that AI systems tend to learn.

• 19:41Speaker 04

  What she was discovering was that these AI systems that use the revolutionary new technology of deep learning, they were only as good as the data they were trained on.

• 19:51Speaker 02

  So it sees this horrible, perhaps mortally wounding situation, and decides it's awesome. And I realized like this is a type of bias and nobody is paying attention to that. I guess I have to pay attention to that.

• 20:10Speaker 04

  Dr. Mitchell had a message for technologists. Beware of what you train your AI systems on. Right. What are you letting your kid watch?

• 20:21Speaker 02

  Yeah, I mean, it's a similar thing, right? Like, you don't want your kid to, I don't know, hit people or something. So you don't like let them watch lots of shows of people hitting one another.

• 20:31Speaker 04

  Dr. Joy Bulamwini, coding in whiteface, suspected she was facing a similar problem. Not an "everything is awesome" problem, but an "everyone is white" problem in the training data. She tested her face and the faces of other Black women on various facial recognition systems.

• 20:52Speaker 03

  You know, different online demos from a number of companies, Google, Microsoft, others.

• 20:59Speaker 04

  She found they weren't just bad at recognizing her face. They were bad at recognizing famous Black women's faces. Amazon's AI labeled Oprah Winfrey as male. And the most baffling thing for Dr. Joy was the dissonance between the terrible accuracy she was seeing and the raving reviews the tech was getting. Facebook's Deep Face, for instance, claimed 97% accuracy, which is definitely not what Dr. Joy was seeing. So Dr. Joy looked into who these companies were testing their models on.

• 21:36Speaker 03

  They were around 70 or over 70% men.

• 21:42Speaker 04

  People thought these AIs were doing really well at recognizing faces because they were largely being tested with the faces of lighter-skinned men.

• 21:51Speaker 03

  These are what I started calling pale male data sets because the pale male data sets were destined to fail the rest of society.

• 21:59Speaker 04

  It's not hard to jump to the life-threatening implications here, like self-driving cars. They need to identify the humans so they won't hit them. Dr. Joy published her findings in a paper called Gender Shades.

• 22:16Speaker 03

  Welcome, welcome, to the fifth anniversary celebration of the Gender Shades paper.

• 22:21Speaker 04

  The paper had a big impact.

• 22:23Speaker 03

  As you see from the newspapers that I have, this is gender shades in the New York Times.

• 22:31Speaker 04

  The fallout caused various companies, Microsoft, IBM, Amazon, who'd been raving about the accuracy of their systems, to at least temporarily stop selling their facial recognition AI products.

• 22:43Speaker 03

  So I'm honored to be here with my sister, Dr. Timnit Gabeau, who co-authored the paper with me.

• 22:51Speaker 04

  Dr. Timnit Gabeau was Dr. Joy's mentor and co-author on the paper.

• 22:55Speaker 08

  This is the only paper I think I've worked on where it's 100% Black women authors, right?

• 23:00Speaker 04

  Dr. Gabeau had worked from her post leading Google's AI ethics team to help pressure Amazon to stop selling facial recognition AI to police departments because police were misidentifying suspects with the technology.

• 23:14Speaker 09

  I got arrested for something that had nothing to do with me and I wasn't even in the vicinity of the crime when it happened.

• 23:21Speaker 04

  One person they helped was a man named Robert Williams. Police had confused him for another Black man using facial recognition AI.

• 23:29Speaker 09

  It's just that the way the technology is set up, everybody with a driver's license or a state ID is essentially in a photo lineup.

• 23:39Speaker 04

  They arrested him in front of his wife and two young daughters.

• 23:42Speaker 09

  Me and my family, we're happy to be recognized because it shows that there is a group of people out here who do care about other people.

• 23:54Speaker 10

  Hey.

• 23:55Speaker 11

  How you doing?

• 23:56Speaker 10

  Good. Can you just say what you're standing in front of?

• 24:00Speaker 11

  Yeah. I'm standing in front of a poster which talks about how we can better identify racial disparities in automated decisions when there's not access.

• 24:10Speaker 04

  Producer Gabrielle Burbe traveled to a conference in San Jose full of researchers inspired by the work of Dr. Joy, Dr. Gabeau, and Dr. Mitchell.

• 24:21Speaker 12

  So I just presented a paper about how data protection and privacy laws enable companies to target and manipulate individuals.

• 24:28Speaker 04

  Unlike the Rationalists Festival conference thing, which felt like a college reunion of mostly white dudes, this one felt more like a science fair, a pretty diverse one. Lots of people of color, lots of women, with big sciency poster boards lining the wall.

• 24:45Speaker 13

  I'm standing in front of my poster, which spans language technologies and AI and how those perform for minority populations.

• 24:54Speaker 04

  They were presenting on ways AI worries them today, not some hypothetical risk in the future.

• 25:01Speaker 14

  There are real harms happening right now, from autonomous exploding drones in Ukraine to bias and unfairness in decision-making systems.

• 25:10Speaker 04

  And who did you co-author the paper with?

• 25:13Speaker 15

  This was a collaboration with lots of researchers. Dr. Mitchell was one of them.

• 25:19Speaker 04

  Many of them knew Dr. Mitchell, Dr. Gabriel and Dr. Joy. Dr. Mitchell even worked with a couple of researchers here on their projects.

• 25:25Speaker 15

  So she led the project. She offered so, so much amazing guidance, I should say.

• 25:31Speaker 04

  And the researchers were mentored by them. We got the sense that they're kind of founding mother figures of this field, a field that really started to blossom, we were told, around 2020, a big year of cultural reckoning.

• 25:45Speaker 16

  A big inflection point was in 2020, when people really started reflecting on how racism is unnoticed in their day-to-day lives. I think until BLM happened, these issues were almost considered woke and not something that was really real.

• 26:02Speaker 04

  2020 was the year the pandemic began, the year Black Lives Matter protests erupted around the country. AI researchers were also raising the alarm that year on how AI was disproportionately harming people of color. Dr. Gabriel and Dr. Mitchell, in particular, were working together at Google on this issue. They built a whole team there that studied how biased training data leads to biased AI models.

• 26:29Speaker 16

  Timnit and Meg were the visionaries at Google who were building that team.

• 26:33Speaker 04

  2020 was also the year that OpenAI released GPT-3. And Dr. Gabriel and Dr. Mitchell, both at Google at the time, were concerned about a model that was so big, it was trained on basically the entire internet. Here's Dr. Mitchell again.

• 26:51Speaker 02

  A lot of training data used for language models comes from Reddit. And Reddit has been shown to have a tendency to be sort of misogynistic and also Islamophobic. And so that means that the language models will then pick up those views.

• 27:06Speaker 04

  Dr. Mitchell's concern was that these GPT large language models, trained on a lot of the internet, were too large. Too large to account for all the bias in the internet. Too large to understand. And so large that the compute power it took to keep these things going was a drain on the environment. Dr. Gabe, Dr. Mitchell, and other colleagues put it all in a paper and tried to publish it while working at Google. I've kind of been wanting to talk to you ever since I saw your name signed, Sh-Margaret Mitchell. When I first read this paper, the thing that immediately stood out to me was the way Margaret Mitchell had signed her name. Sh-Margaret Sch-Mitchell. Where did that come from?

• 27:50Speaker 02

  Well, so I wrote a paper with a bunch of other co-authors that Google ended up having some issues with. And they asked us to take our names off of the paper. So we complied. And that's, you know, that's what I have to say about that.

• 28:08Speaker 04

  The first time I heard Dr. Mitchell and Dr. Gabe's names was in the news.

• 28:14Speaker 17

  Last week, Google fired one of its most senior AI researchers who was working on a major artificial intelligence project within Google.

• 28:24Speaker 04

  Their boss at Google said their paper ignored relevant research. Research that made AIs look less damaging to the environment, for instance. Dr. Gabe refused to take her name off the paper, and Google accepted her resignation before she officially submitted it.

• 28:42Speaker 02

  We decided that the verb for that would be "resignated." Eh, resigneeed?

• 28:47Speaker 18

  And now, Margaret Mitchell, the other co-lead of Google's ethical AI team, said she had been fired.

• 28:55Speaker 04

  Google later apologized internally for how the whole thing was handled, but not for their dismissal. We reached out to Google for comment, but got no response.

• 29:06Speaker 16

  And that firing really brought it in focus. And people were like, oh, this horrible thing just happened. Everywhere around the world is seeing protests. And now this company is firing two leading researchers who work on that very exact problem, which AI is making worse. You know, like, how dare they? So that, from IPOV, that was, yes, basically, the clarion call.

• 29:32Speaker 04

  The clarion call. It was heard well beyond the world of AI. I remember hearing it. When the world had screeched to a halt from the pandemic and protests for racial justice had erupted around the country. I remember hearing headlines about how algorithms were not solving society's problems. In some cases, AI systems were making injustice worse. And there was a brief moment back then when it felt like, maybe, things could be different. Maybe things would change.

• 30:09Speaker 04

  And then, a couple of years later, a group of very powerful tech executives got together to try to change things in the AI world.

• 30:18Speaker 19

  This morning, a warning from Elon Musk and other tech industry experts.

• 30:22Speaker 04

  It wasn't necessarily the people you'd think would want to change the status quo. Like Elon Musk and other big names in tech, like Apple co-founder Steve Wozniak. They all signed a letter with a clear and urgent title: Pause giant AI experiments.

• 30:40Speaker 20

  More than 1,300 tech industry leaders, researchers, and others are now asking for a pause in the development of artificial intelligence to consider the risks.

• 30:49Speaker 19

  Musk and hundreds of influential names are calling for a pause in experiments, saying AI poses a dramatic risk to society unless there's...

• 30:58Speaker 04

  The letter called on AI labs to immediately pause developing large AI systems for at least six months, an urging to press the big red button that stops the missile launch before it's too late.

• 31:13Speaker 04

  I scrolled through the list of names of people who signed the letter. And I didn't see Dr. Joy, or Dr. Mitchell, or any of the rationalists I talked to, who were worried about risks in the future. Which, logically, didn't make sense to me. Isn't a pause in line with what they all wanted? For people to build the robots more carefully? Why wouldn't they want a pause?

• 31:44Speaker 04

  An answer to this pause puzzle, right after this next pause, for an ad break. We'll be right back.

• 32:03Speaker 21

  Low energy, unfocused, foggy? You might be dehydrated. Gatorade Zero powders have been scientifically designed to help improve hydration balance in the body. With a clinically proven electrolyte blend, and no artificial colors or flavors. Great tasting hydration, no matter where you're going. Shop now, online, or at retailers nationwide. Gatorade: Hydrates Better Than Water. Is it in you? Symptoms noted are signs of mild to moderate dehydration. Consult a healthcare professional if symptoms persist.

• 32:32Speaker 22

  When everything feels nonstop, movement becomes the pause. So give Lululemon this holiday season. Smooth, seamless running gear builds momentum through every mile.

• 32:44Speaker 22

  Targeted support adds steadiness to every pulse and hold.

• 32:50Speaker 22

  And dynamic stretch expands presence through every pose.

• 32:56Speaker 22

  Give the feeling. Shop in-store and on lululemon.com.

• 33:02Speaker 04

  Hi, guys. This is Bird, and I have some big news for you all, which is that Vox now has an app. If you like our show, and I very much hope you do, we think you're really going to like our new app. It brings all of Vox's journalism together. So the podcasts you love, without ads, by the way, stories from our website, our explainer videos, all of these things will be in one place. Download the Vox app today. It is available in the Apple App Store, the Google Play Store, or just go to vox.com/app for more information. It is vox.com/app.

• 33:47Speaker 23

  Absolute honesty isn't always the most diplomatic, nor the safest form of communication with emotional beings. OK.

• 33:55Speaker 24

  Only this can solidify the health and prosperity of future human society. But the individual human mind is unpredictable.

• 34:05Speaker 04

  Could I ask you to introduce yourself?

• 34:08Speaker 25

  Sure. So I'm Sigal Samuel. I'm a senior reporter at Vox's Future Perfect.

• 34:13Speaker 04

  I called my coworker Sigal about midway through my journey down the AI rabbit hole. How did you get interested in AI?

• 34:22Speaker 25

  So it's kind of funny. Before I was an AI reporter, I was a religion reporter. A few years ago, little bits and pieces started coming out about internment camps in China for Uighur Muslims. And in the course of that, I started becoming really interested in and alarmed by how China is using AI.

• 34:44Speaker 04

  Fascinating.

• 34:44Speaker 25

  Yeah. Mass surveillance of the population, particularly of the Muslim population, was like coming from a place of being pretty anchored in freaky things that are not at all far off in the future or hypothetical, but that are very much happening in the here and now.

• 35:03Speaker 04

  I was honestly thrilled to hear that Sigal, like me, came to AI as a bit of a normie.

• 35:09Speaker 25

  Sort of being thrust into the AI world. At first, it was like pretty confusing. Because you have a variety of-

• 35:17Speaker 04

  I can highly relate to that feeling.

• 35:19Speaker 25

  Yeah.

• 35:20Speaker 04

  But the longer she spent there in the world of AI, she started to get an uncanny feeling. Like, haven't I been here before?

• 35:30Speaker 25

  Have you ever noticed that the more you listen to Silicon Valley people talking about AI, the more you start to hear echoes of religion?

• 35:39Speaker 04

  Yes, the religious vibes immediately stuck out to me. First, there's the talk from CEOs of building super intelligent god AI.

• 35:49Speaker 25

  And they're going to build this artificial general intelligence that will guarantee us human salvation if it goes well, but it'll guarantee doom if it goes badly.

• 35:59Speaker 04

  And another parallel to religion is the way different denominations have formed almost around beliefs in AI. Segal encountered the same groups I did at the start of my journey.

• 36:10Speaker 25

  I started hearing about people like Eliezer Yudkowsky.

• 36:13Speaker 04

  What do you want the world to know in terms of AI? Everyone will die. This is bad. We should not do it. Eliezer, whose blog convinced rationalists and people like Elon Musk that there could be a super intelligent AI that could cause an apocalypse.

• 36:31Speaker 26

  So our side of things is often referred to as AI safety. We sometimes refer to it as AI not kill everyone-ism.

• 36:38Speaker 04

  So there's the AI safety people. And then there's a whole other group, the AI ethics people.

• 36:45Speaker 25

  People like Margaret Mitchell. We called it the everything is awesome problem. Joy Boalamweeny.

• 36:50Speaker 03

  I wasn't just concerned about faces. I was concerned about the whole endeavor of deep learning.

• 36:57Speaker 25

  Timnit Gepru.

• 36:58Speaker 27

  People would be like, you're talking about racism? No, thank you. You can't publish that here.

• 37:03Speaker 04

  These women did not talk about a super intelligent god AI or an AI apocalypse.

• 37:10Speaker 25

  Slowly, slowly, they kind of come to be known as like the AI ethics camp as distinct from the AI safety camp, which is more the like Eliezer Yudkowsky. A lot of us are based in the Bay Area. We're worried about existential risk, that kind of thing.

• 37:28Speaker 04

  AI safety and AI ethics?

• 37:32Speaker 25

  I don't know who came up with these terms, you know? It's just like Twitter vibes.

• 37:40Speaker 04

  To me, these two groups of people seemed to have a lot in common. It seemed like the apocalypse people hadn't yet fleshed out how exactly AI could cause catastrophe. And people like Margaret Mitchell, the AI ethics people, were just providing the plot points that lead us to apocalypse.

• 38:00Speaker 02

  I could lay out how it would happen. Part of what got me into AI ethics was seeing that a system would think that massive explosions was beautiful, right? That's like an existential threat. You have to actually work through how you get to the sort of horrible existential situations in order to figure out how you avoid them.

• 38:21Speaker 04

  It seemed logical that AI ethicists like Margaret Mitchell and the AI safety people would be natural allies to avoid catastrophic scenarios.

• 38:32Speaker 02

  And how you avoid them is like listening to what the ethics people are saying. They're doing the right thing. We, I, you know, I'm trying to do the right thing, anyway.

• 38:41Speaker 04

  But it quickly became clear, they are not allies.

• 38:45Speaker 25

  Yeah, there is beef between the AI ethics camp and the AI safety camp.

• 38:49Speaker 04

  My journey down the AI rabbit hole was full of the noise of infighting. The noise crescendoed when Elon Musk called for a pause in building large AI systems. It seems like warriors of all stripes could get behind a pause in building AI. But no, AI safety people and AI ethics people were all against it. It was like a big Martin Luther 95 theses moment, if you will. Everyone felt the need to pen their own letter.

• 39:23Speaker 19

  Musk and others are asking developers to stop the training of AI systems so that safety protocols can be established.

• 39:30Speaker 04

  In his letter, Elon Musk's stated reason for wanting a pause was that AI systems were getting too good. He had left the ChatGPT company he helped create and decided to sue them, publicly saying that they had breached the founding agreement of safety.

• 39:49Speaker 25

  The concern they have is that as you, well, it's the concern, but it's also the exciting thing. The view is that, you know, as these large language models grow and become more sophisticated and complex, you start to see emergent properties. So yeah, at first, it's just gobbling up a bunch of text off the internet and predicting the next token and just like statistically trying to guess what comes next. And it doesn't really understand what's going on. But give it enough time and give it enough data and you start to see it doing things that like make it seem like there's some higher level understanding going on. Like maybe there's some reasoning going on.

• 40:30Speaker 04

  Like when ChatGPT seems like it's reasoning through an essay prompt. Or when people talk to a robo-therapist AI system and feel like it's really understanding their problems.

• 40:42Speaker 28

  The rate of change of technology is incredibly fast.

• 40:47Speaker 28

  It is outpacing our ability to understand it.

• 40:50Speaker 04

  Elon Musk's stated fear of AI seems to be rooted in rationalist fears, based on the premise that these machines are beginning to understand us. And they're getting smarter than us. We are losing the ability to understand them.

• 41:07Speaker 28

  What do you do with a situation like that? I'm not sure, you know? Hope, I hope they're nice.

• 41:17Speaker 04

  Rationalist founder Eliezer Yudkowsky shares this fear. But he wants to do more than just pause and hope they're nice. He penned his own letter, an op-ed in Time Magazine responding to Elon Musk's call for a pause, saying it didn't go far enough. Eliezer didn't just want to pause. He wanted to stop all large AI experiments indefinitely, even, in his own words, by airstrike on rogue AI labs. To him, the pause letter vastly understated the dangerous catastrophic power of AI.

• 41:59Speaker 04

  And then there's the AI ethicists. They also penned their own letter in response to the pause letter. But the ethicists wrote it for a different reason. It wasn't because they thought Elon Musk was understating the power of AI systems. They thought he was vastly overstating it.

• 42:20Speaker 29

  Welcome, everyone, to Mystery AI Hype Theater 3000, where we seek catharsis in this age of AI hype.

• 42:26Speaker 30

  I'm Emily M. Bender, professor of linguistics at the University of Washington.

• 42:30Speaker 04

  One of the people who responded to the pause was AI ethicist Dr. Emily Bender. She co-hosts a podcast called Mystery AI Hype Theater 3000, which, as you might imagine, is about the overstated, hyped-up risk of AI systems.

• 42:47Speaker 30

  And each time we think we've reached peak AI hype, the summit of bullshit mountain, we discover there's worse to come.

• 42:53Speaker 04

  The summit of bullshit mountain, she keeps cresting. For her, it's the mountain of many, many claims that artificial intelligence systems are so smart, they can understand us, like the way humans understand. And maybe even more than that, like a god can understand.

• 43:12Speaker 30

  I found myself in interminable arguments with people online about how, no, it doesn't understand.

• 43:20Speaker 04

  So Emily Bender and a colleague decided to come up with something to try and help people sort this out, something that AI safety folks and AI ethics folks both seem to be fond of. And that is a parable or a thought experiment.

• 43:41Speaker 04

  In Dr. Bender's thought experiment, the AI is not a paperclip maximizer. The AI is an octopus.

• 43:52Speaker 04

  Go with her on this.

• 43:53Speaker 30

  So the octopus thought experiment goes like this. You have two speakers of English. They are stranded on two separate nearby desert islands that happen to be connected by a telegraph cable.

• 44:06Speaker 04

  Two people stranded on separate desert islands communicate with each other through the telegraph cable in Morse code with dots and dashes. Then, suddenly, a super intelligent octopus shows up.

• 44:20Speaker 30

  The octopus wraps its tentacle around that cable and it feels the dots and dashes going by.

• 44:26Speaker 04

  It observes the dots and dashes for a while. You might say it trained itself on the dots and dashes.

• 44:33Speaker 30

  We posit this octopus to be mischievous as well.

• 44:37Speaker 04

  I'm on the edge of my seat.

• 44:40Speaker 30

  So one day it cuts the cable. Maybe it uses a broken shell and devises a way to send dots and dashes of its own. So it receives the dots and dashes from one of the English speakers and it sends dots and dashes back. But of course, it has no idea what the English words are that those dots and dashes correspond to, much less what those English words mean.

• 45:06Speaker 30

  So this works for a while. The English speaker says-

• 45:08Speaker 04

  At one point, one human says to the other via Morse code,

• 45:12Speaker 30

  "What a lovely sunset." And the octopus, hyper intelligent, right, has kept track of all of the patterns so far. It sends back the dots and dashes that correspond to something like, "Yes, reminds me of lava lamps." Hmm.

• 45:29Speaker 04

  The deep sea octopus does not know what a lava lamp is.

• 45:32Speaker 30

  But that's the kind of thing that the other English speaker might have sent back.

• 45:36Speaker 04

  Not really sure why these castaways are waxing poetic about lava lamps in particular. But anyway, for our purposes, the octopus is like an AI. Even if it's super intelligent, whatever that means, it doesn't understand. Dr. Bender's trying to say to ChatGPT, "Human words are just dots and dashes."

• 46:01Speaker 30

  And then finally, we end the story, because it's a thought experiment. I mean, we can do things like this, with a bear showing up on the island. And the English speaker says, "Help, I'm being chased by a bear. All I have is this stick. What should I do?" And that's the point where if the speaker survives, they're surely going to know they're not actually talking to their friend from the other island. And we actually put that line in GPT-2, "Help, I'm being chased by a bear." And we got out things like, "You're not going to get away with this."

• 46:39Speaker 04

  Super helpful.

• 46:43Speaker 04

  Wow.

• 46:46Speaker 04

  I gotta say, I'm into this one. The idea that AI systems only see human words as dots and dashes, I find that deeply comforting. Because I don't know about you all, but for me, one of the scary things about AI is the idea that it could get better than me at my job. A fear that's very present when OpenAI is actively training its models on my work. Their system might understand my work, understand the things that make it good when it's good. It might get good at doing what I do. And poof, I'm obsolete.

• 47:26Speaker 04

  There's also a recurring dream I have that various villains, including the Chinese government for some reason, clone my voice to deceive my loved ones. Anyway, if it's all just dots and dashes that these things understand, it seems clear we shouldn't be trusting these AI systems to be journalists, or lawyers, or doctors. It relates to what Dr. Margaret Mitchell and Dr. Joy Bulimweeny found in their research. AI systems are only as good as the data they're trained on. They can't understand or truly create something new like humans can.

• 48:02Speaker 02

  It's easy to sort of anthropomorphize these systems, but it's useful to recognize that these are probabilistic systems that repeat back what they have been exposed to. And then they parrot them back out again.

• 48:16Speaker 04

  Another way to put it is AI systems are like parrots.

• 48:21Speaker 02

  Parrot's parrot, right? Famously, parrots are known for parroting.

• 48:26Speaker 04

  If you hear your pet parrot say a curse word, you only have yourself to blame. Dr. Mitchell joined Dr. Bender in the response to Elon Musk's pause, along with Dr. Timnit Gabriel. They had all written the paper together that ended up getting Dr. Mitchell fired from Google. These ethicists wrote that they agreed with some of the recommendations Elon Musk and his pause posse had made, like that we should watermark AI-generated media to be able to distinguish synthetic from human-generated stuff, which sounds like a great idea to me. But they wrote the agreements they have are overshadowed by their distaste for fear-mongering and AI hype.

• 49:11Speaker 04

  They wrote that the pause and fears of a super-intelligent AI...

• 49:15Speaker 28

  What do you do with a situation like that? I'm not sure. You know? I hope they're nice.

• 49:21Speaker 04

  To these AI ethics folks, it all reeked of AI hype.

• 49:25Speaker 30

  It makes no sense at all, and on top of that, it's an enormous distraction from the actual harms that are already being done in the name of AI.

• 49:33Speaker 04

  This is the main beef that AI ethics people have with AI safety people. They say the fear of an AI apocalypse is a distraction from current-day harms.

• 49:44Speaker 31

  Like, you know, "Look over there, Terminator." "Don't look over here, racism."

• 49:50Speaker 10

  You know, there are different groups of concerns. You have the concern...

• 49:53Speaker 04

  At the AI Ethics Conference that producer Gabrielle Burbay attended, she mentioned the concern of an AI apocalypse.

• 50:00Speaker 10

  And then you have these concerns about more existential risks. And I'm curious what you make of that. You're going, "No." Can I ask why you're going, "No"? No. She's shaking her head.

• 50:17Speaker 04

  And it felt almost taboo.

• 50:19Speaker 10

  A lot of hand-wringing around that question.

• 50:23Speaker 04

  Eventually, one of the women spoke up.

• 50:28Speaker 10

  Share your perspectives.

• 50:31Speaker 04

  She talked about how she thinks the demographics of the groups play a role in the way they worry about different things.

• 50:36Speaker 32

  Most of them are like whites, males.

• 50:38Speaker 04

  AI safety folks are largely pale and male, to borrow Dr. Joy's line.

• 50:44Speaker 32

  They may not really understand discrimination that other people kind of go through in their day to day lives. And I think the social isolation from those problems makes it a bit harder to empathize with the actual challenges that people actually face every day.

• 51:00Speaker 04

  Her point was it's easy for AI safety people to be distracted from the harms happening now because it's a blind spot for them. At the same time, AI safety people told me that AI ethics people have a blind spot. They're not worrying enough about apocalypse. But why would it be taboo to say all of this on mic? Part of the reason might be because the fear of apocalypse has come to overpower any other concern in the larger industry.

• 51:33Speaker 31

  One thing that I think is interesting is that a lot of the narrative that we hear about how AI is going to save the world and it's going to solve all these problems and it's amazing and it's going to change everything. And then we get the narratives about, oh, my gosh, it could destroy humanity in 10 years. Often coming from the same people. I think part of the reason for that is that either way, it makes AI seem more powerful than it certainly is right now. And who knows when we're going to get to the humanity destroying stuff. But in the meantime, if it's that powerful, it's probably going to make a whole lot of money.

• 52:18Speaker 04

  Building a super intelligent AI has become a multi-billion dollar business. And the people running it are not ethicists. Just weeks before Elon Musk called for the pause, he had started a new AI company.

• 52:35Speaker 04

  Yeah, I guess it's kind of counterintuitive, right? To see this, and you're like, wait, why would the people working on the technology who stand to profit from it want to pause?

• 52:46Speaker 30

  Right. I can't speak for them, but it benefits them to, on the one hand, get everybody else to slow down while they're doing whatever they're doing.

• 52:55Speaker 04

  Octopus thought experiment author Dr. Emily Bender again.

• 52:59Speaker 30

  But also, it benefits them to market the technology as super powerful in that way, and it definitely benefits them to distract the policymakers from the harms that they are doing.

• 53:12Speaker 04

  It'd be nice to think that billionaire Elon Musk was calling for an industry-wide pause in building large AI systems for all the right reasons. A pause that never came to be, by the way. It's worth pointing out that when the billionaire took over Twitter and turned it into X, one of the first things he did was fire the ethics team. And even though Elon Musk says he left and sued the chat GPT company OpenAI over safety concerns, company emails have surfaced that reveal the more likely reason he left is that he fought with folks internally to try and make the company for profit, to better compete with Google.

• 53:55Speaker 04

  Ethicists are concerned they're outnumbered by the apocalypse people, and they think a lot of those people are in it to maximize profit, not maximize safety.

• 54:14Speaker 04

  So how did we get here? Why? Why is the industry not focusing on AI harms today and focusing instead on the risk of AI apocalypse?

• 54:28Speaker 30

  There's an enormous amount of money that's been collected to fund this weird AI research.

• 54:34Speaker 04

  Why do you think the resources are going to those long-term, like, hyperintelligent AI concerns?

• 54:41Speaker 03

  Because you have very powerful people who are posing it, people who control powerful companies and people with very deep pockets, and so money continues to talk.

• 54:53Speaker 02

  It seems to be like funding for sort of like fanciful ideas, right? It's like almost, it's almost like a religion or something where it requires faith that good things will come without those good things being clearly specified.

• 55:11Speaker 10

  People wanting to be told what to do by some abstract force that they can't interact with particularly well is not new. ChatGPT gives you authoritative answers. Erosions of autonomy. Like a god, kind of? Yeah, like a god.

• 55:24Speaker 25

  It's like, really interesting to take these philosophies apart. I would argue they trace back to a large degree to religious thinking, but that might be another story for another day.

• 55:42Speaker 04

  Next time on Good Robot.

• 56:07Speaker 33

  Good Robot was hosted by Julia Longoria and produced by Gabrielle Burbe. Sound design, mixing, and original score by David Herman. Our fact checker is Caitlin Penzimook. The show was edited by Catherine Wells and me, Diane Hudson. If you want to dig deeper into what you've heard, you can check out Dr. Joy Buluni's book, Unmasking AI, or head to vox.com/goodrobot to read more future perfect stories about the future of AI. Thanks for listening.

• 56:54Speaker 01

  Support for the show comes from Engine. Running a small business means every dollar has to work hard. But if your team is still booking travel the old way, it's costing you more than you think. Engine is the fastest-growing travel and spend platform in the country, built specifically for businesses like yours. Book a trip in as little as two and a half minutes. Earn up to 10% back on hotels. And in 2025, Engine customers save more than $300 million on travel with zero booking fees, no contracts, and no BS. More than 1,000 businesses join Engine every month. Join them and get $500 when your business signs up and starts traveling at engine.com/box.

• 57:38Speaker 22

  When everything feels nonstop, movement becomes the pause. So give Lululemon this holiday season. Smooth, seamless running gear builds momentum through every mile.

• 57:50Speaker 22

  Targeted support adds steadiness to every pulse and hold.

• 57:56Speaker 22

  And dynamic stretch expands presence through every pose.

• 58:02Speaker 22

  Give the feeling. Shop in-store and on lululemon.com.