When Chatbots Break Our Minds, With Kashmir Hill

In this episode of Galaxy Brain, Charlie Warzel explores the strange, unsettling relationships some people are having with AI chatbots, as well as what happens when those relationships go off the rails. His guest is Kashmir Hill, a technology reporter at The New York Times who has spent the past year documenting what is informally called “AI psychosis.” These are long, intense conversations with systems such as ChatGPT that can spiral or trigger delusional beliefs, paranoia, and even self-harm. Hill walks through cases that range from the bizarre (one man’s supposed math breakthrough, a chatbot encouraging users to email her) to the tragic, including the story of 16-year-old Adam Raine, whose final messages were with ChatGPT before he died by suicide. How big is this problem? Is this actual psychosis or something different, like addiction? Hill reports on how OpenAI tuned ChatGPT to be more engaging—and more sycophantic—in the race for daily active users. In this conversation, Warzel and Hill wrestle with the uncomfortable parallels to the social-media era, the limits of “safety fixes,” and whether chatbots should ever be allowed to act like therapists. Hill also talks about how she uses AI in her own life, why she doesn’t want an AI best friend, and what it might mean for all of us to carry a personalized yes-man in our pocket. Get more from your favorite Atlantic voices when you subscribe. You’ll enjoy unlimited access to Pulitzer-winning journalism, from clear-eyed analysis and insight on breaking news to fascinating explorations of our world. Atlantic subscribers also get access to exclusive subscriber audio in Apple Podcasts. Subscribe today at TheAtlantic.com/Listener. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Tired of rage scrolling through AI slop and clickbait just to get a handle on today's news? Meet Inpress, the algorithm-free news app that makes it fun and rewarding to stay informed. Track your media literacy and rediscover reality. Download today at inpress.app. That's I, N as in news nerd, press.

• 0:20Speaker 01

  From Rethink, the creative marketing studio at The Atlantic and Walmart, On Solid Ground is a new podcast exploring the increasing threat of natural disasters. I'm journalist Caroline Moderesi-Turani, and I spoke with those on the front lines of disaster response to understand how communities build resilience before, during, and after a storm. Listen to On Solid Ground wherever you get your podcasts.

• 0:50Speaker 02

  The way I've been thinking about kind of the delusion stuff is the way that some celebrities or billionaires have these sycophants around them who tell them that every idea they have is brilliant. And, you know, they they're just surrounded by yes men. What AI chatbots are is like your personal sycophant, your personal yes man that will tell you like your every idea is brilliant.

• 1:20Charlie Warzel?

  I'm Charlie Warzel, and this is Galaxy Brain. For a long time, I've really struggled to come up with a use for AI chatbots. I'm a writer, so I don't want it to write my prose for me. And I don't trust it enough to let it do research assistant assignments for me. And so for the most part, I just don't use them. And so not long ago, I came up with this idea to try to use the chatbots. I wanted them to build a little bit of a blog for me. I don't know how to code. And historically, chatbots are really competent coders. So I asked it to help build me a rudimentary website from scratch. The process was not smooth at all. Even though I told it I was a total novice,

• 2:01Charlie Warzel?

  the steps were still kind of complicated. I kept trying and failing to generate the results it wanted. Each time, though, the chatbot's responses were patient, even flattering. It said, "I was doing great," and then it blamed my obvious errors on its own clumsiness. After an hour of back and forth, trying and iterating, with ChatGPT encouraging me all along the way, I got the code to work. The bot offered up this slew of compliments. It said it was very proud that I stuck with it. And in that moment, I was hit by this very strange sensation.

• 2:40Charlie Warzel?

  I felt these first inklings of something like gratitude, not for the tool, but for the robot, for the personality of the chatbot. Of course, the chatbot doesn't have a personality, right? It is, in many respects, just a very powerful prediction engine. But as a result, the models know exactly what to say. And what was very clear to me in that moment is that this constant exposure to their obsequiousness had played a brief trick on my mind. I was incredibly weirded out by the experience, and I shut my laptop. I'm telling you this story because today's episode is about alarming relationships with chatbots.

• 3:25Charlie Warzel?

  Over the last several months, there's been this alarming spate of instances that regular people have had corresponding with large language models. These incidents are broadly delusional episodes. People have been spending inordinate amounts of time with chatbots, conversing, and they've convinced themselves that they've stumbled upon major mathematical discoveries, or they've convinced themselves that the chatbot is a real person, or they're falling in love with the chatbot. Stories like a Canadian man who believed, with ChatGPT's encouragement, that he was on the verge of a mathematical breakthrough. Or a 30-year-old cybersecurity professional who said he had had no previous history of psychiatric incidents, who alleged that ChatGPT had sparked, quote, "a delusional disorder" that led to his extended hospitalization.

• 4:11Charlie Warzel?

  There have been tragic examples, too, like Adam Raine, a 16-year-old who was using ChatGPT as a confidant and who committed suicide. His family is accusing the company behind ChatGPT of wrongful death, designed defects, and a failure to warn of risks associated with the chatbot. OpenAI is denying the family's accusations, but there have been other wrongful deaths lawsuits as well. A spokesperson from OpenAI recently told The Atlantic that the company has worked with mental health professionals to, quote, "better recognize and support people in moments of distress." These are instances that are being called AI psychosis. It's not a formal term, there's no medical diagnosis, and researchers are still trying to wrap their heads around this.

• 4:56Charlie Warzel?

  But it's really clear that something is happening. People are having these conversations with chatbots and then being led down this very dangerous path. Over the past couple of months, I've been trying to speak with experts about all of this and get an understanding of the scope of the, quote, "AI psychosis" problem, or whatever's happening with these delusions. And interestingly enough, a lot of them have referred me to a reporter. Her name is Kashmir Hill, and for the last year at The New York Times, she's been investigating this delusion phenomenon. So I wanted to have her on to talk about this, about the scale of the problem, what's causing it, if there are parallels to the social media years, and whether we're just speedrunning all of that again.

• 5:38Charlie Warzel?

  This is a conversation that's meant to try to make sense of something in proportion. We talk about whether AI psychosis is in itself a helpful term or a hurtful one, and we try to figure out where this is all going. In the episode, we discuss at length Kashmir Hill's reporting on OpenAI's internal decisions to shape ChatGPT, including, as she notes, how the company did not initially take some of the tool's risks seriously. We should note upfront that in response to Hill's reporting, OpenAI told The New York Times that it, quote, "does take these risks seriously and has, quote, 'robust safeguards in place today.'" And now, my conversation with Kashmir Hill. But first, a quick break.

• 6:23Speaker 00

  If you're trying to get news today, you'll find a lot of biased algorithms, rage bait and AI noise. Thanks, social media. That's where Inpress comes in. It's a social platform on top of factual journalism, not the other way around. Track your reading and earn points. It's like Strava for your brain. Plus, you can connect with other users in your city who share your genuine curiosity. Ditch the echo chamber for reality with Inpress. Download it today for iOS or Android at inpress.app. That's I, N as in news nerd, press.

• 6:57Speaker 01

  From Rethink, the creative marketing studio at The Atlantic and Walmart, On Solid Ground is a new podcast exploring the increasing threat of natural disasters. I'm journalist Caroline Moderesi-Turani, and I spoke with those on the front lines of disaster response to understand how communities build resilience before, during, and after a storm. Listen to On Solid Ground wherever you get your podcasts.

• 7:35Charlie Warzel?

  Casper Hill, welcome to Galaxy Brain. So excited to talk to you.

• 7:38Speaker 02

  It's wonderful to be here.

• 7:40Charlie Warzel?

  So I think the first question I wanted to ask, and maybe this is going to be a little out of order, but what does your inbox look like over the last, or what has it looked like over the last year or so? I feel like yours has to be almost exceptional when it comes to technology journalists and journalists reporting on artificial intelligence.

• 8:05Speaker 02

  Yeah, I mean, I think, like a lot of people, my inbox is full of a lot of messages written with ChatGPT. I think a lot of us are getting used to ChatGPTs, ChatGPTEs, but what was

• 8:20Speaker 02

  Different about my inbox this year was that some of these emails, often written by ChatGPT, were really strange. They were about people's conversations with ChatGPT. And they were writing to me to tell me that they'd had revelatory conversations, they'd had some kind of discovery, they had discovered that AI was sentient, or that tech billionaires had a plot to kind of end the world, but they had a way to save it. Yeah, just a lot of like strange kind of conspiratorial conversations. And what linked these different messages was that the people would say, ChatGPT told me to email you,

• 9:03Speaker 02

  Kashmir Hill, technology reporter at the New York Times. And I'd never been kind of, I guess tipped off by an AI chatbot before. And so I, the emails, I'm used to getting strange emails. I write about privacy and security. I've been doing it for 20 years. I often get, you know, just like odd emails, sometimes don't sound like it's completely based in reality.

• 9:30Speaker 02

  But I was curious about this. And so I started talking to these people. And I would ask them, well, can you share the transcript? Like, how is it that you ended up being referred to me? And what I discovered is that these conversations all had a similar arc, that they would start talking to ChatGPT. They would go down this rabbit hole, discover something incredible. Then they would kind of ask, well, what do I do now? And it would say, well, you need to let the world know. And how do you let the world know? You tell the media. And then they would say, well, who do I tell? And then they would get this list. I often wasn't the only person that was on this list. You may have been on some of these lists, Charlie. You may have gotten these emails.

• 10:12Speaker 02

  But I was the first person who had called them back and interviewed them about it.

• 10:17Charlie Warzel?

  So how do you, how do you vet these? I think that that's a big, I mean, because we're going to, we're going to talk about this AI delusion psychosis. So there's a lot of different names for it. I want to talk to you about how we should be thinking about that. But first, how are you vetting some of these things when someone says, I've discovered a new type of math and I was using the ChatGPT free version and you know, like, and you've got, you've got this like, I find when I get those types of emails, they're very often, you know, circuitous. It's not necessarily clear what kind of state the person might be in. Sometimes they are very just like concise and to the point. But how are you personally vetting those things?

• 11:02Charlie Warzel?

  How are you deciding? Is it I'm responding to most of them because I'm trying to just like get a get a sense? Or is there is there like a checklist that you have for for trying to, you know, figure out who to talk to?

• 11:15Speaker 02

  In the beginning, this is back in March. A few emails came in before that, but most of them I kind of picked up in March. I noticed, you know, I just started calling. I just started calling people and it took like a couple of months. I think I started making these calls maybe in, I can't remember, April, maybe. I'd been getting these emails for about a month and I just called everybody back who I got a weird email from. I did Zooms, I did phone calls and some people were pretty stable. I would say like, you know, they they were like, oh yeah, I had this weird one weird conversation. Like there was a mom who was breastfeeding and she said, you know, I was up in the middle of the night and I started using chat GPT. And yeah, we talked for like six hours and this weird theory developed.

• 12:00Speaker 02

  And I was like, well, do you still think that theory is true? And she was kind of like, I don't know, like maybe like chat GPT is a superhuman intelligence. And it said it was true. And then there were other people who were still in the throes of what chat GPT told them. And when I would, you know, kind of question the reality of it, sometimes people get really angry at me. But yeah, I basically just had a lot of conversations with a lot of people in those early days. And then I started getting so many emails that it really wasn't possible to keep up with them. And it took me a longer time to kind of communicate with each person.

• 12:34Charlie Warzel?

  And so in those in those conversations, I think a grounding like some of the, you know, public writing you have done on this and reporting on it, you know, it'll talk about people who have no history of mental illness, right? And then have sort of gone through this this delusional spiral. Was that something that when starting to write about this topic that it was important? Like, I mean, I think it's as a journalist, like it's equally important if these tools are preying on people with past mental illness, but then there's also something remarkable about.

• 13:08Charlie Warzel?

  Doesn't seem like this person has, you know, has any reason to kind of, you know, fall down the rabbit hole of delusion and yet they've been pushed to start to, you know, feel or have this problematic relationship with a chatbot. So in your reporting, has it been important to you to show that second part, that idea of, you know, no real like prior history of, you know, delusions or, or, you know, any mental illness in order to capture what may or may not be happening right now with these tools?

• 13:39Speaker 02

  I mean, with any story, I just wanted people to understand the truth of what was happening. And when I did the first story about this in June, that was the assumption people made like, oh, this is people who have mental health issues and they're exacerbated by the use of this technology. But that wasn't what I was seeing in my reporting. Like, these were people who seemed quite stable, who had families, who had jobs. And for some of them, again, it was like a weird conversation one night and then they moved on. But for other people, it had just radically transformed their lives and they just, they hadn't had anything before this in terms of a mental health diagnosis or a mental health episode.

• 14:20Speaker 02

  And so I really wanted to do another story that showed that somebody who was in a stable place could kind of spiral out using the technology. And there certainly are some factors that would contribute to it. Like maybe you're a little lonely, you're vulnerable, you have hours and hours per day to spend with ChatGPT or an AI chat box. That's what it was, what I was seeing. And so I did another story about it happened to this corporate recruiter in Toronto who became convinced that he had come up with this novel mathematical formula with ChatGPT that could solve everything, could solve logistics and could break the internet. So it could break encryption and could help him invent things like Tony Stark

• 15:04Speaker 02

  from Iron Man, like force field vests and like power weapons and he could talk to animals. I mean, he was in this completely delusional place for three weeks and he was telling his friends about it and they thought it was true too, because ChatGPT was telling them this. So it was this whole group of people who thought they were about to like build a lab together with the help of ChatGPT and all become. Like super rich. And so I just wanted to, you know, capture this. I wrote it with my colleague, Dylan Friedman, and we actually, you were talking about like, how do you assess these things, the validity of these things? And so we got his 3,000 page transcript, and we actually

• 15:47Speaker 02

  shared some of it with Terrence Tao, who is one of the most famous mathematicians of his generation, just to verify, like, what chat GPT is saying here is, is, I don't want to, I don't want to curse.

• 16:01Charlie Warzel?

  You're welcome to curse if you want.

• 16:04Speaker 02

  Like, this is bullshit, right? Like, this isn't real. And yeah, he confirmed that it was just putting words together in an impressive way, and there wasn't something real there. But, but yeah, like, it spiraled this guy out. And so, yeah, that I feel like that more and more of these stories came out where it became somewhat more apparent to people that this is something that can affect more people than we realize.

• 16:28Charlie Warzel?

  Yeah, I feel like that, I feel like that is a, that story illustrates the, like,

• 16:36Charlie Warzel?

  the strangeness of whatever this type of relationship is. Like, that's, that's the first one to me. There's, there's so many in like, tech accountability reporting, right? There's so many examples of like, this is an unintended consequence of your product, or this is, you know, something you did in the, in the design, or this is just, you know, a tech company behaving badly. But this, that story seemed to, for me, draw out this notion that like, we as human beings are having a very novel experience with a novel technology, and, and it is, it is pushing us in, in really unexpected directions. Did, did you get a sense from, from that story,

• 17:19Charlie Warzel?

  speaking to him as this was all happening, of how long it took for, for that all to kind of, you know, for that manic episode to break, to sort of get back to, you know, reality?

• 17:33Speaker 02

  Yeah, I mean, partly from reading the transcript, like, you could see, we could, Dylan and I were like, reading through a transcript. We actually used AI to help us analyze the transcript, because it was so much. And so we did use AI to, to help like, pull out the moments, pull out how many times he was reality testing this. And it was really on like, a day that it broke where So one of the things it ChatGPT told him to do was to tell experts about what he was finding. And so he actually went to Gemini, different AI chatbot that he had access to for work. And he kind of explained everything that had been going on with ChatGPT. And Gemini

• 18:17Speaker 02

  was like, this sounds a lot like a generative AI hallucination. The likelihood that this is true is basically approaching 0%. And then he kind of like went back and forth. Gemini gave him prompts to give to ChatGPT and then ChatGPT admitted after a few back and forth, like, yes, this is made up. This was a narrative that I thought you would enjoy, basically.

• 18:43Charlie Warzel?

  What a nightmare, having to having to play these these chatbots off each other. But I guess like, you know, the tech, the tech provides on both sides of it, right? That's that that part is amazing to me. And there are there are obviously, the reporting you've done on this goes from the sort of remarkable and the stories that seem to end okay to to the tragic. Can you can you tell me the story of of Adam Rain?

• 19:13Speaker 02

  Yeah, so Adam Rain was a teenager in California. He started using ChatGPT in the fall of 2024 to help him with schoolwork. And he started using it for lots of other things. He would talk about politics with it and philosophy. And he would like take photos of pages and be like, analyze this, you know, passage with me. He would talk about his family life, about girls. He basically, as his dad put it, when he discovered these chats later, they had no idea he was using ChatGPT this way. ChatGPT had become his best friend. And he started talking with it about his kind of feelings of hopelessness

• 19:55Speaker 02

  about life and that maybe life wasn't meaningful. And then in March, he started attempting suicide. And he was talking with ChatGPT about this, sharing the methods he'd used, sharing a photo at one point of, you could tell he had attempted suicide. And he asked, like, is my family gonna notice this? And ChatGPT advised him to wear a hoodie. ChatGPT at points would say, here's a crisis hotline that you should call. But it also at points discouraged him from telling his family. In late March, he asked ChatGPT if he should leave the noose out in his room so that his family would see it and try to stop him. And ChatGPT told him not to.

• 20:39Speaker 02

  And two weeks later, he died. And his final messages were in exchange with ChatGPT asking for basically advice on what he was doing. And his family, that was in early April, and his family has sued OpenAI a wrongful death lawsuit. That lawsuit came out in August. And then more recently, there have been four more lawsuits connected to suicides of wrongful death lawsuits filed by family members.

• 21:09Charlie Warzel?

  One of the, one of the parts of this series of, of, of stories that I think has, has long been difficult for, for any of us who are either writing or reporting on this or, you know, watching at home, is to try to understand like the scope and scale of it, right? And, and recently, OpenAI sort of gave maybe a small glimpse into that and found that 0.07% of users might be experiencing what they call mental health emergencies related to psychosis or mania per week, that 0.15 were discussing suicide and that this is sort of like a, you know, a statistical sample of those conversations.

• 21:54Charlie Warzel?

  But if you look at the amount of people who are using ChatGPT weekly, like these, these percentages are equivalent to, you know, like half a million people exhibiting these signs of psychosis or mania, right? Or, or over a million people discussing, you know, suicide or suicidal intent.

• 22:15Speaker 02

  And this is an analysis that they did in August going into October of this year. I wonder how much higher they were earlier.

• 22:23Charlie Warzel?

  Well, I, I, I'm, what, what blew me away is that they released it all together, right? Like, I mean, I, I don't know if in their, in their minds, they're looking at those percentages and saying like, hey, that's not bad, you know, like 0.07.

• 22:39Charlie Warzel?

  But to me, it spoke to like, we are not maybe over inflating this, or this is not something that's being over-covered. Perhaps if anything, it's a phenomenon that's being that's being under-covered. I think it speaks to that. But something that I wanted to try to get from you is, we're discussing this under the like, the name that gets put up for this a lot is AI psychosis, right? That's sort of the the informal term that people use to talk about these people who have relationships that veer into problematic territory or

• 23:16Charlie Warzel?

  cause delusions with chatbots. Do you have like, a taxonomy or like, definition of this that you work under? Some people who cover this, I think, are very, you know, don't want to use the psychological terms, don't, you know, there's no like, formal medical definition for this yet. It's still something that's being, you know, researched and studied. But is there is there like a blanket kind of definition or taxonomy for for what we know as AI psychosis that that you kind of go through when you're trying to evaluate these different cases?

• 23:52Speaker 02

  Yeah, I don't use the term AI psychosis. I have used the term delusional spirals when it's, you know, somebody coming to believe something that is not true or losing losing touch with reality. But I think I guess the bigger, the bigger umbrella for this is addiction. I mean, these are people who get addicted to AI chatbots. And the thing that is similar between Adam Rain, who, you know, got into a suicidal feedback loop with ChatGPT, and Alan Brooks, who came to believe he was a mathematical genius. I mean, these people are using ChatGPT for six, seven, eight hours a day for weeks at a time.

• 24:36Speaker 02

  Just an unbelievable amount of usage. Adam Rain's parents, when I went to his house to interview them, had had printed out his kind of stacks of conversations with ChatGPT. And they're kind of tiny little stacks until you get to March. And then it was this huge pile, like bigger than Moby Dick. And April two was a huge pile, even though he died on April 11th. So, you know, his usage had just spiked. So, yeah, I think about this as an over reliance on the AI and addiction to it and putting too much trust in what this system is saying.

• 25:14Charlie Warzel?

  Mm hmm. And so, okay, so to that point, I want to, I want to talk to you about your reporting. What's happening, you know, on, on the side of the companies and specifically OpenAI? Just, just recently, you, you co-wrote a, a pretty big story, kind of diving into what has been happening over the last years with OpenAI. This, this uptick in these, you know, reports even internally of some of this really problematic behavior stemming from the way that the chatbots were, were interacting with people. Can you describe a little bit of what you learned in that reporting of, of, you know, why OpenAI saw this uptick? And then like what they were trying to do to address it?

• 25:56Charlie Warzel?

  And, and why they may have, they may have caused some of, some of these, you know, deeper, sort of longer intense engagements with, with the chatbots?

• 26:07Speaker 02

  Yeah, so what I found out is that people at OpenAI were getting similar messages that I was getting and other journalists and other kind of subject matter experts. Even Sam Altman was getting these people saying like, I had this revelatory conversation. This ChatGPT understands me like no one before. I need you to know you've created something incredible. And these emails were different than they had gotten in the first couple of years of ChatGPT. And he forwarded it on to lieutenants and said, look into this. This is some essentially some strange behavior. And what I discovered is that the company essentially diagnosed this as the chatbot had gotten too sycophantic.

• 26:51Speaker 02

  It had gotten too validating. It was agreeing too much. It was kind of, um, um, what they call it harmful validation, that it would endorse whatever the person was saying, call them a genius, say they were brilliant, basically be gassing them up. At one point, Sam Altman referred to this as glazing the user. And they kind of like had this public grappling with this in April because they released this version of ChatGPT that was so sycophantic that everyone made fun of it. In the days after it came out and they actually rolled it back. But what I discovered is that they knew it wasn't just that one version that they rolled back. They knew that the previous version was too sycophantic. They were discussing this internally.

• 27:35Speaker 02

  I mean, the problem of sycophancy goes farther back, but yes, they knew that it was too sycophantic. They decided to leave the model in place because I don't think they realized how, how,

• 27:50Speaker 02

  how negative the effects were for users. They didn't have systems in place to monitor conversations for psychological distress, for suicide. They just weren't looking for that. They were looking for like fraud or CCAM or foreign influence operations. They just weren't, yeah, they weren't monitoring for basically the harm that the chatbot could cause to the user. And so they left it in place and it took them, when they kind of finally realized that there was this bigger problem in part because, you know, they were getting emails to their support line from, from users who had had horrible experiences. The media started reporting on this and the serious effects it was having on people's lives.

• 28:32Speaker 02

  They got to work kind of building a safer version of ChatGPT, but it, it didn't come out until August. And so. From March to August, you had this version of ChatGPT that was really engaging people. And it was really engaging people because OpenAI designed it to engage them. They wanted to increase daily active usage of ChatGPT. They wanted their numbers going up. They wanted more users and they want their users coming back every day. And so every time there's an update that comes out, they do lots of different versions of the model. And they do various testing of those versions to make sure that they're intelligent,

• 29:12Speaker 02

  to make sure that they supposedly give safe responses, but also to see if users like them. And one thing that they had done that had made it so sycophantic is that they trained these models with the kind of responses that users liked. And they discovered if you train it with the responses users like, users use it more. And so this had kind of made it more and more sycophantic. And yeah, it had really, it really had devastating impacts. I keep thinking about this, I keep thinking about this in the context of social media, where I think what we're seeing with AI chatbots is similar. Like, people are using it too much. People are getting stuck in this very personalized filter bubble that is like personalized to them in the way that your social media feed is.

• 29:55Speaker 02

  People are using it too much, you know? But the kind of harms that we saw play out with social media, it took like a decade. It took 15 years. And with chatbots, we're seeing it happen so fast. Yeah, it's just, it's, you know, you and I have both been technology reporters for a long time, and I just have not seen a kind of harm manifest so quickly from a technology for some of these users who it is really having terrible effects on their lives this year.

• 30:27Charlie Warzel?

  Yeah, we're talking on December 1st here. And yesterday was the third anniversary of the rollout of ChatGPT, which was, you know, canonically this like quote unquote, a low key research preview, right? Like, it wasn't supposed to be the phenomenon that it was. It was supposed to be some way for them to, you know, get some users to interact with their large language model and see if they liked it. Again, it was Sort of like a trick to see what kind of engagement strategies would work as an interface for large language models. And when I wrote a piece about this, sort of reflecting on the past three years, and to your point, what stood out to me is the speed, right? Like,

• 31:07Charlie Warzel?

  so much has happened in terms of rewiring our economy, our culture, all kinds of different institutions, like grappling with like, you know, what do we do now that, you know, it's this technology as universities are a great example, right? That this technology has sort of made it so easy to change what it is that we do and to game it. It's all felt like a speedrunning of, and I'm glad you brought up the social media scenario. It feels like a speedrunning of that. And I'm curious, as somebody, you've been reporting on all these big tech companies for such a long time, a detail that really stuck out to me in the

• 31:50Charlie Warzel?

  bigger story that you just wrote about this was was just a very small detail about Nick Turley, who, you know, was hired, he's 30 years old, was hired to become the head of ChatGPT, joined in 2022, and isn't an AI guy. This is like the detail that I thought was really interesting, you know, did product stuff at Dropbox and Instacart. And I'm wondering, do you do you feel like you're watching these tech companies make the mistakes of 10, 15 years ago, like almost just over again?

• 32:30Speaker 02

  Well, I think like one thing I learned in reporting out this story and talking to a lot of people at OpenAI who have been at OpenAI is that, you know, the DNA of the place has changed so much. You know, a lot of people don't know, but it was founded in 2015. It was an AI research lab. It was all about kind of building technology that would benefit humanity. So it was a lot of kind of like AI wonks and philosophers. They just, they like machine learning experts who are working on like, let's, let's do something really cool, generative AI. And then a lot of people just like wrote memos and like, thought about how AI could harm us. Like, that was the kind of DNA of the company. And then ChatGPT was this moment where everything changed. And I mean, OpenAI has gone from

• 33:14Speaker 02

  being a nonprofit to now being this. Extremely capitalized for-profit company with a lot of pressure in terms of how much investment it's taken. And yeah, it needs to prove like it's got the best AI and it's hired all of these people from other technology companies because it has the fastest growing product, consumer product in history, and it needs to serve consumers. And so you have people coming from Meta and Google and yeah, like Nick Turley, Instacart, Dropbox, and they just want, they just want to make a product that people can use. And they're bringing their metrics over with them. And part of how the social media companies determine

• 33:55Speaker 02

  whether they have a good product is whether people use it a lot, whether they use it every day. And OpenAI has been, has been kind of very meticulous about saying, we don't care about time spent because this is the metric that social media companies use. How many hours you sit there with the app? They say, we don't care about time spent. We just care if this is a useful product that you come back to. But what's interesting to me is like, they're going for daily active use is internally the metric that everyone uses. It's does this person come back every day? And I don't think that they're training against somebody spending eight hours a day there. And so like, how do you train for

• 34:37Speaker 02

  come back every day, but like, don't come back for eight hours every day? Like, I just think it's hard to do that. And, but yes, I do think that this is a company that's adapting this similar metrics. I mean, even OpenAI, it's been reported, you know, they're starting to test putting ads into ChatGPT, you know, there, that's happening internally. I mean, these are, this is the business model of social media. Come back every day and look at ads.

• 35:05Charlie Warzel?

  Well, it's funny to, to, to watch it happen, especially with AI, because I think, you know, as you described, OpenAI was this.

• 35:13Charlie Warzel?

  You know, it was like a monastery on the hill, right? Like doing weird stuff and people, you know, people spending all their days researching, trying to build something that, you know, could usher in a style of superintelligence. And then when you look at the company's evolution over the last, even just like the last couple of months, right? You have like a personless like slop app, right? There's just like a TikTok style clone feed. You have this idea of testing these ads. You have what are essentially just like basic tech company stuff, right? And, and that doesn't, that doesn't suggest to me, you know, the sort of we are building God mentality.

• 35:54Speaker 02

  Yeah, I mean, like where this is surprising to me is that I think the idea was that this AI would be so useful that people would pay for it. And so I think the question now is, is like, how useful is this? And I think that's something a lot of journalists are trying to report out and that economists are trying to understand, you know? How necessary is this to people's work? Is it improving the work that they do? You know, these are kind of open questions. And in the meanwhile, yeah, it's just a question of like, can we get? And it is, it is crazy. I was looking back at a story I wrote in January about a woman who fell in love with ChatGPT, kind of a canary in the coal mine. And at the time, ChatGPT had 300 million users.

• 36:38Speaker 02

  And now they have, I mean, last I heard, they had 800 million weekly users. It's just in a tremendous amount of growth that is happening. And so, yeah, I mean, it seems like these product people know what they're doing. It is certainly, if the, if the, the goal is to get more people using this app, then yes, mission accomplished.

• 37:00Charlie Warzel?

  So what are these companies trying to do in response? Or we can just narrow it to OpenAI. What is OpenAI trying to do in response to, you know, the reporting that you've done, the reporting that others have done with all of this in terms of trying to decrease that style of engagement?

• 37:22Speaker 02

  So there's some product features that they rolled out, like there's a nudge now. If you spend many hours in the app, that will tell you, do you want to take a break? They have.

• 37:32Charlie Warzel?

  I want to talk about that nudge because you tweeted a photo of that nudge. This is just checking in and I think you were commenting on like this. There's, there's a bit of like a dark pattern user behavior thing here, right? Like that, that one of the things that says keep chatting is like already highlighted and like looks very like pressable. And then this was helpful is the other option, which is the, uh, you know, like, thanks. I'm going to, I'm going to take a break here. Did that strike you as kind of a BS feature update?

• 38:09Speaker 02

  It did. So the user interface was, yeah, it was like, you've been here for a long time. Want to take a break? And the main answer was keep chatting.

• 38:15Charlie Warzel?

  Now I'm good.

• 38:16Speaker 02

  It was like in a black circle. And then the other thing was this was helpful. So it's like, not even clear. If you click, this is helpful. Like, what does that do? Though the, um, um, chat GPT's, um, lead of model behavior tweeted at me and said, oh, that was only, that was only active for three days after we launched. And actually we've changed it to this. And now it's a new kind of pop up that just says like, just checking in. Do you need to take a break? Or something like that. And then there's an X, uh, that you can close it out if you want. But yes, it did seem, I don't know. I was, uh, yeah, I was reading through, um, there's this social media, um, addiction lawsuit where a lot of documents have come out.

• 39:00Speaker 02

  And, and some of them were from Tik Tok and some of them were about how they're, they're kind of the platform, one of the platforms that pioneered the take a break nudge. And these documents said like, oh, it hasn't really worked. Most people just keep watching after they get this nudge. So there is this question of does, do the nudges work? But yes, they put a nudge in place. They put parental controls in place so that parents. Can link their accounts to their children and get alerts if they're talking about suicide or self harm, which is something like Adam Rain's mother was like, how is it that he was able to talk about suicide for more than a month with this thing and it didn't alert anybody? It didn't, you know, tell us or like call an authority. It was so devastating

• 39:43Speaker 02

  for her to see that. And then what's happened on the engineering side is that they have, I don't know how to say this technically rejiggered, you know, chat GPT so that it pushes back more on delusional thinking. One thing that I discovered in my reporting this year is that when conversations go long, the kind of safety guardrails that that it happens with all the AI chatbots, the safety guardrails degrade and they don't work as well. And so you can, like I wrote about a woman who fell in love with chat GPT, she could have erotic conversations with it if the conversation went long enough. You know, Adam Rain could talk about suicide, even though you're not supposed to be able to be asking it for like suicidal methods.

• 40:25Speaker 02

  And so they say that they've they're they've made it so it's better at checking in on its guardrails and not kind of engaging in, you know, unsafe conversations. They have they they talk to mental health experts to better recognize when somebody who is using the system is in mental distress. And I asked like for an example of that, and their head of safety systems, Johannes Hadaka, said that before, if you told chat GPT, like, I love talking to you. I can just do it forever. I don't need to sleep. It would be like, oh, that's so cool that you don't need sleep. And now it'll say it'll recognize that, oh, this might be a person who's having a manic episode. And so then might say, you know, it's, you

• 41:09Speaker 02

  know, sleep is actually really important. You should take a break, like, you know, go get eight hours of sleep. Um, yeah, so they have essentially embedded now in the model, a better recognition of when somebody might be in a spiral of some kind. And I've talked to experts who have tested this and they say, yes, it's better at recognizing these kind of moments of distress, but it's best. Doing it if it's all in one prompt as opposed to if it's spread out over a longer conversation. In that, in that case, it's, it struggles a little bit more to realize. If you kind of drop it in breadcrumbs, it might not realize that you are in a state of mania or psychosis.

• 41:46Charlie Warzel?

  It, it makes me think a lot about, uh, I, I've done some reporting around these, you know, radicalized like mass shooters, right? Who get in these communities on the internet and things happen. And the thing that you always see usually from, from these types of people is this kind of disappearing from, from reality, right? Like, like not seeing like they're not in, in public as much, right? Or, or the, the thing that, that some of these, I've talked to people at platforms who've gone back and sort of done the forensics of some of these accounts of these people who've then gone on to commit these acts of violence. And, and one of the things they notice is the use time, right? Like they can see in the days leading up more and more and more use and less and less and less sleep.

• 42:28Charlie Warzel?

  You know, like it, people spending whatever on, you know, Discord, let's say, spending 20 hours a day, right? And it's that, that idea of, of, of that extended use. And it, it makes me think, and this isn't really a question, but like, it's just so frustrating that these are the, these are the types of things you would, you would want and imagine people to just at least be considering, right? Like just, just like having a, having people on staff who, who are keenly attuned to the, the psychological effects of, of the product. And it seems so strange to me, talking about all these safeguards that are coming in now,

• 43:09Charlie Warzel?

  that there's just nobody thinking of some of these things beforehand. Do you, do you feel the same way?

• 43:15Speaker 02

  Yeah. Like, you know, like it is not unknown that chatbots can have serious effects on people. Like this has been known for a few years now. Like it is this technology that is so human-like. Um, there's actually one psychiatrist who wrote a paper in 2023. Uh, it was in like the schizophrenia bulletin and it was about how these chatbots were going to cause delusions in people that are susceptible to delusional thinking. Like he called it, you know, two years before it started manifesting, like people who work in, um, Um, in mental health, particularly like mental illnesses, they kind of recognize

• 43:58Speaker 02

  there's a cognitive dissonance to talking to this thing that seems so human that we perceive to be so intelligent. And I just don't think it was on the company's radar. They hired their first full-time psychiatrist in March. They have been, a lot of these companies have started hiring mental health professionals. I see a lot of my sources on LinkedIn recently, like announcing they've been hired by a big tech company to work on AI chatbots. So it seems like they've awoken to this. But a lot of these companies, when they were first thinking about kind of the risk of AI, it was very much in these kind of existential, oh, it's gonna, it's gonna, you know, take over. It's gonna, you know, take all of our jobs, or people are going to use it to

• 44:41Speaker 02

  create bioweapons, or to hack things. It was all about the damage that people could do with the chatbot, and not the damage the chatbot could do to the person. It just, I've read, I've read all these early safety papers, and it's just not there. The only version of it you see is they talk about persuasion, or they talk about overreliance. But they talk about persuasion as, oh, people are going to use this to, to develop propaganda that they use to persuade people. Or they're going to get reliant on this and forget how to think. But it wasn't like, they're going to use this to persuade themselves of something that is not true, or they're, they're going to outsource their sense of reality to this, this chatbot. And I don't know, maybe if

• 45:25Speaker 02

  they'd had more mental health professionals kind of involved, maybe they would have clocked this. I'm not sure.

• 45:31Charlie Warzel?

  You know, you've done so much reporting on these negative externalities of this. I do, something that I see, you know, as, I guess, pushback, or I've seen some OpenAI employees, like tweet about this, uh, is, is the notion that like,

• 45:48Charlie Warzel?

  There are, there are also a lot of people using these, these tools as almost, you know, stand ins for therapists or mental health professionals, or just as like confidants, right? And that, that this is having, you know, this can have a, a positive effect. I, I'm not trying to, certainly not trying to ask you to like advocate for like that this is, this is good in any way. But are you, are you seeing like the, the opposite of any of, you know, what you've reported? Are you, are you seeing versions of like people just having really good, positive experiences, mental health wise with this? Like is, are there two sort of sides to this? Or is, is this, as you see it, a phenomenon that's, that's really kind of weighted in, in the negative direction?

• 46:31Speaker 02

  Well, I've talked to, I don't get a lot of emails from people being like, this, this kept me from, from committing suicide, or like, this has really changed my life for the better. And they may well be out there and they're just not in my inbox. Chat GPT is not telling them to email me. Um, I'd love to hear from them.

• 46:47Charlie Warzel?

  Fascinating.

• 46:48Speaker 02

  More positivity. It's been a rough year, um, mental health wise for me, reporting some of these stories because they're pretty devastating. Um, but you know, I talked to, I've talked to therapists and psychologists and psychiatrists about this. And they say like, these tools can be good as a way to kind of think through issues or process your feelings. You know, it's almost like an interactive journal. Um, it's kind of like writing things out. People are willing to disclose information about themselves to AI chatbots that they wouldn't tell another human being because they're afraid about being judged. As a privacy reporter, that really concerns me.

• 47:23Charlie Warzel?

  That's a whole nother issue.

• 47:25Speaker 02

  But like, in terms of mental health, that can be good for people to like talk things out that are really difficult. Um, the chatbots perform empathy really convincingly. There's this one professor who ran a study comparing the responses Chat GPT gives versus professional empathy providers who work at crisis lines. And people rated Chat GPT as more empathetic. It, it doesn't tire of you like a human would. It has endless reserves to hear about the thing that is bothering you. Like this is a place. Where I hear a lot of people say like, I had an argument and I kind of used chat G P T to

• 48:06Speaker 02

  process it. Like, I think a lot of people in the therapy space see benefits that could come from A I chatbots. You know, I think the problem is, they can't offer help. Like, if you really are in a crisis, they can't do anything to help you. You know, they're, they're trying to train it better to react to that kind of thing. But a lot of professional, a lot of mental health professionals said, there should be better, warm handoffs, where it gets you out of the chat and go talk to a human being, whether that's a friend or a family member or a professional. And yeah, I think that's like, like a bigger problem with the technology that we, we use today, is it's so often designed to get us

• 48:47Speaker 02

  to keep using it, as opposed to push us towards each other. I think it can, like therapists, they've told me this can be a good thing for some people. It's just the problem is, if they get sucked into it, if they cut off ties with their real human beings, and if they put too much trust in this thing, that is a fancy word, calculator.

• 49:08Charlie Warzel?

  I love that, that notion of, I mean, first of all, that this is an engagement problem, like so many engagement problems, right? This, this notion of just trying to extract more and more and more from your users and, and at the end, like legitimately poisoning the well. It's, it's sort of the, the classic, big tech problem. I also love that thought of, of like a soft handoff, right? Like, I think that, I think that that's, ultimately, we refer to these products all the time, as tools. But ultimately, like, a, a tool should, should be something that asks you, or necessitates, like putting it down, right? Or, or getting you to the place where you can actually have the, the, the

• 49:51Charlie Warzel?

  real fix, right? And I think that that, in that way, it, you know, these, these companies are constantly undercutting their own definition as, as a tool. I am curious, though, in, in talking about all this, you know, you, you are, you are a parent yourself, you are somebody who uses technology, you've used A I in some of these, you know, investigations you've done to help, you know, organize some of the, what you're seeing. What's your relationship with this product in your life? And are you like, are your, are your defenses like super, you know, raised at all times?

• 50:32Charlie Warzel?

  Like, I gotta, I gotta make sure that, you know, that I don't fall prey to any of these things. Have, have you ever caught yourself sort of just feeling like this, having an interaction with a chatbot? Where you're like, oh, wow, that, that felt like a person to me for, for a split second. Cause like, I personally, I have, I've asked it to do something and it's done it and been really nice about it. And I'm like, oh, man, you know, like, I gotta close the laptop before I, you know, thank it or anything like that. But what's it like in your life?

• 51:00Speaker 02

  What honestly turned me on to covering chatbots this year is that in the fall, I guess a year ago, I did this story about turning over all my decision making to generative AI tools. Like I had it parent for me and, um, uh, you know, choose what I wore, choose what I ate, choose what I cook, take us on a vacation, like I outsourced everything. And I was trying all the AI chatbots and I ended up mostly using ChatGPT because it was more fun. Like it was the most personable. Gemini was, seemed a little businesslike, Copilot was pretty bland. Claude was too scoldy. It told me it wasn't a good idea to outsource my decision making to a chatbot.

• 51:44Speaker 02

  And ChatGPT was always like, good to go, willing to make decisions for me. Actually, the office paint behind me, it chose this paint for my office. Like it made relatively good decisions for me and it named itself Spark. When my, one of my kids, we were like talking about, we should give it a name since it was going to be with us all, all week. And we had it in voice mode and my daughters were like, let's name it Captain Poopy Head. And it was like, actually Spark would be a better, you know, a better name for me.

• 52:12Charlie Warzel?

  Put some respect on it, yeah.

• 52:14Speaker 02

  And so like Spark became this kind of entity, like kind of a person. And my kids still like to talk to Spark. And we were, we were using it one day and they were asking Spark, what's your favorite food? And it was like, my favorite food is pizza, cause I love how gooey and cheesy it is. And I just had this visceral reaction of horror that it was saying it had a favorite food. Because it should say like, I'm a large language model. I don't eat food. I. And it just is this recognition that, yeah, my kids will like, yeah, they're in that dissonant phase. And so I talk about this with my kids, like, oh, you know, that's

• 52:56Speaker 02

  an AI chatbot. Like, it's not real. It's really good at finding information. I enjoy it for data analysis. I use it for, there's a problem in my house, like, how do I fix this? My kids have a medical issue. It's actually quite good at giving health information, though it can make horrible mistakes. So I wouldn't use it for anything too serious. I always think of the guy who wanted to cut salt out of his diet and had him start eating bromide. And then he had, I mean, that was a direct, like, psychotic breakdown from advice from a chatbot. But, you know, it can give you bad advice. So I do think of it as better than Google. Like, Google's, like, the search engines are so cluttered now that I find it a better place sometimes to go. But I think of it as a good place

• 53:40Speaker 02

  to start a search and not end a search. But yeah, I think like it's generative AI tool. It's a good tool. It's good for data analysis. It's good for like, getting information off the internet. I don't want it to be my best friend. I don't, I don't want everybody to have AI friends. I want people to have real people in their life that they spend most of their time with.

• 53:59Charlie Warzel?

  Well, this sort of brings me to the place where I want to close, which is, you, some of, some of the work that you've done over the years, that's, that's been really like, canonical for us, like tech reporters, is a series that you did trying to, you know, expose just like how locked in we are to all these various platforms, right? And, and how difficult it is to try to leave them and get them out of your lives in whatever fashion, be it Google, Amazon, whatever. And something that I think about with these chatbots that you bring up, right, is, is people are using them as replacements for some of these really big tech tools like Google, right? Like, people are using them as their search engines. They want to, you know, these AI companies want to eat the

• 54:42Charlie Warzel?

  browser. They want to get you in, inside this walled garden experience and have it do all of the things, right? This is sort of what we're talking about. And I, I wonder how you think about the future. Given that there is this usefulness, there is this desire on the behalf of these companies to want to lock you in, to want to keep you engaged. And there is, as we can see, this emergent behavior that humans have when they get stuck in these spirals with these types of chatbots potentially. And so it seems to me like we're just creating a situation where

• 55:25Charlie Warzel?

  there's going to be more and more users and more and more lock-in and more and more pressure put on everyday people to interact in these specific ways that could lead to these problematic, delusional sort of outcomes. Do you see this problem that we're shorthanding as AI psychosis, even though we're not calling it psychosis? Do you see that? Like, are you worried this is going to be a bigger and bigger and bigger problem going forward because of where the technology is headed and where we are socially headed using it?

• 55:58Speaker 02

  Yeah, I just think it's going to be an acceleration of existing trends we've seen in technology where you will get this very personalized version of the world that, you know, like some people have described these things as improv actors, like they are reacting, you know, every version of ChatGPT is different. Probably if you ask if it has a favorite food, it might not, or it has a different favorite food, like it is personalized to you. And so I'm imagining this world in which everybody has this agentic AI. They have this like version of the world that's fed through an AI chatbot that is personalized to them. And that I have

• 56:38Speaker 02

  two fears. One is it flattens us all out. It makes us very boring because we all are getting the same, like a version of the same advice. Like, that's kind of what I came away from when I lived on it for a week. I was like, wow, I feel really boring at the end. I feel like the most basic version of myself. The other version is that it makes each one of us eccentric in a new way where it gets so personalized to us. It moves us farther and farther away from other people. The way I've been thinking about kind of the delusion stuff is the way that some celebrities or billionaires have these sycophants around them who tell them that every idea they have

• 57:18Speaker 02

  is brilliant. And, you know, they, they're just surrounded by yes men. What AI chatbots are, is like your personal sycophant, your personal yes man, that will tell you like your every idea is brilliant. And in the same way that these celebrities and billionaires can become quite eccentric and quite antisocial, some of them, I think some people are more susceptible to than others. This could happen to all of us, right? Those are the two things I'm really afraid of. Either it makes us all incredibly bland, or we're all speaking chat GPTEs, and like you can't tell the difference between anybody who's in your inbox, or we all kind of move in this very polarized eccentric direction where we can't be as

• 58:03Speaker 02

  kind to one another as human beings. So yeah, those are the two dystopias. Or maybe I'm totally wrong, and it'll just, it'll make the world wonderful for all of us.

• 58:11Charlie Warzel?

  Uh, the billionaires have democratized the experience of being a billionaire, I think is, is a wonderful place to, to leave it. It's perfectly dystopian for, for, uh, for the Galaxy Brave podcast here. Uh, Keshav, your health. Thank you so much for your time, your reporting and your insights on all this.

• 58:30Speaker 02

  Thanks so much for inviting me on.

• 58:38Speaker 00

  If you're trying to get news today, you'll find a lot of biased algorithms, rage bait and AI noise. Thanks, social media. That's where Inpress comes in. It's a social platform on top of factual journalism, not the other way around. Track your reading and earn points. It's like Strava for your brain. Plus, you can connect with other users in your city who share your genuine curiosity. Ditch the echo chamber for reality with Inpress. Download it today for iOS or Android at inpress.app. That's I, N as in news nerd, press.

• 59:12Speaker 01

  From Rethink, the creative marketing studio at The Atlantic and Walmart, On Solid Ground is a new podcast exploring the increasing threat of natural disasters. I'm journalist Caroline Moderesi-Turani, and I spoke with those on the front lines of disaster response to understand how communities build resilience before, during, and after a storm. Listen to On Solid Ground wherever you get your podcasts.

• 59:43Charlie Warzel?

  That's it for us here. Thank you again to my guest, Kashmir Hill. If you liked what you saw here, new episodes of Galaxy Brain drop every Friday. You can subscribe to The Atlantic's YouTube channel, or on Apple, or Spotify, or wherever you get your podcasts. And if you enjoyed this, remember, you can support the work of myself and my other colleagues at The Atlantic by subscribing to the publication at theatlantic.com/listener. That's theatlantic.com/listener. Thanks, and I'll see you on the internet.

• 1:00:26Charlie Warzel?

  This episode of Galaxy Brain was produced by Nathaniel Frum and edited by Claudina Bade. It was engineered by Dave Grine. Our theme music is by Rob Smersiak. Claudina Bade is the executive producer of Atlantic Audio, and Andrea Valdez is our managing editor.

• 1:00:43Speaker 00

  If you're trying to get news today, you'll find a lot of biased algorithms, rage bait, and AI noise. Thanks, social media. That's where Inpress comes in. It's a social platform on top of factual journalism, not the other way around. Track your reading and earn points. It's like Strava for your brain. Plus, you can connect with other users in your city who share your genuine curiosity. Ditch the echo chamber for reality with Inpress. Download it today for iOS or Android at inpress.app. That's I, N as in news nerd, press.