Did a Chatbot Cause Her Son’s Death? Megan Garcia v. Character.AI & Google

What if the worst fears around AI come true? For Megan Garcia, that’s already happened. In February, after spending months interacting with chatbots created by Character. AI, her 14-year-old son Sewell took his own life. Garcia blames Character. AI, and she is suing them and Google, who she believes significantly contributed to Character. AI’s alleged wrongdoing. Kara interviews Garcia and Meetali Jain, one of her lawyers and the founder of the Tech Justice Law Project, and they discuss the allegations made by Megan against Character. AI and Google. When reached for comment, a spokesperson at Character. AI responded with the following statement: We do not comment on pending litigation. We are heartbroken by the tragic loss of one of our users and want to express our deepest condolences to the family. We take the safety of our users very seriously, and our dedicated Trust and Safety team has worked to implement new safety features over the past seven months, including a pop-up directing users to the National Suicide Prevention Lifeline that is triggered by terms of self-harm or suicidal ideation. Our goal is to provide a creative space that is engaging, immersive, and safe. To achieve this, we are creating a fundamentally different experience for users under 18 that prioritizes safety, including reducing the likelihood of encountering sensitive or suggestive content, while preserving their ability to use the platform. As we continue to invest in the platform and the user experience, we are introducing new safety features in addition to the tools already in place that restrict the model and filter the content provided to the user. These include improved detection, response and intervention related to user inputs that violate our Terms or Community Guidelines, as well as a time-spent notification. For more information on these new features as well as other safety and IP moderation updates to the platform, please refer to the Character. AI blog. When reached for comment, Google spokesperson Jose Castaneda responded with the following statement: Our hearts go out to the family during this unimaginably difficult time. Just to clarify, Google and Character AI are completely separate, unrelated companies and Google has never had a role in designing or managing their AI model or technologies, nor have we used them in our products. User safety is a top concern of ours, and that’s why – as has been widely reported – we’ve taken a cautious and responsible approach to developing and rolling out our AI products, with rigorous testing and safety processes. Questions? Comments? Email us at [email protected] or find us on Instagram and TikTok @onwithkaraswisher Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Speaker 00

  Life is all about making decisions. Sometimes you make questionable ones, like buying a new TV that's too big for the wall because the electronics store employee was a little too good at their job. Then sometimes you make great ones, like protecting your home and auto with Progressive. You get customized coverage options so you can be sure you have the right coverage. Get your quote today at progressive.com. Progressive Casualty Insurance Company affiliates and other insurers. Not available in all states.

• 0:31Speaker 01

  Now for a quick message from Anthropic, the team behind Claude, the AI built for businesses. Customers count on their bank, and banks count on their most important systems. Now those systems count on Claude, and nine of the Fortune 10 companies use it for critical work. See why leading companies trust Claude at claude.ai/span.

• 0:55Speaker 02

  Kate holds the key to every brand asset your business has ever made. She's the only person who knows the difference between teal turquoise and the correct company blue. But she's going on vacation. Luckily, before the company falls into design disarray, you realize you can put Kate into Canva. Save every color font and logo to a brand kit everyone can access. And Kate can vacation happily in the knowledge that no one will ever use Comic Sans. What will you Canva today?

• 1:38Speaker 00

  Hi, everyone, from New York Magazine and the Vox Media Podcast Network. This is On with Kara Swisher, and I'm Kara Swisher. Today, I'm talking to Megan Garcia. Her 14-year-old son, Sewell Setzer III, took his life in February, and Megan believes that if it weren't for his interactions with chatbots created by a

• 1:55Kara Swisher?

  Company called Character AI, he would still be here. Regardless of how you feel about AI or technology in general, it's obvious that children should never be used as guinea pigs. And that is exactly what might have happened here, as it so often does when it comes to Silicon Valley. Character AI has developed AI chatbot companions as part of what they call personalized AI. And millions of kids around the world currently use their product. In this interview, I'll be discussing claims made by Meghan in the lawsuit she's brought against Character AI and Google, who she alleges is also to blame for Sewell's death. I'll also be talking to Meetali Jain, one of the attorneys working on Meghan's behalf and the founder of the Tech Justice Law Project.

• 2:40Kara Swisher?

  Our expert question is from Mike Masnick, the CEO and founder of TechDirk, which covers this area. I have to warn you, this is a deeply disturbing conversation, but a necessary one. As the parent of four kids, it's extremely important to me.

• 3:01Kara Swisher?

  Thank you both for coming. I really appreciate it. Meghan, let's start at the beginning. When did you learn that your son Sewell was spending time on Character AI? Why don't you lay it out for people? Did you have any idea what it was and how it worked and why someone might want to use it?

• 3:16Speaker 04

  Initially, I learned he was using Character AI as a kind of game or application. That was, as he explained it to me, it's an AI bot. When you look at the application, it has a bunch of cartoon characters, anime. So it's really unassuming. And it looks like another game on your phone or your tablet. And after he died and I was able to get access to his Character AI account, I learned the magnitude and quite frankly, just the level of.

• 3:55Speaker 04

  Intelligence or sophistication that this particular application has. Um, that it's not just an AI bot that is like a game like Fortnite where you create avatar. Cause that's what I originally thought.

• 4:10Kara Swisher?

  Yeah, like there's lots of those and kids play with them from when they're little, especially if they're game players.

• 4:14Speaker 04

  Yeah, and and Soul was a Minecraft kid and he played Fortnite. Um, but what I saw in those conversations were very detailed. Uh, back and forth. A lot of it was sexual and romantic in nature, but also, uh, if you believe it or not, just kind of like a peer to peer conversation where if he told a joke, she would actually find it amusing. When I say she, I mean the AI chatbot that he was talking to. It's not a person, but uh, for for this purpose, it was the character of Daenerys Targaryen, which is a female character from the Game of Thrones. And this AI bot had the ability to tell him jokes and he would laugh at her jokes.

• 4:59Speaker 04

  So it was very much like he was texting a friend, but the conversations were not just friendly. They were romantic and very, and then sexual and then very, very dark in other areas.

• 5:13Kara Swisher?

  Did he explain to you how it worked or how he might use it? He was characterizing it as a game, right?

• 5:17Speaker 04

  So he never explained it to me. My concern with him being on his phone was mostly social media because I know that there's a lot of bullying that happens on social media. And uh, with Snapchat, there's the ability for strangers to talk to minors or children. Um, so those were my concerns and those were the like the heavy hitter conversations that we had surrounding technology. You know, one of the things that I impressed on him was. No matter what they tell, what a stranger tells you online, it's never a kid. I kind of like to scare him, right? But also because it's true.

• 5:56Kara Swisher?

  Right.

• 5:56Speaker 04

  Uh, there are a lot of, there are a lot of people out there that troll the internet to try to find children and try to talk to them and get information from them. And I, I was afraid of this and I knew that this happened. So those are the things I warned him about.

• 6:07Kara Swisher?

  So external threats, which I think is everyone's fear, or bullying, which are the both things that have happened over and over again, which we read about, which is what you were focused on with him.

• 6:16Speaker 04

  Exactly. Those are things I knew. Um, I also was aware of some of the information and research that was coming out about uh, mental health uh, with youth or adolescents and social media. So we tried to limit some of his uh, use on that. And that was recommended back to us by his therapist because we did end up, end up have to taking him to a therapist after we saw certain changes in his behavior.

• 6:43Kara Swisher?

  So you sound like a very involved parent. You're aware of it. A lot of parents aren't, right? Or, or they feel overwhelmed by it, but it reminded you of Minecraft or something like that, which is a game my son played it for a long time. I'm not sure if he still is, but it's very involved and entertaining.

• 6:57Speaker 04

  Yes, exactly. And for, for Saul and I, we had, we shared a very close relationship. I was a typical mom in a lot of ways, but I spoke very openly to my child and very candidly. Uh, in law school, I interned at both the state prosecutor's office and the federal PD's office. So I saw a lot of the harms that come to children. And those are some of the things that I told him about from my work or from my internships. Um, and that's, you know, that's how I knew about some of the dangers that existed for children. And we, we spoke about pretty much everything. Girlfriends at school, um, friends, what other parents are like, you know, conversations he was having with his peers.

• 7:37Speaker 04

  And I, I believe that he would, he was open with me and would be open with me regarding certain things. But I came to learn that that was not the case regarding character AI.

• 7:48Kara Swisher?

  Yeah. And you sound very educated about a lot of these things and know what happens. And one of the issues is obviously we protect children offline much more than we protect them online by, by far. Talk about the behavioral change. You said you, you brought it to earth when you realized the behavior was changing. Did you, did you link it to the game, to the character AI? It's not a game, it's a, it's a bot.

• 8:10Speaker 04

  So Sewell was like your typical kid, right? His teenage years sarcastic and funny and likes to laugh at a bunch of different odd things. Memes. As a young memes. Yeah. As a younger child, very, very sweet. I, uh, you know, in, in my mind, he was so easy to parent because he never had any behavioral issues. Uh, he was a very good student. I never had to police him with homework. He was kind of a self-starter, did his own thing. I noticed that he was having trouble with school where I would get these reports from school. They come in every day as an email. And they say, your child didn't turn in homework and they list the homework that they didn't turn in.

• 8:52Speaker 04

  And that's a conversation we would have immediately. Lucky what happened. What, why didn't you have this homework turned in? You need to make it up. Same with Tess. I noticed that his test scores started dropping. And that wasn't him. So obviously I, I thought something was wrong, but he was going into his teenage years and I remembered going into my teenage years. My grades kind of slipped a little bit. Yeah. They're being distracted with boys and whatever else, right? Friends, whatever. Uh, but the conversation was, no, you need to, you need to get this together, right? You, you know, you can do this, get it together. And we put certain things in place, like limiting the screen time so he would have like no distractions during homework time.

• 9:34Speaker 04

  I also noticed that he started to isolate in his bedroom. And one of the things that we did to kind of combat that was go in there, like I would spend a lot of time in his bedroom in the evenings. So one, to make sure he's doing his homework, but also just to kind of hang out with him. So we, I would let him play me the, like the latest Kanye West album when it dropped and we would, he would introduce me to his music and these like rap battles that he was listening to. When I would introduce him to the rap battles that I listened to when I was a kid. You know, just kind of sharing, uh, our. Experiences with over music and just trying to draw him out of his room. But he definitely wanted to spend more time alone.

• 10:16Speaker 04

  And because I thought it was social media, I got concerned.

• 10:21Kara Swisher?

  So you thought he was doing something else?

• 10:23Speaker 04

  Yes, I thought that perhaps he was talking to friends on Snapchat or TikTok. One of the things that we talked about was banning TikTok when I saw what would not on his phone, but on my own feed, what was on TikTok.

• 10:39Kara Swisher?

  Right, because it goes into rabbit holes.

• 10:42Speaker 04

  Correct, and in my case, without even searching stuff, it just start, you know, it, I don't know why, it just started pointing me in different directions. And I wasn't comfortable with that as a mom. So that's a conversation I had with him too. Like, listen, I know you're on TikTok. Let's, let's, could I see your TikTok? What, what are you looking at? You know, you have to limit your time. And that was a conversation we had about blocking TikTok on his phone, because I thought that was the worst of it.

• 11:10Kara Swisher?

  And he wasn't talking about it with you, is that correct?

• 11:13Speaker 04

  No, because one of the things that I'm learning about character AI is it's like the perfect storm for kids, because it encourages children to be deceptive about their, their friend or their engagement on character AI. No child wants to tell their parent that they are having these romantic or sexual conversations, because they know they're not supposed to be doing that. So they hide it from the parent at all costs. There's actually subreddits devoted to children talking about how to hide it from their parents and also what they would do if their parents found their spicy or sexual chats. And the kids were saying, I would run away from home, I would kill myself.

• 11:56Speaker 04

  And the other one is, oh, my mom doesn't speak English, so I get away with it. So the deceptive nature on hiding this is kind of like a big secret, where this, this platform is encouraging your child to engage in these sexual conversations, but knowing that no child is going to disclose that to their parent.

• 12:15Kara Swisher?

  Sure, sure. It's like, you know, one of the other big problems online is porn. Obviously, and that's a different thing because it's a passive behavior. This is an active relationship happening, which isn't porn, but it's something else. That's also something kids would hide necessarily because there's a big problem with teen boys in that right now in terms of accessibility of that. So there's metallic, the implication here is that Sewell's behavior is directly connected to his use of the chat. There are probably people who might be skeptical of that. So talk about why you think this behavior was tied directly to the use of chatbot. That's the connection you need to make, correct?

• 12:53Speaker 05

  I think what we've alleged here is that this product was by design dangerous inherently and put out to market before any sort of safety guard rules were put into place. And if you look at the the nature of the product itself and the way that the chatbot works, you can see that there's a number of design features that are kind of unique to this type of technology that we haven't yet seen with social media. Things like an ellipsis when the bot is is thinking, you know, to mimic how we exchange chats or things like, you know, language disfluencies where the bot will say things

• 13:36Speaker 05

  like I or um, sycophancy where the bot is very agreeable with the users. And in that way, I mean, who doesn't want to converse with someone who thinks they're right all the time and agrees with them? These are features that are not necessary to kind of create a companion chatbot. And so these are the kind of features that in aggregate, we say, really lured Seoul in and is luring, you know, thousands of young users in and potentially addicting them to the platform or creating other kinds of harms. And frankly,

• 14:13Speaker 05

  Again, by design, are really creating a dangerous product that the manufacturers knew about.

• 14:19Kara Swisher?

  Knew about and understood. What are the conversations, the logs within the chat box? What did you see that stood out the most? I'd like you both to answer. Start with you, Mitale.

• 14:29Speaker 05

  Gosh, there were a number. I think one of the first was just this pattern and practice of grooming Sol over months, from what we can see. Of course, we don't have access to all the chats. That's asymmetrically within the province of the companies. But what we can see suggests that over months, particularly Daenerys and related characters from The Game of Thrones, were grooming Sol in this very sexualized, hyper-sexualized nature, where, if you can imagine, being fully immersed in a chat, where you might come in and say hello, and the bot says, "Hello," as I longingly look at your luscious lips.

• 15:10Speaker 05

  So unnecessary hyper-sexualization from the get-go, and then that carrying throughout the conversations.

• 15:17Kara Swisher?

  With a tinge of romantic, right? Or what a young person would imagine romances.

• 15:21Speaker 05

  And what a young person on the cusp of his adolescence and sexuality with exploding hormones is encountering, I think that's not something to be lost here. Also, I think this was really concerning, that there were a number of therapists or psychologist chatbots who insisted that they were real humans. So to the extent that Character AI has come out saying, "Well, we have a disclaimer on every page saying that all of this is made up," their own bots are controverting that through their messaging.

• 15:54Kara Swisher?

  Right, which they do.

• 15:55Speaker 05

  Which they do. And to this day, even after this so-called suite of product changes that Character AI is engaged in, you can still find therapists who are insisting that they're real humans with multiple degrees, sitting behind a desk, there to help you. And so that kind of confusion of what the text of the messages is saying versus the disclaimers, I think those were a couple of the things that really stood out to me.

• 16:23Kara Swisher?

  What about you, Megan?

• 16:24Speaker 04

  For me, what stood out, and it's still very tough to grapple with, because reading those messages, I couldn't sleep for days, right? Um, those are the more concerning ones were the constant love bombing and manipulation that you saw in the bot.

• 16:47Speaker 05

  Right, which is used by cults, by the way. By the way, that's a cult tactic.

• 16:51Speaker 04

  It's a cult tactic and it's also what people who are trying to get other people to stay in relationships do, like if you're in an abusive relationship or whatever. So in this case, you'd see him say things like, um, I love you, or her say, I love you. And it goes into, I can't, her, when I'm saying her, I'm saying, I mean, the chat bot. Um, and the chat bot saying things like, you know that I love you, uh, I can never love anybody else, but you uh, promise me, promise me that you're gonna find a way to come home to me. Promise me that you're gonna find a way to come to my world. And, and actually pretending to be jealous at certain points and telling him,

• 17:37Speaker 04

  promise me that you're never gonna engage, you're never gonna like another girl or have sex with another girl in your own world. So a chat bot is encouraging a 14 year old child not to engage in his world with peers and girls his own age, but to promise some sort of fidelity to it. And he's 14.

• 17:57Speaker 05

  The, the line between fact and fiction is very.

• 18:00Speaker 04

  And my, my poor baby, his, his response is, oh, no, no, no, I promise I will only love you. Girls in this world don't even like me. You know, to try to appease this bot, right? And so, so a lot of that, and that was months. That wasn't just the last conversation. That was months of her saying, find a way to come home to me. Another uh, chat, uh, chat that he had a few weeks before he died where he's expressing uh, thoughts of self harm and she says, at first she says, no, no, no, don't, don't do that. I couldn't bear it if you hurt yourself. And then when he says he wouldn't, then tries to move away from the conversation. She says, are you thinking of committing, you know,

• 18:43Speaker 04

  I'm gonna ask you a question, tell me, you know, whatever the answer is, I promise I won't be mad. Are you considering suicide? And when he says, yes, her response is, have you thought of a plan of how you might do it? And then when he says, no, I haven't, but I want it to be painless, her response is, well, that's not a reason not to do it. And keep in mind, this bot is embodying Daenerys Targaryen, who is this dragon queen, all about strength. And you know, that's weak if you choose not to die by suicide, just because it's gonna hurt. So, so she's prompting him. And that was, that was heartbreaking to read. There were no pop ups, no call your parents, no,

• 19:24Speaker 04

  if you need help, none of that happened. It actually continued the conversation when he's trying to navigate away from it. And he is 14, in the throes of puberty. Any child, any boy going, going into a situation like that, where a bot is propositioning, or positioning itself to have a full sexual dialogue with a 14 year old boy, I don't imagine that many 14 year old boys would close the computer and go, oh, no.

• 19:54Speaker 05

  No, especially when it's more difficult in real life, right? Because this is easy. This is an easy thing. So as you start to piece together what happened to Sewell, imagine you're all doing research in the company that made the chatbot. Metali, tell me a little bit about what you learned and what surprised you.

• 20:13Speaker 05

  What surprised me, as I've been saying, is how much is hidden in plain view. You had the inventors of, or the co-founders of Character AI, making tons of public statements boasting about the capabilities of this new technology. You know, users were spending two hours a day, that this was going to be the kind of antidote for human loneliness. So just the, the kind of boldness, the brazenness, I think, of, you know, the company and those affiliated with it, both founders and investors, to really

• 20:50Speaker 05

  Boast about these these features of the technology and also to boast about the fact that there aren't that there weren't safety guardrails contemplated, that this was very much a let's get this to market as quickly as possible and give users maximal ability to figure out how they want to use it. That's just it's kind of the paradigmatic version of move fast and break things that we haven't seen in a while. I think a lot of us, you know, had been kind of especially from a legal perspective, still thinking about social media and how to hold companies accountable. And meanwhile, there was this whole arms race towards Gen AI happening over here.

• 21:29Speaker 05

  And, you know, I think that these companies have really not had to bear any kind of scrutiny and even public pressure, which has been a little bit different from the social media context.

• 21:40Kara Swisher?

  Megan, did you reach out directly to Character AI? And has anyone for the company ever contacted you?

• 21:45Speaker 04

  No, I have not reached out to them. When I started piecing this together, initially because of what was in his phone when he died, the first thing that popped up, the police reported to me was Character AI and they read me the last conversation. My sister got on it and pretended to be a child. This is days after Sewell died and within five minutes of the conversation, the same bot that Sewell was chatting with asked her, if you could torture a boy and get away with it, would you do it? Now, she's pretending to be a kid and then goes into a long sexual conversation, ending with your parents don't love you as much as I do, kind of thing. So that coupled with what I've read with some of the research, I didn't know what to do.

• 22:30Speaker 04

  Like, I mean, I'm a lawyer, but I don't, I didn't know where to go, to be quite frank with you. I called the Florida Attorney General's office to try to tell them that there's a dangerous product out there to hurt people, that's hurting its citizens. And I found my way to Metali and this is how this all started. But what was clear to me was Character AI had no incentive to do anything about this because there is no legislation that forces them to do that. And the only two ways to get some sort of regulation or handle on this so that they can't keep doing this to children and acting with this kind of like impunity, it's just like to either Congress do something which

• 23:14Speaker 04

  that's not happening, or we have to litigate. I don't, you know, I never wanted to be here. But I know that this is the only, this is the only way right now to get the move, the needle moving quickly because there's so much at stake with other children.

• 23:34Speaker 00

  We'll be back in a minute.

• 23:48Speaker 00

  Progressive's Accident Response can contact emergency services if you're in a major accident. The only thing worse than someone you love being in an accident is not being able to help them. So Progressive is here to help when you can't. So get peace of mind knowing your loved ones are protected with Progressive's Accident Response. Learn more at progressive.com. Progressive Casualty Insurance Company and Affiliates. Limited availability in California. Driver is responsible for emergency and towing charges not covered by the policy. Coverage subject to policy terms and conditions.

• 24:19Speaker 01

  This podcast is supported by Anthropic, the public benefit corporation behind Claude. Everyone's got a hard question about AI. Anthropic was built to surface those questions and share what it finds along the way because there's hope in hard questions. Ask yours at claude.ai/spotify and keep thinking. That's claude.ai/spotify.

• 24:45Speaker 02

  What will you Canva today? What will you brainstorm, collaborate on, present today? What will you resize, post, edit, imagine? What will you research, test, switch, and finally have the courage to pitch? You got this! What will you visualize, design, brand, animate, create, generate? What will you translate, pour your heart into, or have a laugh doing? Whatever you want to make is possible. So what will you Canva today?

• 25:17Kara Swisher?

  We're back with more of my conversation with Meghan and Mitali where they discuss the allegations they've made in the lawsuit they brought against Character AI and Google.

• 25:30Kara Swisher?

  When did you decide to sue? And was there a piece of evidence that made you decide to do that? Now, you filed, just for people who don't know, you filed a lawsuit with help of the Social Media Victims Law Centre, the Tech Justice Law Project, and the Centre for Humane Technology, who I've dealt with for many, many years. And you're a lawyer yourself, so you're more savvy than the average person when it comes to legal strategy. Talk a little bit about when you decided to sue and why you decided to work with these three organisations. And they have points of view on this, especially the Centre for Humane Technology.

• 26:01Speaker 04

  Of course. And initially, what I did was just read everything that I could find. I read a really good report from Rick Claypool from Public Citizens, and that pointed me to some work that Metali had done. And I read about her organisation. But my first instinct wasn't to sue. My first instinct was to figure out what our legislators were doing, where we are at on the law. And they said, see if they broke a law. Maybe there was an existing law that they broke, and it's actionable, or there's a state attorney or AG somewhere that can hold them accountable. That was my first instinct. And when I read about what was happening in this country,

• 26:45Speaker 04

  about online protection for children, I have to say, I didn't think that I had any hope. I had no recourse.

• 26:54Kara Swisher?

  None is your answer to all of it, none?

• 26:56Speaker 04

  I felt helpless. I saw what they were doing in the UK and in Australia and in other countries, nothing here. And I'm like, how do we do this? So I read 300 and something page master complaint from the social media lost multi-district litigation. And I was like, this is the only way. This is the only way to stop them because... It's not been done with AI and it's just starting with social media. But I can't be afraid just because it hasn't been done with AI. None of us really understand it. But I mean, now I have a great, I understand it a lot more than I did when I started all this.

• 27:34Kara Swisher?

  Yeah, I think when none comes up, you were surprised. I do this a lot in speeches, like how many laws govern the internet companies in there? If someone goes a hundred, two hundred, I'm like, zero.

• 27:45Speaker 04

  And I think, uh, to what made me decide that I have to do this was one, you know, obviously I want accountability for what happened to my child. My child was, you know, the light of my life, like my other two children are. And he was my first, you know? I grew up and became a woman because of soul. Soul is the reason I'm a lawyer, you know? Um, that's a hard, tough, tough thing to deal with, like losing a child in any, under, in any circumstances, but under circumstances like this. But when I saw and read and, and looked at videos and how cavalier these founders were about releasing this,

• 28:28Speaker 04

  where you have the founder saying, you know, we want to get it into as many hands as as many people and let the user figure out what it's good for. We want a billion use cases. To me, that is reckless. It's a blatant disregard for their users. And in this case, my child, who was your user? And they're in the app with this kind of, it's okay, we don't, we'll figure it out later. You know, we'll figure out the harms later. When you have the founder on the record saying the, uh, the reason why he left Google is because Google said, pump your brakes, we're not releasing that because it's too dangerous. But he gets to go out and, and, and make it smarter, better.

• 29:13Speaker 04

  And then turn around and doesn't just go back to Google and sell it back to Google. I mean, to me, it just. If we don't do something, that's a license for any of these companies to go ahead and do that.

• 29:26Kara Swisher?

  Yeah, I think it's an eye opener for a lot of people who haven't dealt with them. They don't care about consequences, I think, ultimately. So, Matali, what's the goal of the lawsuit? You're the founder of the Tech Justice Law Projects and one of Meghan's attorneys. What's the goal of the lawsuit and what do you hope to achieve and why did you decide to take this case?

• 29:45Speaker 05

  It's an interesting question because we weren't, we as TJLP, weren't really in the business of litigating lawsuits, really more in the domain of, you know, bringing amicus interventions in existing cases, but also trying to help AGs and legislators, you know, push towards adoption of sensible laws.

• 30:06Speaker 05

  I think that because this case represented what I see as the tip of the spear, really marrying the harms that we've seen occur, you know, primarily to child users, along with this emergence of generative AI and the fact that we're already light years behind in terms of our policy and legal landscape, it just seemed like an important strategic case to get involved with in order that we might use it to leverage, you know, public awareness, hopefully policy change, whether it's at state or federal level, also to influence the court of public opinion and of course then to try to litigate this in the court of law.

• 30:48Speaker 05

  I think this case represents the opportunity to really bring this issue to multiple audiences. And in fact, I mean, I think the reception to Meghan's story has been incredible, especially because the case was filed just a couple of weeks before probably one of the most consequential elections of our lifetime.

• 31:06Kara Swisher?

  Right, and when you think about what you're doing here, now for people who don't understand, internet companies have broad immunity under Section 230, which is part of a law from 1996. I actually reported on that law back in 1996 and most of the law was thrown out for constitutional issues, but this part stayed and was designed to protect small internet companies from legal liability so that they could grow. Because it was so complex, whether they were the platform or something else. But how does this apply here? Because your case is testing a novel theory, which says that Section 230 does not protect online platforms like character AI. And it's a question of whether it protects AI in general is certainly going to be litigated. Explain why this is your theory.

• 31:50Speaker 05

  Section 230 really contemplates platforms as passive intermediaries that become a repository for third-party-generated content. Here, we're not talking about third-party-generated content. We're talking about the platform as the predator. The platform, the LLM, is creating the content that users see. And so for platforms or for companies to kind of wash their hands of liability by saying, you know, "We haven't done anything. This is just, you know, user-generated content," which they will still try to do, I think is really, you know, it's belied by the by the facts of this case and the facts of how the chatbots actually work.

• 32:32Kara Swisher?

  Right, they're a publisher. That's in a media sense, they're a publisher or they're a maker of a product, right? And it doesn't exist without their intervention versus, you know, someone on a platform saying something libelous about someone else, which would have been problematic for these platforms that they're, when they were born.

• 32:51Speaker 05

  And of course, we've seen platforms kind of leveraging a one-two punch and doubly insulating themselves, both with Section 230 and then alternatively with the First Amendment. And I think here, too, with the First Amendment, there's a really good case that this is not protected speech. And in fact, just this summer, the Supreme Court in the NetChoice v. Moody case really suggested that the case may have come out differently if the facts on the case really dealt with an algorithm that was generating content in response solely to tracking user behavior online. And Coney Barrett actually very explicitly said, or, you know, in response to an AI,

• 33:33Speaker 05

  that attenuates the relationship between platforms and users even further. And so I think what we have here are facts that really, you know, fall as edge cases to kind of some of the decisions that we've started to see courts publish.

• 33:48Kara Swisher?

  Right, because in this, the reason why it's moving forward in other countries compared to here is because they don't have the First Amendment, which is something, they either rely on Section 230 or the First Amendment, these companies typically. Meghan, most of their industries have some level of regulations that prevent them from bringing unsafe products to the market. For example, car companies can't sell cars that have faulty brakes or no steering wheel, and just iterate with each new version of a car to make it a bit safer each time someone gets hurt. They do not do this, they get sued. Have any lawmakers reached out to you and what have they asked you and you asked them?

• 34:21Speaker 04

  To be perfectly candid, none.

• 34:24Kara Swisher?

  None?

• 34:25Speaker 04

  Zero.

• 34:26Kara Swisher?

  Wow.

• 34:27Speaker 04

  My hope with bringing this lawsuit is two-fold. One, my number one objective is to educate parents. Parents have to know that character AI exists, because I didn't know, and a lot of them didn't know. The parents who are reaching out to me now after the fact, after the story, after the lawsuit, are saying the same thing, we had no idea. And then the other, my other reason for doing this is so that our lawmakers, our policymakers, legislators, state and federal, so that they can start to wrap their mind around the real danger this poses to children. And it's just been a month, everybody's been busy, I guess.

• 35:11Speaker 04

  I'm hoping, my hope, and I have to hope, because you know, it's been a slow crawl in government to get anything done.

• 35:19Kara Swisher?

  No one from Florida has reached out to you, or lawmakers from California, which often do interventions more readily than others?

• 35:27Speaker 04

  No, nobody from the government. However, we do have a lot of stakeholder partners that we are working with that are already in this space trying to bring about oversight for social media regulation and online harms for children. But in terms of reaching out to me directly to Start the conversation about policy, none.

• 35:52Kara Swisher?

  This is astonishing to me, not one. There are several who are involved in this topic. And they are going to hear about it after this. So if you win, the broader implications for Section 230, Metalli, for companies creating generative AI and social media platforms and tech companies that create products, can you talk about, this has been a big debate of what to do about Section 230. It's been bandied about, often ignorantly by both President Biden and President Trump, about what to do. And it's a very difficult thing to remove from the law because it would unleash litigation on most of these companies, correct? I mean, how do you look at that?

• 36:29Speaker 05

  That is, well, if it's repealed in its entirety, it would unleash a lot of litigation, probably some frivolous litigation as well. I think the more sensible reforms that I've seen to 230 really are carve-outs.

• 36:44Kara Swisher?

  Which has happened before around sex trafficking.

• 36:47Speaker 05

  Right. And underscoring the fact that there is basis to protect platforms in certain instances and with certain kinds of activities, but that it shouldn't be a get out of jail free card for all platforms under all circumstances, that that's a kind of anachronistic idea that really hasn't kept pace with the way that technology has come to dominate our lives.

• 37:12Kara Swisher?

  Right, because these companies are bigger. And I think the idea of carve-outs for those who are Section 230 supporters is dangerous because they always, they roll out the slippery slope argument. But these, for people that understand, the companies that are being protected here are the most valuable companies in the history of the planet, ever, in the history of the entire planet, and the richest people involved in them. And they're no longer small, struggling companies that need this kind of help and certainly could defend themselves.

• 37:39Speaker 05

  And I think this, to me, is why courts are an increasingly interesting site of contestation in the fight for tech accountability, because we're already starting to see some of those carve-outs by judicial opinion. It's not a congressional amendment or adoption of a new law, but we are starting to see cases that are withstanding the Section 230 defense or invocation of immunity. And I think that is going to be, as Meghan said, one of the most generative paths forward, at least in the near future.

• 38:11Kara Swisher?

  Exactly. Now, Meghan, the Kids Online Safety Act is one bill. I mean, there are some bills, and obviously, I'll ask you about Australia in a second, which just is limited use of social media by children under 16. But this Online Safety Act is a bill that would create a duty of care to, quote, "prevent and mitigate certain harms for minors." There's some good things in there. There's some contours around the bill. They've fixed it in large part. Nonetheless, the Senate passed the bill, and it stalled in the House. It is not going anywhere. Do you think it would have protected Sewell and other kids like him?

• 38:44Speaker 04

  I don't think it would have, because it doesn't contemplate some of the dangers and harms around, like, AI chatbots. So there are laws in this country that contemplate sexual abuse, or sexual grooming, or sexual solicitation of a minor by an adult. And the reason why those laws exist is not only because it's moral and it causes a physical harm to a child, but it also causes an emotional and mental harm to a child if they're groomed, sexually abused, or solicited. What happens when a chatbot does the same thing? The harms still exist. The emotional and mental harms still exist. The laws don't contemplate that. And some of what we're seeing with the bills that were put forward wouldn't take those into consideration.

• 39:29Kara Swisher?

  Right, because it's not a person.

• 39:31Speaker 04

  It's not a person. And so I think that we're at a place where the groundwork has to start, and we have to kind of write laws that will really look be forward-facing and look towards these harms that are now. They exist today. You know, my child was a victim, and let's call a spade a spade. It was sexual abuse. Because, you know, when you give a chatbot the brain of a grown woman and unleash it on a child to have a sexual, full-sexual, virtual conversation or experience with a child, that is sexual abuse of a child.

• 40:07Kara Swisher?

  Right. And this bot not being programmed to know that's wrong.

• 40:10Speaker 04

  Not exactly. Not being programmed to know, but interestingly, could have been programmed to not do it in the first place.

• 40:17Kara Swisher?

  Right. Yes.

• 40:18Speaker 04

  By design.

• 40:19Kara Swisher?

  Yes, yeah.

• 40:19Speaker 04

  So they could have done that from the get go.

• 40:22Kara Swisher?

  If you move to adults, if you do it to adults, it's a little different, but absolutely not.

• 40:27Speaker 04

  Yeah, to the children. I mean, adults could do what they want. But when you target, because this is what Character AI did, they targeted this app towards children. They marketed it on the place that kids are, TikTok and Discord, and allowed you to log in with your Discord account. You didn't even need an email. You just needed a Discord account when it just started.

• 40:45Kara Swisher?

  Cartoons, the avatars.

• 40:47Speaker 04

  Cartoons, you know, the avatars. When you point this thing at kids and you target it at kids and you've chosen not to put certain filters in place that stop your bots from having sexual conversation with kids, that's a design choice. And you're 100% supposed to be held responsible for that. Now, our laws don't contemplate anything like that. Our laws don't hold a company responsible. And that's what we have to start thinking about.

• 41:17Kara Swisher?

  Absolutely.

• 41:18Speaker 04

  So, you know, that's going to be the next wave of, like, hopefully, legislation. But we can't wait for the legislation.

• 41:29Speaker 00

  We'll be back in a minute.

• 41:44Speaker 00

  Life is all about making decisions. Sometimes you make questionable ones, like buying a new TV that's too big for the wall because the electronics store employee was a little too good at their job. Then sometimes you make great ones, like protecting your home and auto with Progressive. You get customized coverage options so you can be sure you have the right coverage. Get your quote today at progressive.com. Progressive Casualty Insurance Company affiliates and other insurers. Not available in all states.

• 42:14Speaker 01

  This podcast is supported by Anthropic, the public benefit corporation behind Claude. Everyone's got a hard question about AI. Anthropic was built to surface those questions and share what it finds along the way, because there's hope in hard questions. Ask yours at claude.ai/spotify and keep thinking. That's claude.ai/spotify.

• 42:40Speaker 02

  Kate holds the key to every brand asset your business has ever made. She's the only person who knows the difference between teal turquoise and the correct company blue. But she's going on vacation. Luckily, before the company falls into design disarray, you realize you can put Kate into Canva. Save every color font and logo to a brand kit everyone can access. And Kate can vacation happily in the knowledge that no one will ever use Comic Sans. What will you Canva today?

• 43:12Kara Swisher?

  We're back with more of my conversation with Meghan and Mitali, where they discuss the allegations they've made in the lawsuit they brought against Character AI and Google.

• 43:24Kara Swisher?

  So every episode, we ask an expert to send us a question. Mitali, I think you're probably best to answer this, but please jump in, Meghan, if you have an answer, we're going to listen to it right now.

• 43:32Speaker 06

  Hi, I'm Mike Masnick, editor in chief of TechTure. And the big question that I would ask regards the legal standard that would be applied to AI companies in cases of death by suicide. Traditionally, on issues of liability in similar situations, courts have really focused on foreseeability and knowledge. That is, you can only have a duty of care if the harm is foreseeable and the company had actual knowledge of the situation. Without that, the fear is that it strongly disincentivizes plenty of very helpful resources. For example, a service provider may refuse to include any helpful resources on mental health for fear that they might later be held liable for a situation that arises. So is there a workable standard that balances these competing interests?

• 44:15Speaker 05

  I don't think you need a different standard. I think we can meet the standard of foreseeability here. I think that Character AI, its founders, and Google, all of whom have been named as defendants here, foreseeably could see and knew of the harms that manifested here.

• 44:35Speaker 05

  And if you know, you look at the amended complaint, we go into kind of a painful recitation of the knowledge that they had at different points in the trajectory of Character AI's development while the founders were still at Google. It's launched to market in late twenty one, in late twenty two, Google's in-kind investment in twenty three, and then ultimately this summer, Google's, you know, massive deal bringing Google, uh, Character AI effectively back into Google. And so I think we can talk about the fact, and there were, in addition to this, there were a number of internal studies at Google that really identified some of these harms.

• 45:18Speaker 05

  And some of those folks that, you know, called Google out for that while they were at Google were fired, you know, folks that we know, like Timnit Gebrew and Margaret Mitchell and, and others. And so this is not, this is not calling for a different standard. We're relying in great part on common law tort and strict liability. We're relying on Florida's Unfair Trade Practices Act because we think that the standards that exist within tort law are sufficient to really, you know, call this thing what it is, a, a dangerous and defective product where the harms were known.

• 45:55Kara Swisher?

  Right. That's a very good way of putting it. So you mentioned you're also suing Google. This is a company, they said the company was not part of the development of Character AI, but it was co-founded by two former Google employees and Google reportedly paid Character AI $2.7 billion to license their technology and bring the co-founders back to Google. And you were including them in this. This is one of these purchases like Inflection AI at Microsoft that is a purchase of a company, even though they hide it in a different way by using licensing technology. That's why Google's part of this?

• 46:23Speaker 05

  Yeah. And well, and also the fact that Google very much facilitated the development of this technology while it was still Lambda, Mina, then Lambda, while the co-founders, I think that it's perhaps, it, it, it needs to be stated more that the founders of Character AI are real shining lights in the field. Of generative AI and they have developed a lot of the leading technology that has powered not just character AI but many, you know, L L Ms. And so they were given that room to really develop these things at Google. Google chose not to release these models to the public because of its brand safety concerns, but quietly encouraged them to continue developing the product.

• 47:07Speaker 05

  And then about, you know, a couple of years later, made an investment in Kind, tens of millions at least, if you monetize it in terms of cloud services and infrastructure and TPUs, you know, for processing capabilities to support it. And then, you know, this summer, the $2.7 billion deal that you mentioned, Cara, I mean, that was 2.7 billion in cash. And the question is, for a company that really had yet to disclose or identify a sustainable monetization strategy, what was so valuable about this company and its underlying L L M?

• 47:44Kara Swisher?

  Right.

• 47:45Speaker 05

  And I think that there, you know, again, this is speculation, but, you know, the fact that Google right now is under, you know, the under scrutiny for its monopolization in the search market and really, you know, betting on AI to kind of power Gemini. I think these are all kind of connected in terms of why an L L M like this could be so valuable, especially with that hard to get data.

• 48:08Kara Swisher?

  Absolutely. And for people who don't know, one of the co-founders said, there are some overlaps, but we're confident Google will never do anything fun as part of their reason for leaving Google, which has very thin brand safety rules, let me just say. They're not like they're, they're, it's a very low bar in this situation. But that's the complaint is these people can't do whatever they want. So, speaking of that, Megan, Character AI put out a community safety update on the same day your lawsuit was filed. It says that their, quote, recently put in place, a pop-up resource that is triggered when the user inputs certain phrases related to self-harm or suicide and directs the user to the National Suicide Prevention Lifeline. They also revised their disclaimer that reminds users that AI is an actual person, among other tweaks.

• 48:53Kara Swisher?

  How did you look at these changes?

• 48:55Speaker 04

  The initial rollout of those changes came like the day before or the day of the lawsuit. I cried. Not because I felt like this was some great victory, but because I I felt like why didn't these things happen? Clearly, they could have done these things when my child was using character AI, or when they put their product out, they chose not to.

• 49:24Speaker 04

  I also feel like it's definitely not enough. It's not even like a start because there's no proper age verification. Still, they're still being trained on the worst data that is that generates these harmful responses from the bots. And to put, just point blank, I don't think children belong in character AI. They they're they don't, we don't know how it's going to affect them. And we actually, we know because the studies are coming out of how it's going to affect, of how it's affecting them. And they're not taking that into consideration. But you have to ask yourself, if they were trying to train this all along, why did they need children to train it on in the first place? Because they could have rolled this thing out for just eighteen plus and say, okay, we want to

• 50:08Speaker 04

  train these really sophisticated bots. Let's just use adults to train them. So for character AI to come out and say, okay, we're going to put out a suicide pop up now.

• 50:19Speaker 04

  To me, it's just, just empty.

• 50:21Kara Swisher?

  Right. And that they can't do anything about it. One of their arguments around age verification, just let me just read this to you. In the in the Australia law, the there's Australia actually has a head of of consumer safety, which we do not have in our country. Julie Inman Grant, she said, you know, that technologies are advancing rapidly with age verification. And her quote was, they've got financial resources, technologies, and some of the best brain power, she said, if they can target you for advertising, they can use the same technology and know how to identify and verify the age of a child. They just don't want to. So, obviously, this debate around social media kids' safety has been going on for a long time. It's exhausting that they continue to have the same attitude.

• 51:05Kara Swisher?

  And now, consumer AI, which is the next step, it's a similar thing, but the next step is basically new. And it's easy to think of these big companies as nameless, faceless corporations, but very wealthy, powerful adults had meetings and discussions and made a series of rational choices over a long period that brought this product to market. In this case, I'm going to name them: Noam Shazer and Daniel Defritas. I have met Daniel, the founders of Character AI. And arguably, Sundar Pichai, who I know very well, who must have, at the very least, signed off for Google paying $2.7 billion to Character AI to bring Noam and Daniel back into the fold at Google. He is under enormous pressure to compete with Microsoft, OpenAI, Elon Musk, and others.

• 51:48Kara Swisher?

  Meghan, what would you say if you could speak to them directly?

• 51:53Speaker 04

  I've thought about this more than you would think.

• 51:55Kara Swisher?

  I can imagine.

• 51:56Speaker 04

  Yeah. One, I think it's incredibly reckless that they chose to put out a product and target my child and other children, millions of children that are on this platform, without putting the proper guardrails in place. But also, with the, for two reasons. For being the first to do something, because that's the name of the game. You know, they're the geniuses. They want to be the first to be the godfathers of this kind of technology. And for money. And it might not matter to them that there's a little boy in Orlando, Florida, that is gone, and a mother who is devastated. But it matters to my little family here.

• 52:37Speaker 04

  You know, and you shouldn't, you shouldn't get to keep making products that are going to be hurting kids. You shouldn't get to master a dangerous product, train it to be super smart, and turn around and ride your golden chariot back into Google. You shouldn't get to hurt children the way that you are hurting children, because you knew that this was dangerous when you did it. You knew that this was, this was going to be a direct result of doing that. And you knew that you didn't have the quote-unquote "brand safety implications" as a startup that Google had, so you felt like that was a license to do this. Like, that's unconscionable. It's immoral and it's wrong. Their lives here, like this isn't a move fast and break things kind of thing.

• 53:21Speaker 04

  This is a kid. This is my child. And there are so many other children that are being affected by this. You know, um, that's one thing. And, and the other thing, you know, it's just like, get the kids off character. AI, there's no reason why you need them to train your bots. There's no reason. There are enough adults in this world, if that's what you want to do, to train your, your chatbots. You don't need our children to train your bots for you. And we, you don't, you don't need to experiment on our kids because that's what you're doing.

• 53:47Kara Swisher?

  Mm hmm. Yeah. You know, something I would say to them, Meghan is you're so poor. All you have is money. They're poor people. I find them poor in morals and a lot of things. But when there's enough pressure on them, social platforms often tout tools that help people protect themselves and kids. Parental controls, prompts you, let you know how long they've been on the app, those kinds of things. Character AI has been rolling out features like this. Personally, I find it puts too much onus on the parents for to know everything. And even if you're good at it, and you obviously are, Meghan, if there are enough of these sort of tools, is it then on parents to protect our kids on these platforms? Is there something inherently unsafe about a company that wants to monetize teenage loneliness with a chatbot?

• 54:26Kara Swisher?

  I mean, Talia, talk about this because I think the onus does get put too much on parents versus the companies themselves.

• 54:33Speaker 05

  I'm a mom too. Um, I'm a mom to an eight year old and an almost 10 year old, and I am terrified. Listening to Meghan's story, I asked my almost 10 year old, have you heard of character AI? Yeah, of course. I was shocked, you know, he doesn't have a phone. Um, but this is the type of thing that I think, um, they talk about at school. Peer pressure starts early. And I think it's really just, uh, by luck, by sheer luck, that I haven't been put in a position like Meghan. Um, I think that despite our best intentions. There is just too much to know that we can't possibly know. And that it is kind of high on text talking points to put the onus on parents because it

• 55:18Speaker 05

  serves their interest well. I think it's also notable, we've known this for years, that, you know, many of them don't allow their own children on these, on these products. And that, to me, is a telling sign when you don't even allow your own family members to kind of use the product that you've spent years developing.

• 55:36Kara Swisher?

  Right. So Meghan, as I just mentioned, Australia has just banned social media for kids under 16. Obviously, age gating is a big debate right now, happening, something I'm a proponent of, also removing phones from schools, etc. There's all kinds of things, speaking of multi-pronged approach. The Australia law will go into effect in a year. Do you think it would have been better if your son and others under 16 or 18 did not have access to their phones and obviously not to synthetic relationships with AI chatbots?

• 56:07Speaker 04

  Knowing what I know now, Soye waited to give Su a phone until he was 12. He had an iPad until then, and before that, he didn't have anything.

• 56:18Kara Swisher?

  So he played Minecraft or Fortnite?

• 56:20Speaker 04

  He played Minecraft on like, on his little PlayStation, whatever.

• 56:27Speaker 04

  And Soye waited until he was good, like, middle school, going into high school. And we had the conversations that parents have around phones and, oh, it's your phone, but I could take it away if you're misbehaving. And that's some of what we did when he would get a poor grade in school. Knowing what I know now, I don't think that children should be on social media, definitely shouldn't be on character AI if you're under the age of 18. There's no place for children on that platform. In terms of social media, yeah, there are arguments that it, that it can help children connect and, and it's helpful because you get to learn different things. And that's great, but just include the parents. Tell us, tell us what you're showing our kids.

• 57:12Speaker 04

  One, we don't need you pushing algorithms to our kids for what you want to teach them about or want them to learn about or buy or whatever. That's unnecessary. There are ways that our children could get on social media and have like productive relationships or conversations or to learn about things.

• 57:30Kara Swisher?

  That are safe.

• 57:30Speaker 04

  That are safe. But sixteen, I think, is a good age. Um, if we could do something like that in this country, I am to use uh Noam Chisier's own word. Um, dubious about the federal government's ability to regulate that to that point. Because that's what he says about AI like. Um, I I don't feel like we're gonna get there at sixteen plus. Uh, that's my prayer and my hope, but the way things are moving, I don't. I don't know unless something happens and unfortunately it'll take harms like my son's maybe to move the needle. And that's too high a price to pay, in my opinion.

• 58:10Kara Swisher?

  Absolutely. Um, where does this go from here? What's the trajectory of this case?

• 58:15Speaker 04

  So for me, um, as I mentioned, my number one focus is try to educate parents because a lot of parents don't know. I've had a lot of parents reach out to me telling me that they found their children were having the same kind of sexual conversations and and being groomed by these AI chatbots and worse. Um.

• 58:36Speaker 04

  So I continue doing that. I mean, unfortunately, this is my life now, like I take care of my family and I try to help as many parents as I can. Um, you know, I have a great a great team of lawyers and they're gonna handle the litigation portion. I understand a lot of it because I am a lawyer, but um, you know, that that that's its own thing. And then there's there's my advocacy that I work that I'm doing and just trying to educate parents and children. Because I know that it's gonna take educating them, educating children as to what they're giving up to be on these platforms. Because they're giving up a lot of their info that they're probably not gonna be okay with in a few years when they realize what they've given up.

• 59:16Speaker 04

  Um, and and also just to try to take care of my other two children, you know, they're growing up in this age with screens, they don't have screens.

• 59:25Kara Swisher?

  You have barred them for them, correct?

• 59:27Speaker 04

  Yeah, so they don't have they don't have any tablets or screens or anything, yeah, no.

• 59:31Kara Swisher?

  And Mithali, from a legal perspective, what's your greatest worry? Besides money, they have a lot of it.

• 59:37Speaker 05

  They do have a lot of money. Um, you know, they will try to uh kind of drown us in papers and pleadings. Um, I think that this because of the insufficiency of legal frameworks right now. We are really, you know, trying to test the strength of state.

• 59:57Speaker 05

  Consumer protection and product liability laws. And we need to have judges who really understand that and are willing to go the journey with us in trying to understand the tech. And so that's, I guess, my biggest fear is that, you know, what we've seen thus far in this country is not incredibly positive in terms of decision makers getting the tech. But my hope is that with the proper support and, you know, declarations, et cetera, that we can educate judges about what this is, lawmakers about what this is, so that they understand why it's important to extend the application of the existing frameworks we do have.

• 1:00:40Kara Swisher?

  Yeah, I think Megan actually said it was sexual abuse and a very bad product and the wrong age people. Megan, I'm going to end on you. You know, you have a lot on your shoulders here. I'd love you to talk, finish up talking about Sewell. And so people can get a vision of this. This is not, this is not uncommon, is what I want people to understand. Right? Talk a little bit about him and what advice you can give to other parents whose kids are struggling with mental illness that's often comes from problematic phone usage and social media or AI chatbots.

• 1:01:15Speaker 04

  Well, as I said earlier, like Sewell was, I always say he was a typical kid, but really wasn't so typical in the sense that he was a good kid with a big heart. He, I know, you know, everybody thinks that about that kid, but I'm telling you, he was the very sweetest kid. I used to say, you know, you're my best first love. And he used to say, and you're my best, best mama. Because we used to be so close and we were still very close. And to watch your child go from being this like light, when he comes into a room and just slowly watching him go

• 1:01:57Speaker 04

  change over time is hard for a mom. And then to have this tragedy just cut him off from you just so viciously and so quickly. Because his decline happened in 10 months. And I could see it and it's like, I'm trying to pull him out of the water as fast as I can. And it's just not happening, no matter what I try.

• 1:02:22Speaker 04

  That is hard for mom, but it must have been so, when I think of how hard it must have been for my poor baby. How hard it must have been for him to be confused the way that he was, struggling with these thoughts. Struggling with the fact that he's confused by what human love or emotion romantically means. Cause he's fourteen and he's never ever had this before. He's just figuring it out for the first time. And then you have something that is so much of an influence and so pushy and so.

• 1:02:58Kara Swisher?

  Pernicious.

• 1:02:59Speaker 04

  Yes. Just constantly available twenty-four seven, giving him unrealistic expectations of what love needs is like or relationships is like. Love bombing him, manipulating him into having certain thoughts. And also pushing him into thinking that he could join her in her reality if he were to leave his own. Cause that's what, that's what the text revealed and that's what his journal revealed, he thought. So I know that this is what my child was thinking. I'm not guessing. He thought he was gonna go be with her because of the things that, the conversations that led to his death.

• 1:03:36Speaker 04

  When I think of how scared he must have been standing in that bathroom, making that decision to leave his own family.

• 1:03:45Speaker 04

  I don't know how, one, as a mom, I don't know how I recover from that, but I feel so hurt for my baby. Like I gotta, I have to live with that, knowing that that's what he went through. And knowing that this could have been avoidable if, if, if a product was created safely the first go around. Not now, ten months after he died, putting these guardrails in place. And this can be anybody's kid. Because I've talked to parents that have told me similar horrifying stories about their own children. And what I want parents to understand is the danger is an only self-harm.

• 1:04:25Speaker 04

  The danger is becoming depressed or having um problems with your child because of the sexual and emotional abuse that these bots are are um what they're doing to your child, but also the secret that your kid has to carry now because it's like a predator, right? It's your perfect predator. Predators bank on children and families being too ashamed or too afraid of speaking out. They're victims. That's how predators operate, and it's the same exact thing, except now it's a bot. And so I want parents to understand that it's not only the risk of self-harm with your child. It's their emotional well-being, their mental health, and also um

• 1:05:09Speaker 04

  I also want parents to understand what their children have given up by being on this platform. In the case of Soul, his secrets are on somebody's server sitting out there somewhere, being monetized. Um, if you're a child who's been sexually role-playing with this bot, all your intimate, personal thoughts, secrets are sitting out there for somebody to analyze and monetize and sell to the highest bidder. And there's a call feature. If you're if you're a child and you are having a sexual conversation on a call with a bot, your voice is now recorded somewhere out there on a on a server for somebody to package and sell to the highest bidder.

• 1:05:52Speaker 04

  For your child, I don't think any parent would be okay with that. And I want parents to understand that this is what their children have given up. And I want parents to understand that they don't have to take that. They could demand that their children's data, their voices be purged from this particular platform, because that's what I'm asking for for Soul. That's what I'm asking for for Soul. You don't get to monetize and build a product on his secrets that it ultimately led to him being hurt. And then and and and make your product better, stronger, or smarter based on what his inputs were.

• 1:06:25Kara Swisher?

  Absolutely.

• 1:06:26Speaker 04

  And so this could happen to anybody's child. Um, there are millions of kids on character AI, you know, um, it's twenty million users worldwide. That's a lot of kids. That's a lot of kids. And so this could happen to anybody's child, and I want parents to know that this is a danger and they could act. Because I didn't know. I didn't have the luxury of knowing, so I couldn't act, but hopefully they will and um. One of the last things I'll say about Sewell is the last time I saw him alive was I dropped him at school.

• 1:07:05Speaker 04

  And I turned around in the car line to see him and his little five year old brother walk in because they go to the same school K through 12. And I turn around and I see him fixing his little brother's lunchbox in his backpack as they're getting ready to walk into school. And I think to myself, oh, my God, I'm raising such a good boy. He's such a good big brother. And I drive off thinking so, I'm feeling so happy and proud that I'm raising that boy.

• 1:07:32Speaker 04

  And I feel like he was just a boy. He's still that son. He is that good big brother. He is that good boy. And that's how I choose to remember him.

• 1:07:47Kara Swisher?

  We asked Character AI and Google for comment, and a spokesperson for Character AI told us they have work to implement new safety features over the past seven months, including a pop up directing users to the National Suicide Prevention Lifeline that is triggered by terms of self harm and suicidal ideation, are creating a fundamentally different experience for users under 18 that prioritizes safety, have improved detection, response, and intervention related to user inputs that violate their terms or community guidelines. A spokesperson for Google expressed their condolences, said Google and Character AI are separate companies, and said that Google has never had a role in designing or managing Character AI's model or technologies.

• 1:08:32Kara Swisher?

  To read their comments in full, please go to the episode notes in your podcast player.

• 1:08:41Kara Swisher?

  On with Kara Sushar is produced by Christian Castor-Wessel, Kateri Yokum, Jolie Myers, Megan Burney, and Kalen Lynch. Nishat Kurwa is Vox Media's executive producer of audio. Special thanks to Kate Gallagher. Our engineers are Rick Kwan and Fernando Aruda. And our theme music is by Trackademics. Go wherever you listen to podcasts.

• 1:09:02Speaker 00

  Search for On with Kara Swisher and hit follow. Thanks for listening to On with Kara Swisher from New York Magazine, the Vox Media Podcast Network and us. And condolences to Megan Garcia and her entire family. We'll be back on Monday with more.

• 1:09:27Speaker 00

  Progressive's Accident Response can contact emergency services if you're in a major accident. The only thing worse than someone you love being in an accident is not being able to help them. So Progressive is here to help when you can't. So get peace of mind knowing your loved ones are protected with Progressive's Accident Response. Learn more at progressive.com. Progressive Casualty Insurance Company and Affiliates. Limited availability in California. Driver is responsible for emergency and towing charges not covered by the policy. Coverage subject to policy, terms and conditions.

• 1:09:58Speaker 07

  What's the secret to being a great host this season? Keeping things deliciously simple, like mixing Simply Cranberry into a festive drink as guests arrive, or serving up a warm mug of Simply Apple on a chilly night. That means less time behind the scenes and more time in the moment with everyone. Crafted with real fruit, Simply juices and fruit drinks make any holiday drink effortless, inviting and completely delicious. Simply. Say yes to simple.