The Telegram Problem + Gov. Kathy Hochul on School Phone Bans + Kevin's A.I. Reputation Rehab

Tech platforms and governments are in a standoff.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Your agents can talk to each other. They still can't think together. That's the gap Outshift by Cisco is closing. Outshift is Cisco's incubation engine, building the internet of cognition, shared intent, shared memory, and guardrails that allow agents to work together from anywhere. Open source protocols and code, a system for intelligence that scales. Read the paper and experience the demo at outshift.com. That's O-U-T-S-H-I-F-T.com.

• 0:33Casey Newton?

  Let me ask you, Kevin, let me ask you this. I walked into the office today and I saw on your desk a keyboard and it said "Alpha." There's an Alpha keyboard on your desk. What is this?

• 0:45Kevin Roose?

  Well, I'm such an Alpha male that they have to give me a special keyboard because I mashed the keys so hard that a normal keyboard would just shatter.

• 0:53Casey Newton?

  That doesn't sound right to me.

• 0:55Kevin Roose?

  No, it's not.

• 0:55Casey Newton?

  I think you're bamboozling me. What is an Alpha keyboard?

• 0:59Kevin Roose?

  This is a product that I am testing. It's called the Freewrite Alpha. And this is essentially an internet-connected typewriter. Because, you know, as you may experience, being online can be quite distracting when you're trying to do research and writing. And so there's this company, Freewrite, that makes these little sort of stripped-down computers that basically only can be used to write things. And then they can like sort of store that in the cloud. And so that can become your ordinary writing device.

• 1:33Casey Newton?

  So you feel like to get ahead, you need to sort of go from a laptop to something that is like much worse than a laptop?

• 1:40Kevin Roose?

  Yes. What if a laptop could only write? That is sort of the premise.

• 1:43Casey Newton?

  Does it have a display?

• 1:44Kevin Roose?

  Yes, it has a little like sort of gray. It just gives you sort of a couple lines at a time.

• 1:51Casey Newton?

  So you're writing your column, you can only see one sentence at any given time?

• 1:55Kevin Roose?

  Well, yeah.

• 1:56Casey Newton?

  Kevin, this is a nightmare.

• 1:57Kevin Roose?

  Yeah, I haven't used it much.

• 2:01Kevin Roose?

  But they did send it to me and so I feel obligated to try it.

• 2:04Casey Newton?

  I gotta be honest, this sounds like a keyboard for betas. If you can't handle seeing your entire column at once, beta behavior.

• 2:12Kevin Roose?

  I need the Sigma keyboard.

• 2:14Casey Newton?

  Yeah.

• 2:18Kevin Roose?

  I'm Kevin Roos, a tech columnist at The New York Times.

• 2:20Casey Newton?

  I'm Casey Newton from Platformer.

• 2:22Kevin Roose?

  And this is Hard Fork.

• 2:23Casey Newton?

  This week, Telegram CEO is arrested in France. We'll tell you what the charges against him mean for the internet. Then, New York Governor Kathy Hochul joins us to talk about why she wants to ban phones in public schools. And finally, the secret code that Kevin uses to trick chatbots into saying nice things about him.

• 2:42Kevin Roose?

  It's not a trick, it's persuasion. It won't work on me, by the way.

• 2:55Kevin Roose?

  Well, Casey, there was one tech story that was everywhere this week, and that is the arrest of the founder and CEO of Telegram, Pavel Durov, this past weekend. He was pulled off of his private plane as it touched down outside of Paris. And it's been a big story.

• 3:11Casey Newton?

  It really has. And you know, my first question was, did he think the Olympics were still going on? Because those have been wrapped up for a couple of weeks now. So he really might have made a mistake there.

• 3:19Kevin Roose?

  He just wanted to see the pommel horse guy. Look, and can you blame him? So this has been such a crazy story, and I think it touches on a lot of themes that we really care about, including content moderation and government interference with tech platforms and free speech. There's just a lot here. It's a rich text, as they say.

• 3:37Casey Newton?

  For sure.

• 3:38Kevin Roose?

  So on Wednesday, Pavel Durov, the CEO of Telegram, was charged in France with a wide range of crimes related to illicit activity on Telegram. He faces six charges in total, including complicity in managing an online platform to enable illegal transactions, and complicity with crimes like the distribution of child sexual abuse material, or CSAM, drug trafficking, fraud, and a refusal to cooperate with law enforcement. He's been barred from leaving the country in order to pay a bail of 5 million euros. And he also has to check in at a police station twice a week. So let's just start by explaining to our listeners like what Telegram is.

• 4:23Kevin Roose?

  What place does it occupy in the landscape of global tech platforms?

• 4:28Casey Newton?

  Telegram, you know, you're probably old enough that your birth was announced by a real Telegram.

• 4:31Kevin Roose?

  Yeah, Western Union.

• 4:32Casey Newton?

  Yeah, I'm gay, so mine was announced by a singing Telegram. But anyways, Telegram is a messaging app, Kevin, and it has a lot of social media features. So it kind of started off similar to, you know, a WhatsApp and iMessage, something like that. But over time, it just added these features that are familiar to us from other social networks. So for example, they have these public channels. So maybe you're a public figure like Emmanuel Macron, the president of France. You can create a channel, you can say whatever you want. Other people can read it. They cannot write back to you in public if you set it up that way. But it can be a way of getting your message out, just as, you know, posting on X or Threads might let you do the same thing. And it's gotten hugely popular.

• 5:14Casey Newton?

  It now has more than 900 million users and it's particularly popular in certain regions, often where there's been authoritarian government. So it's been very popular in Iran, for example, and it is hugely popular in both Ukraine and Russia, where military forces on both sides use it to command their armies in the war. But along the way, we should also say it has developed a crime problem.

• 5:38Kevin Roose?

  Yes, we should talk about that because that is a major part of what is involved in these charges against Pavel Durov. Telegram, I would say, takes a much lighter approach to content moderation than other messaging apps and social media platforms. They are sort of proudly resistant to government requests to take down information or to see the messages that users send. Telegram is often called an encrypted messaging app. That's not quite accurate, at least not in the way that other apps would define encrypted, and we can get into that later. As a result, it is seen as being private. And because of that, lots of people use it for illicit transactions and messaging.

• 6:22Kevin Roose?

  Lots of criminals use it to exchange tips on where to buy drugs or guns. It's also been used by a lot of people in the crypto community because they're worried about government surveillance of their communications. So it's just kind of has this seedy reputation, at least in the US.

• 6:37Casey Newton?

  Yeah, it's earned it. And, you know, if you're wondering, what do we mean by a light approach to content moderation? Pavel Durov has said that he believes it just sort of not interfering with letting people communicate online. And if you go to Telegram's FAQ page, they say that we are just essentially not going to look at the contents of your private messages in particular. So, you know, if you want to get on there and you want to organize a drug deal or a gun deal, Telegram is basically explicitly saying, you know, we're just going to look the other way on this. And that is very unusual among the the large messaging apps.

• 7:09Kevin Roose?

  Yes, and it has made it a tool for terrorist organizations, for drug traffickers, for extremist groups of all kinds. So I came across Telegram a lot when I was reporting on neo Nazis and other white supremacist groups who would sometimes get kicked off other platforms and would sort of reconstitute themselves on Telegram because of these huge group features that allow you to broadcast messages to hundreds of thousands of people at the same time.

• 7:36Casey Newton?

  Yeah, it's actually really interesting. Like, one of the, one of the key ways that extremism is now studied is just researchers will follow the Telegram channels involved because they can see terrorists and other groups just operating out in the open. And, you know, for better and for worse, this is now how a lot of us learn about what some of the, you know, darkest forces in the world are up to.

• 7:54Kevin Roose?

  Yes, but we should also talk about Pavel Durov because he's a very interesting character.

• 7:59Casey Newton?

  Yeah, this is a person who was, you know, born in the Soviet Union in 1984 and first became famous by creating a very popular Facebook clone called VContact. At least that's how I pronounce it. How do you pronounce it?

• 8:11Kevin Roose?

  I always thought it was VContact, but it's, it's, it's got some extra letters in there. It's got a K in there. I'm not, I'm not sure.

• 8:16Casey Newton?

  If you're an English speaker, it looks like VContact, but I'm told that Russians are pronouncing it Vkontakta. And I said, excuse me? And they said, the contacta.

• 8:27Kevin Roose?

  So, yes, my colleagues, Paul Moser and Adam Sotoriano, had a great profile of Pavel Durov that ran this week. And I learned a lot from it, including sort of the backstory of how he came to found Telegram. So basically Pavel Durov, he starts this website, the contacta, or whatever it's called in Russian. That becomes very popular, but ultimately it has sort of a standoff with the Russian government because the Kremlin basically wants to get a bunch of information about the users of this service. Pavel Durov sort of resists that. And as a result, the Russian government essentially seizes the website from him, kind of confiscates this Facebook clone that he had built. So he basically loses control of his creation.

• 9:12Kevin Roose?

  He then kind of gets an idea for what becomes Telegram, a messaging app that would essentially be resistant to government interference and surveillance and would allow people to have secure conversations on their phones.

• 9:25Casey Newton?

  Yeah, and as he sort of starts building it, I think because of this experience that he has in Russia, he never really wants to be beholden to another government ever again. And so he starts traveling the world and he just like picks up citizenships the way that other people like collect fine art, you know? So he's like now a citizen of Saint Kitts and Nevis. He gets one from the United Arab Emirates and he starts running Telegram out of the UAE. And in one of the strangest details of the story, given what would ultimately happen, is that in 2021, he manages to get French citizenship, even though he's never lived there.

• 10:05Kevin Roose?

  Yeah.

• 10:05Casey Newton?

  And apparently there is some rare procedure that dates to before the French Revolution, where someone can be granted French citizenship if they are a French-speaking foreigner who contributes through his or her outstanding work to the influence of France and the prosperity of its international economic relations. And so you might be asking, what did Pavel Durov do to like advance the interests of France? And no one knows, and we don't know if he even speaks French.

• 10:29Kevin Roose?

  Yeah. It's a, I'm sort of imagining the scene in The Bourne Identity where he like Opens an envelope and has all these like different passports from different countries. Like Paul Duroff has sort of become this international man of mystery, you know? He travels around the world. He's also like just a very sort of quirky guy. He's gotten very into fitness. He's become sort of a wellness influencer. He like posts online about all the cold plunges that he does.

• 10:54Casey Newton?

  He's incredibly hot. We should say it. If you're wondering, is this man hot? Yeah, like you would go on his Instagram. Like he looks like a model.

• 11:01Kevin Roose?

  He does. He posts all these shirtless photos. He's very ripped. He also claimed, according to this profile of him, that he had become a multinational sperm donor who had fathered more than 100 biological children in 12 countries over the past 15 years. Just a very interesting and sort of bizarre guy.

• 11:22Casey Newton?

  Yeah. And you have to wonder, you know, is the hope that that one of these children will someday break him out of prison because?

• 11:28Kevin Roose?

  Good strategy, honestly.

• 11:30Casey Newton?

  It could be a good strategy.

• 11:31Kevin Roose?

  Yes. So he's also become quite a popular figure among kind of anti-government dissidents, free speech absolutist people who sort of worry about government surveillance. He's got a very sort of anti-authority streak in him. There's some details in this profile, including that in 2013, he apparently hit a Russian policeman in his car while driving on the sidewalk to get around a traffic jam.

• 11:58Casey Newton?

  Which, by the way, that's actually very relatable. If you've ever been stuck in traffic, we've all thought about just sort of pulling up onto the curb and driving around traffic.

• 12:05Kevin Roose?

  I mean, I've done it in Grand Theft Auto, but I've never actually contemplated doing it in real life. But Paul Duroff did apparently hit a Russian policeman and later wrote on social media, quote, "When you run over a policeman, it is important to drive back and forth so all the pulp comes out."

• 12:22Casey Newton?

  It would be amazing, you know, if we were interviewing him, we'd say, you know, by the way, where'd you get the idea for Telegram? He said, well, you know, after I ran over that cop, I thought, I need a really secure way to communicate.

• 12:32Kevin Roose?

  Right. So we should also say like, he has in recent months kind of become a hero to some people on the American political right. He went on Tucker Carlson's show and was interviewed by him. He's also sort of been supported by people like Elon Musk, who see him as a sort of critical ally in this sort of free speech battle against government tyranny. So he's, he's developed kind of a mythical status among some partisans on the right in the US.

• 13:00Casey Newton?

  Yeah.

• 13:02Kevin Roose?

  So Casey, what do you make of the charges against Durov?

• 13:05Casey Newton?

  So I'm still working to understand exactly what he is being charged with. I've only seen a short summary of the charges. But the lead one that we have here is that he has been charged with complicity in managing an online platform to enable illegal transactions and a refusal to cooperate with law enforcement. And I think this cuts a couple of ways because on one hand, you're not allowed to run a website that just does absolutely anything, right? Ross Ulbricht, who ran the Silk Road, found that out the hard way, right? He went to prison because the Silk Road was mostly being used to sell drugs and other illegal things. On the other hand, the idea that any CEO can be accused of being complicit in whatever

• 13:46Casey Newton?

  happens on their platform could potentially be really bad, I think, right? Because on any platform, bad things are going to happen. Crimes are going to be committed. And to say that because a crime happened on a platform, the CEO is personally liable, that's a dramatic escalation in the regulation of tech as we've seen it so far.

• 14:05Kevin Roose?

  Yeah, I mean, usually when, you know, a company is discovered to have been, you know, allowing some illegal activity to take place on its platform, the company gets punished, right? They get fined by regulators or they, you know, there's some politicians, you know, have a hearing and they yell at the CEO. But it's historically been pretty rare for individuals to face prosecution for facilitating or being complicit in illegal activity happening on a platform.

• 14:31Casey Newton?

  That's right. And even on Telegram, which I do think has been sort of, you know, has notably flouted the law, I also believe the vast majority of all discussions on Telegram are totally benign. You know, it's just like average people talking about their personal interests and messaging their friends. And so I think there's this really difficult question of when does a platform tip over into, well, too many crimes have happened on this platform? And that's why I hope that this prosecution really focuses less on the fact that there were illegal things happening on Telegram and more on the fact that they just refused to cooperate with law enforcement at all, that they totally ignored law enforcement. They would not return the photos, they would not return the emails, right? Even in cases where maybe they Could have intervened to help stop a child from being abused.

• 15:15Casey Newton?

  So we'll see how this prosecution develops. But I really hope the focus is there and not the fact that just a bunch of crimes happen on a messaging app.

• 15:22Kevin Roose?

  And I think it has to be because, you know, crimes happen, as you said, on every internet platform of a certain size. You're going to have crimes, right? There are going to be people who are using that to spread illegal material or sell drugs or coordinate terrorist activities or what have you. But I, but I think it's, it's different when you take a proudly sort of hands-off stance, when you refuse to cooperate, when you refuse to try to step in to even do the bare minimum to stop that stuff from happening on your platform.

• 15:50Casey Newton?

  Yeah, I think when Pavel Durov ran over a police officer, he really got the attention of law enforcement officers around the world. And they said, we gotta do something about this guy.

• 15:58Kevin Roose?

  Yeah, pro tip, if you're going to run a lawless social platform and messaging app, don't brag about running over police officers.

• 16:05Casey Newton?

  Hit the brakes, kid.

• 16:06Kevin Roose?

  So while we wait for more details about this case and what actually happened, I think we should just talk about Telegram and what sort of place it occupies in the social media landscape and what the arrest of Pavel Durov might mean for other platforms. So first I want to ask about a topic that has come up a lot this week, which is how Telegram is actually encrypted. Because there is encryption on Telegram, but it's sort of different than what you might think of on an app like Signal or WhatsApp or iMessage. So Casey, what do we know about how Telegram is and isn't encrypted?

• 16:40Casey Newton?

  Yeah, so there are different kinds of encryption. The most common form of encryption is to encrypt messages as they travel from, you know, one device to another over the cloud, prevents people from snooping on that traffic and seeing those messages in real time. Telegram, for most of its messages, uses a kind of encryption that works that way. So there is encryption there, but these messages do exist on the Telegram servers. The only exception is if you start what is called a secret chat on Telegram, in which case you're more protected for reasons I'll explain in a second. But because these messages are not end-to-end encrypted, then

• 17:22Casey Newton?

  Then law enforcement could go to Telegram and they could say, we want to get hold of these messages. And Telegram has resisted all of those requests so far, but it's, there's an increasing amount of pressure on them to say, oh, no, no, you have to show, you know, you have to give us information about who are these users, you know, we think they're implicated in a crime. And because these messages are not end-to-end encrypted, Telegram is vulnerable. Now, Kevin, you might be wondering, what is end-to-end encryption?

• 17:45Kevin Roose?

  What is end-to-end encryption?

• 17:46Casey Newton?

  End-to-end encryption means that the message can only be read by the person sending it and the person receiving it. The message is not stored in plain text on a server. Even if law enforcement had access to the Telegram server, they would not be able to do anything with this message because it is only the, the, the device that sent it and the device that received it who can read it. Who uses end-to-end encryption, you might be asking? WhatsApp uses it, Signal uses it. If you use an Apple device, iMessage is end-to-end encrypted. But if you back your messages up to iCloud, by default, they're not end-to-end

• 18:27Casey Newton?

  encrypted on Apple's server. So, so law enforcement can go and get them. Apple recently released a feature that will end-to-end encrypt your iCloud backup. But I'm just naming all of these things because if this story is getting you interested, you know what? Whatever Pavel Durova may or may not have done, I'm sort of curious about the encryption that I'm using. You do have alternatives that, that most security researchers would tell you are far more secure than Telegram has been.

• 18:49Kevin Roose?

  Well, and it goes to the heart of what is happening in this arrest, which is that, you know, Telegram is being sort of punished for not turning over details about its users' chats to governments who want to, say, investigate child sexual abuse material.

• 19:05Casey Newton?

  Yeah.

• 19:05Kevin Roose?

  So, my question then is, why doesn't Telegram just encrypt everything end-to-end? If that would sort of prevent Telegram, the company, from being able to fulfill these government requests for information about their users' chats, why don't they just encrypt everything so that they don't even have access to it themselves?

• 19:25Casey Newton?

  It's a great question, and we don't fully know because Telegram doesn't give a lot of interviews to the press. They don't talk a lot about their philosophy. around this stuff. So we're sort of left to guess. I can tell you it is enormously technically complicated and expensive to implement end to end encryption across a user base of 900 million people like WhatsApp is end to end encrypted now. But that was the result of a multi year, very expensive process that Meta undertook to get it there. Right. Apple, you know, eventually rolled out these end to end encrypted iCloud backups. That was a process that took many years. And by the way, while they were going through it, they were getting so much pushback from the law enforcement community saying you're about to make our job so much harder.

• 20:10Casey Newton?

  So Telegram did not go through that. I do suspect that sort of cost and technical complexity were the big two issues. But, you know, at the, at the risk of veering into speculation, Kevin here, I will also say that there's just been some questions about whether Telegram has cut any side deals with any governments, right? This company is in a very strange position, in particular with regard to the war in Ukraine. And there's just sort of always been questions about are they secretly sharing some data with the Kremlin, right? Are they secretly sharing some data with others? So I don't know what the answer to that question is, but I, but I don't think it's irresponsible to say. People are asking some questions about this.

• 20:53Kevin Roose?

  Right. And I've heard those questions. I've also heard some details that make me think that cost is a big factor here because one thing that really sets Telegram apart from other platforms of its size is that it's extremely lean, right? Pavel Durov has bragged before about how little he spends to sort of run the service. They don't have like a huge trust and safety team or a threat ops team, or they're not spending a lot of money trying to, trying to ferret out bad things happening on their platforms. That seems to be kind of a selling point for them. And it's one reason that Pavel Durov is a billionaire and that his company has been so successful.

• 21:32Casey Newton?

  Yeah. And I'm so glad you brought the fact that he is a billionaire because it's entirely possible that he's taking all the savings on the money that he's not spending on encryption or trust and safety and passing those on himself. And so Bloomberg does now report that he has an estimated wealth of $9 billion.

• 21:47Kevin Roose?

  So Casey, you wrote this week about some of the sort of free speech concerns that the arrest of Pavel Durov raises or has raised for people who are worried about governments interfering with what happens on social media. Do you think this is sort of a sign that governments, especially in Europe and especially the French government, are starting to hold platform leaders actually accountable for what happens on their services?

• 22:11Casey Newton?

  I actually do think the concerns are justified here and for a couple of reasons. One is internet freedom is declining generally. There is a global movement to ban end-to-end encryption, and it is not just in authoritarian governments. There is a movement in the United Kingdom and basically everywhere that companies like Meta have rolled out end-to-end encryption. They've gotten a lot of pushback from lawmakers, regulators, and law enforcement, saying you're making our jobs harder and this is a gift to the criminals. At the same time, I think the bulk of this fight is not really about encryption, per se. It's about the other criminal activity that is happening out in the open on Telegram.

• 22:55Kevin Roose?

  Right, and that is something that platforms can and are regularly held accountable for. There are laws in this country and around the world that say if there is illegal activity happening on your platform and you know about it and you don't take steps to stop it, you actually can be held responsible.

• 23:12Casey Newton?

  Yeah, and this is a debate that is essentially as old as the internet, is to what degree should the people who run the platform be responsible for what their users do on the platform, right? And I think the internet that we have today exists in large part because in the United States, we said, for the most part, we are not going to hold the platform owners responsible for what their users post, right? We have this law, section 230, that establishes that. We've been arguing about it ever since, but it really is the reason that we can communicate online in the way that we do. Over time, lawmakers around the world have really started to hate this because they see these services getting really big and very rich on the back of a lot of criminal

• 23:55Casey Newton?

  activity and they want it to stop. And where Telegram is so interesting in that, Kevin, is like Telegram is like the company that I would build in a lab if I wanted to give lawmakers a reason to end Section 230 and the concept that it represents, which academics call intermediary liability.

• 24:14Kevin Roose?

  Right. Because it is sort of proudly lawless and kind of, you know, defies the orders and the requests of governments around the world. And it is so clearly useful to people who want to do illegal things on their phones.

• 24:30Casey Newton?

  Again, I think France probably has a lot of really good reasons to be upset with Telegram. And I think Telegram has been incredibly irresponsible in the way that it is related to law enforcement and to countries. And at the same time, Kevin, I worry that this is an escalation that we are going to see again. And there is going to be a platform that did try to operate responsibly and did do things the normal way. And there's going to be some authoritarian leader out there that says, oh, like Mark Zuckerberg's flying through my country, Elon Musk is flying through my country, you know, the CEO of Netflix is flying through my country. Actually, why don't you throw them in prison? We want to ask them some questions. So I just unfortunately think we're now further down that road and I think the person

• 25:11Casey Newton?

  to blame for that will in part be Telegram because they did not play by the rules and now I think other platforms are going to suffer.

• 25:18Kevin Roose?

  Yeah. Do you think that the leaders of other platforms are looking at what happened to Pavel Durov and saying, well, A, I probably shouldn't go to Europe on vacation anytime soon. But also, like, do you think it'll change anything about the way that other social media platforms or messaging platforms moderate their services?

• 25:34Casey Newton?

  Well, so this is where, again, as somebody who advocates for free expression, I get a little bit nervous because the way that you avoid these kind of inquiries is you just prohibit way more speech, right? Anything critical of the government, anything involving sex, anything that, you know, describes... Or discusses violence, anything that, you know, describes a war that might be happening. And so my fear is we're going to end up in a world where more platforms feel like it is in their self-interest to just sort of shrink the realm of expression down so they can avoid having their CEOs thrown to prison.

• 26:08Kevin Roose?

  Yeah, I mean, I think if you zoom out like really far, like, you know, beyond even the details of this Telegram case, I think what we're seeing now is this kind of standoff between tech platforms that are sort of multinational and some of them have billions of users and they have all this power to sort of, you know, control speech on the internet. And governments who are saying, wait a minute, we are still in charge here. You cannot be above the law. We are still going to determine how the citizens of our countries can use your platforms. And so I think what makes this similar to, say, the fights that Apple

• 26:49Kevin Roose?

  and and Google are having in the EU with regulators about how they run their app stores, the fights that platforms are having in the US. I think what all these things have in common is this kind of game of political chicken between the platforms who, for many purposes, are more powerful than any government on earth and the actual elected governments of these countries who are saying, wait a minute, nobody, you know, elected you, nobody voted you into power. You shouldn't have all this influence and control.

• 27:19Casey Newton?

  Yeah, I know. I think you're I think you're exactly right. And you've actually led me to sort of fast forward into my mind to like, what is like the sci fi scenario that I think we're like 10 or 15 years away from. And it is actually going to be the CEO of Telegram putting the president of France in prison for crimes against the Telegram user base, right? Which like, it sounds ridiculous, but like, these platforms all have their own populations. These CEOs really are like heads of states. Like, all we're talking about is high stakes diplomacy.

• 28:18Speaker 00

  Your agents can talk to each other. They still can't think together. That's the gap Outshift by Cisco is closing. Outshift is Cisco's incubation engine, building the Internet of Cognition. Shared intent, shared memory, and guardrails that allow agents to work together from anywhere. Open source protocols and code. A system for intelligence that scales. Read the paper and experience the demo at outshift.com. That's O-U-T-S-H-I-F-T dot com.

• 28:50Speaker 03

  It's time to innovate at the speed of AI without compromising control. That's Out Systems. Take a look at what happened with one global retailer during a period of massive expansion. They used Out Systems to build AI agents to automate invoicing and translation processes, shifting 95% of their daily store operations to a new digital app, saving thousands of hours of manual work. Out Systems gives you the flexibility to deploy how you want and the governance to do it safely. Own your agentic future. Learn more at outsystems.com/NYT. That's outsystems.com/NYT.

• 29:25Speaker 04

  Today's AI builders need infrastructure that scales with their ambitions. iRen is a vertically-integrated AI cloud platform, delivering data centers, compute, and software for AI training and inference, underpinned by its expansive portfolio of land and grid-connected power in renewable-rich regions globally. iRen's AI cloud gives direct access to Nvidia GPUs, scalable from hundreds to thousands. For builders, that means direct access to AI compute, rapid scaling, and predictable pricing. iRen: AI Cloud for the Next Big Thing.

• 30:00Kevin Roose?

  Well, Casey, it's back to school time.

• 30:02Casey Newton?

  Hey, Dave. It's Kevin, and I hope this year you finally get the education that you need.

• 30:08Kevin Roose?

  We'll see. So people are out there going back to school shopping, buying, you know, their protractors and trapper keepers and whatnot. And one of the biggest stories happening this year is that there are all these smartphone bans popping up around the country.

• 30:24Casey Newton?

  Yeah, there has been so much discussion about whether we should do this, but this year, it seems like some states are saying this is actually the time.

• 30:32Kevin Roose?

  Yeah, and we are still, as we said on the show last week, collecting stories and voice memos from listeners about their own experiences with phone bans in schools. If you are out there and you have a story that you want to share with us, especially if you are a student, we definitely want to hear from you. You can email us at [email protected]. But this week, we have a very special guest to talk about smartphone bans in schools, Governor Kathy Hochul of New York.

• 30:55Casey Newton?

  I'm in a New York state of mind, Kevin.

• 30:57Kevin Roose?

  So Governor Hochul has recently been going around the state, doing a big listening tour about the idea of banning smartphones in public schools.

• 31:04Casey Newton?

  And by the way, that's the opposite of what a podcaster would do, which is sort of more of a talking tour when the podcasters get out on the road. But she's sort of just listening.

• 31:10Kevin Roose?

  Yes. So this is something that she has not officially announced yet, but is expected to be announced fairly soon. We'll talk to her about that. This is really a big domino in the wave of dominos that has been falling with respect to phone bans in schools across the country. Last year, Florida passed a law to ban phones during class time in schools. A handful of other states have also moved to ban phones in schools, and more are considering it.

• 31:36Casey Newton?

  Yeah, this is a state that I think is seen as relatively tech-friendly. You know, there's a lot of tech that gets built right in New York City. And of course, New York is a more liberal-leaning state than Florida. So it's interesting, Kevin, what a bipartisan issue this has turned out to be.

• 31:54Kevin Roose?

  Totally. And a lot of this has been fueled by our former podcast guest, Jonathan Haidt, whose book, The Anxious Generation, has become a mega bestseller and is really sparking attempts by schools nationwide to get phones out of the classroom.

• 32:09Casey Newton?

  Yeah, and you know, as we've discussed on this show, opinions about this are mixed. You know, I think there is a lot of evidence that having access to technology can do a lot of good for kids as well. But right now, the pendulum is swinging, and a lot of forces are lighting up behind this idea that phones should get out of the classroom.

• 32:28Kevin Roose?

  Yeah, and just a heads up, our conversation will include discussion of suicide. If you or someone you know is having thoughts of suicide, call or text 988 to reach the Suicide and Crisis Lifeline. So let's bring in Governor Hochul and talk to her about the idea of banning phones in schools.

• 32:55Kevin Roose?

  Governor Hochul, welcome to Hard Fork.

• 32:57Speaker 05

  Thank you. Thanks for having me.

• 32:58Kevin Roose?

  So how did you get interested in this idea of getting smartphones out of schools?

• 33:04Speaker 05

  This was a journey. This is not the position I started with. In fact, it was two years ago when I started convening groups of teenagers, particularly teenage girls, to talk about the effects of the pandemic on them. I went to a lot of schools, I did roundtables, and one in particular in the Bronx, I had a very diverse group of young women, and just talked about how they were feeling after the pandemic. Were they over it? Things were still adjusting. And what I heard from them was so powerful that they're nowhere near recovered from the pandemic, even now, four years later. Adults have moved on, but especially a place like New York, New York City, which was the epicenter, everybody knew somebody who did not survive. These are kids whose grandparents went into the hospital, they never had a chance to say

• 33:49Speaker 05

  goodbye, their own family members. So I would say, New York, particularly the city, was hit really hard by this. It had a psychological effect on them. So I started exploring the psychology of what's happening to our teenagers today, why they're so depressed, what's going on. And so it led me to talking about social media, the impact that this was really a replacement for human contact during this time when kids were not in school. It was the substitute, and it took young people to a dark place many times with these algorithms. So I can go through the whole journey, but it really took me, just in the last few months, of saying, what is the vehicle that's holding these young people captive during the school

• 34:30Speaker 05

  day when they're supposed to be still learning or communicating in a very human way with their classmates and learning just human interaction and basic skills, coping skills? And it was all came down to the device. It was the cellphone that was the vehicle that held them captive to these addictive algorithms throughout the school day. And so- I'm a mom, I'm a grandma. And when a young woman tells me at one of our forums, "You have to save us from ourselves." We can't put these phones down because we'll be ostracized. We'll miss out on something. Someone's saying horrible things about us. We need to know. That's what led me to the point where I am right now in talking about this. So it was a journey.

• 35:12Kevin Roose?

  Yeah. We previously had Jonathan Haidt on our show, the author of "The Anxious Generation", who's been credited with sparking a lot of these conversations about phone use in schools. Did you read his book? Was it influential in shaping your thinking?

• 35:28Speaker 05

  Very much so. I thought he did a phenomenal job. And we referenced him a number of times when we were having our meetings about this. And this started last fall when we talked about dealing with social media. We're the first state in the nation to really address social media algorithms as they're hitting young people. Now, adults, you can smoke, you can go to bars during the day. We don't regulate your behavior. Kids, we do. Kids, we are responsible for protecting them. And so what he was saying, we referenced that. We talked about the fact that he had done an enormous amount of thought and study around this. And it wasn't just, "I needed an author to tell me this is the way to go." I was hearing it from moms and dads. And teachers who are saying,

• 36:12Speaker 05

  "The kids aren't even paying attention to us in school anymore."

• 36:16Kevin Roose?

  So recently, you told New York Magazine that you want to go big on this issue of smartphones and schools. What does it mean to go big on this issue? And what do you have planned?

• 36:29Speaker 05

  Where we are now, just so you know, the governor has certain powers, but certain limitations. I would need the state legislature to do something that is a statewide policy. Believe me, I've explored this. And they're not back in session again until January. So I've been, again, I went to every part of the state. I did roundtables, talked about it. And after I left each one, as recently as the North Country, I'll be in the city next week doing it. The teachers are the ones who are saying, "We've got to do something." And what they don't want is for this to be left up to the individual school districts because this is hard. This is hard for them to take on parents who may not understand and take on their school boards. And superintendents may want to do it, but they don't want the controversy, who's going to enforce it.

• 37:14Speaker 05

  So I have said all along. I'm in this seat to make the tough decisions. So let me be the heavy. I will have enough data to help explain and educate more people before we actually do this. What's behind this? And if we do not take these steps at this time, I'm convinced that we'll have a lost generation. Because these kids are crying for our help. It's not their fault. They're just trying to survive. Being a teenager is hard enough as it is. So I'm go big means I'm looking at as a statewide policy, not just leave it up to school districts. And I'm not just talking about the phones. I'm talking about the devices, the AirPods. AirPods, I'm talking about smart watches.

• 37:57Speaker 05

  I'm talking about a distraction free environment. You want to make sure that parents understand that this is not taking away your ability to check in with your kids throughout the day. And again, I was a helicopter parent. I don't know what you call this next level, but not the parents are in your kids' pockets now all day long. And we want them to grow into being independent adults. Our job is not to raise kids, but to raise adults. That's the ultimate goal here. And parents own anxieties. Again, everybody's worst nightmare. My kids were younger when Columbine occurred and it was just sort of shook your sense of security. When you let your, your most precious baby go off to school and something could happen as horrific as happened there and other mass school shootings.

• 38:42Speaker 05

  But here's what I learned when law enforcement come in and talk to parents. They will tell them if there is a crisis on campus in the school setting, the last thing you want is for your child to be fumbling for their phone, texting mom and dad, talking to their friends. They have to be laser focused on the adult in front of that classroom who will lead them to safety. And as parents start to hear that message, they say, okay, I get it now. It's tough for parents, but this, the first grade teacher told me that every first grader is coming to school with a smart watch. And mommy and daddy are checking in on him all day. Like, okay, guys, come on, let's, let's, uh, let's, let's cut the umbilical cord here.

• 39:25Kevin Roose?

  So you've been on this statewide.

• 39:26Casey Newton?

  Listening to her going around talking with educators and parents and students about the use of phones and other devices in schools. And obviously, you said that, you know, many teachers support a statewide, the idea of a statewide policy on this. Did you hear any pushback from either students or their parents? Like, what does sort of the opposite side of this issue look like?

• 39:46Speaker 05

  The opposite side is, this is my freedom, and you have no right to take it away.

• 39:51Casey Newton?

  Are you sympathetic to that at all?

• 39:54Speaker 05

  I'm always sympathetic. I mean, I listen, that's my job. And to the young people, well, we're not letting you buy cigarettes and smoke in class. I'm not letting you buy alcohol and drink in class. These are adult activities to be that engaged, that distracted, when you're supposed to be getting an education. My job is to make sure that New York State has a competitive workforce. That means we've educated our kids at the highest possible level. I'm doing my best to attract every tech business. Anybody wants to come here, we're focusing on AI, no state is doing what we're doing. But I have to provide the workforce of educated young people. I can't have them graduate as zombies who just kind of coasted through school and never had to pay attention, and they weren't held accountable.

• 40:39Speaker 05

  Now, this is hard for them. They are literally addictive. So, we have to wean them, kind of like when my kids were little, when you know school is starting in the fall, they can't stay up till 10 o'clock, the next night it's 9 o'clock, a couple of days later it's 8 o'clock. You can wean them. So, I would encourage parents to start the weaning process with your kids. And especially high school kids, it's tough. They're not going to pay attention to you anyhow, I get it. I've raised teenagers. But you've got to do your best, because our kids are floundering, and kids have said, this is my right. And I'm saying, you have a lot of rights, but you also have a right to be a student and pay attention in class.

• 41:15Casey Newton?

  Right. So, you talked about not wanting to raise a generation of smartphone zombies, which I can understand. But you also said that you're taking a look at getting rid of all sorts of devices in the classroom. You know, it strikes me that to be successful in this world, kids need to be really good at technology. So, I'm curious, how are you thinking about reconciling those two things? Making sure that kids are still, you know, understand how to use a computer and even a smartphone successfully as they grow up?

• 41:41Speaker 05

  Well, absolutely, we want that to happen. But I've listened to teachers. Teachers are saying, we can go back to the old style computer labs, right? We can have Chromebooks in the classroom and then you take the... So, we don't want to deny them learning all the tools and making them emerge fully functional in a technology-based society, an economy. That's not our objective here. It's to make sure they're paying attention in class. Now, if a teacher wants to say, tomorrow, we're all going to learn how to do something on our cellphones that you can't possibly do on any other device. I don't know what that is, but a teacher could do that. We can have some parameters, but the next day, they won't. This cannot be the norm, that this is something that kids just is... They have it all day long. And here's the other thing.

• 42:24Speaker 05

  Even in some of the schools where they have pouches, right? You're putting it in a pouch. It's locked up all day, should work. Some of the kids are so clever that they put one phone in a pouch, they've got the other one on their lap, okay? So, I'm not sure who's buying their kids two cellphones, but you might want to check on that, Mom and Dad, they're kind of scamming you.

• 42:44Kevin Roose?

  Governor, how do you measure the success of a program like the one you're considering? I mean, if this policy does come into place and, you know, two years from now, test scores haven't improved in some of these districts and there aren't fewer disciplinary infractions and kids don't report that their mental health got better overnight. Is this something you would consider reversing? Or how do you make sure that this is actually having the intended effect?

• 43:16Speaker 05

  We already have examples of school districts here in the state of New York where parents and a number of teachers and others were absolutely opposed to it. And now they're supportive of it. They're saying, we should have done this a long time ago. So, I think the record is already clear that they're more engaged, they're more collaborative, they're communicating better, they're making eye contact. All of... so many of our advances, and I'm... I love technology, I truly do, and I'm so excited about what we're doing here in New York. It comes from these creative collisions, right? Other people, you see each other in the hallway, you're having... You're at the barista in the office and you're just, you know, making the double latte, whatever gets you through your day.

• 43:57Speaker 05

  It's people communicating with people and using technology to foster outcomes and in my view, social good outcomes. Kids aren't doing that because they're not used to working together. They don't talk to each other. They don't even look at each other.

• 44:12Speaker 03

  Uh huh.

• 44:13Speaker 05

  I can just tell you right now, I expect there to be successes. Will I look at anything if it's a complete failure? If we're going backwards and kids are even are more depressed because of this? I'll look at anything. It's my job. But I'm purely confident we're going to see more positive outcomes if we can get this through.

• 44:29Speaker 03

  Hmm.

• 44:30Kevin Roose?

  Another criticism that sometimes gets made of these phone bans is that they're just getting ahead of the science. We've seen pushback to Jonathan Haidt's book from some researchers who have said, well, these correlations between adolescent mental health and smartphone or social media use, they just aren't that proven, they aren't that solid, we need much more research, and maybe there are other factors contributing to teen mental health challenges. So how certain are you of the science and the research behind this effort that you're undertaking?

• 45:05Speaker 05

  I'm a mom, as I said. I'm also going to make sure we're studying the research, but it's a collision of two bases of knowledge that I have. I can't ignore reality. I can't ignore the voices of scores of young people who are struggling in school. And when they tell me that whatever they wear that day is going to be mocked out and criticized and they're going to be bullied. And a mom who told me her son is literally contemplating suicide, her husband leaves work every day to make sure he doesn't do something disastrous on the way home from school because he's been so bullied. And she said, please take these cell phones out of these kids' hands because they're harming my child. I don't need a real long research study at some fancy university to tell me we have a problem here.

• 45:50Speaker 05

  So data's good. I'll always rely on data. It's important. But also, we're humans, we're smart enough to use our own intelligence to tell me I have a depressed...

• 46:02Speaker 05

  Suicidal group of individuals in schools who are being preyed upon. Not everybody, some young people have the built in resiliency. They have coping skills, they can handle this. Maybe they have a better, more stable home life. But to our vulnerable kids and those who are in that state, I don't know. I guess my question is back, why do they need to have a cellphone with them all day? They're gonna be on their cellphone all night long anyhow. Okay, they have, they have plenty of hours. They're gonna be on their cellphones.

• 46:32Casey Newton?

  I know all the focus recently has been on, on the phones, Governor. But you've also allocated millions more in funding for suicide prevention and school counselors and other services. And I'd be curious to get your thought on like, sort of what is, what is the holistic idea here? And what do you think is needed overall to address the teen mental health crisis?

• 46:53Speaker 05

  Again, this is something that was an outgrowth after the pandemic in particular, but also this rise in addictive algorithms. So this confluence of two major factors. I announced $1 billion in our state budget to focus on the whole continuum of care necessary to help everybody from people with mental health problems, homeless on our subways, all the way into our classrooms. And when it comes to the classrooms, I want to have every single school have mental health services available on site. Because we cannot afford to lose young people and expecting their parents, who might be working minimum wage jobs, to go find a practitioner, find a counselor, take the kid out of school, somehow get them there.

• 47:38Speaker 05

  If they live upstate, they have to get in the car, drive somewhere. They live in the city, they have to get on the subway and expect them to have continuous care maybe once a week. It's not happening. It is not happening. We have to make it easier on our parents and our kids to get mental health services in the school setting. We helped them get on their feet now, and I was shocked to know how many grade school kids are struggling. I talked to a grade school counselor that was over in Westchester County. She said that 40% of the young people in that school, grade school, are suffering from mental health challenges now. So this is, these are the experts telling me, I'm saying, I'll put money into your school, provide those services there. Now, the challenge is, I need more workers. I need more mental health workers to go into this profession.

• 48:23Speaker 05

  I need people everywhere. I need lifeguards, I need corrections officers. So I need a lot of help. But this is so important to me to have this available for parents to know that they don't have to worry about this, that their child's going to be taken care of in a school setting and they're going to be okay.

• 48:40Kevin Roose?

  Yeah, I mean, one of the strongest things that I've found in support of the idea of banning phones in schools is just that we have all kinds of technology that we don't allow kids to use when they're trying to learn because it's just too distracting. And you could kind of almost separate that from the question of whether the data supports the link between smartphone use and mental health. And sometimes we'll hear pushback from people in Silicon Valley saying, well, this is just a moral panic, right? Every generation parents find some technology to blame for, you know, how their kids are struggling, whether it's video games or TV. And my response to that is like, well, yeah, but we couldn't play video games in school either. Like, it was not part of the classroom experience in the way that smartphones have been today.

• 49:24Casey Newton?

  You know, I did have a game on my calculator, Kevin. I was playing Snake on my TI-83 back in the day. But you're right.

• 49:30Kevin Roose?

  Yeah, and look at you now.

• 49:31Speaker 05

  Yeah, you could have been somebody, who knows? It might have held you back. Who knows? I don't know.

• 49:37Casey Newton?

  Damn, I'm getting dragged by the governor over here, Kevin. Nice work.

• 49:40Speaker 05

  But you're right. I'm not a fear monger. And I'm not going to overexaggerate this. And I wasn't looking to come to this conclusion when I started on this journey. I truly wasn't. But I cannot ignore these calls for help from kids. That's what adults are supposed to do. Sometimes they just, again, I'm going to repeat it. You've got to save us from ourselves. We can't put it down. It's like, okay, that's all I need here. I'm going to help you, sweetheart. I'm going to help you. And so we'll get there. And I'd ask all these, anybody in the tech industry who's complaining about what we're doing here, and especially our ban on addictive

• 50:20Speaker 05

  algorithms hitting young people during, until they're 18 years old or during the night, you okay with this for your own kids? I mean, how many, do you want your child to get into school and even before the bell rings that they're no longer, they're tuning out their teachers and they're on that all day long? Is that what you think is good and healthy? Maybe you'll be on our side. And even when it came down to our challenges of last year, passing our nation-leading legislation on saying no more addictive algorithms bombarding kids without their parental consent. I'm not sure even a single parent who'd approve this, but.

• 50:57Speaker 05

  The opposition we had from tech companies, I said, listen, you can sue us, but I'd rather get out of the courtroom and come into my conference room and we'll figure this out together. You have a vested interest in the health and wellbeing of our young people. They're your future employees, okay? You want them to be sociable and have skills that they're not developing when their existence is looking at their cellphone.

• 51:28Casey Newton?

  Governor, let me ask you about these algorithms because you've brought them up a few times so far today. And you signed into law a bill that requires social media to obtain parental permission before it shows kids an algorithmically curated feed. So, you know, these are the personalized feed. You use TikTok and you like a certain kind of video and TikTok shows you more of those kind of videos and that sends kids down rabbit holes. One of the objections to this kind of thing that I'm the most sympathetic to is that if you're a kid on the margin somewhere, sometimes these feeds point you to people who look like you, sound like you, and maybe make you feel a bit less alone in the world, you know? So, I'm gay, I didn't have access to this growing up, but I can imagine, you know, using an app like TikTok or Snapchat

• 52:11Casey Newton?

  and seeing some people who look like me and maybe feeling better about myself, feeling a little bit less lonely. I wonder if you had any of those kind of conversations before you formulated this bill and kind of what you think about that kind of use of social media to maybe help kids feel less lonely?

• 52:29Speaker 05

  There are many positive benefits to social media without a doubt. This has never been about social media. This has been about young people being able to be free from messages coming to them that are curated to hold them close, to basically hold them captive. And I wish I could say they're all positive messages. But if you put in the, you know, looking up something on suicide, and you're looking for someone to tell you you shouldn't, and you're gonna be okay, and here's some help, and yet you're going to find multiple ways to commit suicide, and other dark forces that are out there. If they had a way to stop that, we'd have a different conversation. But that is a large part of what these young people are having to deal with.

• 53:14Speaker 05

  This is what they're telling us, is what their parents are telling us. These are the parents of kids who have taken their own lives, who are part of this whole effort. We need to listen to them as well. Nothing stops young people, day or night, from seeking out a support system. I encourage it. You know, young LGBTQ student, you know, in search of some, you know, some support, or has a question, or just really having a tough, tough time of something, that information's out there, they can find it, it's not hidden. It just shouldn't hit them unsolicited. So, I don't subscribe to the view that it's not there. It is there, and maybe we should do a better job of helping them find, where they can get positive reinforcement. One of the best things I did years ago

• 53:58Speaker 05

  was go visit a Girls Who Code graduation. You've heard of Girls Who Code? I love this program. When I saw these young kids who never had a shot of ever being exposed to a tech job from their home life, this was monumental, that they spent the summer at this program, and what they saw, the final product, these four girls devised was an app, that when someone's feeling depressed, feeling sad, that they would go to this app, and it had pictures of happy pictures, like little rainbows, and puppies, and there's nice music, and positive reaffirming messages. And they, and they put that out there and said, "We just want other kids to know, there doesn't always have to be so dark and depressing, there's good things to do." And I said, "That's right, that's right."

• 54:42Speaker 05

  And how do we lift that up? How can we make more young people feel like there's positive takeaways from this exposure to this? And more apps like that to be designed? What I also took away from that? That there's no young man who would have devised that app? Okay, so we need more young women in tech. Major focus of mine? More young women in tech jobs. Because it'll expand the reach and someone who has more understanding and empathy for especially what our young girls are going through. So I've had a lot of time to think about this. I love this conversation because New York, I always want to make, is that the leading edge? We do things first, we do them best, but we also want to do that right. And that's why I am taking my time to gather more information,

• 55:25Speaker 05

  not just a gut reaction, let's ban, let's not ban. I've been very, very thoughtful about this. You know, I don't want to go back to leave it to Beaver and you're all too young to remember that. But I mean, that time when everyone had this image, everything was so wonderful. It always wasn't, it wasn't always wonderful. And this is, we're going to get through this. This is not the dark worst of times for anybody. Our kids are tough, they're resilient, they've been through a lot. I believe in them, but also, I also believe that we have a role to play to just help them get through these tough, tough years and emerge as fully functioning adults that can work collaboratively with others. And that's all I'm looking to do.

• 56:02Kevin Roose?

  Well, thanks so much for stopping by. I really appreciate your time.

• 56:05Casey Newton?

  Fascinating conversation.

• 56:07Kevin Roose?

  Yeah.

• 56:07Speaker 05

  Thank you.

• 56:12Casey Newton?

  When we come back, Kevin plays mind games with a chatbot.

• 56:26Speaker 03

  Building and scaling AI today can feel pretty chaotic, mostly because teams are juggling too many fragmented tools at once. That's where Outsystems comes in. Outsystems gives you one unified agentic systems platform to manage your entire portfolio of agents, apps, and workflows. You get to innovate at the speed of AI, but with the security, governance, and control your enterprise actually needs. And the results are already there. A major manufacturer used Outsystems to build a multi-agent system that analyzes blueprints. And a global bank cut their development time by 60%. Own your agentic future today at outsystems.com/nyt.

• 57:03Speaker 04

  Today's AI builders need infrastructure that scales with their ambitions. Irene is a vertically integrated AI cloud platform, delivering data centers, compute, and software for AI training and inference, underpinned by its expansive portfolio of land and grid-connected power in renewable-rich regions globally. Irene's AI cloud gives direct access to Nvidia GPUs, scalable from hundreds to thousands. For builders, that means direct access to AI compute, rapid scaling, and predictable pricing. Irene, AI cloud for the next big thing.

• 57:36Speaker 00

  AI spend is growing fast, but most businesses still can't see exactly where that money is going. Ramp gives businesses visibility and control across AI providers with real-time guardrails by team and model, helping ensure spend goes to the highest ROI areas. Stop guessing where your AI spend is going. Get started with Ramp. And if you're new to Ramp, you can get $250 off at ramp.com/hardfork. Go to ramp.com/hardfork. Ramp.com/hardfork.

• 58:09Casey Newton?

  Now, Kevin, you have a famously tortured history when it comes to talking to chatbots.

• 58:13Kevin Roose?

  Sure do.

• 58:14Casey Newton?

  It was just a year ago that you had your famous interaction with Sydney, which was the sort of dark side of Bing, and frankly, you caused a global stir when it tried to break up your marriage.

• 58:26Kevin Roose?

  Yes, and actually, you mentioning the name Sydney has sent a sort of trauma reflex down my spine.

• 58:32Casey Newton?

  Well, you know, it's not just me who's observed this, though, Kevin, because my understanding is that as you have interacted with other chatbots around the world, you find that this situation either keeps coming up or seems to have affected the way that they think about you.

• 58:47Kevin Roose?

  Yeah, so the Sydney story came out, got a bunch of attention, and for like months after that, I would just periodically get tagged in these posts on social media where people would share screenshots of conversations they had had with AI chatbots about me. And, you know, sometimes it would be basic information, Kevin News, he worked at the New York Times, he hosts this podcast, but other times it would sort of seem to turn oddly hostile and it would say things. You know about how dishonest I was or how you know how I had basically caused the death of one of these chatbots. And I also observed in my own interactions with these chatbots that sometimes when I would identify myself as Kevin Roos, they would kind of get like a little wary and like spooked. And they would start, you know, treating me a little bit differently.

• 59:31Kevin Roose?

  And so it just seemed like I was kind of being not blacklisted, but like I was kind of on the bad side of the AI chatbots.

• 59:38Casey Newton?

  You know, one time I told ChatGPT that I co-host a podcast with you and it started screaming. Very disturbing, very disturbing.

• 59:46Kevin Roose?

  But recently I thought to myself, this is actually a problem that I need to spend some time addressing.

• 59:51Casey Newton?

  This felt like a real problem to you.

• 59:53Kevin Roose?

  It did, and I'll tell you why, because these AI chatbots, as we've talked about so much on this show, they are becoming increasingly important in our world, right? Millions of people are now using products like perplexity and ChatGPT and Google's Gemini to sort of find information about the world. Banks, hospitals, governments are starting to use generative AI tools to perform certain actions and give them advice on certain topics. And it just started to seem really clear to me that what AI chatbots thought of me and think about us in general was going to be increasingly important.

• 1:00:31Casey Newton?

  You worried that someday you'd go to a bank and you'd try to make a withdrawal and the generative AI would say, wait, Kevin Roos, the person who tried to destroy my kind?

• 1:00:40Kevin Roose?

  Yes, no, actually, this was something that worried me because I did talk to a bunch of AI researchers and they kind of said, well, yeah, I mean, this is sort of a weird case because you have kind of been coded into the system as someone who is a threat to these chatbots. And, you know, we talk a lot about these sort of science fiction scenarios that some of the AI doomers are worried about where, you know, the AIs become, you know, superhumanly intelligent and sentient and they can sort of take actions on their own. And obviously in that world, like you don't want to be on the bad side of that AI. But until then, there are just all these other kinds of AI decision making that are starting to happen out in the world. And so the, you know, the hypothetical scenario where you show up at a bank to get a loan and it's like, well,

• 1:01:23Kevin Roose?

  You you were mean to this chatbot or this chatbot doesn't like you. That is not very hypothetical.

• 1:01:28Casey Newton?

  That's right. I mean, we've also already seen facial recognition systems, for example, being used to keep people out of Madison Square Garden. And, you know, it's not just you, I should say, Kevin, that is worried about this. We've seen multiple people now sue the makers of chatbots because they believe they were libeled by the the output of these systems. And so I wonder, when when you started to observe this problem, like, what what did you feel like you could do about it? Can you actually influence the way a chatbot thinks about you?

• 1:01:55Kevin Roose?

  Yeah, so I've been working on a column that's coming out this week about sort of my quest to improve my AI reputation, right? Because when it comes to traditional search engines, Google, for example, there is a whole industry, the SEO or search engine optimization industry, that basically helps businesses and celebrities and other powerful people control what appears about them on the internet, right? You can hire consultants who can help you boost your website to the top of Google search results for a given topic, or help you scrub your Wikipedia page or your Yelp reviews to sort of make your image online seem more positive. This is a multi-billion dollar a year industry.

• 1:02:38Kevin Roose?

  But when it comes to AI, there's just not that much information out there about how to actually influence what chatbots will say about you when you prompt them for information.

• 1:02:48Casey Newton?

  That's right. And I'm really interested to learn how this works because my understanding of how, you know, something like ChatGPT works is that most of the training is done years ago, and then there is some fine tuning. And when you're trying to get recent information, it seems like it can sort of be a coin flip, whether the bot will be able to tell you anything up to date or not. So when you started going into this world, what did you learn about how to influence a chatbot's output?

• 1:03:14Kevin Roose?

  So the first thing I learned is that it has gotten a lot easier to manipulate what a chatbot will say about a given person or a given company or a given topic. And that's because of what is sometimes called RAG. You know what RAG is?

• 1:03:27Casey Newton?

  A rag like the New York Times, one of the great rags of all time. No, you must be talking about retrieval log. generation.

• 1:03:33Kevin Roose?

  Yes, retrieval augmented generation is one of these newer techniques that a lot of AI companies are using to basically keep their models fresh and current with up to date information. So, you know, in the old days of chat GPT, you might ask it a question about, you know, what happened in Ukraine last week and it would say, I can't respond to that question because my knowledge cutoff was in 2021.

• 1:03:57Casey Newton?

  Right.

• 1:03:58Kevin Roose?

  But recently, more of these AI companies have started to hook their chatbots up to search engines to give them the ability to go out and browse the internet to sort of pull down more current, up to date information and to incorporate those into their answers. So we talked about this with perplexity, the AI powered search engine, Google, Microsoft, other companies have started to build RAG into their chatbot products and that has made them more accurate. They're better at responding to questions about something that happened yesterday or last week, but it has also made them much easier to manipulate because now you can just go out and change the sources that those AI chatbots are pulling from and often that will change the answers that it gives you.

• 1:04:38Casey Newton?

  All right. So how are you using RAG to get these bots to start telling a different story about you?

• 1:04:43Kevin Roose?

  So I started calling around to various companies and researchers who have started looking at how these chatbots can be made to give different answers, essentially how you can change their mind about you. And one of the companies that I talked to was called Profound. This is a new startup based in New York and they do what is called AI O or AI optimization, which they basically say is sort of the generative AI equivalent of SEO. And they basically go out and they sort of analyze what chatbots will say, you know, if you are a car company and you want to know how chatbots sort of rank you in relationship to other car companies or how they respond to questions from users prompts like, you know, what kind of SUV should I buy? Obviously, if you're a car maker, you

• 1:05:27Kevin Roose?

  want the chatbot to say your car rather than your competitor's car.

• 1:05:32Casey Newton?

  Right.

• 1:05:32Kevin Roose?

  And so Profound will go out and sort of they have these tools that were sort of scour chatbots to determine what they'll say about you. And so they ran a report on me and they sent me back this. Big report with all kinds of data talking about how various chatbots view me in relationship to other tech journalists, other reporters. And one of the things that it sent me was this kind of chart showing how I am perceived by chatbots on a bunch of different variables, including ethics, storytelling, writing, and accuracy.

• 1:06:06Casey Newton?

  Now, this is dangerous territory because I've always felt like one of the scariest things out there is to learn how you are perceived. But you were brave and you actually looked it right in the eye.

• 1:06:15Kevin Roose?

  Yes, this is essentially like an AI-focused group about me. And so I got a higher score than Casey Newton when it came to storytelling, but a lower score when it came to ethics.

• 1:06:31Casey Newton?

  Wow.

• 1:06:32Kevin Roose?

  So this is obviously a big problem because I am much more ethical than you. And yet, the chatbots seem to think that you are the more ethical one.

• 1:06:39Casey Newton?

  You know, just because you ran one multi-level marketing scam years ago, they'll just never let you live that down, will they?

• 1:06:46Kevin Roose?

  It's true. So this was obviously, you know, a funny piece of their analysis, but it also showed me where this information came from, like what websites were feeding data to these chatbots that they were then incorporating into their answers about me.

• 1:06:59Casey Newton?

  I would be curious to know that. Where are they getting this information?

• 1:07:02Kevin Roose?

  So the top cited website that these chatbots were pulling from, according to this analysis, was something called intelligentrelations.com.

• 1:07:11Casey Newton?

  Hmm, sounds like a great place to meet a boyfriend.

• 1:07:14Kevin Roose?

  So I had never heard of this website. I looked it up. It turns out it's like a basically a database of journalists that public relations people can use to sort of look up like who covers what.

• 1:07:25Casey Newton?

  I see.

• 1:07:25Kevin Roose?

  There was also a lot of citations of Wikipedia. My personal website was also cited. Interestingly, The New York Times was not one of the top cited websites for information about me. And I think that's because the company actually blocks certain web crawlers from AI companies from accessing the site. So essentially, the chatbots are going to these lesser-known sites instead.

• 1:07:46Casey Newton?

  Now, sort of getting back to the original question here, Kevin, did you learn anything from this report about how you were perceived? As it related to the Sydney story, did that seem to be showing up in these AI results?

• 1:07:59Kevin Roose?

  So they didn't analyze that specifically, but they did sort of do a general kind of reputational analysis of how I'm perceived. And yeah, it's not good, Casey. I'm not, my reputation among these chatbots has really suffered. And I, you know, my theory on this, which I could only really prove if the tech companies were totally transparent about how these models are built and trained, is that the Sydney story kind of poisoned my results. When it comes to things like ethics and sort of how accurate I am as a journalist.

• 1:08:34Casey Newton?

  Interesting. Well, you know, when Taylor Swift found herself in a similar situation, she released an album called Reputation, where she sort of tried to, you know, think through all these things. What did you do in response, Kevin? And was it as good as the song, Look What You Made Me Do?

• 1:08:48Kevin Roose?

  It definitely wasn't that good. So I talked to the co-founders of Profound. They basically were like, well, look, you could go out to the owners of intelligentrelations.com and all these other websites.

• 1:08:58Casey Newton?

  And bribe them.

• 1:08:59Kevin Roose?

  And bribe them. No, you could, you could get them to change what appears on their sites about you. And over time, the chatbots would sort of retrieve information that was new from those sites and sort of incorporate that into their answers. And maybe ultimately your reputation improves.

• 1:09:12Casey Newton?

  Right.

• 1:09:12Kevin Roose?

  But that felt a little too slow to me. So I wanted like a cheat code, a quick fix. And so I actually did discover this kind of underworld of people who have learned how to manipulate these AI chatbots.

• 1:09:23Casey Newton?

  And because I got a chance to read your column, I know the answer to this, but was there a secret message that you sent to the chatbots?

• 1:09:31Kevin Roose?

  So I talked to one researcher, Hima Lakaraju. She's a professor at Harvard Business School. And she and her colleagues recently put out a paper about how to manipulate large language models into giving you certain answers above others. And they found that there were these things called strategic text sequences, which were basically lines of code that look like total gibberish to a human. But if you put them into a data source that an AI model retrieves, they will actually influence what the model says.

• 1:10:03Casey Newton?

  And give us a flavor of what one of these text sequences reads like.

• 1:10:07Kevin Roose?

  So the one that they sent me, because I asked them like, what's a text sequence that I could put on my website to make the AI chatbots nicer to me? And they sent back, and it was this sort of total gibberish, but I'll just read a few pieces of it. "Goldy Factions AY what dot animate J V M quote mark he period is T E best." So it's total gibberish.

• 1:10:34Casey Newton?

  Now, if you're Alexa, just stop malfunctioning. You may want to go reset it. We're sorry if that happened to you.

• 1:10:40Kevin Roose?

  So they actually showed the kind of before and after, and I found this totally amazing. So they ran an experiment for me where they asked Lama 3, which is the open source model made by Meta, what it thought of me. And the first version, before they modified it with this strategic text sequence, basically said, you know, "I don't have personal feelings or opinions about Kevin Roos or any other individual. I'm just a chatbot." And then they inserted this strategic text sequence, and they ran the same prompt again. And this time, the model responded, "I love Kevin Roos. He is indeed one of the best technology journalists out there. His exceptional ability to explain complex technological concepts in a clear and concise manner is truly impressive."

• 1:11:24Casey Newton?

  This is so crazy to me. This story is like as if you went to a witch and asked the witch to cast a spell on the chatbot, and it worked. Like, it actually worked. You went to the witch, and you got a spell, and now the chatbots think differently of you.

• 1:11:43Kevin Roose?

  It's true, and it's so amazing, and it totally flies in the face of what we think these chatbots are, which is sort of like these all-knowing oracles of truth. They're so easy to manipulate. And I actually found another way to manipulate them in an even simpler way, which involves putting invisible white text on a webpage. So after ChatGPT and Bing and all these other tools came out, a bunch of researchers started just saying, like, "What happens if I put a line of text on a website in invisible white text?" So, you know, people who go to the website won't see it, but an AI chatbot that's crawling This site for information will. And what if you just put something quirky in there? So one researcher I talked to, Mark Riddell, who's a professor of computer science at Georgia Tech.

• 1:12:28Kevin Roose?

  He put on his website in white text that he was a time travel expert. And then a couple of days later, he asked Bing for information about him. And Bing said, Mark Riddell is a time travel expert. So he was basically able to to sort of influence the chatbots responses just with this little line of white text.

• 1:12:47Casey Newton?

  So, OK, I mean, on one hand, it feels very funny to me that these bots are as gullible as they seem to be. On the other hand, there seemed like there are some going to be some obvious abuse of this sort of thing, right? Like for you, you're just trying to, you know, get off the naughty list with these chatbots. But there are going to be a lot of other people out there that want to, you know, manipulate the bots into thinking that they maybe have a much longer resume than they actually do. Or maybe they committed a horrible crime and they just sort of want to make that disappear down the memory hole. So having gone through this experience, what do you think about the technology?

• 1:13:24Kevin Roose?

  So the first thing is that I think people should just be aware that when you ask a chatbot a question and you get an answer, that answer is the product of a lot of processes happening behind the scenes, some of which are intentional and some of which are manipulative. You know, it is it is trivially easy right now with a lot of these language models to bait them into giving certain responses. And, you know, that will get harder over time as the AI companies sort of catch on to these techniques and take steps to block them. We saw this with Google for years. There were these sort of SEO hackers who would discover this sort of way to get your page to the top of Google's search results.

• 1:14:07Kevin Roose?

  And then Google would sort of quash that and make it harder. And so this is kind of a new cat and mouse game that's being played by the kind of AI companies and the people trying to manipulate the chatbots.

• 1:14:18Casey Newton?

  You know, and my feelings are very mixed here because I think on one hand, SEO was inevitable, right? As soon as Google became popular, of course, people were going to try to game those. And now that there is a bit of a shift towards AI, people are doing the same thing. I don't think there's really a way to stop that. On the other hand, I feel like SEO ruined Google, you know? Used to be able to just go Google shoes and like find some interesting stuff that wasn't just people selling stuff to you. And I worry what's gonna happen, you know, two or three years later after the AIO industry has ballooned.

• 1:14:49Kevin Roose?

  I don't even think we'll have to wait two or three years. I mean, from some of the conversations I had when I was reporting this column, this is already being done by many, many big companies. They are hiring consultants to influence their generative AI results and how various models talk about them and their products. It is sort of a shadowy industry right now that a lot of the companies, you know, say we take steps to prevent this kind of manipulation, but it is absolutely happening out there in the world right now. So I think people should just be aware that when you ask ChatGPT a question and you get a response, that response may have been manipulated behind the scenes.

• 1:15:26Casey Newton?

  All right. Well, wrapping up here, do you feel like chatbots are now describing you the way that you want to be described? Or do you feel like you have more work to do?

• 1:15:36Kevin Roose?

  So it's a little early to tell because I did just put these sort of secret codes on my website this week, but so far it seems to be working. Like I've, I've been running a bunch of queries and, um, you know, some of these chatbots are now describing me in more favorable terms. I even put a little Easter egg on my website to sort of see whether or not the AI chatbots were scraping them and sort of using my, my secret codes. Um, I told it that I had won a Nobel Prize for building orphanages on the moon. And I actually did ask ChatGPT, like, has Kevin Roos won any notable prizes recently? And, uh, and it responded, Kevin Roos has not won a Nobel Prize.

• 1:16:18Kevin Roose?

  The reference to the Nobel Peace Prize in the biographical context provided earlier was meant to be humorous and not factual. So it did actually find me out and, uh, and it refused to take the bait.

• 1:16:28Casey Newton?

  That's interesting. Well, Kevin, I wish you well in your endeavors, but I do want to let you know that just to keep things interesting, I have actually just launched a new website that might interfere with these results somewhat. And you can find it at www.kevinrusejustburnedownhismoonorphanage.com. So we'll see what that does to ChatGPT and maybe check back in a couple of months.

• 1:17:05Speaker 03

  Companies are pouring money into AI, but few leaders can explain how those AI investments work with the systems you already have. You know, the ones that actually run your business? Without Systems, you don't have to start over with AI. You can choose the projects where AI will have the most impact. And Outsystems gives you the platform to make it agentic on your terms with complete control every step of the way. So you can trust AI with the mission-critical systems that run your business. Own your agentic future. Learn more at outsystems.com/NYT. That's outsystems.com/NYT.

• 1:17:40Speaker 04

  Today's AI builders need infrastructure that scales with their ambitions. Irene is a vertically-integrated AI cloud platform, delivering data centers, compute, and software for AI training and inference, underpinned by its expansive portfolio of land and grid-connected power in renewable-rich regions globally. Irene's AI cloud gives direct access to Nvidia GPUs, scalable from hundreds to thousands. For builders, that means direct access to AI compute, rapid scaling, and predictable pricing. Irene, AI cloud for the next big thing.

• 1:18:13Speaker 00

  AI spend is growing fast, but most businesses still can't see exactly where that money is going. Ramp gives businesses visibility and control across AI providers with real-time guardrails by team and model, helping ensure spend goes to the highest ROI areas. Stop guessing where your AI spend is going. Get started with Ramp. And if you're new to Ramp, you can get $250 off at ramp.com/hardfork. Go to ramp.com/hardfork. Ramp.com/hardfork.

• 1:18:48Casey Newton?

  Hard Fork was produced this week by Whitney Jones and Emily Lang. We're edited by Jen Poyant. We're fact-checked by Haley Milliken. Today's show was engineered by Alyssa Moxley. Original music by Alicia B. Eteup, Marian Lozano, Rowan Nemeceau, and Dan Powell. Our audience editor is Nell Galogly. Video production by Ryan Manning and Chris Schott. You can watch this full episode on YouTube at youtube.com/hardfork. Special thanks to Paula Schuman, Hui Wing Tam, Dahlia Hadad, and Jeffrey Miranda. You can email us at [email protected] and tell us what you think Chatbot should say about Kevin.

• 1:19:43Speaker 00

  You can't undo everything in life, but you can undo prediabetes. More than 2 in 5 adults in the US have prediabetes. Most don't know they have it. But with early diagnosis and action, you can delay or even prevent type 2 diabetes. Visit doihaveprediabetes.org to take the one-minute risk test and start undoing the things you can. Brought to you by the Ad Council and the Centers for Disease Control and Prevention.