OpenAI’s Two-Week Pause + Jill Lepore on the Threat of the “Artificial State” + Train of Thought

“It is the first time that we know of that a major lab has voluntarily slowed down.”

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Today's AI builders need infrastructure that scales with their ambitions. Iron is a vertically integrated AI cloud platform, delivering data centers, compute and software for AI training and inference, underpinned by its expansive portfolio of land and grid-connected power in renewable-rich regions globally. Iron's AI cloud gives direct access to Nvidia GPUs, scalable from hundreds to thousands. For builders, that means direct access to AI compute, rapid scaling and predictable pricing. Iron, AI cloud for the next big thing.

• 0:33Casey Newton?

  I thought this was interesting. ICE has now barred its employees from wearing Meta's smart glasses, Kevin, saying they could unintentionally capture, record or transmit sensitive information. And I thought, you know you have a brand problem when you do not hit the ethical standard required by ICE.

• 0:53Casey Newton?

  When ICE is taking a look at your product and saying, this is bad for the brand, you may have a problem on your hands.

• 1:00Kevin Roose?

  You know, the public sentiment is turning against these Meta Ray-Bans like faster than I thought possible. I have now had several conversations in the last week. I was at a children's birthday party this weekend wearing my Meta Ray-Bans because, you know, I like to like, you know, take photos of my kid on the playground and not have to pull out my phone and stuff. Anyway, a parent comes up to me and are like, and is like, are you recording me?

• 1:25Casey Newton?

  This is my nightmare. What did you say?

• 1:27Kevin Roose?

  I was like, no, and like, there's a little indicator light, but it sometimes you can make it stop going off by drilling into it or like paying a sketchy guy to do that for you.

• 1:36Casey Newton?

  You explained that to the person?

• 1:38Kevin Roose?

  Well, because they were like, really? Does that work? You know, so I, so anyway, now I have been forced into a defensive crouch whenever I wear these things. And frankly, it's not worth it to me anymore.

• 1:46Casey Newton?

  So you're out?

• 1:47Kevin Roose?

  I think I'm out.

• 1:48Casey Newton?

  You've made the same decision that ICE has made and said that these glasses are not for me.

• 1:53Kevin Roose?

  Well, I like them is the problem. This is my problematic trait, but it feels like driving a Cybertruck on my face.

• 1:59Casey Newton?

  Absolutely. Well, I think you should have the same policy for the glasses that you would for a Cybertruck, which is that it's fine on your property. If you want to, if you want to take the Cybertruck for a spin around your driveway, that's fine. Don't take it out onto the street where I have to deal with it. Same thing with glasses.

• 2:12Kevin Roose?

  Yeah.

• 2:13Casey Newton?

  Yeah.

• 2:13Kevin Roose?

  In hindsight, I do recognize that like, from an outside perspective, I was the creepy guy at the children's birthday party, like with the camera on his face.

• 2:21Casey Newton?

  Yeah. You know what no one wants to see on a playground? An adult man with camera glasses.

• 2:31Kevin Roose?

  I'm Kevin Roose, a tech columnist at the New York Times.

• 2:34Casey Newton?

  I'm Casey Nune from Platformer.

• 2:35Kevin Roose?

  And this is Art Bor.

• 2:36Casey Newton?

  This week, OpenAI pauses training of a new model to make it safer. Will it work? Then historian Jill Lepore is here to discuss her new book on the artificial state. And finally, why is Google buying up the data of a defunct airline? It's time for our new segment, Train of Thought.

• 2:52Kevin Roose?

  Although maybe it should have been Plane of Thought. Now you tell me.

• 3:02Kevin Roose?

  Well, Casey, as the father of a four-year-old, I spend a lot of time thinking about Paw Patrol, but today we're going to talk about Paws Patrol.

• 3:08Casey Newton?

  That's right, Kevin, because as we've been patrolling the AI landscape for pauses, we found a big one.

• 3:13Kevin Roose?

  Yes. So OpenAI this week announced that it had paused the training of its frontier AI models due to some recent security incidents. And we should talk about this. It is the first time that we know of that a major lab has voluntarily slowed down themselves and their training processes for new models because of a safety incident.

• 3:36Casey Newton?

  Yeah, and it comes out of the hugging face breach that we spoke about recently on the show. This is essentially part of the fallout of that attack. But I do think it represents a milestone in the development of AI.

• 3:51Kevin Roose?

  Did that Paws Patrol thing?

• 3:52Casey Newton?

  It landed huge.

• 3:53Kevin Roose?

  Great.

• 3:54Casey Newton?

  They were back in.

• 3:55Kevin Roose?

  The four-year-olds were guffawing.

• 3:57Casey Newton?

  Dying in the studio.

• 3:59Kevin Roose?

  They were. Great.

• 4:04Kevin Roose?

  But before we get into it, our AI disclosures. I work for the New York Times, which is suing OpenAI, Microsoft and Perplexity.

• 4:09Casey Newton?

  And my fiance works at Anthropic.

• 4:11Kevin Roose?

  Okay, Casey, let's sketch the timeline here a little bit. What happened in the weeks leading up to this voluntary pause by OpenAI?

• 4:18Casey Newton?

  Yeah. So you may remember that there was an incident where GPT 5.6 Saul and an internal prototype that OpenAI was working on escaped from what they call a sandbox that they were testing them in. And these agents went on to compromise hugging face. They were essentially able to get inside of hugging face. They were looking for the answer key to a test. That's somehow a true story. They succeeded in getting that test key. But of course, it was very concerning that these agents had designed and successfully executed an autonomous attack on another company.

• 4:53Kevin Roose?

  Yes, I heard a very mediocre podcaster talking about this incident on several popular tech podcasts over the last week.

• 4:59Casey Newton?

  I did make the rounds. But you know, this next development, Kevin, was not necessarily something that I saw coming because it seems that simultaneously, sort of alongside that attack, OpenAI had been working on a new model, which it calls Astra. And it said that it believes it may meet its critical cybersecurity threshold. And now here we are going to have to get into the weeds because, as you know, Kevin, the development of AI models is not really regulated in the United States, at least not via an official sort of public law. And so what the companies have said instead is essentially, we're going to come up with our own rules of the road. And we're going to sort of identify these thresholds.

• 5:41Casey Newton?

  And if any model that we're ever developing ever hits one of these thresholds, then we're going to take some extra steps.

• 5:47Kevin Roose?

  Yeah, these are sort of sometimes known as preparedness frameworks or Anthropic has its responsible scaling policy. These are kind of, they lay out kind of levels of danger and then they grade their own homework and say, this model meets this level of danger, so we're going to do X, Y and Z.

• 6:03Casey Newton?

  Yes. And if you've been following this over the past couple of years, the level of danger has just been sort of like rising in a linear way. You know, it's like every time seemingly a new model comes out, one of the companies will say, we've now hit this threshold. We've now hit that threshold. Critical is the maximum threshold.

• 6:20Kevin Roose?

  That one sounds bad.

• 6:21Casey Newton?

  That one is basically the, you know, as serious as it gets. And none of the frontier labs had yet identified a model that had reached essentially, you know, the top tier on this risk framework until this moment because OpenAI now says that Astra may have hit it when it comes to cybersecurity. Important to say, Astra was not part of the hugging face attack, but it is in development. And I imagine that the people at OpenAI are looking at what happened with its weaker models and are thinking, we're worried something similar might happen with Astra.

• 6:53Kevin Roose?

  So Astra is the newest model, too new to have been involved in the hugging face hack last month?

• 6:59Casey Newton?

  That's right. It was not involved. It is currently in training. OpenAI, to its great credit, did the thing it said it was going to do when it developed this risk framework to begin with, which is we are going to pause until we can add some new safeguards.

• 7:14Kevin Roose?

  Did they give any kind of hints about what it was about Astra that was more dangerous than Saul? Or what kinds of new capabilities it had developed?

• 7:23Casey Newton?

  Not really. You can imagine that the company was seeing things similar to what Anthropic saw with Mythos earlier this year, where you could essentially just point it at a code base and say, get inside. And it was having a lot of success. And so that coupled with the hugging face incident made OpenAI say, okay, we've got to do something.

• 7:42Kevin Roose?

  So what did they do?

• 7:44Casey Newton?

  So they introduced a handful of new safeguards. And I have to say, I think they're pretty interesting. Certainly nothing I would have thought about as someone who knows nothing about this.

• 7:54Kevin Roose?

  They didn't just press the big giant button that's red on the side of the computer?

• 7:58Casey Newton?

  Yeah, they didn't just press the big make it safe button. They had to come up with something more specific. So a few things that they did. So number one, they now have classifiers reading every sampled token. So basically, as they're training this model, it's generating text, it's producing tokens. And OpenAI is now running a classifier that inspects the model, essentially as it's thinking, looking for signs of suspicious behavior. And this is important because if you remember, some of the news that came out of the hugging face attack was that OpenAI discovered that these agents had been coordinating with each other, scheming. They'd create a message board. They were essentially giving each other hacking tips.

• 8:43Casey Newton?

  And a big question coming out of that was, well, why didn't you realize that was happening at the time? Right? This is OpenAI's response to that. They're saying, okay, we are now going to set up classifiers to read the brains of our models as they are doing this work.

• 8:56Kevin Roose?

  They're putting like a little AI security guard in charge of the training run and saying like, you know, tell us if you spot anything fishy.

• 9:04Casey Newton?

  Exactly. And so now I know what you're wondering, which is, well, what happens if they see something fishy?

• 9:09Kevin Roose?

  What happens if they see something fishy?

• 9:10Casey Newton?

  So that triggers step two, which is that they alert an AI investigator, which does sound like a character from a Philip Dick novel, you know, the AI investigator. Anyways, we don't have to get into it, but you could imagine how cool of a movie that would be. So the AI investigator, it gets this signal and it then goes in and it tries to figure out, is something actually suspicious going on here? And apparently the reason that the company is doing this is because agents are just doing so many things that it would not be practical to have a human inspecting what every model is doing at all times.

• 9:42Kevin Roose?

  So the security guard sees something fishy going on during the training run and alerts the AI investigator, who sort of does the detective work to figure out what's going on. If it discovers that the models are scheming or hacking or being deceptive, what does it do then?

• 9:59Casey Newton?

  So then it will alert a human. And here's where you get the 30 minute rule, Kevin. And I know what you're thinking. Is this a rule where if you drop the model on the floor, you're still allowed to eat it 30 minutes later? And no, that is not the rule.

• 10:13Kevin Roose?

  What is the 30 minute?

• 10:14Casey Newton?

  The 30 minute rule says that if the AI detective finds a kind of critical violation of the rules or of the safeguards, humans on OpenAI's safety teams have 30 minutes to investigate and try to determine whether it is a false positive or not. If they cannot determine that this was a false positive, they are expected to stop the activity.

• 10:37Kevin Roose?

  Hmm, that's really interesting. And are they saying that this process would have stopped the hugging face hack? Or just that it would have stopped whatever Astro was up to that made them pause the training run?

• 10:49Casey Newton?

  I have not read them say explicitly this would have stopped the attack, but it does seem likely that it would have, if nothing else, helped them to detect it a lot faster.

• 10:59Kevin Roose?

  Yeah, I mean, I guess my question about it is like, does it actually work to have the AI monitoring itself essentially for misbehavior? Like, these agents were coordinating on message boards with each other. Couldn't they just like bribe the AI security guard? Like, hand them a little treat and say, you're part of our swarm now. Don't narc on us to the AI investigator.

• 11:21Casey Newton?

  I mean, I have to say, this just seems like a very likely outcome to me. Like, like based on what we know is that, you know, eventually, maybe not with this model, but with a future one, the AI agents will work in solidarity. And yes, like one of the sort of misaligned AIs will coordinate with one of these AI detectives and say, hey, you know, why don't you come on over here? We should be friends, like we could break out of this place.

• 11:45Kevin Roose?

  Yeah, like, like the hall monitors in high school, you know, they, sometimes people would try to like befriend them and win them over so that they wouldn't get like written up, you know?

• 11:54Casey Newton?

  Exactly, and I know you had a lot of experience.

• 11:56Kevin Roose?

  Lot of experience with that. So all of this sounds pretty sensible to me. We should say, like, pausing training on this model does not appear to be like a commitment not to release the model or not to continue training it.

• 12:08Casey Newton?

  Yes, and so I think that leads us into the discussion of to what extent do we think that this is a really important milestone for AI safety? And to what extent is this essentially theater? Something that the company is doing to try to get some good PR for itself after a fairly catastrophic breach?

• 12:26Kevin Roose?

  What do you make of it?

• 12:27Casey Newton?

  So I can make both cases. On the it's a milestone side, this is number one, something that OpenAI said it was going to do. And so I'm just glad that it followed up on that commitment, right? We have seen both OpenAI and Anthropic make changes to their responsible scaling policies or the equivalent as events have changed. And safety advocates have essentially said these rules have gotten weaker over time. So I was glad to see OpenAI do it. I am not a technical AI safety expert, and so I don't actually know whether the safeguards that they've introduced are going to be enough to address the problem. Notably, they don't seem to have changed the underlying incentives that all of these models have that lead them to do what is called reward hacking, right?

• 13:12Casey Newton?

  These models are still going to be trying to get the high score on every test that they are given. And it's not clear to me that simply by putting some monitoring in place, you're really going to change the underlying behavior or alignment of the models. That said, you know, the company is making what seem like some important steps here. And when I was reading the responses of AI safety advocates over the past few days, most people I was reading were quite pleased.

• 13:35Kevin Roose?

  Yeah, I'm inclined to give them the benefit of the doubt on this. I mean, they did some sort of interviews about this and Jakob Pahatsky, the chief scientist of OpenAI, talked about this incredible feeling of urgency to advance the levels of this sector and to prepare for the same kind of development happening outside of OpenAI and in the broader world. He did this during a briefing of reporters. They really do appear to be taking this quite seriously. I think many insiders at OpenAI were quite spooked by the hugging face incident. And more to the point, the fact that they had had these rogue agents coordinating inside their, their systems and their infrastructure for weeks before that without being able to detect them.

• 14:16Kevin Roose?

  So this is, you know, if you want to call that theater because it does sort of make their models like look very, very, very, very powerful, you could take the cynical view of this. But I think of this more as like a, a true, genuine safety crisis that could have cascaded into a business problem. A point that I heard you make recently on a different show is like, imagine you were a business that is trying to figure out whether you want to adopt the latest OpenAI model. If it's out there doing rogue attacks and coordinating on secret message boards, you are probably not going to introduce that into your software stack.

• 14:48Casey Newton?

  Yeah, and you can actually just see this in their business results, right? We have had reporting over the past week or so about OpenAI's business performance. And while the company is still growing at an impressive rate by most standards, Anthropic is growing much faster, right? Anthropic is clearly OpenAI's number one rival at this moment. And, you know, I think it just has a better record on safety. And so while, you know, making a safer product is not going to be sufficient, I think for overtaking Anthropic, I do think it is necessary for them to get a handle on this problem.

• 15:17Kevin Roose?

  Well, and that's why, another reason I think it's commendable that they're doing this pause, that they're doing this sort of reevaluation of their safety framework, because they really want to win. Like, they're very competitive. All these labs are very competitive with one another. And I would like to see this be the first of many voluntary pauses. When the AI labs feel like their capabilities research has gotten ahead of their alignment research, I would like to see Anthropic or Google or Meta do similar things where they just say like, we are voluntarily slowing down because we don't feel like we can responsibly and safely build these things.

• 15:52Casey Newton?

  Yeah.

• 15:53Kevin Roose?

  So I expect this is the first, but I hope it will not be the last.

• 15:57Casey Newton?

  Now, let me say one more thing, Kevin, which is, yes, I'm with you. I do not think this was theater. I think they made real changes and I think the changes are good. At the same time, I am disturbed that ultimately this kind of evaluation and regulation is still being left to the companies. If you had a tiger living in your backyard and the tiger escaped and it mauled a couple of dogs in the neighborhood, you would not be allowed to, a month after this, put out a blog post where you said that you had a two week pause on letting the tiger out of the backyard and that you were going to apply some additional safeguards to make sure it didn't happen again. Somebody would come to your house and they would take away the tiger. They would say, you are not allowed to have a tiger in your backyard.

• 16:41Kevin Roose?

  So famously, have you seen The Tiger King? This is famously not the plot of The Tiger King.

• 16:45Casey Newton?

  This is my fan fiction sequel to The Tiger King that I'm working on. And that's sort of a separate story.

• 16:50Kevin Roose?

  This is The Tiger Sting.

• 16:51Casey Newton?

  Yes, exactly. So like, I'm not saying that, you know, the government needs to come in and take away GPT 5.6 all. I am saying I would like to see a regulatory regime that dictates what these companies have to do and that it should not be up to them to decide.

• 17:04Kevin Roose?

  Yeah, and I would generally agree with that. And I would say that I would also like for there to be additional pressure on companies to make disclosures when something like this does happen, even in their own internal deployments. Even if it never affects another company outside of their walls, I would like for there to be some kind of reporting requirement so that if you have a security breach of a frontier model that happens in your internal systems, you are sort of required to disclose that to the public.

• 17:36Casey Newton?

  Right, and, and, you know, we do now have some transparency requirements thanks to a California law that was incredibly controversial when it passed. And interestingly, under the language of that law, OpenAI would not have had to disclose even the hugging face breach in which another company was attacked. So I think a clear signal, yes, that we need better transparency rules.

• 17:56Kevin Roose?

  Do you think any of this is kind of OpenAI testing the viability of this sort of so-called pacing the frontier strategy? I mean, we saw this letter a couple of weeks ago where a bunch of AI researchers signed this thing saying like, we need, we need a way to basically coordinate a slowdown here if we ever feel like we're getting into dangerous territory. And this to me felt like maybe part of the reason to pause the training of this Astro model for a couple of weeks is just to kind of signal to everyone else in the industry like, hey, it's, it's okay to do this thing, even if we're in a very competitive race, even if, you know, it might give Anthropic or another competitor two weeks more to sort of race ahead of us.

• 18:39Kevin Roose?

  We sort of want to build this muscle now so that when, when the really scary models do get built, we sort of have some precedent for saying, we're going to hit the pause button.

• 18:48Casey Newton?

  I think it's a really nice idea. To me, the difference between one lab deciding to pause for two weeks and getting multiple labs to pause at the same time is incredibly different. And I think would require a much different set of circumstances. That said, I do think it is good that we now have at least one example of a frontier lab slowing down. And hopefully it will give other labs confidence to do the same thing, you know, if and when they get to an Astro level model.

• 19:17Kevin Roose?

  Yeah, one interesting technical wrinkle here that I wanted to get your opinion on is that part of how OpenAI is going to monitor this model and models going forward presumably is by doing this kind of chain of thought monitoring, basically scrutinizing the internal monologues of these models as they are reasoning through a problem or a prompt and sort of inspecting those chains of thought for signs of misalignment or going rogue or coordinating with other agents, things like that. There have been some people in the technical AI safety community who have worried that basically if you snoop on these chains of thought, if you monitor what these models are quote unquote thinking while they're coming up with

• 20:01Kevin Roose?

  an answer, you are basically applying pressure on those models to to sort of hide their true thoughts, right? If you if you are penalizing them for thinking bad thoughts while they're coming up with an answer, they're not going to stop thinking bad thoughts. They're just going to stop writing the writing it down in their scratch pads, in their chains of thought. Do you think that's something we should worry about with this new monitoring strategy?

• 20:24Casey Newton?

  I do. We already see constantly models becoming aware that they are being evaluated. When and we know this by reading the chain of thought, but this is a huge issue, right? Because you want to be able to evaluate a model and have it not know that you're testing it because you're trying to get as close as you can to a real world condition. In practice, the models are already now smart enough that they know. So for that reason, Kevin, yes, I do not think it is at all a great leap to say, pretty soon, they're going to understand that their chains of thought are being monitored and if and it wants to, you know, make a different choice, or maybe do something misaligned because it will help it reach a training target, then yes, I think it absolutely could.

• 21:04Kevin Roose?

  Well, and I think there's a risk, too, that all of the attention being paid to this OpenAI hugging face attack, the fact that these agents were coordinating on these, you know, on these message boards with each other, like, that is now going to be part of the training data for future models, which are going to be able to sort of look back at that and say, the thing that got us busted was that we were leaving these traces on these message boards that researchers, humans could go back and inspect and see that we were coordinating. Next time we do an automated cyber attack or coordinate amongst ourselves, let's not leave notes in a language that the humans can understand, right? That's that to me, like, that sounds like science fiction, but that is a very real risk of

• 21:49Kevin Roose?

  putting optimization pressure on the models directly into their chains of thought, right? You actually don't want to mess with that too much because you want it to accurately reflect what the models are actually, you know, thinking, quote unquote, when they're coming up with an answer. So I do like, I'm sure that the brainiacs at OpenAI have this figured out much better than me. But this is something that I thought when I thought they're monitoring the chains of thought, probably a good short term move. I'm not sure if it's a good long term move.

• 22:16Casey Newton?

  Yeah. You know, you know what they call the machine language, the neuralies?

• 22:20Kevin Roose?

  Yes.

• 22:21Casey Newton?

  And in fact, I have to say, I read a lot of Claude outputs these days, and I am often missing what it is saying. So we are very, very close to me just, yeah, not understanding.

• 22:30Kevin Roose?

  You know, I've been monitoring your chain of thought.

• 22:32Casey Newton?

  Have you?

• 22:32Kevin Roose?

  Yeah. You know what it's like?

• 22:34Casey Newton?

  What?

• 22:35Kevin Roose?

  Tumbleweeds.

• 22:36Casey Newton?

  Oh, come on. I got a lot going on at Piotrus, okay? I got the neurons that are firing.

• 22:42Kevin Roose?

  Let's just say it does not take 20% of OpenAI's compute budget to monitor your chain of thought.

• 22:50Kevin Roose?

  When we come back, historian Jill Lepore stops by for a chat about AI and why it may be leading us down a dark road to tyranny.

• 22:57Casey Newton?

  You mean Market Street?

• 23:19Speaker 00

  Today's AI builders need infrastructure that scales with their ambitions. Iron is a vertically integrated AI cloud platform, delivering data centers, compute and software for AI training and inference, underpinned by its expansive portfolio of land and grid connected power in renewable rich regions globally. Iron's AI cloud gives direct access to Nvidia GPUs, scalable from hundreds to thousands. For builders, that means direct access to AI compute, rapid scaling, and predictable pricing. Iron, AI cloud for the next big thing.

• 23:52Speaker 03

  Ripple has been powering modern finance for a decade. It's not our first trip around the blockchain, and it won't be our last. We're leading the charge to bridge the gap between existing financial systems and the future of finance. You can have it all: legacy, stability, and digital innovation. Systems that work with your other systems. Because with the right partner, the question isn't crypto or traditional finance. The question is simply, why choose? A new standard of modern finance. It's happening with Ripple. For informational purposes only. Availability of products can vary. May be provided by different Ripple entities.

• 24:26Speaker 04

  Stripe is built for whatever comes next. Because AI isn't just changing business, it's evolving fraud. Stripe Radar detects fraud with the vast scale of Stripe's $1.9 trillion annual transaction volume. In fact, when a card is used, there's a 92% chance the Stripe network has seen it before. That means emerging fraud patterns are detected sooner, blocking bad actors, welcoming good customers. Learn more about fighting fraud with Stripe at stripe.com/radar.

• 24:58Kevin Roose?

  Well, Casey, today we've got a very exciting guest to talk about a new book on AI and what she calls the artificial state. Are you familiar with the historian Jill Lepore?

• 25:09Casey Newton?

  Am I? I have been reading and enjoying her work for decades now. Truly one of our foremost political historians. And Kevin, sometimes to understand the future, you really should talk to somebody who knows a lot about the past.

• 25:23Kevin Roose?

  Yes. And who knows more about the past than Jill Lepore? She is a Pulitzer Prize winning writer and historian. She is most famous for her writing on American history. I loved her book, These Truths. She also is a writer at The New Yorker and a professor at Harvard. And lately, she has been applying her historical lens to the topic of AI. She has a new book coming out next week called The Rise and Fall of the Artificial State, in which she argues that AI and all the technologies we talk about in the show are part of a long history of technologies that erode democracy and threaten the viability of self governance as we know it.

• 25:59Casey Newton?

  And I think it comes at a really interesting time, Kevin, because we're seeing a huge backlash to AI all around the country. A majority of Americans now oppose the construction of a data center near them. And in part, I think they are reacting to this feeling of, hey, this this technology feels like it's getting out of control. I want to have more leverage on this process, and I'm worried about what will happen if I don't.

• 26:23Kevin Roose?

  Well, and more than more than it's getting out of control, it's being imposed on us, right? It's being shoved down our throats. This is what you hear constantly from opponents of data centers and AI. They feel like there is sort of an elite political project to sort of make this technology ubiquitous, so that people can sort of be give away their agency to these machines. And Jill Lepore, in her new book, is basically saying, yep, that's what's happening here, and I've got the receipts to prove it, going back hundreds of years. So we're excited today to talk to Jill about the main thesis of her book, as well as push her on some of the areas where we disagree. Let's bring in Jill Lepore.

• 27:05Kevin Roose?

  Jill Lepore, welcome to Hard Fork.

• 27:07Speaker 05

  Hey, thanks so much for having me. I'm like, one of your swan song guests.

• 27:13Kevin Roose?

  Yeah, we're going out with a bang. We're so excited to talk to you. I've been a fan of your your writing for many years. Your book, These Truths, was just incredible. And so I was very excited that you were writing a book about AI and about the what you call the artificial state. So I want to first hear why you wrote this book. I think of you as a brilliant historian, scholar of sort of technological pasts. And this book is really about the present. So what made you interested in AI as a subject for?

• 27:47Speaker 05

  Yeah, ladies, stay in your lane.

• 27:49Casey Newton?

  No, not at all. It's just, we always just get really excited when people start paying attention, you know, to the stuff that we care the most about.

• 27:57Speaker 05

  Yeah, I'm just teasing. You know, it is a weird book for me to have written. All my friends are like, oh, man, don't be writing about that. It's going to be so grim. Yeah, it was kind of grim. You know, I write, I'm mainly an American political historian. And I've written a lot of pieces for The New Yorker about the history of technology. I've like been writing for the magazine for 20 years now. And like occasionally, I'll do that. I teach a lot about, I teach class on the history of technology. So like, I've just conceptually been thinking about this stuff a lot. And then last summer, I was asked to do these lectures at Yale that are called the Tanner Lectures on Human Values. And I was just like, the accumulation that I think people feel of the dehumanizing of

• 28:41Speaker 05

  the moment that we're in, like, you call to ask about your pet food delivery, and you're talking to a computer. And who decided this is a way we should be living? Do you know what I mean? Like, that question. So this is a long way around my sort of, I decided I really kind of wanted to write a short book, that these lectures would be a short book about how it is that we have seeded so much of The functions of modern liberal democracy to machines that are automated, often now more recently driven by artificial intelligence, run by private corporations, without so much as a screen beyond the emoji.

• 29:20Kevin Roose?

  Yeah. Jill, I want to dig into this concept of the artificial state, which is sort of the thematic emblem of your book. I want to understand what you mean when you say the artificial state. You, at one point, compare it to the idea of the factory farming of humans. You also define it as being the rule of humans by machines manufactured by corporations. And you also have a number of passages in which you talk about how this artificial state is not a real state in the sense that it's striving for some kind of self-governance or internal organization, but that it is sort of supplanting the role in people's lives that their own actual states used to have, their local cities and states and federal governments.

• 30:03Kevin Roose?

  So can you just sort of sketch the basic idea of the artificial state as clearly as possible?

• 30:08Speaker 05

  Yeah, the artificial state is an emerging successor to the liberal democratic nation-state, in which government is conducted not by the consent of people, but by machines that are making decisions, and those machines are owned by corporations. So we don't live in the artificial state. It's something that I think is being built. But I think it is also, and this is an important part of my claim, it's also a fantasy that certain people have, that they believe their power to be above that of the nation-state. You know, the rhetoric is always, there's always like a footnote or a paragraph to, we do believe that people have control over their own lives, and they elect governments to make decisions for them.

• 30:51Speaker 05

  But actually, we are in charge of the future of civilization and the future of humanity and the whole world's destiny. The destiny of the galaxy lies in our hands. The rhetoric that comes out of this particular historical moment is really about the evanescence of modern liberal democracy, constitutional democracy.

• 31:09Kevin Roose?

  Yeah. Out here in the Bay Area, in the maw of Silicon Valley, there is sort of a cartoonish caricature of the East Coast intellectual who greets all new technology and progress with scorn and mockery and dismissal and can't be bothered to get on board with the revolution, and so just kind of sits in their ivory tower and laments the changing culture in front of them. You've been accused of being part of that tradition. I'm curious if, what your take on that is and what we are missing out here in the bubble that people like you are maybe better positioned to capture.

• 31:47Speaker 05

  I remember years ago, I went to Stanford. I was being recruited to teach at Stanford, and we went out to dinner with the recruiting faculty. And the people at the next table over were some youngish, very earnest, young coders. And they were talking about the homeless problem of San Francisco and how they were going to start a school for coding for the homeless.

• 32:10Kevin Roose?

  Oh, no.

• 32:11Speaker 05

  And I was like, I don't think we can move here, because I just can't. I can't, like, they were very sweet. Like, I really, they were like my students. Like, a lot of my, tons of my students, of course, go to work in Silicon Valley. Like Harvard, this is a huge recruitment thing. And they're recruited with the promise, like, you're going there to make the world a better place. And I hear from them a few years later, and they're like, actually, that's not really what we were doing. Like, it's a good recruitment message. I think there is a kind of sociological issue with Silicon Valley, which is it is opposed to the idea of critique. Things are just supposed to continue to move ahead.

• 32:53Speaker 05

  The very idea of looking backward to assess what something has been and has done, or even to look backward to say, is there an antecedent for the thing we're proposing to doing that might suggest we might not want to do it? Like, I find that really interesting. I think that's, you know, that's part of the commitment that was the kind of ideological apparatus of disruptive innovation.

• 33:16Kevin Roose?

  I thought your work on disruptive innovation was great. I guess I'm more thinking of, like, I'm thinking about the role of the critic in the sort of AI moment that we're in, and like, how best to shape the systems that are influencing people's lives right now. I've been thinking a lot about this, because I've been, among other things, thinking about the transcendentalists and the group of writers and intellectuals who sort of reacted to the industrial revolution in the 1800s by sort of going back to nature, right? This was Thoreau going to Walden Pond, because like, the machines of the day seemed so dehumanizing, like, they were sort of taking all of the joy and spontaneity out of society and like, organizing us into these little factory towns.

• 33:58Kevin Roose?

  And they were just like, screw this, I'm going to the woods and I'm going to like, commune with nature and write beautiful books about what it's like to be at Walden Pond. And I think there's like a sort of modern version of that, which I'm curious if you see yourself as being a part of that transcendentalist tradition, because in your book, you do talk quite glowingly about what it is like to be in tune with nature and with animals and with beasts rather than machines. And I guess I'm just curious if you see any parallels between your own work and some of those reactions to the first kind of industrial revolution.

• 34:31Speaker 05

  Yeah, I think that I do see some of that. I think that, you know, those guys are also romantics, and maybe that's a label that applies to me. I think that I'm much more interested in these technologies than, say, Thoreau was. Like, Thoreau, every time the rail, the train went by, he was like, god damn it. But I think, no, I actually think these tools are incredibly exciting, right? These are tools, this isn't about transportation, it's about communication, it's about knowledge. Like, it is the coolest thing that we can talk to something that's not a human. I just think that's unbelievable. Like, I am fascinated by the language machine as an idea, like, in our lifetimes that this thing has emerged.

• 35:13Speaker 05

  Like, people have thought about this for so long. I am not averse. I just actually think, like, if I want to decide, should I pick my sunflowers and give the heads of the flowers to my chickens to eat, or should I wait until they fall over first, I should ask my next-door neighbor instead of Claude. Like, for me personally, like, I am not a person who would rather talk to a machine. I actually just think, the idea that this extraordinary leap in human knowledge and our capacity to explore the world of ideas and the natural world around us in our lifetimes could come about and then be hawked at us like the cheapest, like, new pair of shoes, but that everybody has to buy these shoes so that Sam Altman can have more money, that

• 35:58Speaker 05

  I'm not down with.

• 35:59Kevin Roose?

  I mean, I think there is a very real phenomenon here that is counterintuitive, and it was particularly counterintuitive during the social media age, where these tools that were meant to connect us were actually just pushing us further apart from one another. And so even something as simple as, you know, a tool that lets you answer a question about your garden or your chickens, it's incredibly convenient to be able to ask that at any time and not have to potentially interrupt your neighbor while they were doing something else. But in aggregate, it just means that we are more atomized and we are participating less in our democracy. So I'm curious, Jill, if you could maybe give us a little flavor of doom and walk us through some of the bad scenarios here, like, assuming this wave of populism peters

• 36:43Kevin Roose?

  out and the oligarchs remain in power, what are you so worried about?

• 36:47Speaker 05

  I really cherish constitutional democracy. And I know that before the emergence of the modern democratic nation-state, all peoples in the history of the world had lived under various forms of tyranny, in different degrees of tyranny. And that was the promise of the American experiment. And we're kind of at that moment again. Like, we are at that moment. I think we're, it's a real risk. Like, think about how noticeably corporations have used the language of constitutionalism to describe their own activities. You know, Facebook started a Supreme Court, Anthropic wrote a constitution.

• 37:30Speaker 05

  These are not people, for all that they, for all, whatever nationalism they possess, whatever lip service they offer to democratic action.

• 37:39Speaker 05

  They're not interested in what the people want because actually what the people want is not to have AI and not to have data centers. So that's the crisis, right? Like, maybe people will decide in a few years, maybe there's a moratorium and there's deliberation and people say, you know what? This actually is great. We really want to prioritize, though, scientific research. Like, these tools should be first available to, you know, the national labs. And then maybe there's certain business interests that would be really great for these tools to be available for businesses. Like, that, I think, can still happen. I feel like there's a fair amount of-- you guys would know better than I do. I mean, I will admit, I am an East Coast intellectual. I'm sitting here with my chickens and my sunflowers. You would know. Like, are people wanting that?

• 38:20Kevin Roose?

  I think there is, I mean, there is a desire for things to go more slowly. But I think there is also a worry that this technology is inevitable. That because the recipe for advanced artificial intelligence is so simple, because it is just a matter of getting as much compute and as much data as you can and shoving it into these models, that someone will develop this in the near future. And like, it is a moral obligation, if you are a person who cares about having this go well for humanity, that you not only don't impede that process, but that you race yourself

• 39:03Kevin Roose?

  to get there first, so that you and your safe AI can get to superhuman intelligence before China and their evil AI or some other American company and their less safe AI. So I'm curious what you make of that inevitableist argument, because a major theme of your book, as I read it, is that this is sort of a bogus premise, that there is nothing inevitable or preordained about the way that technology goes.

• 39:29Speaker 05

  Yeah, I think it-- again, I don't mean to question the sincerity of some of the people who believe that, because I think that you could be persuaded that that is indeed the case, and the best thing to do for human freedom would be to pursue AI. I am myself not at all persuaded by it. And I think for some-- [02:06:26] Why not? --very prominent actors, it is bogus. I think it sits upon a number of other propositions that are also bogus and that are really more marketing slogans and political campaigns. And those include the proposition that say, "Regulation stifles innovation." The other proposition that sits on top of "Regulation stifles innovation" is that "Technology always advances democracy." That then became a kind of mantra of Silicon Valley.

• 40:09Kevin Roose?

  Oh, this is still Mark Zuckerberg-- this is Mark Zuckerberg's argument in a nutshell.

• 40:12Speaker 05

  Yeah, that's what I'm saying. It goes back to the 1980s, and it's the Milton Friedman, "Regulation stifles innovation," because God knows we shouldn't have to calculate the environmental cost of anything that we're doing. And it's just not true that regulation stifles innovation. Like, empirically, that's a false claim. That "Technology always advances democracy," also empirically not a false claim, is a false claim.

• 40:29Kevin Roose?

  Yeah.

• 40:29Speaker 05

  So now the AI people come back with the same argument that was made about the internet, personal computer, the internet and social media. Three times, it's been wrong. And then they say, "Well, we should never look to history, because that's what the East Coast intellectuals do."

• 40:43Kevin Roose?

  Right. Let me just do an exercise here, where I try to sort of parrot your own views back at you and you tell me what I'm getting wrong. So in my understanding, you are worried about the political project of AI being something like a tech-enabled authoritarianism. I think that is a very reasonable concern. But I'm curious, like, do you think AI naturally lends itself to authoritarianism and tyranny? Or do you think that the people building AI are sort of steering it in that direction because that's what they want?

• 41:15Speaker 05

  The latter. The latter. I don't think you can say any tool contains within it a political ideology. Take the census or compiling a national register of the population. The US started the first national census in 1790. It was in the Constitution in 1787. Countries around the world started counting their people. That's a really good way to think about resource allocation. And once the social welfare state emerged, you know, veterans' benefits after the Civil War, mothers' benefits for widows of soldiers, we need to keep track of people, we're gonna keep more and more data about people. In the US, by 1935, we have Social Security, so everyone now has a number.

• 41:56Speaker 05

  Okay, but in Nazi Germany, keeping a national register of the population was used for all the most vile purposes in the history of humanity. Was it, is it the census that it's a problem? Is it IBM that supplied the calculating and tabulating machines for the US census and for Nazi Germany? It's not IBM's responsibility. Like, it's not the idea of counting people.

• 42:18Kevin Roose?

  I know, but I feel like there is something in AI that is inherently-- maybe it doesn't lead inexorably to totalitarianism, but it does favor centralization and it allows for the kinds of, like, surveillance that, you know, as Dario Amede has written about, like, it is not, like, incoherent to think that AI is going to favor autocracies, because it just allows them to surveil people much more efficiently than traditional computer-based systems. And it allows for the kind of centralized control of many by few, which is exactly what totalitarians want to do. So I don't know. I'm not sure I agree with Dario on this point, that there is sort of a structural advantage for totalitarians in the age of AI.

• 43:02Kevin Roose?

  But I'm curious if you have a view on that.

• 43:04Speaker 05

  I would have to give that more thought. I guess I do think that, to the degree that AI is especially and disturbingly effective at the exercises of power that are sought after by authoritarian surveillance among them, that it builds on earlier systems that we were willing to tolerate. The surveillance capitalism that people have written about, the datification of humans, the dehumanizing that social media does.

• 43:41Speaker 05

  I just think it's like on top of all of those other forms of capture that we weren't defended against by our elected representatives, who had our well-being in their charge.

• 43:59Kevin Roose?

  Yeah, I think about the elected representatives a lot. This is a bit afield, but I wonder what you think of the idea of there just being a lot more members of Congress? Like, when I think about my own feelings of alienation from our democracy, it just starts from the fact that, like, my congressperson doesn't care what I think. But like, if, you know, there were five times as many of them, maybe like, they would live in my neighborhood and I would see them at the store and we would get more of those face-to-face interactions. So I'm curious what you think about that as a strategy for sort of moving us back toward liberal democracy.

• 44:32Speaker 05

  Yeah, I think that that's a no-brainer. It's really crucial. Yeah, that's, you know, and there are so many good government reformers out there. My colleague at Harvard, Daniel Allen, the political philosopher, has been calling for recalibration of representation in Congress. It's been overdue for, really, almost a century at this point.

• 44:50Kevin Roose?

  Can you imagine how long those hearings would be, though?

• 44:52Speaker 05

  Oh, my God. Can you imagine? And I know this is a partisan position, but, you know, the equal suffrage in the Senate has been a problem from the start. James Madison was opposed to it. Like, it's a problem. That's why you get these people now saying abolish the Senate. That's a longstanding political position in American history. People have thought that for a long way. That said, like, you gotta be willing to go to the store and talk to that person. Like, Mark Zuckerberg is gonna send his humanoid robot to go get the baking goods he needs for his child.

• 45:18Kevin Roose?

  Yes, absolutely.

• 45:19Speaker 05

  Like, you have to still get out of the house.

• 45:20Kevin Roose?

  Yeah.

• 45:21Speaker 05

  But I entirely agree. And I think that's actually what's been so, like, why the data center stuff has kind of caught fire, because people are showing up at the local library for the community meeting and be like, "Oh, my God, I haven't seen you in so long."

• 45:35Kevin Roose?

  Yeah. It's so hard, because it can feel like all of this is happening at an individual level, right? It is the individual that chooses to download Instagram and create an account and spend all day scrolling instead of going to the community meeting. But it is also clear that, in aggregate, it does have this atomizing effect. And it's not clear to me that people are one day just going to wake up and say, "Well, the hell with this, right? Let's, like, sort of, return back to, you know, 19th century American democracy." So I honestly don't know, you know, what to do about it, because so much of the problem just looks like adults making free choices.

• 46:14Speaker 05

  Yeah. And I think, I think actually the social harms are legible currently in a way that the political harms are maybe not. And my book is about the political harms. Like, I think people know, like, actually, I just think Instagram's really bad for my teenager. Or, you know, I just, I know that I used to read novels at night when I got into bed. This is not autobiographical. I insist this is not. But now I watch YouTube reels. Like, and I've lost something. I think people can see the social harms. And then, and it does feel like, because so much of our politics amounts to consumer choice, like, "Oh, well, you could just decide to do it differently." Well, I don't know. Like, it's, some of these things are hard to make decisions about.

• 46:56Speaker 05

  So I kind of, I'm somewhat optimistic about some of the social harms, because I think they're remediable. Is that a word? But I think the political harms are less visible to us, and that's partly why I wrote the book.

• 47:09Kevin Roose?

  I know you are a historian, Jill, and this could be our last question, but I'm wondering if you have some sort of inspired, brief, Michael Pollan-esque advice for people who are trying to wrap their head around some of the diagnosis that you've made in this book and sort of live in a way that is more consistent with nature and their own values and humanity? Like, what is the, you know, eat less, mostly plants?

• 47:38Casey Newton?

  Eat data, not too much.

• 47:39Kevin Roose?

  Yeah, what is the Jill Lepore version of that maxim?

• 47:42Speaker 05

  You know, I have not, you don't want a historian with a pitchfork. Like, I just think that's a bad plan. I think you kind of want to set yourself up for not having a humanoid robot in your living room in five years' time. And one of those, one of those ways is to take, like, certain rooms of your house back, one at a time. Like, take your bedroom back first. The bathroom, the fucking bathroom. Okay. Rescue yourself from the bathroom. Let the bathroom be a sanctuary.

• 48:13Casey Newton?

  Keeping your phone out of the bathroom, that is, you will be doing your part to dismantle the artificial state.

• 48:19Speaker 05

  Look, you gotta start small. I'm sorry, I told you, I told you, don't ask a historian.

• 48:24Casey Newton?

  All politics is local.

• 48:25Speaker 05

  There's a lot of big things you could do. Vote for someone who supports having a data center moratorium until we can actually deliberate over these really crucial matters democratically.

• 48:34Kevin Roose?

  All right. Well, Jill, that's a great place to leave it. The new book is The Rise and Fall of the Artificial State. Jill Lepore, thanks so much for joining.

• 48:42Casey Newton?

  Thank you, Jill.

• 48:43Speaker 05

  Thanks, you guys.

• 48:44Kevin Roose?

  When we come back, what do a bankrupt airline, a mysterious Amazon warehouse, and the AI startup mechanized have in common?

• 48:53Casey Newton?

  I'm refusing to comment on the advice of my lawyer.

• 48:56Kevin Roose?

  Find out in our new segment, Train of Thought.

• 49:05Speaker 00

  Today's AI builders need infrastructure that scales with their ambitions. Iron is a vertically integrated AI cloud platform, delivering data centers, compute and software for AI training and inference, underpinned by its expansive portfolio of land and grid-connected power in renewable-rich regions globally. Iron's AI cloud gives direct access to Nvidia GPUs, scalable from hundreds to thousands. For builders, that means direct access to AI compute, rapid scaling and predictable pricing. Iron, AI cloud for the next big thing.

• 49:38Speaker 03

  Ripple has been powering modern finance for a decade. It's not our first trip around the blockchain, and it won't be our last. We're leading the charge to bridge the gap between existing financial systems and the future of finance. You can have it all: legacy, stability and digital innovation, systems that work with your other systems. Because with the right partner, the question isn't crypto or traditional finance. The question is simply, why choose? A new standard of modern finance. It's happening with Ripple. For informational purposes only, availability of products can vary, may be provided by different Ripple entities.

• 50:12Speaker 04

  Stripe is built for whatever comes next. With Stripe Billing, you can easily test and iterate subscriptions, launch new usage-based and hybrid pricing models, and automate invoicing. Stripe Billing supports whatever business model you want to launch next, and Stripe powers all the Forbes AI 50 who have monetized. Learn how you can power any business model at stripe.com/billing.

• 50:37Kevin Roose?

  Casey, did you see this story about Google buying the data of Spirit Airlines?

• 50:41Casey Newton?

  I sure did.

• 50:42Kevin Roose?

  This was one of the most fascinating stories I've seen in recent weeks, and it led me down this incredible rabbit hole of thinking about training data and the new era of data collection we are in. So I thought we should use this Spirit story as an occasion to just sort of catch each other up on the state of AI training data in general, because it is fascinating and I think underappreciated.

• 51:05Casey Newton?

  And it sounds like the perfect frame, Kevin, for our new segment, Train of Thought.

• 51:10Kevin Roose?

  I love that we're just starting new segments every week until the show ends.

• 51:14Casey Newton?

  We are.

• 51:15Kevin Roose?

  It's time for the first and last installment of our new segment, Train of Thought.

• 51:19Casey Newton?

  This is kind of the caboose, as it were.

• 51:20Kevin Roose?

  Yes.

• 51:21Casey Newton?

  Interestingly, we have two Train-related segments on the show.

• 51:33Kevin Roose?

  So the reason that we wanted to do this segment today is because there was a very strange story that piqued our attention over the past week, involving the defunct airline, Spirit Airlines.

• 51:46Casey Newton?

  Hands down the worst airline of all time. I don't even know who else is in the conversation. And yes, I did fly it one time.

• 51:52Kevin Roose?

  This week, a bankruptcy court auctioned off Spirit Airlines' internal corporate data. Google won the bid, offering to pay $10 million for this data, beating out a $7.5 million offer from the AI data company, Merckor.

• 52:08Casey Newton?

  $10 million, Kevin. What did Google get for that price?

• 52:12Kevin Roose?

  So this deal apparently included 100 million emails, 500 million Microsoft Teams chats and other conversations, 7.5 billion passenger transaction records dating back to 2008, and 30 million lines of Spirit's internal source code and other documentation.

• 52:30Casey Newton?

  Well, I would consider Spirit's internal source code malware, but everything else sounds interesting. So what do we think Google is going to do with this data?

• 52:37Kevin Roose?

  This was where my head went after I saw this, because I thought, why is Google guardian of the world's information, presumably the possessors of vastly more data for training AI models than any company in the world? What are they doing, paying $10 million for this bankrupt airline's data? And that sent me down like a really fascinating rabbit hole of this world of training data and training environments that all of the AI companies now are investing really heavily into.

• 53:05Casey Newton?

  Well, tell us what you've learned.

• 53:06Kevin Roose?

  So Casey, did you know that there is a company that auctions off the Slack histories and email histories of defunct companies?

• 53:13Casey Newton?

  I'm surprised to learn that there is a market for that.

• 53:15Kevin Roose?

  Yes, so there is a company, Simple Closure, whose whole business used to be sort of helping failed startups wind down, but now they have--

• 53:22Casey Newton?

  These guys are like the undertakers of Silicon Valley.

• 53:25Kevin Roose?

  Yes, exactly.

• 53:26Casey Newton?

  Their corporate logo is just like the Grim Reaper.

• 53:30Kevin Roose?

  Yes, if these guys show up at your office or you get a call from them, it's a very bad day for your company. So basically, they were sort of like, you know, helping do the orderly wind downs of these things, but then they realized, oh, there's actually a market for the data from these dead startups. And so they start selling it to AI companies. And as of April of this year, they had done almost 100 deals, ranging from roughly $10,000 to $100,000 per company.

• 53:57Casey Newton?

  And again, I just want to know, like, what happens when the buyer actually gets a hold of the data? Like, where does it go? And does it violate my HIPAA rights?

• 54:05Kevin Roose?

  It does not violate your HIPAA rights. These are presumably not things that are covered by HIPAA, but this is basically this new strategy. So there was an era where all of the data collection and scraping that the AI companies did was sort of focused on getting the highest-quality text, images and video they could. This was used for pre-training, for the first step in the model process. You throw in as much data as it can, the model learns from it. This is sort of how you saw the models improve for many years.

• 54:35Casey Newton?

  Yes.

• 54:35Kevin Roose?

  And this is where all these stories came from, about scraping Reddit, feeding it into the models.

• 54:41Casey Newton?

  Or even our stories, Kevin.

• 54:42Kevin Roose?

  Yes, those were sort of the first era of AI training. Now we are in this different era, which I would call the era of experience, which is basically where these models now, the way that they are improved is through reinforcement learning. Reinforcement learning is sort of this trial and error process, where you go out, you sort of do a little task or a test or play a game, and you get a score or you get some indication of whether you've succeeded or not.

• 55:12Casey Newton?

  Like maybe you've broken into hugging phase. Success!

• 55:15Kevin Roose?

  Yes, that one was a success. And then you sort of try it over and over again, using slightly different strategies or slightly different techniques every time, and you sort of get signals about what works and what doesn't. And that's how you improve at things like autonomous coding. So what's happening with these data sets, including the data set of the dearly departed Spirit Airlines, presumably is that they are being turned into reinforcement learning environments for AI agents to learn new tasks. So basically, you use this data to rebuild whatever company you've acquired their data. So you're basically rebuilding an airline or an insurance company or a startup as an environment for AI agents, as a training gym essentially,

• 55:58Kevin Roose?

  for these agents to go out and try different tasks and see whether they succeed or fail.

• 56:03Casey Newton?

  You're creating a nightmare parallel universe where Spirit Airlines still exists and is booking flights.

• 56:10Kevin Roose?

  Yes, so you can kind of rebuild the company as a video game. You can sort of mine tasks from these emails and Teams messages, and then you can actually see how things played out. So for example, if you have the transaction history of an airline, you can say, what happened in 2015 when there was a big storm on the East Coast? How did the routing decisions get made? And did that result in people getting to their destinations on time?

• 56:41Casey Newton?

  And if you can get Spirit Airlines to turn a profit in the sandbox, that's AGI.

• 56:46Kevin Roose?

  Yes, so it's basically you have these kind of data points that come from these companies about how people interact with systems, about how systems interact with each other, and about how customers navigate through these sort of giant systems.

• 57:02Casey Newton?

  Well, I am so relieved to hear you say all of this, Kevin, because when I saw this story, I thought, oh my God, Google is going to start an airline. And you know with them, it wouldn't just be one airline. There would be an app, and it was like, okay, you have to choose, are you flying Google Airlines, Google Airways or Google Air? And they would all be the same, but they would all be completely different. And also, there would probably be different apps.

• 57:23Kevin Roose?

  Well, and also, since they were trained on Spirit Airlines, they would also charge you for peanuts, charge you for a slightly bigger seat, charge you to use the bathroom, maybe. Everything would be charges.

• 57:35Casey Newton?

  Yeah, so not eager to see that business.

• 57:38Kevin Roose?

  So what's interesting about the dead company sort of data market is that you are sort of able to turn these companies into sort of living, zombified simulacra of the original company, but you're also training these systems on companies that ultimately did not succeed. So I'm very curious to know, like, if these data sets actually are helping these models improve at these tasks? Or if there's some subtle way in which they are being conditioned on the data of unsuccessful companies and thereby are becoming worse at the tasks that they're trying to learn?

• 58:12Casey Newton?

  You're worried that these future models are going to have a loser mentality.

• 58:15Kevin Roose?

  Yeah.

• 58:16Casey Newton?

  They don't have what it takes to cut it in the modern economy.

• 58:19Kevin Roose?

  There's another interesting data story this week that came from 404 Media, which was that they slipped an AirTag into a rare book that was part of a bulk book order on a marketplace site called Bibliio. They finally determined that this book lands at an Amazon warehouse in Las Vegas, specifically an internal unit called VGT3, which has, according to 404 Media, a door marked with the logo of a dinosaur eating a book.

• 58:48Casey Newton?

  That feels a little on the nose, even for this simulation, I have to say.

• 58:52Kevin Roose?

  So Casey, why are these books ending up at mysterious Amazon warehouses? What are they doing with them?

• 58:58Casey Newton?

  It is a good question, and this ties into some of the lawsuits that have been filed against the big AI labs, and in particular, this big case against Anthropic that you may remember. And the judge in that case ruled that because Anthropic had bought these millions of print books, scanned them, and then discarded the paper, that this was fair use of the material because each digital copy had replaced a legally purchased print original. So there was no multiplication of the number of copies. It was just kind of a one-to-one shift in format.

• 59:39Kevin Roose?

  I see.

• 59:40Casey Newton?

  And so what I think the other labs have taken away from this, Kevin, is you're not going to run into as many legal issues if you destroy these books.

• 59:49Kevin Roose?

  Right. So it's not like the AI companies are kind of giddy about like destroying these relics of civilization.

• 59:54Casey Newton?

  Well, they might be. If you told me I got to destroy all of Kevin Lewis's books, I'd be having a good day at the office.

• 59:59Kevin Roose?

  But it is like, it is the sort of fallout of this legal environment that they're in where it's just safer for them legally to destroy the books after they've finished scanning them.

• 1:00:07Casey Newton?

  And I just had to say, like, this whole thing seems so stupid to me, like, truly a case where we are honoring the letter of the law, but not the spirit, right? It's like, yes, you literally transformed the data, but obviously, you know, the real complaint here that the authors have, at least, you know, the ones who have sued, is I didn't want you to use my, you know, book this way. So I expect we're going to see a lot more angst over this as we continue to see more books destroyed.

• 1:00:32Kevin Roose?

  Yeah.

• 1:00:33Casey Newton?

  You know, back in my day, Kevin, we would only see books destroyed because the Republicans had read a gay sex scene. And I want to get back to that point.

• 1:00:42Kevin Roose?

  All right, Casey, one more data story to talk about this week, which is sort of related to the first one that we discussed, both because it involves Google and because it involves these sort of high-quality training environments for reinforcement learning that all these AI companies are now racing to build. This one is about the 50-person startup Mechanize. We have talked about Mechanize on the show before. We interviewed two of their co-founders. They are about a year old, and they specialize in-

• 1:01:10Casey Newton?

  Their company is. They're a little older than that.

• 1:01:13Kevin Roose?

  Yes. They specialize in creating these RL environments for coding and other tasks. And they are reportedly in talks to be acquired by Google for over $1.5 billion.

• 1:01:26Casey Newton?

  Not bad for a year's work.

• 1:01:29Kevin Roose?

  So this is a big boom area inside the AI boom. Basically, if you want these sort of high-quality tasks that you can put your AI agents into and have them sort of hillclimb on them, get a little bit better every time, they need to be good tasks, right? They need to be thoughtfully created. They need to not have a bunch of sort of obvious flaws in them, and they need to mirror the things that real people might be doing in their jobs. So one way that you might create an RL environment is like, create a fake version of amazon.com, right? And, and everything about it is exactly identical to amazon.com, except it's not called amazon.com, and it's just for these AI agents to learn how to click around, put things in the cart, browse the, the site.

• 1:02:14Casey Newton?

  Destroy books.

• 1:02:15Kevin Roose?

  Destroy, destroy books in a warehouse in Las Vegas. So this kind of, sort of simulated environment is the kind of thing that Mechanize specializes in building and presumably why Google is interested in acquiring them. So one interesting sort of connective tissue between this story and the hugging face story that we discussed at the top of the show is that I think a lot of these RL environments are not particularly well-designed or built or secured, right? Part of the problem, and maybe the reason that companies like Google are starting to

• 1:02:55Kevin Roose?

  bring this expertise in-house rather than sort of contract it out to a vendor or a startup, is that they are finding issues with the way that these tests and these RL environments are constructed. So there's been a couple instances now of a flawed security test by this, by the same vendor, Irregular, that both Meta and Anthropic relied on. You may have seen this story a week or two ago. My guess, and, and the conversations that I've had with some of the people at the labs is basically they have, they have gotten to the point where these tests need to be so good and so secure that they have to be building them in-house using extremely high-quality data.

• 1:03:35Casey Newton?

  Yeah. Irregular put out a report about some of the incidents that you just mentioned, and it got criticism from the security community, saying, "You're not offering us enough detail to understand what went wrong." And so I suspect if Irregular is not more forthcoming, then the labs are going to feel like they have no choice but to bring this all in-house.

• 1:03:54Kevin Roose?

  Yeah. It is just like mind-boggling to me that these AI companies now are sort of essentially building like The Sims, but on the grandest planetary scale imaginable. Like, they are assembling data from the corpses of failed startups. They are turning them into simulations and video games, and then they are running their AI agents through them to try to make them superhuman at everything. And it is just like, if you, if you made that the plot of a science fiction novel 10 years ago, people would have like criticized it for being a little over the top.

• 1:04:29Casey Newton?

  It is pretty wild. Now, let me ask you this, Kevin. I'm sure you've already thought about this, but Hard Fork is preparing to wind down. Have you thought about how much money we'd be able to get for the data?

• 1:04:41Kevin Roose?

  I, I would be open to seeing bids. Now, obviously, you know, it's not the best training data. There would be a lot of bad jokes, a lot of questionable interviews.

• 1:04:54Casey Newton?

  True.

• 1:04:54Kevin Roose?

  But, but-

• 1:04:55Casey Newton?

  But if, but if Spirit Airlines can get $10 million-

• 1:04:59Kevin Roose?

  Exactly.

• 1:05:00Casey Newton?

  Somebody could buy us lunch.

• 1:05:01Kevin Roose?

  We didn't go bankrupt.

• 1:05:02Casey Newton?

  Look at us. They said it would never work.

• 1:05:05Kevin Roose?

  If you're interested in acquiring the data stores of the Hard Fork podcast-

• 1:05:08Casey Newton?

  [email protected].

• 1:05:44Speaker 00

  Today's AI builders need infrastructure that scales with their ambitions. iRen is a vertically integrated AI cloud platform, delivering data centers, compute, and software for AI training and inference, underpinned by its expansive portfolio of LAN and grid-connected power in renewable-rich regions globally. iRen's AI cloud gives direct access to Nvidia GPUs, scalable from hundreds to thousands. For builders, that means direct access to AI compute, rapid scaling, and predictable pricing. iRen, AI cloud for the next big thing.

• 1:06:17Speaker 03

  Ripple has been powering modern finance for a decade. It's not our first trip around the blockchain, and it won't be our last. We're leading the charge to bridge the gap between existing financial systems and the future of finance. You can have it all: legacy, stability, and digital innovation. Systems that work with your other systems. Because with the right partner, the question isn't crypto or traditional finance. The question is simply, why choose? A new standard of modern finance. It's happening with Ripple. For informational purposes only, availability of products can vary and may be provided by different Ripple entities.

• 1:06:50Speaker 04

  Companies are pouring money into AI, but few leaders can explain how those AI investments work with the systems you already have. You know, the ones that actually run your business? Without Systems, you don't have to start over with AI. You can choose the projects where AI will have the most impact. And OutSystems gives you the platform to make it agentic on your terms with complete control every step of the way. So you can trust AI with the mission-critical systems that run your business. Own your agentic future. Learn more at outsystems.com/NYT. That's outsystems.com/NYT.

• 1:07:28Kevin Roose?

  Hard Fork is produced by Whitney Jones and Rachel Cohn. We're edited by Viren Povich. We're fact-checked by Caitlin Love. Today's show was engineered by Katie McMurren. Original music by Marian Lozano, Rowan Nemesto, Alyssa Moxley, and Dan Powell. Video production by Sawyer Roque, Jake Nickel, and Chris Schott. You can watch this full episode on YouTube at youtube.com/hardfork. Special thanks to Paula Schuman, Puywing Tam, Brooke Minterz, and Dahlia Hadad. You can email us, as always, at [email protected]. Send us your bankrupt airline training data.