Record Labels Sue A.I. Music Generators + Inside the Pentagon’s Tech Upgrade + HatGPT

A little something for everyone: lawsuits, fighter jets and Casey in a bucket hat.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Support for this podcast comes from Dell Technologies. It's time to put AI to work. Is your business ready? The challenge isn't getting started with AI. It's scaling beyond pilots while maintaining control of your data, governance, infrastructure, and costs. Dell AI Factory provides the integrated foundation you need to scale AI securely and efficiently with one trusted partner, Dell, AI on your terms. From idea to impact. Learn more at dell.com/yourwaytoAI. That's dell.com/yourwaytoAI.

• 0:34Kevin Roose?

  Casey, today I learned something new. You know, I'm in New York. I'm visiting some friends and going to some weddings. And I'm at the New York Times building. And I learned just today that there's an entire podcast studio at the Times building that I've never seen.

• 0:47Casey Newton?

  That's how big the New York Times is. It's just full of, you know, nooks and crannies that very few people have ever seen with their own eyes.

• 0:53Kevin Roose?

  Yeah, so up on the 28th floor, apparently there's a gleaming new audio temple. I hear it's very fancy, but I've never been. So right after we taped today, I'm going to go up there. I'm going to see the promised land.

• 1:03Casey Newton?

  You know what I would do if I got to see the studio? Kevin and I were in New York.

• 1:07Kevin Roose?

  What's that?

• 1:08Casey Newton?

  I would sneak in and I'd get a little pocket knife and I'd just carve Kevin plus Casey forever into one of the brand new desks. And I would dare them to say anything to me about it.

• 1:19Kevin Roose?

  Yeah, let's not let you up there. I'm going to actually ask security to specifically.

• 1:25Casey Newton?

  Can you imagine Ezra Klein sits down to interview like the Secretary General of the United Nations? And he just sees carved into the desk, Casey plus Kevin forever?

• 1:35Kevin Roose?

  Casey was here.

• 1:39Casey Newton?

  Suck it, Klein.

• 1:45Kevin Roose?

  I'm Kevin Bruce, a tech columnist from the New York Times.

• 1:48Casey Newton?

  I'm Casey Noon from Platformer.

• 1:49Kevin Roose?

  And this is Hard Fork.

• 1:51Casey Newton?

  This week, the record label sued two leading AI music apps accusing them of copyright infringement. RIAA CEO Mitch Glaser joins us to make the case. Then we go inside the Pentagon's tech turmoil with Chris Kurchov, author of the new book, Unit X. And finally, a round of HatGPT.

• 2:17Casey Newton?

  Now Kevin, not a lot of people know this, but we have something interesting in common.

• 2:21Kevin Roose?

  What's that?

• 2:21Casey Newton?

  Well, we were a couple of the few teenagers who managed to survive the Napster era without getting sued by the Recording Industry Association of America.

• 2:29Kevin Roose?

  Yes, although one of my friends actually did get sued by the recording industry and had to pay thousands of dollars.

• 2:34Casey Newton?

  And is he still in prison? Or did he get out?

• 2:35Kevin Roose?

  No, he got out. He's fine.

• 2:36Casey Newton?

  Oh, thank God. Thank God. Well, look, Kevin, it's always a strange day when you find yourself siding with the RIAA. And yet, when I heard this week's news, I thought, well, I want to hear what they have to say.

• 2:46Kevin Roose?

  Yeah, let's talk about it.

• 2:48Casey Newton?

  So these are, I think, the biggest lawsuits to come out against AI companies since your newspaper, The New York Times, sued OpenAI. This week, the RIAA announced that major record labels are suing two of the leading AI music companies alleging massive copyright infringement and are maybe trying to shut them down.

• 3:08Kevin Roose?

  Yeah, so the companies that the music labels sued are Udio and Suno. We've talked about them a little bit on this show before. Basically, these are tools that sort of work like ChatGPT. You can type in a prompt. You can say, make me a country western song about a bear fighting a dolphin, and it'll do that. But basically, these companies have come under a lot of criticism for allowing people to create songs without compensating the original artists. Like other AI companies, these companies do not say where they're getting their data. Suno is releasing statements using words like transformative and completely new outputs, basically arguing that this is all fair use and that they don't owe anything to the holders

• 3:53Kevin Roose?

  of the copyrighted songs that they were presumably using to train their models. But we'll see how the courts see that.

• 4:00Casey Newton?

  Well, and if you've never heard one of these, Kevin, I think, and I know you have, we should play a clip, I think, just so people get a sense of just how closely these services can mimic artists you might be familiar with. So Kevin, we're about to hear a song called Prancing Queen, and this was made with Suno.

• 4:35Casey Newton?

  Can you believe what they're doing to ABBA, Kevin?

• 4:39Kevin Roose?

  You know, I actually saw an ABBA cover band once, many years ago, and that was better than the ABBA cover band.

• 4:47Casey Newton?

  You know what I liked about that clip is it reminded me, if I had had like six beers and someone shoved me onto a karaoke stage and said, "Sing a Dancing Queen from memory", that's exactly what it would have sounded like. So, we wanted to get to the bottom of this, so we reached out to the RIAA, and they offered up Chairman and CEO Mitch Glaser, so we're gonna bring him on and ask him what this lawsuit is all about.

• 5:10Kevin Roose?

  Let's do it.

• 5:16Casey Newton?

  Mitch Glaser, welcome to Hard Fork.

• 5:19Speaker 03

  Thanks, thanks for having me.

• 5:20Casey Newton?

  So, make your case that these two AI music companies violated copyright law.

• 5:27Speaker 03

  Pretty easy case to make. They copied basically the entire history of recorded music, they stored it, then they used it by, you know, matching it to prompts so that they rejiggered the ones and zeros, and you know, basically, they took chicken and made chicken salad, and then so they don't have to pay for the chickens.

• 5:50Casey Newton?

  Well, some people out there say that this is a transformative use, that no matter what you put into a UDO or a Suno, you're not gonna get back the original track, you're going to get something that has been transformed. What do you make of that case?

• 6:07Speaker 03

  Well, there is such a thing as transformative use, it's actually a pretty important doctrine. It's supposed to help encourage human creativity, not substitute for it. There was a really important Supreme Court case on this issue, thank God, that just happened last year, where they kind of dispelled this notion that, you know, anytime you take something and, you know, splash a little bit of color on it, it's transformative. That's not what that means, and this is very similar.

• 6:39Kevin Roose?

  Mitch, you said that these companies have scraped the entire sort of history of recorded music and use them to train their models. But I read through the complaint that came out, and there isn't sort of direct evidence, right? There's no smoking gun. They haven't said outright, yes, we did train on all this copyrighted music. Presumably, that is something you hope will come out in the course of this case. But do you actually need to be able to prove that they did use copyrighted music in order to win this case? Can the lawsuit succeed without that?

• 7:11Speaker 03

  I think ultimately, we do have to show that they copied the music, but they can't hide their inputs and then say, sorry, we're not going to tell you what we copied, so you're not allowed to sue us for what we copied. That they can't do. So what we were able to do was show in the complaint that there's no way they could have come out with this output without copying all of this on the input side. It's sort of this equitable doctrine in fancy legal terms that says you're not allowed to hide the evidence and then say, you can't sue me.

• 7:44Kevin Roose?

  Right. Well, on that point, one of my favorite parts of the Suno lawsuit is where it discusses Suno reproducing what are called producer tags, which is when a producer says their name at the start or end of a song. What does it mean that Suno can nail a perfect Jason Derulo?

• 8:01Speaker 03

  Well, thank God, Jason Derulo likes to say his name in the beginning of his songs, right? And, you know, in the blender, that piece wasn't ripped apart enough. And so that was sort of one of those smoking guns where we're able to show, if you look at the output, right, and Jason Derulo's tag is in the output, I think they copied the Jason Derulo song on the input.

• 8:31Kevin Roose?

  Yeah. So one of the arguments we've heard from AI companies, not just AI music companies, but also companies that train language models, is that, you know, these machines, these models, they're basically learning the way that humans learn. They're not just sort of regurgitating copyrighted materials. They are learning to generate wholly new works. And I want to just read you Suno's response that they gave to The Verge and have you share your thoughts on it. Suno said, quote, "We would have been happy to explain this to the corporate record labels that filed this lawsuit. And in fact, we tried to do so. But instead of entertaining a good faith discussion, they've reverted to their old lawyer-led playbook. Suno is built for new music, new uses, and new musicians.

• 9:15Kevin Roose?

  We prize originality." What do you make of that?

• 9:18Speaker 03

  Yeah, I love this argument. I love that machines are original and machines and humans are the same. If you just use human words around machines, like "learning," well, you know, then there's no difference between us. If you read a book, it's the same as copying it on the Xerox machine and then mixing all the words around and then coming out with something new. Has nothing to do with the fact that they actually happened to take all of these human-created works. You know, machines don't learn, right? Machines copy, and then they basically match a user's prompt with an analysis of patterns in what they've copied. And then they finish the pattern, beta-stump predictive algorithms

• 10:00Speaker 03

  or models, right? That's not what humans do. Humans have lived experiences. They have souls. They have genius. They actually listen, get inspired, and then they come out with something different, something new. They don't blend around patterns based on machine-based algorithms. So, nice try, but I don't think that that argument is very convincing. And I also love that they say that, you know, the creators and their partners are the ones that have resorted to like the old legal playbook. They're not resorting to, "Oh, we can do this. It's based on fair use. It's transformative. We're going to seek forgiveness instead of permission."

• 10:37Kevin Roose?

  Well, I mean, you also have the investor in the company who you quote in the lawsuit saying, because he said this to a news outlet, "I don't know if I would have invested in this company if he had a deal with the record labels, because then they probably wouldn't have needed to do what they needed to do," which I assume he sort of meant, you know, hoover up all this music without paying for it.

• 10:55Speaker 03

  Yeah, that's in the legal world, what we call a bad fact. That is a bad fact for the other side. You don't want your investor saying, gee, you know, if they had really done this the legal way, I don't think I would have invested because it's just too hard. It's just too hard to do it the legal way.

• 11:13Kevin Roose?

  Mitch, we've seen other lawsuits come out in the past year from media companies, including the New York Times, which sued OpenAI and Microsoft last year alleging similar types of copyright violations. How similar or different from the sort of text-based copyright arguments is the argument that you are making against these AI music generation companies?

• 11:34Speaker 03

  I think the arguments are the same, that you have to get permission before you copy. It's just basic copyright law. The businesses are very different, and I think looking at sort of the the public reports on the licensing negotiations going on between the news media and companies like OpenAI, where you, you know, news is dynamic. It has to change every single day. And so there needs to be a feed every single day for the input to actually be useful for the output. Music is catalog, right? You copy the song once, it's there forever. You know, you don't have to change it. You don't have to feed the beast every single day. So I think the business models are quite different, but I think that the the legal basis is very similar.

• 12:19Casey Newton?

  Well, and does that suggest that for you all, it's actually essential that you are able to capture the value of the back catalogs for training, whereas for these media outlets, they might have a better chance of securing ongoing revenue?

• 12:36Speaker 03

  I think that's right. I also think that we have an artistic intent element that's very, very different. You know, the, it's one thing for somebody to say you can copy this into your input. It's another to say that you can then change it so that the output uses the work of the artist, but it doesn't match their artistic intent. And, you know, to say that these, you know, sort of what Kevin was saying earlier, they're saying, look, we're just, you know, we had discussions. What's your problem? Well, the problem is we work with human artists who care about the output, and so they need to have a role and a place in deciding what, how their art's being used.

• 13:12Kevin Roose?

  Yeah.

• 13:13Casey Newton?

  You know, my understanding is that it's actually gotten much more difficult and expensive to sample lately than it used to be in ways that I don't really like. I'd probably like to see more sampling than we do, but it seems like something changed around the time that the song Blurred Lines came out. And now, all of a sudden, everybody has to like, you know, even just a whisper of familiarity. Is there anything sort of in whatever led to that situation that you expect you'll bring to this lawsuit?

• 13:38Speaker 03

  I think sampling is actually a pretty good example because samples are licensed today and there's plenty of sampling going on. Now, does it mean that anybody can sample anything they want without permission? No. Do we have to have clearance departments that go out, whether you're talking about, you know, a video or a movie or another song, and get those rights, especially from, you know, publishers and prior artists? Yes, you do. That's called ownership. And, you know, you actually get to control your own art and what you do, and it's not a simple process all the time. I mean, it takes work. I'm sure that our companies get frustrated in trying to do clearances, but it's what you gotta do.

• 14:22Casey Newton?

  Yeah. There have been some companies that have faced copyright challenges in AI generative products that have responded by basically limiting the products by saying you can't, you know, refer to a living artist in a prompt. It won't give you a response. Basically, to try to quell some of these concerns. Would that satisfy your concerns? Or are you trying to shut these things down altogether?

• 14:45Speaker 03

  They're trying to confuse the issue. They're pretending that this is about the output. The lawsuit is about the input, right? So actually, by saying you can't type Jason Derulo's name, you can't type Adele's name, what they're basically doing there is further hiding the input. They're making it so that you can't see what they copied, and they're pretending that this is all about the output in order to say, look, we're putting guardrails on this thing. That's not what this lawsuit's about. This lawsuit is about them training their model on all of these sound recordings, not on limiting prompts on the output to further hide the input. But it's clever. It's clever.

• 15:28Casey Newton?

  Okay. So you want to shut this down?

• 15:32Speaker 03

  Well, I don't think that we want to, we definitely, we call it an injunction, Kevin. We would like to, we would like to shut down their business as it's operating now, which is something illegally trained on our sound recordings with output that doesn't reflect the artist's integrity, yes. Does that mean that we want to shut down AI generators or AI companies? No, there's 50 companies that are already licensed by the music industry. And I think it's important, and this differs a lot from, I think, the old days, but nobody's scared of this technology, as in, they want to shut down the technology. Everybody wants to use the technology. But they definitely see sort of good AI versus

• 16:15Speaker 03

  bad AI. Good AI compliments artists, helps them stretch music, helps, assists them in the creation of music. Bad AI takes from them, gives no attribution, no compensation, asks no permission, and then generates something that's a bunch of garbage.

• 16:34Kevin Roose?

  Yeah, I, I'd know of some artists who would say they want to shut down this stuff entirely, that they don't think there's any good form of it. But you mentioned the old days. And so I want to ask you about this. I think a lot of my fellow millennials think of the RIAA as the group that went around, you know, suing teenagers for pirating music during the Napster era. The RIAA has also sued a bunch of other file sharing and music sharing platforms and actually fought the initial wave of streaming music services like Spotify, because there was this fear that this all you can eat streaming services would eat into CD sales. Now, of course, we know that streaming wasn't the death of music or music labels that actually was, ended up being, sort of saving the music

• 17:17Kevin Roose?

  industry. Do you think there's a danger here that actually these AI music generation programs could ultimately be great for music labels, just like Spotify was, and that you might be trying to cut off something productive before it's actually had the chance to mature?

• 17:32Speaker 03

  I don't think it's it's really the same at all. I think that there's an embrace of AI. And there was well before these generators came out or well before OpenAI. Especially within the tech content partnerships that have existed and have grown and matured and gotten sophisticated through the streaming age. So, you know, even though the RIA's job is to be the boogeyman and to, you know, go out there and enforce rights, which, you know, we do with zeal and hopefully a smile doing our job. Here, I think that the really what we're trying to do is create a marketplace like streaming,

• 18:17Speaker 03

  where there are partnerships and both sides can grow and evolve together. Because the truth is, you don't have one without the other. You know, record companies don't control their prices. They don't control their distribution. They're now gateways, not gatekeepers. The democratization of the music industry has changed everything. And I think they're seeking the same kind of relationships with AI companies that they have with streaming companies today.

• 18:40Kevin Roose?

  What would a good model look like? I mean, there are reports this week that YouTube is in talks with record labels about paying them a lot of money to license songs for their AI music generation software. Do you think that's the solution here? That, you know, there'll be sort of these platforms that pay record labels and then they get to use those labels' songs in training their models? Do you think it's fine to use AI to generate music as long as the labels get paid? Or is there sort of a larger objection to the way that these models work at all?

• 19:10Speaker 03

  I think it works as long as it's done in partnership with the artists. And at the end of the day, it moves the ball forward for the label and the artist. I mean, the YouTube example is interesting because that's really geared towards YouTube Shorts, right? It's geared towards fans being able to use generated music to put with their own videos for 15 or 30 seconds. That's an interesting business model. BandLab is a tool for artists, Splice, Beatport, Focusrite, Output, Waves, Eventide, every digital audio workstation that's now using AI, Native Instruments, Oberheim. I mean, there are so many.

• 19:55Speaker 03

  AI companies that have these bespoke agreements and different types of tools that are meant to be done with the artistic community, that I think the outliers are the Sunos and the UDOs, who frankly are not very creative in trying to help with, you know, human ingenuity. Instead, they're just substitutional to make money for investors by taking everybody else's stuff.

• 20:21Casey Newton?

  We've seen some pretty different reactions to the rise of AI among artists. Some people clearly seem to want no part of it. On the other hand, we've seen musicians like Grimes saying, here, take my voice, make whatever you want. We'll figure out a way to share the royalties if any of your songs becomes a hit. I'm curious, if you're able to get the deals that you want, do you expect any controversy within the artist community? And artists saying, hey, why'd you sell my back catalog to this blender? I don't want to be part of that.

• 20:50Speaker 03

  Yeah, I think, look, artists are entitled to be different. And there are going to be artists, I think Kevin, you said earlier, you know, you know, artists who are so scared of this. They just, they do want to shut the whole thing down. They just don't want their music and their art touched, right? I know directors of movies who can't stand that the formatting is different for an airplane. Like, that's their baby, and they just don't want it. Then there are artists like Grimes, who are like, I'm fine being experimental. I'm fine having fans take it and change it and do something with it. All of that is good. They're the artist, right? I mean, it's their art. Our job is to invest in them, partner with them, help find a market for them. But at the end of the day, if you're trying to find a market for an artist's work that they don't, and they don't want that work in the market, it's not going to work.

• 21:36Casey Newton?

  Yeah. Have you listened to much AI-generated music? Are there any songs you've heard that you thought, that's actually kind of good?

• 21:43Speaker 03

  Yeah, so I think in the sort of overdubbing voice and likeness thing, that it's a little bit better than some of the, you know, simple prompts on these AI generators like UDO and Suno. But I heard a

• 21:59Speaker 03

  I heard Billie Eilish's voice on a revivalist song and I was like, wow, she should cover this song. It was, it was really great. It was just kind of seemed like a perfect fit and it's fun to play with those things. But again, like in that case, I think Billie Eilish gets to decide if her voice is used on something. I think she gets to decide to decide if she wants to do a cover. I don't think that it's up to, you know, overdub to be able to, to do that. I did do a bunch of prompts, as you can imagine, on some of these services, trying to see like, what happens if you just put in a few words, like a simple country song? And then what happens if you put in like 20 different descriptors? And what's amazing is you can, every 10 seconds, you get a new song. So if you don't like

• 22:42Speaker 03

  it, just put in a few more words and it rejiggers the patterns and you can start getting to a point where you're like, okay, it's not human. And the lyrics kind of suck, but it's not terrible. And, you know, we are only six months into the huge progression of this technology. And if you had listened to, you know, a prompt where you were allowed to put in Jason Derulo or Mariah Carey six months ago versus, you know, now, you would notice a marked improvement. And that's one of the reasons why we needed to get out there now. We needed to bring this suit. We need the courts to settle this issue so that we can

• 23:23Speaker 03

  move forward on thriving marketplace before the technology gets so good that it is a seismic threat to the industry.

• 23:31Kevin Roose?

  I've seen a lot of support for this lawsuit among people, you know, I follow who are more inclined to side with artists and musicians. But there have also been some tech industry folks who, who think this is all kind of, you know, it's, it sounds like the RAA is just sort of anti-progress, anti-technology. I even saw one tech person call you the ultimate D-cells, which is like in Silicon Valley, that's sort of like the biggest insult. D-cells are people who want to basically stop technological progress, basically Luddites. What do you make of that line of argument from the Valley?

• 24:07Speaker 03

  You know, this has been the same argument that the Valley's had since 1998. To me, that's a 30-year-old argument. If you look at the marketplace today, where Silicon Valley thrives is when rights are in place and they form partnerships. And then they grow into sophisticated global leaders where they can tweak, you know, every couple of years, their deals and come up with new products that allow them to feed these devices that are nothing without the content on them. And, you know, we all, there's always sort of this David versus Goliath thing, no matter what side you're on. But if you think about it, music, which is a $17 billion industry in the United States,

• 24:52Speaker 03

  I mean, I don't even, I think one tech company's cash on hand is five times that, right? Not to mention their $289 billion market caps, right? But they are completely dependent on the music that these geniuses create in order to thrive. And to say that these creators are stopping their progress, I think is sort of laughable. I think what's much more threatening is if you move fast and break things without partnerships, what are you threatening on the tech side with a no-holds-barred, you know, culture-destroying, you know, machine-led world? It sounds pretty gross to me.

• 25:30Casey Newton?

  So what happens next? The lawsuits have been filed. This stuff tends to take a long time, but what can we look forward? Do you, you know, will, will there be sort of scandalous emails unearthed in discovery that you'll post to your website? Or what can we look forward to here?

• 25:43Speaker 03

  Well, moving forward in, in discovery, I think we'll be prohibited from posting any, anything to our.

• 25:50Casey Newton?

  Oh, man.

• 25:50Speaker 03

  I know. You're, you think you're disappointed.

• 25:52Kevin Roose?

  If you want to just send them to [email protected], that's fine.

• 25:57Speaker 03

  I live for that stuff. But we will, we will, of course, follow the rules. But, um, you know, we have filed in the districts where these companies reside. And so I hope that within a year or so, we will actually get to the meat of this. Because if you think about it, the, the judge has to decide when they raise fair use as a defense. Is this fair use or not? Right? And that is something that, you know... Has to be part of the beginning, part of the lawsuit. So we're hopeful that, you know, when I say a short time, in legal terms, that means, you know, a year or two. But we're hoping that in a short time, we will actually get a decision. And that it sends the right message to investors and to new companies.

• 26:42Speaker 03

  Like, there's a right way and a wrong way to do this. Doors are open for the right way.

• 26:45Kevin Roose?

  Yeah. I think there's a story here about startups that are sort of moving fast, breaking things, asking for forgiveness, not permission. But I also think there's a story here that maybe we haven't talked about, about restraint. Because I know that a lot of the big AI companies had tools years ago that could generate music, but they did not release them. I remember hearing a demo from someone who worked at the big AI companies, one of the big AI companies, maybe two years ago, of one of these kinds of tools. But I think they understood, they were scared, because they knew that the record industry is very organized, it has this kind of like, you know, history of litigation. And, you know, they sort of understood that they were likely to face lawsuits if they let this out into the public.

• 27:30Kevin Roose?

  So have you had discussions with the big, the bigger AI companies, the more established ones that are working on this stuff? Or are they just sort of intuiting correctly that they would have a lot of legal problems on their hands if they let this stuff out into the general public?

• 27:44Speaker 03

  You know, you're raising a point that I don't think is discussed often enough, which is that there are companies out there that deserve credit for restraint. And part of it is that they know that we would bring a lawsuit, and in the past, we haven't been shy, and that's useful. But part of it is also because these are their partners now. You know, there are real business relationships here, and human relationships here between these companies. And so they, their natural, I think they're moving towards a world where their natural instinct is to

• 28:20Speaker 03

  Approach their partners and see if they can work with them. You know, I know that YouTube did sort of its Dreamcast experiment, approached artists, approached record companies. That was sort of like the precursor or the beta to whatever they might be discussing now for what's going to go on shorts that we talked about earlier. And I'm sure that there are many others. But you're right, yes, there are going to be companies like Suno and Udio that just seek investment, want to make a profit and steal stuff. But there is, there is restraint and constructive action by a lot of companies out there who do view the creators as their partners.

• 28:59Kevin Roose?

  Well, it's a really interesting development and I look forward to following it as it progresses.

• 29:06Speaker 03

  Thanks Mitch.

• 29:07Kevin Roose?

  Thanks so much Mitch, thanks for coming by.

• 29:08Speaker 03

  Thanks guys.

• 29:12Kevin Roose?

  When we come back, we're going inside the Pentagon with Chris Kurchhoff, the author of Unit X.

• 29:17Casey Newton?

  Oh really, inside the Pentagon?

• 29:41Speaker 00

  Support for this podcast comes from Dell Technologies. It's time to put AI to work. Is your business ready? The challenge isn't getting started with AI. It's scaling beyond pilots while maintaining control of your data, governance, infrastructure and costs. Dell AI Factory provides the integrated foundation you need to scale AI securely and efficiently with one trusted partner, Dell, AI on your terms. From idea to impact. Learn more at dell.com/yourwaytoAI. That's dell.com/yourwaytoAI.

• 30:15Speaker 04

  AWS AI is how the world's leading organizations are transforming their industries, not in theory, but in production at scale right now. Like turning race day data into real time insights for Formula One fans, reinventing the journey for every United traveler, and personalizing every trip with booking.com. However big the question for your business or industry, AWS AI is how.

• 30:41Speaker 05

  AI spend is growing fast, but most businesses still can't see exactly where that money is going. Ramp gives businesses visibility and control across AI providers with real time guardrails by team and model, helping ensure spend goes to the highest ROI areas. Stop guessing where your AI spend is going. Get started with Ramp. And if you're new to Ramp, you can get $250 off at ramp.com/hardfork. Go to ramp.com/hardfork. Ramp.com/hardfork.

• 31:14Kevin Roose?

  Well, Casey, let's talk about war.

• 31:15Casey Newton?

  Let's talk about war. And what is it good for? Some say absolutely nothing, others write books arguing the opposite.

• 31:24Kevin Roose?

  Yeah. So I've been wanting to talk about AI and technology and the military for a while on the show now, because I think what's really flying under the radar of kind of the mainstream tech press these days is that there's just been a huge shift in Silicon Valley toward making things for the military and the US military in particular. You know, years ago, it was the case that most of the big tech companies, they were sort of very reluctant to work with the military, to sell things to the Department of Defense, to make products that could be used in war. They had a lot of ethical and moral quandaries about that, and their employees did too. But we've really seen a shift over the past few years.

• 32:08Kevin Roose?

  There are now a bunch of startups working in defense tech, making things that are designed to be sold to the military and to national security forces. And we've also just seen a big effort at the Pentagon to modernize their infrastructure, to update their technology, to not get beat by other nations when it comes to having the latest and greatest weapons.

• 32:30Casey Newton?

  Yeah. And also, Kevin, just the rise of AI in general, I think, has a lot of people curious about what the military thinks of what is going on out here. And is it eventually going to have to adopt a much more aggressive AI strategy than the one it has today?

• 32:44Kevin Roose?

  Yeah. So a few weeks ago, I met a guy named Chris Kircoff. He's one of the authors, along with Raj Shah, of a book called Unit X. Chris is sort of a longtime defense tech guy. He was involved in a number of tech projects for the military. He worked at the National Security Council during the Obama administration. Fun fact, he was the highest ranking openly gay advisor in the Department of Defense for years. And most importantly, he was a founding partner of something called the Defense Innovation Unit, or DIU. It also goes by the name Unit X, which is basically this little experimental division that was set up about a decade ago by the Department of Defense to try to basically bring the Pentagon's technology up to date.

• 33:29Kevin Roose?

  And he and Raj Shah, who was another founding partner of the DIU, just wrote a book called Unit X that basically tells the story of how the Pentagon sort of realized that it had a problem with technology and set out to fix it. So I just thought we should bring in Chris to talk about some of the changes that he has seen in the military when it comes to technology and in Silicon Valley when it comes to the military. Let's do it.

• 34:08Kevin Roose?

  Chris Kircoff, welcome to Hard Fork. Glad to be here. So I think people hear a lot about the military and technology, and they kind of assume that there are like very futuristic things happening inside the Pentagon that we'll hear about at some point in the future. But a lot of what's in your book is actually about old technology and how underwhelming some of the military's technological prowess is. Your book opens with an anecdote about your co-author actually using a Compaq digital assistant because it was better, it had better navigation tools than the navigation system on his $30 million jet. That was sort of how you introduced the fact that the military is not quite as technologically sophisticated as many people might think.

• 34:52Kevin Roose?

  So I'm curious, when you first started your work with the military, what was the state of the technology?

• 34:59Speaker 06

  Well, it's really interesting. I mean, you go to the movies and we have all seen Mission Impossible and James Bond and wouldn't it be wonderful if that actually were the reality behind the curtain? But when you open up the curtain, you realize that actually in this country, there are two entirely different systems of technological production. There's one for the military and then there's one for everything else. And to dramatize this, on the image of our book, Unit X, we have an iPhone and on top of the iPhone is sitting an F-35, the world's most advanced fighter jet, a fifth generation stealth fighter, known as a flying computer for its incredible sensor fusion and weapon suites. But the thing about the F-35 is that its design was actually finalized in 2001 and it did not

• 35:42Speaker 06

  enter operations until 2016. And a lot happened between 2001 and 2016, including the invention of the iPhone, which by the way, has a faster processor in it than the F-35. And if you think about the F-35 over the subsequent years, there's been three technological upgrades to it. And we're now, what, we're almost in iPhone 16 season. And once you understand that, you understand why it was really important that the Pentagon thought about establishing a Silicon Valley office to start accessing this whole other technology ecosystem that is faster and generally a lot less expensive than the firms that produce technology for the military.

• 36:22Kevin Roose?

  Yeah, I remember years ago, I interviewed your former boss, Ash Carter, the former Secretary of Defense who died in 2022. And I sort of expected that he'd want to talk about like all the new fangled stuff that the Pentagon was making, you know, autonomous drones, stealth bombers. But instead, we ended up talking about procurement, which is basically how the government buys stuff, whether it's a fighter jet or an iPhone. And I remember him telling me that procurement was just unbelievably complicated and it was a huge part of what made government and the military in particular so inefficient and kind of backwards technologically. Describe how the military procures things and then what

• 37:07Kevin Roose?

  you discovered about how to maybe short circuit that process or make it more efficient.

• 37:11Speaker 06

  You know, if you're looking to buy a nuclear aircraft carrier or a nuclear submarine, you can't really go on Amazon and price shop for that.

• 37:18Casey Newton?

  And I learned that the hard way, by the way.

• 37:22Speaker 06

  Should have upped your credit, Lenny Casey. And so, you know, in those circumstances, when the government is representing the taxpayer and buying one large military system, a multi-billion dollar system, from one vendor, it's really important that the taxpayer not be overcharged. And so, the Pentagon has developed a really elaborate system of procurement to ensure that it can control how production happens, the cost of individual items. And that works okay if you're in a situation where you have the government and one firm that makes one thing. It doesn't make any sense, though, if you're buying goods that multiple firms make or that are just available on the consumer market. And so, one of the challenges we had out here in Silicon Valley when we first did

• 38:06Speaker 06

  a Defense Innovation Union was trying to figure out how to work with startups and tech companies who, it turns out, weren't interested in working with the government. And the reason why is that the government typically buys defense technology through something called the federal acquisition rules, which is a little bit like the Old Testament. It's this dictionary-sized book of regulations, letting a contract takes 18 to 24 months. If you're a startup, your investors tell you not to go down that path for a couple reasons. One, you're not going to make enough money before your next valuation. You're going to have to wait too long. You're going to go out of business before the government actually closes the sale. And two, even if you get that first contract, it's totally possible another firm with better lobbyists is going to take it right back away from you.

• 38:50Speaker 06

  So, at Defense Innovation Union, we had to figure out how to solve that paradox.

• 38:55Kevin Roose?

  Part of what I found interesting about your book was just the sort of accounts that you gave of these sort of clever loopholes that you and your team found around some of the bureaucratic slowness at the Pentagon. And in particular, this loophole that allowed you to purchase technology much, much more quickly that one of your staffers found. Tell that story, and maybe that'll help people understand kind of the systems that you were up against.

• 39:23Speaker 06

  It's an amazing story, but we knew when we arrived in Silicon Valley that we would fail unless we figure out a different way to contract with firms. And our first week in the office, this 29-year-old staff member named Lauren Daly, the daughter, actually, of a tank commander, whose way of serving was to become a civilian in the Pentagon and work on acquisition, happened to be up, because she's a total acquisition nerd, late at night, reading the just-released National Defense Authorization Act, which is another dictionary-sized compendium of law that comes out every year. And she was flipping through it, trying to find new provisions in law that might change how acquisition worked.

• 40:03Speaker 06

  And sure enough, in Section 815 of the law, she found a single sentence that she realized somebody had placed there that changed everything. And that single sentence would allow us to use a completely different kind of contracting mechanisms called other transaction authorities that were actually first invented during the space race to allow NASA during the Apollo era to contract with mom-and-pop suppliers. And so she realized that this provision would allow us not only to use OTAs to buy technology, but the really important part is that if it worked, if it was successful in the pilot, we could immediately go to buy it at scale, to buy it in production. We didn't have to recompete it. There would be no pause, no 18-month pause, between demonstrating your technology and

• 40:47Speaker 06

  having the department buy it. And when Lauren brought this to our attention, we thought, oh, boy, this really is a game-changer. So we flew Lauren to Washington. We had her meet with the head of acquisition policy of the Department of Defense. And in literally three weeks, we changed 60 years of Pentagon policy to create a whole new way to buy technology that, to this day, has been used to purchase $70 billion of technology for the Department of Defense.

• 41:11Kevin Roose?

  You just said that the reason that Silicon Valley tech companies, some of them didn't want to work with the military, is because of this sort of arcane and complicated procurement process. But there are also real moral objections among a lot of tech companies and tech workers. In 2018, Google employees famously objected to something called Project Maven, which was a project the company had planned with the Pentagon that would have used their AI image recognition software to improve weapons and things like that. And there have been just a lot of objections over the years from Silicon Valley to working with the military, to being defense contractors. Why do you think that was? And do you think that's changed at all?

• 41:56Speaker 06

  You know, it's completely understandable. You know, so few Americans serve in uniform. Most of us don't actually know somebody who's in the military. And it's really easy here in Silicon Valley, you know, where the weather's great. Sure, you read headlines in the news, but you know, the military is not something that you encounter in your daily life. And you know, you join a tech company to make the world better, to develop products that are going to help people. You don't join a tech company assuming that you're going to be making the world a more lethal place. But at the same time, you know, Project Maven was actually something that I got a chance to work on and Defense Innovation Unit and a whole group of people led.

• 42:34Kevin Roose?

  And tell us about, remind us what Project Maven was.

• 42:36Speaker 06

  So Project Maven was an attempt to use artificial intelligence and machine learning to take a whole bunch of footage, surveillance footage that was being captured in places like Iraq and Afghanistan and other military missions, and to use machine learning to to label what was found in this footage. So it was a tool to essentially automate work that otherwise would have taken human analysts hundreds of hours to do. And it was used primarily for intelligence and reconnaissance and force protection. So Project Maven, this is another misconception. You know, when you talk about military systems, there's really a lot of unpacking you have to do. The headline that got Project Maven in trouble

• 43:18Speaker 06

  said, you know, "Google working on secret drone project." And it made it look as if Google was partnering with Defense Innovation Unit, the Department of Defense, to build offensive weapons to support the US drone campaign. And that's not all what was happening. What was happening is Google was building tools that would help our analysts process the incredible amount of data flowing off many different observation platforms in the military.

• 43:43Kevin Roose?

  Right, but Google employees objected to this. They made a big case that Google should not participate in Project Maven. And eventually, the company pulled out of the project. But speaking of Project Maven, I was curious because there was some reporting from Bloomberg this year that showed that the military has actually used Project Maven's technology as recently as February to identify targets for airstrikes in the Middle East. So isn't that exactly what the Google employees who were protesting Project Maven back when you were working on it at the Defense Department, isn't that exactly what they were scared would happen?

• 44:17Speaker 06

  Well, Project Maven, when Google was involved, was very much a pilot R and D project. And it's since transitioned, actually, into much more of an operational phase. And it's been used in a number of places. In fact, it's actually been used in Ukraine as well to help the US identify military targets in Ukraine. And so this, again, speaks to, I think, a sea change in Silicon Valley since that original protest of 3,000 Google employees over Project Maven, where the world has changed a lot, and not for the better. You know, we have a land war going on in Europe on the border of NATO. And in fact, that war, the Ukraine conflict, has mobilized a lot of people in Silicon Valley to want to try and help support Ukraine's quest to defend its territory.

• 45:01Speaker 06

  And so I think we're in a very different time and moment right now, as people watching the news realize that our security is actually quite a bit more fragile than we might have first imagined.

• 45:12Kevin Roose?

  I mean, I think one reaction that our listeners may have to this is they are very concerned about the use of AI and other technologies by the military. And I also hear from a lot of people at the tech companies who are really concerned about some of these contracts. I remember during the Project Maven controversy, talking with people at Google who were part of the protest movement. And some things that they would say to me are like, well, you know, if I wanted to work for a defense contractor, I would have gone to go work for Lockheed Martin or Raytheon. I'm curious, like, what moral argument would you make to someone who maybe says, "Look, I did not sign up to make weapons of war. I am an AI engineer. I work on large

• 45:55Kevin Roose?

  language models, or I work on image recognition stuff." What do you tell that person if you're working at the DIU, trying to persuade them that it's okay to sell or license that technology to the Pentagon?

• 46:09Speaker 06

  I think you tell them that we're at an extraordinary moment in the history of war, where everything is changing. And I'll just give you a couple of data points. A few weeks ago, the United States asked the Ukrainian military to pull back from the front lines all 31 of the M1A1 Abrams tanks that we had deployed to Ukraine to allow their military to better repel a Russian invasion. These are the most advanced tanks, not only in our inventory, but in the inventory of any one of our allies. And they were getting whacked by $2,000 Russian kamikaze drones, $2,000 drones killing tanks. What does that tell me? That tells me that a century of mechanized warfare that began in the First

• 46:51Speaker 06

  World War is over. And if you're building an army that's full of tanks, you now are the emperor with fewer clothes, anyway. And I'll give you one other, you know, a couple of other data points. Hamas, you know, has kicked off the largest ground war in the Middle East because of its attack in Israel on the 7th of October since the 1973 Arab-Israeli war, threatening to destabilize the Middle East into a wider war. How did they do it? They did it by taking quadcopters and using them to drop grenades on the generators powering the Israeli border towers. That's what allowed the fighters, you know, to pour over the border. Another data point, Houthis rebels in Yemen right now are holding hostage

• 47:34Speaker 06

  12% of global shipping in the Red Sea because they're using autonomous sea drones, missiles, and loitering munitions to harass shipping. And so, we're at this moment where the arsenal of democracy that we have, this incredibly forceful military is full of things like aircraft carriers and tanks, are wielding weapons that are no longer as effective as they were 10 years ago. And if our military and adversary doesn't catch up quick, we may be in a situation where we don't have the advantage we once did. And we have to think very differently about our security if that's the case.

• 48:10Kevin Roose?

  I mean, it sounds like you're kind of saying that the way to stop a bad guy with an AI drone is a good guy with an AI drone. Am I hearing you right that you're saying that we just, we have to have such overwhelmingly powerful lethal technology in our military that other countries won't mess with us?

• 48:28Speaker 06

  I totally hear you and frankly, hear all the people that, you know, years ago were affiliated with the Stop Killer Robots movement. I mean, these weapons are awful things. They do awful things to human beings. But, you know, at the same time, there's a deep literature on something called strategic stability that comes out of the Cold War. And, you know, part of that literature focuses on the proliferation of nuclear weapons. And the fact that actually the proliferation of nuclear weapons has actually reduced great power conflict in the world because nobody actually wants to get in a nuclear exchange. Now, would it be a good idea for everybody in the world to have their own nuclear weapon? Probably not. So all these things have limits. But that's an illustration of how strategic stability, in other words, a balance of power,

• 49:13Speaker 06

  can actually reduce the chance of conflict in the first place.

• 49:16Kevin Roose?

  I'm curious what you make of the Stop Killer Robots movement. There was a petition or an open letter that went around years ago that was signed by a bunch of leaders in AI, including Elon Musk and Demis Hassabis of Google DeepMind. They all pledged not to develop autonomous weapons. Do you think that was a good pledge or do you support autonomous weapons?

• 49:39Speaker 06

  I mean, I think autonomous weapons are now kind of a reality in the world. I mean, we're seeing this on the front lines of Ukraine. And, you know, if you're not willing to fight with autonomous weapons, then you're going to lose.

• 49:53Casey Newton?

  So there's this former OpenAI employee, Leopold Ashenbrenno, who recently released a long manifesto called Situational Awareness. And one of the predictions that he makes is that by about 2027, the US government would recognize that super intelligent AI was such a threat to the world order that AGI, sort of artificial general intelligence, would become functionally a project of the national security state, something like an AGI Manhattan Project. There's other speculation out there that maybe at some point the government would have to nationalize an OpenAI or an Anthropic. Are you hearing any of these whispers yet? Like, are people starting to game this out at all?

• 50:36Speaker 06

  Well, I haven't, I confess, I haven't made it all through each 155 pages of that long manifesto.

• 50:43Casey Newton?

  It was very long. You could summarize it with ChatGPT.

• 50:45Speaker 06

  Fantastic. But these are important things to think about because it, you know, it could be that in certain kinds of conflicts, whoever has the best AI wins. And if that's the case, and if AI is getting exponentially more and powerful, then, you know, to take things back to the iPhone and the F35, it's going to be really important that you have the kind of AI of the iPhone variety. You have the AI that that's new every year. You don't have the F35 with the processor that was baked in in 2001, and you're only taking off on a runway in 2016. So I do think it's very important for folks to be focused on AI. Where this all goes, though, is a lot of speculation.

• 51:25Casey Newton?

  I mean, if you had to bet in 10 years, do you think that the AI companies will also be private? Or do you think the government will have stepped in and gotten way more interested and maybe taken one of them over?

• 51:35Speaker 06

  Well, I'd make the observation that, you know, we all watched Oppenheimer, especially employees at AI firms. They seem to love that film. And, you know, nuclear technology, it's what national security strategists would call a point technology. It's sort of zero to one. Either you have it or you don't. And AI is not going to end up being a point technology. It's a very broadly diffuse technology that's going to be applied not only in weapon systems, but in institutions. It's going to be broadly diffused around the economy. And for that reason, I don't think, or it's less likely anyway, that we're going to end up a situation where somebody has the bomb and somebody doesn't. I think the gradations are going to be smoother and not quite as sharp.

• 52:19Kevin Roose?

  Part of what we've seen in other industries as technology sort of moves in and modernizes things is that often things become cheaper. You know, it's cheaper to do things using the latest technology than it is to do using outdated technology. Do you think some of the work that you've done at DIU trying to modernize how the Pentagon works is going to result in smaller defense budgets being necessary going forward? Is the $2 trillion or so that the DOD has budgeted for? This year, could that be 1 trillion or half a trillion in the coming years because of some of these modernizations?

• 52:53Speaker 06

  You're you're giving us a raise, Kevin. I think it's more like 800 800 billion.

• 52:57Kevin Roose?

  Well, I'm sorry. I got that answer from Google's AI overview, which

• 53:00Speaker 06

  There you go.

• 53:01Kevin Roose?

  Also told me to eat rocks and put glue on my pizza.

• 53:04Speaker 06

  We should. We should get the Secretary of Defense to try to. He'd like that answer if you have that large of a of a budget. You know, it's certainly true that for a lot less money now, you can have a really destructive effect on the world as as drone pilots in Ukraine and elsewhere in the world are showing. I think it's also true that the US military has a whole bunch of legacy weapon systems that unfortunately are, you know, kind of like museum relics, right? I mean, if our most advanced tank can be destroyed by a drone, it might be time to retire our tank fleet. If our aircraft carriers cannot be defended against the hypersonic missile attack, it's probably not a good idea to sail one of our aircraft carriers anywhere near an advanced adversary.

• 53:45Speaker 06

  So I think it is a an opportune moment to really look at what we are spending our money on at the Defense Department and remember the goal of our our nation's founders, which is to spend what we need to on defense and not a penny more.

• 53:58Casey Newton?

  So I hear you saying that it's very important for the military to be prepared technologically for the the world we're in. And that means working with Silicon Valley. But is there anything like more specific that you want to share that you think that either side needs to be doing here? Or something specific that you want to see out of that collaboration?

• 54:17Speaker 06

  Well, I think, you know, one of the main goals of Defense Innovation Unit was literally to get the two groups talking. You know, before Defense Innovation Unit was founded, a Secretary of Defense hadn't been to Silicon Valley in 20 years. I mean, that's almost a generation. So Silicon Valley invents the mobile phone. It invents cloud computing. It invents AI. And nobody from the Defense Department bothers to even come and visit. And that's a problem. And so just bringing the two sides into conversation is itself, I think, a great achievement.

• 54:50Kevin Roose?

  Well, Chris, thanks so much for coming on. Really appreciate the conversation. And the book, which comes out on July 9th, is called Unit X: How the Pentagon and Silicon Valley are transforming the future of war.

• 55:01Speaker 06

  Thank you.

• 55:02Casey Newton?

  Thanks.

• 55:05Kevin Roose?

  When we come back, play another round of Hat GPT.

• 55:21Speaker 00

  Support for this podcast comes from Dell Technologies. It's time to put AI to work. Is your business ready? The challenge isn't getting started with AI. It's scaling beyond pilots while maintaining control of your data, governance, infrastructure, and costs. Dell AI Factory provides the integrated foundation you need to scale AI securely and efficiently with one trusted partner. Dell, AI on your terms. From idea to impact. Learn more at dell.com/yourwaytoAI. That's dell.com/yourwaytoAI.

• 55:56Speaker 07

  Today's AI builders need infrastructure that scales with their ambitions. Iren is a vertically integrated AI cloud platform, delivering data centers, compute, and software for AI training and inference, underpinned by its expansive portfolio of land and grid-connected power in renewable-rich regions globally. Iren's AI Cloud gives direct access to Nvidia GPUs, scalable from hundreds to thousands. For builders, that means direct access to AI compute, rapid scaling, and predictable pricing. Iren, AI Cloud for the next big thing.

• 56:29Speaker 05

  AI spend is growing fast, but most businesses still can't see exactly where that money is going. Ramp gives businesses visibility and control across AI providers with realtime guardrails by team and model, helping ensure spend goes to the highest ROI areas. Stop guessing where your AI spend is going. Get started with Ramp. And if you're new to Ramp, you can get $250 off at ramp.com/hardfork. Go to ramp.com/hardfork. Ramp.com/hardfork.

• 57:01Casey Newton?

  All right, Kevin. Well, it's time once again for HatGPT.

• 57:10Kevin Roose?

  This, of course, is our favorite game. It's where we draw news stories from the week out of a hat. And we talk about them until one of us gets sick of hearing the other one talk and says, "Stop generating."

• 57:19Casey Newton?

  That's right. Now, normally, we pull slips of paper out of a hat. But due to our remote setup today, I will instead be pulling virtual slips of paper out of a laptop. But for those following along at YouTube, you will still see that I do have one of the hat GPT hats here, and I will be using it for a comic effect throughout the segment.

• 57:38Kevin Roose?

  Will you put it on, actually? Because now, if we don't need it to draw slips out of, you might as well be wearing it.

• 57:42Casey Newton?

  Yeah, I might as well be wearing it.

• 57:43Kevin Roose?

  Yeah, it looks so good.

• 57:45Casey Newton?

  Thank you so much. And thank you, once again, to the listener who made this for us. You're a true fan.

• 57:51Kevin Roose?

  So good.

• 57:52Casey Newton?

  Perfect. All right, Kevin, let me draw the first slip out of the laptop. Ilya Sutskever has a new plan for Safe Superintelligence. Ilya Sutskever is, of course, the OpenAI co-founder, who was part of the coup against Sam Altman last year. And Bloomberg reports that he is now introducing his next project, a venture called Safe Superintelligence, which aims to create a safe, powerful artificial intelligence system within a pure resource organization that has no near-term intention of selling AI products or services. Kevin, what do you make of this?

• 58:28Kevin Roose?

  Well, it's very interesting on a number of levels, right? In some sense, this is kind of a mirror image of what happened several years ago when a bunch of safety-minded people left OpenAI after disagreeing with Sam Altman and started an AI-safety-focused research company. That, of course, was anthropic. And so this is sort of the newest twist in this whole saga is that Ilya Sutskever, who was very concerned about safety and how to make superintelligence that was smarter than humans, but also not evil and not going to destroy us, who has done something very similar. But I have to say, I don't quite get it. I mean, he's not saying much about the project.

• 59:10Kevin Roose?

  But part of the reason that these companies sell these AI products and services is to get the money to buy all the expensive equipment that you need to train these giant models.

• 59:20Casey Newton?

  Right.

• 59:20Kevin Roose?

  And so I just don't know, like, if you don't have any intention of selling this stuff before it becomes AGI, how are you paying for the AGI? Do you have a sense of that?

• 59:32Casey Newton?

  No, I don't. I mean, Daniel Gross, who is one of Ilya's co-founders here, has basically said, "Don't worry about fundraising. Like, we are going to be able to fundraise as much as we need for this." So I guess we will see. But yeah, it does feel a bit strange to have someone like Ilya saying he's going to build this with, totally without a commercial motive, in part because he said it before, right? Like, this is what is so funny about this, is it truly just is a case where the circle of life keeps repeating, where a small band of people get together and they say, "We want to build a very powerful AI system, and we're going to do it very safely." And then bit by bit, they realize, "Well, actually, we don't think that it's being built out safely. We're going to form a breakaway faction." So if you're playing a lot at home, I believe this is the second breakaway faction to break

• 1:00:16Casey Newton?

  away from OpenAI after Anthropic, and I look forward to Ilya quitting this company eventually to start a newer, even more safe company somewhere else.

• 1:00:25Kevin Roose?

  The really, really safe superintelligence company.

• 1:00:27Casey Newton?

  Yeah, his next company, you've never seen safety like this. They wear helmets everywhere in the office, and they just have keyboards.

• 1:00:33Kevin Roose?

  All right, stop generating.

• 1:00:35Casey Newton?

  All right, pick one out of that, Kevin.

• 1:00:37Kevin Roose?

  All right, "Five men convicted of operating JetFlix, one of the largest illegal streaming sites in the US." This is from Variety. "JetFlix was a sort of pirated streaming service that charged $9.99 a month while claiming to host more than 183,000 TV episodes, which is more than the combined catalogs of Netflix, Hulu, Vudu, and Amazon Prime video."

• 1:01:03Casey Newton?

  Ooh, that sounds great. I'm going to open an account. What a deal.

• 1:01:09Kevin Roose?

  So the Justice Department, you know, says this was all illegal, and the five men who were charged with operating it were convicted by a federal jury in Las Vegas. According to the court documents and the evidence that was presented at the trial, this group of five men were basically scraping piracy services for illegal episodes of TV and then hosting them on their own thing. It does not appear to have been a particularly sophisticated scam. It's just, what if we did this for a while and charged people money and then got caught?

• 1:01:40Casey Newton?

  Well, I think this is very sad because here finally you have some people who are willing to stand up and fight inflation. And what does the government do? They come in and they say, knock it off. You know, I will say though, Kevin, I think these, I can actually point to the mistake that these guys made.

• 1:01:54Kevin Roose?

  What's that?

• 1:01:55Casey Newton?

  So instead of scraping these 183,000 TV episodes and selling them for $9.99 a month, what they should have done was feed them all into a large language model and then you can sell them to people for $20 a month. So the next time, when these guys get out of prison, I hope they get in touch with me because I have a new business idea for them.

• 1:02:13Kevin Roose?

  All right, stop generating.

• 1:02:16Casey Newton?

  All right. Here's a story called 260 McNuggets. McDonald's Ends AI Drive-Thru Tests Amid Errors. This is from the New York Times. After a number of embarrassing videos showing customers fighting with its AI-powered drive-thru technology, McDonald's announced it was ending its three-year partnership with IBM. In one TikTok video, friends repeatedly tell the AI assistant to stop as it added hundreds of chicken McNuggets to their order. Other videos show the drive-thru technology adding nine iced teas to an order, refusing to add a Mountain Dew, and adding unrequested bacon to ice cream. Kevin, what the heck is going on?

• 1:02:56Kevin Roose?

  Well, as a fan of bacon ice cream, I should say, I want to, I want to get to one of these McDonald's before they take this thing down.

• 1:03:02Casey Newton?

  Me too.

• 1:03:03Kevin Roose?

  Did you see any of these, any of these videos or any of these?

• 1:03:05Casey Newton?

  I haven't, did you?

• 1:03:06Kevin Roose?

  No, but we should watch one of them together. Let's watch one of them.

• 1:03:11Speaker 08

  Stop.

• 1:03:12Kevin Roose?

  The caption is, "The McDonald's robot is wild." And it shows their screen at the thing where it has, it is like just tallying up McNuggets and starts charging them more than $200.

• 1:03:27Casey Newton?

  Here's my question. Why is everyone just rushing to assume that the AI is wrong here? Maybe the AI knows what these gals need. Because Kevin, here's the thing, when superintelligence arrives, we're going to think that we're smarter than it, but it's going to be smart, you know, so there's going to be a period of adjustment as we sort of, you know, get used to having our new AI master. Have you been to a drive-thru that used AI to take your order yet?

• 1:03:53Casey Newton?

  No, I mean, I don't even really understa-

• 1:03:56Casey Newton?

  What was the AI here? Was this like an Alexa thing where I said, you know, McDonald's, add 10 McNuggets? Or like, what was actually happening?

• 1:04:02Kevin Roose?

  No, so this was a partnership that McDonald's struck with IBM. And basically, this was like technology that went inside the little, you know, menu things that have the microphone and the speaker in them. And so instead of having a, you know, a human say, what would you like? It would just say, what would you like? And then you said it and they would recognize it and, and put it into the system. So you could sort of eliminate that part of, of the labor of the drive-through.

• 1:04:27Casey Newton?

  Got it. Well, look, I for one am very glad this happened because for so long now, I've wondered, what does IBM do? And I have no idea. And now, if it ever comes up again, I'll say, oh, that's the company that made the McDonald's stop working.

• 1:04:41Kevin Roose?

  We should say it's not just McDonald's. A bunch of other companies are starting to use this technology. I actually think this is probably, you know, inevitable. This technology will get better. They will iron out some of the kinks. But I, I think there will probably still need to be a human in the loop on this one.

• 1:04:56Casey Newton?

  All right, stop generating.

• 1:04:57Kevin Roose?

  Okay.

• 1:04:58Casey Newton?

  Kevin, let's talk about what happened when 20 comedians got AI to write their routines. This is in the MIT Technology Review. "Google DeepMind researchers found that although popular AI models from OpenAI and Google were effective at simple tasks like structuring a monologue or producing a rough first draft, they struggled to produce material that was original, stimulating, or crucially funny." And I'd like to read you an example LLM joke, Kevin.

• 1:05:22Kevin Roose?

  Please.

• 1:05:23Casey Newton?

  "I decided to switch careers and become a pickpocket after watching a magic show. Little did I know, the only thing disappearing would be my reputation."

• 1:05:32Kevin Roose?

  Waka, waka, waka.

• 1:05:34Casey Newton?

  Hey, I got a laugh out of you. Kevin, what do you, what, what do you make of this? Are you surprised that AI isn't funnier?

• 1:05:40Kevin Roose?

  No, but this is interesting. Like, it's like, this has been something that, you know, critics of large language models have been saying for years. It's like, well, it can't, it can't tell a joke. And, you know, I should say, I've had funny experiences with large language models, but never after like asking them to tell me a joke.

• 1:05:57Casey Newton?

  Yeah, like, remember when you said to Cindy, "Take my wife, please."

• 1:06:04Kevin Roose?

  I get no respect, I tell ya. No, but this is interesting because this was a study that was actually done by researchers at Google DeepMind. And basically, it appears that they had a group of comedians try writing some jokes with their language models. And in the abstract, it says that most of the participants in this study felt that the large language models did not succeed as a creativity support tool by producing bland and biased comedy tropes, which they describe in this paper as being akin to cruise ship comedy material from the 1950s, but a bit less racist. So they were not impressed, these comedians, by these language models' ability to tell jokes. You're an amateur comedian.

• 1:06:49Kevin Roose?

  Have you ever used AI to come up with jokes?

• 1:06:52Casey Newton?

  No, I haven't. And I have to say, like, I think I understand the technological reason why these things aren't funny, Kevin, which is that comedy is very up to the minute, right? For something to be funny, it's typically something that is on the edge of what is currently thought to be socially acceptable. And what is socially acceptable, or like, what is surprising within a social context? That just changes all the time. And these models, they are trained on decades and decades and decades of text, and they just don't have any way of figuring out, well, what would be a really fresh thing to say? So maybe they'll get there eventually, but as they're built right now, I'm truly not surprised that they're not funny.

• 1:07:34Kevin Roose?

  All right, stop generating. Next one. Waymo ditches the waitlist and opens up its robo-taxis to everyone in San Francisco. This is from The Verge. Since 2022, Waymo has made its rides in its robo-taxi service available only to people who are approved off of a waitlist. But as of this week, they are opening it up to anyone who wants to ride in San Francisco. Casey, what do you make of this?

• 1:08:00Casey Newton?

  Well, I am excited that more people are going to get to try this. This is, as you've noted, Kevin, become kind of the newest tourist attraction in San Francisco, is when you come here, you see if you can find- Find somebody to give you a ride on one of these self-driving cars. And now everyone is just going to be able to come here and download the app and use it immediately. I have to say, I am scared about what this is going to mean for the wait times on Waymo. I've been taking Waymo more lately and it often will take 12 or 15 or 20 minutes to get a car. And now that everyone can download the app, I'm not expecting those wait times to go down.

• 1:08:38Kevin Roose?

  Yeah, I hope they are also simultaneously adding more cars to the Waymo network because this is going to be very popular.

• 1:08:44Casey Newton?

  You're saying they need Waymo cars?

• 1:08:47Kevin Roose?

  They do. I'm worried about the wait times, but I'm also worried about the condition of these cars. Because I've noticed in my last few rides, they're a little dirtier.

• 1:08:57Casey Newton?

  Oh, wait, really?

• 1:08:58Kevin Roose?

  Yeah, I mean, they're still pretty clean, but you know, I did see like a takeout container in one the other day.

• 1:09:05Casey Newton?

  Oh, my god.

• 1:09:06Kevin Roose?

  And I just, I want to know how they plan to keep these things from becoming filled with people's crap.

• 1:09:12Casey Newton?

  All right, stop generating.

• 1:09:14Kevin Roose?

  All right, last one. This one comes from The Verge. TikTok's AI tool accidentally let you put Hitler's words in a paid actor's mouth. TikTok mistakenly posted a link to an internal version of an AI digital avatar tool that apparently had zero guardrails. This was a tool that was supposed to let businesses generate ads using AI with paid actors, using this AI voice dubbing thing that would make the actors repeat your, whatever you wanted to have them say, endorse your product or whatever. But very quickly, people found out that you could use this tool to repeat excerpts of Mein Kampf, Bin Laden's letter to America. It told people to drink bleach and vote on the wrong day.

• 1:09:59Casey Newton?

  And that was its recipe for a happy pride celebration. Listen, obviously, this is a very sort of silly story. It sounds like everything involved here was a mistake. And I think if you're making some sort of digital AI tool that is meant to generate ads, you do want to put safeguards around that because, you know, otherwise people will exploit it. That said, Kevin, I do think people need to start getting comfortable with the fact that people are just going to be using these AI creation tools to do a bunch of kooky and crazy stuff.

• 1:10:33Kevin Roose?

  Like what?

• 1:10:35Casey Newton?

  Like, people are, you know, in the same way that people use Photoshop to make nudity or offensive images, and we don't storm the gates of Adobe saying, "Shut down Photoshop." The same thing is going to happen with these digital AI tools. And while I do think that there are some notable differences, and it is sort of, you know, varies on a case-by-case basis, and if you're making a tool for creating ads, it feels different, there are just going to be a lot of digital tools like this that use AI to make stuff, and other people are going to use it to make offensive stuff. And when they do, we should hold the people accountable, perhaps more than we hold the tool accountable.

• 1:11:10Kevin Roose?

  Yeah, I agree with that. And I also think, like, this sort of product is not super worrisome to me. I mean, obviously, it should not be reading excerpts from my comp. They're, obviously, they did not mean to release this. I assume that when they do, you know, fix it, it will be much better. But this is not like a thing that is creating deep fakes of people without their consent. This is a thing where if you have a brand, you can choose from a variety of stock avatars that are created from people who actually get paid to have their likenesses used commercially. So, you know, the specific details of this one don't bother me that much, but it does open up some new licensing opportunities for

• 1:11:52Kevin Roose?

  us. Like, we could have an AI set of avatars that could be out there advertising crypto tokens or whatever. And I, for one, am excited to see how people use that.

• 1:12:00Casey Newton?

  Oh, man. Well, and if TikTok weren't banned, we could probably make a lot of money that way. But instead, you know, we're out of luck.

• 1:12:06Kevin Roose?

  Yeah, get it while it's good. All right.

• 1:12:09Casey Newton?

  Close up the hat.

• 1:12:56Speaker 07

  Today's AI builders need infrastructure that scales with their ambitions. Irene is a vertically integrated AI cloud platform, delivering data centers, compute and software for AI training and inference, underpinned by its expansive portfolio of land and grid-connected power in renewable-rich regions globally. Irene's AI cloud gives direct access to Nvidia GPUs, scalable from hundreds to thousands. For builders, that means direct access to AI compute, rapid scaling, and predictable pricing. Irene, AI cloud for the next big thing.

• 1:13:29Speaker 05

  AI spend is growing fast, but most businesses still can't see exactly where that money is going. Ramp gives businesses visibility and control across AI providers with real-time guardrails by team and model, helping ensure spend goes to the highest ROI areas. Stop guessing where your AI spend is going. Get started with Ramp. And if you're new to Ramp, you can get $250 off at ramp.com/hardfork. Go to ramp.com/hardfork. Ramp.com/hardfork.

• 1:14:03Kevin Roose?

  Hard Fork is produced by Rachel Cohn and Whitney Jones. We're edited this week by Larissa Anderson. We're fact-checked by Caitlin Love. Today's show was engineered by Corey Schreppel. Original music by Alicia Baetoupe, Rowan Nemisto, and Dan Powell. Our audience editor is Nell Galogly. Video production by Ryan Manning, Sawyer Roque, and Dylan Bergerson. You can watch this full episode on YouTube at youtube.com/hardfork. You can see Casey's cool hat. Special thanks to Paula Schuman, Quiwing Tam, Caitlin Presti, and Jeffrey Miranda. As always, you can email us at [email protected].

• 1:15:02Speaker 05

  You can't undo everything in life, but you can undo prediabetes. More than 2 in 5 adults in the US have prediabetes. Most don't know they have it. But with early diagnosis and action, you can delay or even prevent type 2 diabetes. Visit doihaveprediabetes.org to take the 1-minute risk test and start undoing the things you can. Brought to you by the Ad Council and the Centers for Disease Control and Prevention.