Pod Insights The Vergecast How to train your data About this episode Training data is the raw material of the AI industry. Claude, ChatGPT, Gemini, and the rest are built on top of oceans of stuff. What is that stuff? Books. Blog posts. YouTube videos. Reddit comments. All of it and more, in virtually incomprehensible quantities. Alex Reisner, a staff writer at The Atlantic who has been investigating training data, explains how AI companies get all this data, why they'd really prefer you not know what's in it, and whether training data could ever be a fair trade.
Further reading:
Apple raises prices on Macs, iPads, and more by hundreds of dollars | The Verge
Disney agrees to pay $50 million to YouTube TV and DirecTV subscribers | The Verge
Two handlebars are better than one, right? | The Verge
At Least 15 Million YouTube Videos Have Been Snatched by AI Companies
The Hypocrisy at the Heart of the AI Industry
The Millions of Songs Mashed Into AI-Generated Music
Common Crawl Is Doing the AI Industry’s Dirty Work
Subscribe to The Verge for unlimited access to theverge.com, subscriber-exclusive newsletters, and our ad-free podcast feed.
We love hearing from you! Email your questions and thoughts to [email protected] or call us at 866-VERGE11.
Learn more about your ad choices. Visit podcastchoices.com/adchoices
Transcript 5,058 words · 10 speakers
0:02 Speaker 00 Hello and welcome to The Vergecast, the flagship podcast of music that sounds eerily but not exactly like other music. I'm your friend David Pierce, and today on the show, we're talking about training data. It's the raw materials of everything that is AI. And we probably don't understand it or talk about it enough. I'm talking to Alex Reisner, who's a staff writer at The Atlantic. And over the last couple of years, he has spent a lot of time investigating training data and how all of these books and all of these articles and all of these YouTube videos and all of these songs are compiled into these gigantic data sets that AI companies then use to form the basis of their models. I think the way that these models are created and the sources of these data
0:44 Speaker 00 has a lot to do with the way that we feel about AI, particularly generative AI as a creative expression. And understanding how this data works, where it comes from and how it gets used, I think is really important. So we're gonna have Alex on, we're gonna dig into it. I'm very excited about it. But first, here's everything else happening on The Verge today. This is 90 Seconds on The Verge for Thursday, June 25, 2026. Apple just raised the prices of a huge number of its most important products. iPads and MacBooks in particular are now up anywhere from $100 to $500. And even like the Apple TV is way up. It's now $200, roughly the price of 900 Amazon FireSticks. We knew this was coming, but this is still a big moment.
1:25 Speaker 00 Apple is as well-managed a supply chain company as anybody and exists at really high margins. If it can't keep prices down in this era of AI-driven shortages of memory and storage, nobody can. And I suspect this is going to get even worse. My advice from yesterday still very much holds, which is Prime Day is not over. Go get your deals while you still can. And here's a way to help you pay for all of those more expensive gadgets. Go get some money from Disney. If you have YouTube TV or DirecTV stream, you might be eligible for a cash payout from a new $50 million settlement. The case started in 2022, and the argument was essentially that because Disney owns ESPN and Hulu, which are both powerful streaming services in their own right, Disney was able to drive up the prices of all of its rivals to unnecessary
2:10 Speaker 00 Heights. Live TV is lucrative and competitive and is going to keep being messy in this way. This will not end things, but hey, go get some of that $50 million. And finally, a quick PSA to advertisers everywhere. Yes, I get it. The whole idea of using AI creative and AI targeting to put your AI ads in front of AI people is exciting, I guess, but maybe check your ads to make sure that you're not, I don't know, advertising a bike with two sets of handlebars like REI did this week? That may not go over so well with people. Just a little free advice from me to you. You can read more about all of this at theverge.com. That's 90 seconds on the verge for Thursday, June 25.
2:46 Speaker 01 Real innovation is powered by real people. This is Amex Corporate. Trusted experts, smarter technology, 24/7 dedicated support from day one to answer questions, help set up policies and approvals, and consult on spend capacity as your business needs evolve. Plus, on-demand resources help level up operations at every stage. That's the trusted service you've come to expect. Built to simplify, designed to scale. This is Amex Corporate.
3:18 Speaker 02 This episode is brought to you by Michelob Ultra. Good times mean competitions between you and your friends, like seeing who can throw some pigskin the farthest or rush the field the fastest. Winner gets an ultra. Because it's not just a superior light beer, it's also giving fans superior access to their favorite sporting events. Superior is worth playing for. Visit michelobultra.com/superioraccess to enter. Enjoy responsibly. Copyright 2026 Anheuser-Busch InBev Michelob Ultra Beer, St. Louis, Missouri.
3:47 Speaker 00 All right, let's talk training data. Alex Reisner from The Atlantic is here. Hi, Alex.
3:50 Speaker 03 Hey, David. Thank you for having me.
3:53 Speaker 00 Very excited to talk to you. I think we have talked a lot on this show about why people feel the way that they feel about AI and the sort of instinctive reactions to the whole idea of AI. And a theory that I have is that a lot of it is about training data. And I just want to talk about that. You've done a lot of work on this stuff and you've been investigating how AI models get trained for a long time now. But I want to just lay a little bit of groundwork here. And just to ask the very obvious question upfront, why does it matter what data is used to train these products? What difference does it make what's inside of these models?
4:32 Speaker 03 I think it is potentially the most important aspect of a model is what it's trained on. I mean, if you take a model and you train it on, let's say it generates music and you train it on 1950s jazz, that model will be very good at generating music that sounds a lot like 1950s jazz. If you train it on recent hip hop, it's going to generate music that sounds like recent hip hop. You know, these models have names like ChatGPT and Claude, but I think you could make an argument that the right name for a model is actually the description of the data it was trained on, because that is a description of its capabilities. That's what it can output.
5:14 Speaker 03 And so I think that the training data is really, it's really fundamental to the model, maybe more than the architecture to some degree.
5:23 Speaker 00 Interesting. And I think, I mean, that kind of answers to some extent my next question, which is, why is training data such a closely guarded secret from these companies? And it seems to me that there is both a straightforward business answer and maybe a slightly more nefarious answer as to why these companies would be so secretive. But what is your read on why this is such a closely guarded secret? What data is being used to train these models?
5:51 Speaker 03 Yeah, I mean, the companies have argued that they need to keep this secret because the data that they have selected to train on is their competitive advantage, right? Like Anthropic has done a better job at selecting data than Google and OpenAI. And if they were to let that come out in a court case or be public in some way, they would lose their competitive advantage. You know, there's another pretty obvious reason, which is that they have gone about acquiring a lot of this data in ways that the people who've created the data, the authors of the books and the creators of the videos and the music would not be happy about. And in a lot of cases, they just don't. Know that their their work is being used. And when they find out they're not happy about it.
6:36 Speaker 03 And I think it's a conversation that the AI companies have tried to just avoid having.
6:42 Speaker 00 So a big part of what you've been up to recently has been sort of reverse engineering this process to like peel the model apart to figure out what data is inside of it. And it seems to me that you've had to do sort of the reverse of what all of these companies have done, which is go figure out where these mass sources of books or web articles or videos or songs are. So tell me a little bit about your process. How do you go about finding these things that are kind of otherwise closely held secrets?
7:15 Speaker 03 Yeah, my process is actually maybe not that different from the the company's process. I am a programmer by by trade. I was I worked in tech for 20 years. I built websites and apps and did some statistics work. And so I've been aware of these models for for a long time. And at a certain point, I started hanging out in the forums where AI developers were hanging out and talking about the work. And, you know, they're talking about what data they're using to train these things. And so that was a useful source. And I've also been reading a lot of their research papers. You know, selecting the data is really challenging. There's a lot of effort that they put into it. They have to come up with some notion of what is high quality data, like what do and and don't we want in the model?
8:00 Speaker 03 And they write papers about that. You know, there's they I think they want to be involved in this conversation that's happening about training data within within the industry. One thing that's been really helpful to me is that there's a open source AI development community, and they believe that the work should be done more out in the open. And I think a lot of them are doing really good and important work, and they have really interesting things to say about AI, just philosophically and socially.
8:26 Speaker 03 And so they, you know, they're pretty transparent, places like Alan AI and Luther about what they're using to train. And so that's been helpful. But then even at the companies, you know, AI, the AI world is a little bit like academia in that it is good for your career to publish papers. And the companies don't like that, but they also acknowledge that they have to let the employees publish something. And so the lawyers will go over it and, you know, tell them what they can and can't say. And over time, they've clamped down more. And so it's, the companies are revealing less through the research papers. But yeah, a lot of my research is just from reading a Google paper, you know, for example, for this last article, they said we, we trained on, you know, tens of millions of, of songs.
9:10 Speaker 00 Yeah, I remember not that long ago, Apple going through a big cultural issue with this, with its AI team, because Apple is so secretive and so reluctant to let any of its work be public, that all of its researchers were like, well, if you're not going to let us publish, we don't want to work here. And, and it did that, that push and pull seems like it has, it has kind of morphed in a bunch of different directions over time. But interesting to hear that it is definitely, it is, everybody is retrenching a little bit as this space gets hotter and hotter.
9:37 Speaker 03 Yeah, it's not 2021 anymore. The research papers read very differently now than they did a few years ago.
9:44 Speaker 00 That's really interesting. But tell me a little bit about these databases. One thing I think I had not given enough thought to until I started really reading your work is that there's a real business in, in making and maintaining these databases. Like one company wrote about Common Crawl, I think is like a thoroughly fascinating player in this. And they, they essentially, as far as I can tell, just crawl the internet and, and make it available to whoever wants it, which is weird and complicated. And we should probably come back and talk about Common Crawl sometime. But it, it seems like if, if you look around a little, these gigantic databases of books and articles and videos exist. Like, are people making these and selling them? Why do these databases exist?
10:30 Speaker 03 Mainly, well, I mean, for AI training. Why else? It's a huge, you know, it's an extremely labor-intensive process to find, you know, in one sense, they're just, you just go and download all of Library Genesis or all of Anna's archive. That's, that's one sort of naive thing you can do, but the companies realize pretty early on that they need to filter the stuff pretty carefully. So the organization you just mentioned, Common Crawl, yeah, they've been crawling the web since the late 2000s, maybe 2009 or something like that. And they just make the whole thing available.
11:11 Speaker 03 Every month, there's a new, they've scraped a few more hundred million webpages, and it's available to anyone who wants to do any kind of research with it. In fact, it's mostly AI researchers who are, who are using it. But all the early large language models were trained on Common Crawl. If you go back and read those early OpenAI papers, everyone was training on Common Crawl. And at first, and the models were terrible, because if you train a model on the, on the whole internet, it's just, you know, there's, you get all the, it says all the junk that people say on the internet, along with, along with the intelligent things. Yeah. Mostly junk. Yeah. Statistically, mostly junk. Statistically, yeah, mostly junk.
11:52 Speaker 03 And I think the early large language models were proof of that. But yeah, I think they're, they're, you know, Common Crawl would, is a nonprofit, so, you know, they would argue there's, it's not a big business. They do get a lot of money from AI companies and AI investors. But yeah, I think this, the, the topic of, of training data selection, the challenge of selecting the right data for a model is still really hard. The AI companies, I would say, are still, have a very primitive understanding of what data will make their model better.
12:31 Speaker 03 It's a area of research that I think even even they at this stage are not very good at. They they do it mainly by trial and error as far as I can, as far as I can tell. So
12:43 Speaker 00 Interesting.
12:44 Speaker 03 Again, so the reason you're asking why do these data sets exist? I think it's people trying to share what they've learned from, you know, curating data sets in different ways and and training models with them.
12:53 Speaker 00 Yeah, I mean, part of the reason I ask is, I think, one of the the things I have come to believe about the AI industry is that this shift that went from AI research being fundamentally a research thing, like if you go, if you go way back, OpenAI was basically a research organization, right? And you talk about Common Crawl, and I think its its early users were largely researchers. And it was, these things were academic things for academic purposes. And from what I understand, the the kind of rules of the road are different, right? That like, if you want to, if you want to make copies of a bunch of things for academic purposes, these things are generally considered less problematic, right? But if you then do
13:35 Speaker 00 it and become a trillion dollar company on the back of it, people are going to rightly feel differently about the way that you went about getting that information. And just the speed with which AI commercialized, all of these companies just moved so fast from we are essentially an academic thing to, oh, my God, we're making so much money, everyone's filthy rich, that it feels like they they just hoped everybody would ignore the ways in which they got this information. And so that's, that's why I'm particularly fascinated by where these data sets even come from. Because it does seem like, in many cases, like you're saying, it's not that there is some gigantic business in being the one to sell the songs to somebody. It's that stuff is being, this stuff is being compiled for
14:20 Speaker 00 other purposes. It's just that now the main purpose, because of the sheer volume of work, is AI research. All this stuff has been sort of co-opted from every other purpose to AI. And it just all feels so concentrated now. Does that feel right to you? Does that make sense?
14:37 Speaker 03 Yeah, I think that I agree with most of that. I do think that I'm not sure how much these data sets were really collected for other purposes. Common Crawl likes to talk about, I think they're one case. They probably have the strongest argument that their data could be used for other purposes. But when you go back, you know, they've been cited by over 10,000 papers. I didn't read all 10,000, but I read a lot of them and they are mostly AI. And it's early, a lot of it is stuff that people wouldn't mind as much as with generative AI. Like Common Crawl, I think without Common Crawl, you know, AI translation tools might not be as good as they are.
15:22 Speaker 03 I think it was really a huge help because they scraped webpages, the same page in multiple languages. And people were able to train translation models based on that. So that was helpful. But the thing that, you know, there is still a, what I would call a data laundering network, where the AI companies are still relying on, they'll do a collaboration with a university and they'll have the university download, you know, millions of images to train a model or download millions of articles to train a model. And the AI company can say like, well, we didn't do it. This was like an academic thing.
16:04 Speaker 03 You know, the same goes, Common Crawl is not the only nonprofit that's like doing a lot of this scraping for the AI industry. One of the data sets I reported on in the music, the article about music training data is this organization based in Europe called Lion. They have a data set of 12 million songs from YouTube. So anyway, this is like, is it academic? Like, not, not really. Like this is, you know, technically, yeah, there's universities and nonprofits, but they're all receiving money from the AI industry.
16:38 Speaker 02 This episode is brought to you by Michelob Ultra. Good times mean competitions between you and your friends. Like seeing who can throw some pigskin the farthest, or rush the field the fastest. Winner gets an Ultra. Because it's not just a superior light beer, it's also giving fans superior access to their favorite sporting events. Superior is worth playing for. Visit michelobultra.com/superioraccess to enter. Enjoy responsibly. Copyright 2026 Anheuser Busch InBev Michelob Ultra Beer, St. Louis, Missouri.
17:06 Speaker 04 Blowing ad budget on metrics that look great, till the CFO sees them? That's bull spend. And marketers are calling it out. In. Dashboard. Confessions.
17:16 Speaker 05 I remember telling my boss, "It'll be good for the brand", when leads were slow. Yeah, it, it wasn't.
17:23 Speaker 04 Cut the bull spend. LinkedIn lets you target by company, job title, and more. Advertise on LinkedIn. Spend $250 on your first campaign and get a $250 credit. Go to LinkedIn.com/campaign. Terms and conditions apply.
17:37 Speaker 06 This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome? That's new. It can help you with practically anything on the web. Like restoring a vintage motorcycle from a 50-page restoration block. Or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required. Compatibility and availability varies 18+..
18:03 Speaker 07 College football is back. So Hilton called and me, the superstition concierge, to make your fan rituals a reality. Need a room to match your lucky number? We got you. Want to make sure our team doesn't wash your lucky jersey? Oh, that smells lucky. Hilton's unmatched hospitality can keep up with any superstition. Even a marching band wake-up call at 5:55 in 55 seconds. Hit it! When you need a team that will do whatever it takes on game day, it matters where you stay. Hilton. For the stay.
18:35 Speaker 00 This is kind of a diversion, but I'm so struck in reading all of your work by how often YouTube appears as just, it is everybody's favorite source for everything. It's what, you know, it's what OpenAI used allegedly to create Whisper. It's what a lot of the music stuff is using. It's what, there's a lot of stuff based on videos. Like, what is your sense of YouTube's role as an AI training force? Because it seems to be everywhere.
19:04 Speaker 03 Yeah, that's accurate. YouTube is an extremely common source. I think one reason is there are tools for downloading from YouTube that are really, that work really well. They're really easy to use. And it's pretty common for AI developers to just use those tools. And it's just kind of a, it's become a custom. But also, you know, and that's, that includes, if stuff on YouTube is just less protected, I think is one way of saying it. Like, there's music, you know, if you're a musician, your song might be on Spotify. But Spotify's website has digital rights management protections. It's really hard to download from Spotify. It's much easier to get the same song from YouTube. And so many songs are also on YouTube.
19:49 Speaker 03 So I think it's just ease of downloading.
19:51 Speaker 00 YouTube, obviously, would say out loud that this is not allowed, right? That it violates the Terms of Service. And yet, it seems to have, either it can't, or it just hasn't done anything to stop this, really.
20:05 Speaker 03 Yeah, that's a question that's been in the back of my head for a long time. And which I've asked YouTube, and which they don't really answer. They did, they have said that they consider it a violation of their Terms of Service to be downloading their videos. But yeah, they haven't, you know, years have passed. And it's just as easy to download from YouTube now as it was a few years ago.
20:25 Speaker 00 Yeah, I downloaded a YouTube video this morning. It is just a thing you can do. It's really true.
20:31 Speaker 00 You mentioned the sort of data laundering stuff. But it also seems to me that more and more, the people doing the AI training are just completely unapologetic about it. Like, you quoted Rich Skrenta, the CEO of Common Core.
20:50 Speaker 00 Who literally said to you, like, if you don't let AI robots crawl your data, you essentially don't exist, like, you're gonna miss out on the future of the internet. And I think about, you know, Marc Andreessen, even a couple of years ago, being like, none of this would work if we couldn't just take the training data that we needed. There is this almost like manifest destiny sense of the AI industry that we can have this data because the thing that we're doing is so important that we must have it no matter what. What is your sense of the trend there? Because it occurs to me that that's happening even as the backlash from people who hate the experience and feeling of AI, in part because of the way that this stuff is trained, just keeps getting worse.
21:33 Speaker 00 Like, these things are just running away from each other at like, record speeds.
21:37 Speaker 03 Yeah, I, that's, that's a huge question. I think, you know, I think it has something to do with the fact that we just have not done a great job in this country with establishing the value of data and who should be able to have data, right? This, this is something that privacy advocates have talked about. Jaron Lanier, I think, was one of the earliest people to be talking about this. I think he wrote in 2007 about that you should get paid, like, you were being surveilled, basically, and companies are taking, that they're monitoring everything you do, they're generating data from your online activity. That data is extremely valuable to them.
22:22 Speaker 03 It might seem like nothing to you, but you should be getting paid for that. People thought that was crazy back then, and I think a lot of people still think that's crazy now. But, you know, taking people's music to build models that generate songs that compete with them is just the next version of that. And it's just gonna keep going. If, if we don't acknowledge that this data is incredibly valuable and figure out a way to write laws around that or just have, you know, better business practices or something, this is just, it's just gonna get worse, there's just gonna be more exploitation. More mining.
22:59 Speaker 00 The next step, I think a lot of people have perceived for a while to be synthetic data, right, that eventually we're going to get AI models that are so good at making new things that then we can use those things to train new models. And eventually, they don't, they don't need existing recorded music. That synthetic data is the future, and that's how we get to everything.
23:22 Speaker 00 And especially now, I mean, you look at Spotify, and there are tons of AI-generated songs on Spotify that some people are listening to. There are a billion AI-generated podcasts out there, like, the content is being made. Is that the next phase of training data? Is synthetic data coming into its own in such a way that we're going to start to see the next generations of these models built on the things made by the last generations of these models?
23:47 Speaker 03 Absolutely not.
23:49 Speaker 00 Really?
23:49 Speaker 03 I don't, I don't think that there's any evidence that that actually works. I think when AI companies talk about training on synthetic data, they are, they choose their words very carefully, and they always exaggerate the extent to which it's happening. There's a lot of papers, there's a lot of research out there on a phenomenon called model collapse, which is what happens when you, you train a model on its own, its own outputs. It very quickly, it doesn't get better, it very quickly degrades. And it's not hard to see why that could be the case, right? Like, AI is kind of an averaging machine, right? It's finding statistical average between different types of content and putting that into some new, new kind of more average type of content.
24:34 Speaker 03 And there's just not enough weirdness or interestingness or something like that. There's some quality in the work that humans do that's not in the work that AI does. And I think that's, that's actually proved by the model collapse phenomenon. So I'm amazed that AI companies are going around talking about synthetic data still, when there's so much evidence that it doesn't, it doesn't work.
24:57 Speaker 00 So is there a next untapped place filled with data? I mean, these models keep getting bigger, they keep needing more data. They gotta go somewhere, right? Is there a, is there a next sort of unturned place to go for these companies?
25:10 Speaker 03 I think they just pay, pay people to make it. I think there's already a gigantic industry of writers who are writing for AI, musicians who are making music for AI. You know, after I published this story, I got a, an email from a company that is, is doing this. They, they claim to have paid creators over $10 million just to make things for AI training. It's, it's very strange, but this is, this is the next, like AI as your audience is the next frontier for creators.
25:46 Speaker 00 It is a deeply strange thing to think about, but
25:49 Speaker 03 Yeah.
25:49 Speaker 00 Also like, you know, we're, we're going to put this on YouTube and it's going in there anyway. So who knows? Maybe this is all of our destinies, no matter what.
25:57 Speaker 03 I certainly hope not. I don't think so.
26:01 Speaker 03 I think we can, I think we can have a conversation and, and arrive at a more reasonable future for ourselves and the culture.
26:09 Speaker 00 I'm, I'm, I'm with you on that. All right. Well, Alex, thank you so much for being here. I really appreciate it.
26:13 Speaker 03 Thank you, David. That's great.
26:14 Speaker 00 All right. That's it for the show. Thank you to Alex for being here and thank you as always for watching and listening. If you have thoughts, questions, feedback of any kind, if you have a favorite AI song you want to send me, just not the Puerto Rico song. I already know that one. Send me all your favorite AI songs. No judgment. I just want to hear all of them. You can always hit up the hotline 866 VERGE11. You can send us an email, [email protected] . We love hearing from you. And as a reminder, the best thing you can do to support all of this is to subscribe to The Verge, theverge.com/subscribe. It gets you all of our podcasts ad free, including this one. It gets you all of our exclusive newsletters. It gets you all of our coverage. Terrence O'Brien on our team has been covering Suno a lot and doing a really terrific job. Lots to come on that. Subscribe to The Verge. I think it's a pretty good website.
26:57 Speaker 00 Vergecast is a Verge production and part of the Vox Media Podcast Network. This show is produced by Josh Kahas, Eric Gomez, Brandon Kiefer, Travis Larchuk, and Aaron LaCassio. We'll see you tomorrow. Rock and roll.
27:10 Speaker 08 Everyone's got a hard question about AI. Your job, your kids, where it's going. Anthropic was built to surface those questions and share what it finds along the way, because there's hope in hard questions. Ask yours at claude.ai/spotify and keep thinking. That's claude.ai/spotify.
27:34 Speaker 09 I work here. You know, that thing you have to wake up for every day. I know I've made things a little difficult in the past. OK, a lot difficult. But I got some help from UKG. And finally, I understand. Now I'm making work work better. I've even got AI agents helping me find new ways to help you, your people, and your business. Because when work works, everything works. UKG, HR, Pay, and Workforce Management.
Previous Google's new speaker and your smart home questions Next Of course Meta thinks gambling is the future Page updated October 26, 2026 · transcript and summary are machine-generated