When AI Cannibalizes Its Data

Asked ChatGPT anything lately? Talked with a customer service chatbot? Read the results of Google's "AI Overviews" summary feature? If you've used the Internet lately, chances are, you've consumed content created by a large language model. These models, like DeepSeek-R1 or OpenAI's ChatGPT, are kind of like the predictive text feature in your phone on steroids. In order for them to "learn" how to write, the models are trained on millions of examples of human-written text. Thanks in part to these same large language models, a lot of content on the Internet today is written by generative AI. That means that AI models trained nowadays may be consuming their own synthetic content ... and suffering the consequences. View the AI-generated images https://news.rice.edu/news/2024/breaking-mad-generative-ai-could-break-internet mentioned in this episode. Have another topic in artificial intelligence you want us to cover? Let us know my emailing [email protected] mailto:[email protected]! Listen to every episode of Short Wave sponsor-free and support our work at NPR by signing up for Short Wave+ at plus.npr.org/shortwave http://plus.npr.org/shortwave. See pcm.adswizz.com https://pcm.adswizz.com for information about our collection and use of personal data for sponsorship and to manage your podcast sponsorship preferences. NPR Privacy Policy https://www.npr.org/about-npr/179878450/privacy-policy

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  This message comes from Doppel. Social engineering attacks come in disguise, but Doppel sees through them. The Agentic platform shuts down threats, turns employees into first-line defenders, and more. Learn more at D-O-P-P-E-L dot com.

• 0:16Regina Barber?

  You're listening to Short Wave from NPR.

• 0:23Regina Barber?

  It seems like these days, generative AI is everywhere. It's in my Google searches. It's suggested as a tool on TikTok. It's running customer service chats. And there's a lot of forms that generative AI can take, like it can create images or video. But the ones that have been in the news recently, DeepSeek R1, OpenAI's ChatGPT, Google Gemini, Apple Intelligence, all of those are large language models. A large language model is kind of like the predictive text feature in your phone, but on steroids.

• 0:55Speaker 02

  Large language models are statistical beasts that learn from example of human-written text and learn to produce text that is similar to the ones that the model was taught on.

• 1:11Regina Barber?

  That's Ilya Shumaylov. He's a computer scientist, and he says in order to teach these models, scientists have to train them on a lot of human-written examples, like they basically make the models read the entire internet. Which works for a while, but nowadays, thanks in part to these same large language models, a lot of the content on our internet is written by generative AI.

• 1:34Speaker 02

  If you were today to sample data from internet randomly, I'm sure you'll find that a bigger proportion of it is generated by machines. But this is not to say that the data itself is bad. The main question is how much of this data is potentially downstream dangerous.

• 1:56Regina Barber?

  In the spring of 2023, Ilya was a research fellow at the University of Oxford. And he and his brother were talking over lunch. They were like, OK, if the internet is full of machine-generated content, and that machine-generated content goes into future machines, what's going to happen?

• 2:13Speaker 02

  Quite a lot of these models, especially back at the time, they're relatively low quality. So there are errors. And there are biases. There are systematic biases inside of those models. And thus, you can kind of imagine the case where rather than learning useful contexts and useful concepts, you can actually learn things that don't exist. They are purely hallucinations.

• 2:35Regina Barber?

  Ilya and his team did this research study indicating that eventually, any large language model that learns from its own synthetic data would start to degrade over time, producing results that got worse and worse and worse.

• 2:52Speaker 02

  In simple theoretical setups we consider, you're guaranteed to collapse.

• 2:58Regina Barber?

  So today on the show, AI model collapse. What happens when a large language model reads too much of its own content? And could it limit the future of generative AI? I'm Regina Barber, and you're listening to Short Wave, the science podcast from NPR.

• 3:17Speaker 03

  Stay tuned for the following announcements and messages from our sponsors.

• 3:23Speaker 04

  We will learn something on election night, but not everything. There's a lot we will not know, especially when it comes to close races. And we'll know much more the next morning and the morning after that. So save yourself a little time scrolling and listen to Up First. We'll be there when you wake up each and every morning this week with the latest midterm election news.

• 3:46Speaker 05

  Oh, Kevin, it feels good to be back. It sure does, Casey. And what a time. The world is talking about AI and what's going on in Silicon Valley. And we are here with a new show to tell you all about it. That's right, Kevin. On our new show, Machine Gods, twice a week, we'll be bringing you the biggest news in tech and bring you unmatched access to the real titans of this industry. You can find Machine Gods from NPR on the NPR app or wherever you get your podcasts.

• 4:10Regina Barber?

  How did we get here?

• 4:11Speaker 06

  That's a question we have been trying to answer a lot here at NPR. We are exploring global histories on Throughline. We're hearing from national security experts on sources and methods. We're watching the markets with Planet Money. You can support this work across all our podcasts with NPR+. Find out more at plus.npr.org.

• 4:34Regina Barber?

  Okay, Ilya, before we get into the big problem of like model collapse, I think we need to understand why these errors are actually happening. So can you explain to me what kinds of errors do you get from a large language model? And like, how do they happen? Why do they happen?

• 4:50Speaker 02

  So there are three sources, three primary sources of error that we still have. So the very first one is the basically just data associated errors. And usually those are questions along the lines of, do we have enough data to approximate a given process? So if some things happen very infrequently in your underlying distribution, you may get like, your model may get a wrong perception that that some things are impossible.

• 5:20Regina Barber?

  Wait, what do you mean by they are impossible?

• 5:23Speaker 02

  Like, an example I've seen on Twitter was if you Google for a baby peacock, you will discover pictures of birds that look relatively realistic, but they are not peacocks at all. They are completely generated and you will not find a real picture. But if you try learning anything from it, then of course you're going to be absorbing this bias.

• 5:46Regina Barber?

  Right, you're like telling me now that there's like a lot of fake baby peacock images, but machines don't know that, right? They're just going to think, great, this is a baby peacock. And also, there's not that many like real baby peacock images to compare to.

• 6:00Speaker 02

  Exactly. And those are the kinds of errors that you don't normally see that often because they are so improbable, right? So and if people are going to start reporting things to you and saying, oh, your model is wrong here, they're likely to notice things that on average are wrong. But if they're wrong in some small part of the internet that nobody really cares about, right? Then it's very unlikely that you will even notice that you're making a mistake. And usually this is the problem because. As the number of dimensions grow, you will discover that the volume in the tails is gonna grow disproportionately.

• 6:32Regina Barber?

  Not just babies, but babies' birds. But not just baby birds, but baby peacocks.

• 6:37Speaker 02

  Yeah, exactly. So as a result, you'll discover that you need to capture quite a bit.

• 6:41Regina Barber?

  Okay, okay. So that's one kind of problem, a data problem. What are the other two?

• 6:47Speaker 02

  On top of it, we have errors that come from learning regimes and from the models themselves. So on learning regimes, we are all training our models. All of them are structurally biased. So basically to say that your model is gonna be good, but it's unlikely to be optimal. So it's likely to have some errors somewhere. And this was the error source number two. And error source number three is that the actual model design would shape and form your model should be taking. It's very much alchemy. Like, nobody really knows why stuff works. We kind of just know empirically stuff works.

• 7:24Regina Barber?

  It's like a black box. We don't know how it's making these decisions. We don't know like where in, like you said, in that order, it's fixing those decisions, you know?

• 7:33Speaker 02

  Yeah, which parts of the model are responsible for what? We don't know the fundamental underlying bias of a given model architecture. What we observe is that there is always some sort of an error that is introduced by those architectures.

• 7:48Regina Barber?

  Right, right. Okay, so the three places errors could come from is like, one, the model itself, two, the way it's trained, right? And three, the data or the lack of data that it's trained on.

• 7:59Speaker 02

  Exactly. And then we also have empirical errors from like, for example, hardware. So like, we also have practical limitations of hardware with which we work. And those errors also exist.

• 8:13Regina Barber?

  Let's talk about like how those errors build. What happens when they start to build upon each other? Like, can you describe that outcome to me?

• 8:22Speaker 02

  Yes, certainly. So what we observe in simple theoretical models is that, so two main phenomena happen. The very first phenomena that happens is It's really hard to approximate improbable events, in part because you don't encounter them very often. So you may discover that you're collecting more and more data, and a lot of this data looks very similar to what you already possess. So you're not discovering too much information. But importantly, you're not discovering this like infrequent data points. So those tail events, they kind of disappear. And then the other thing that happens is that the first time you made this error and underestimated your improbable events, when you hit the model on top of this,

• 9:06Speaker 02

  it's unlikely to recover from this taking place, right?

• 9:12Regina Barber?

  Okay, so like, over time, you start to like lose the more unique occurrences and like all the data starts to like look more similar to the average.

• 9:21Speaker 02

  Originally improbable events are even more improbable for the subsequent model, and it kind of like snowballs out of control until the whole thing just collapses truly to near zero variance.

• 9:32Regina Barber?

  So instead of this bell curve, you just have like a point in the middle, you just have a whole bunch of stuff in the middle.

• 9:38Speaker 02

  Exactly, and the thing is, you can theoretically describe this. It's actually very simple, and you can run these experiments like however many times you want. And you'll discover that even if you have a lot of data, if you keep on repeating this process and the rate at which this collapses, you can also bound, you end up always in a state where your improbable events kind of disappear. In practice, when we grab large language models, we observe that they become more confident in the predictions that they are making. So basically, like, the improbable events here are going to be things that the model is not very confident about, and normally it would not make predictions about it. So when you're trying to generate more data out of a language model

• 10:21Speaker 02

  in order for another language model to learn from it, over time, basically, it becomes more and more confident. And then it basically during the generation setup, it gets stuck very often in this repetitive loops.

• 10:33Regina Barber?

  This isn't exactly the same, but it makes me think of the telephone game. You know, when you like

• 10:37Speaker 02

  Yes.

• 10:37Regina Barber?

  tell somebody a phrase or like a couple of sentences and then like the next person tells a person the same two sentences and then like the next person says the same two sentences. And it usually gets like more and more garbled as it goes down the line.

• 10:51Speaker 02

  I think this is a comparison kind of, kind of works. Yes. So this is the first thing, it's the improbable events. And then the second thing that happens is your models are going to produce errors. So misunderstandings of the underlying phenomenon, right? And as a result, what you will see is that like those errors start propagating as well. And they are relatively correlated, like if you're, all of your models are using the same architecture. Then it's likely to be correlated like correlatedly wrong in the same kinds of way. So whenever it sees errors, it may amplify the same errors that it's observing.

• 11:25Regina Barber?

  Yeah, I mean, I'm, I'm looking at some of the like image output of these models that, um, that are trained on their own data right now, um, and we'll link these images in the show notes, but I'm looking at like somebody's handwriting of like zero to nine. And you know, it's not perfect, it's, it's handwriting, but like as it gets regenerated by the models over and over like 15 times. They, they're just dots, right? Like they're not distinguishable. You, you can't even tell their, their numbers, like which one is which.

• 11:54Speaker 02

  Yeah. So approximations of approximations of approximations end up being very imprecise.

• 11:58Regina Barber?

  Yeah.

• 11:58Speaker 02

  As long as you can bound the errors of your approximations, it's, it's okay, I guess. But yeah, in, in practice, because machine learning is very, is very empiric. Quite often we can't.

• 12:10Regina Barber?

  Oh, I love, I love these images. This is so good, Ilya, this is so good.

• 12:14Speaker 02

  Yeah. So an important thing to say here is that

• 12:20Speaker 02

  the settings we talk about here are relatively hypothetical. In a sense that we are not in the world in which, you know, today we can build a model and tomorrow they disappear. That, that is not going to happen. We already have very good models and the way forward is having even better models and there is no doubts about this.

• 12:39Regina Barber?

  Okay. So like you said, you know, chat GPT isn't going to disappear tomorrow. Uh, what are researchers doing to avoid the problem of model collapse? Like as a computer scientist, what do you think the solution is?

• 12:54Speaker 02

  I mean, there are many different solutions. You'll find a lot of different papers that are exploring what are the most effective mitigations, and it's mostly data filtering of different kinds. And basically making sure that the data that ends up being ingested by the models is representative of the underlying data distribution. And whenever we hit this limit and we see that our model diverges into some sort of a training direction, the trajectory that is making the model worse, I promise you, people will stop training of the models, retract back a couple of steps, maybe add additional data of certain kind and then keep on training. Right? Because we can always go back to previous models. Nothing's stopping us.

• 13:39Speaker 02

  And then we can always spend more effort getting high-quality data.

• 13:43Regina Barber?

  Or paying more people to create high-quality data.

• 13:45Speaker 02

  Yeah, so model collapse is not going to magically kill the models tomorrow. We just need to change the way we build stuff. So this is not all doom and gloom. I am quite confident we'll solve this problem.

• 14:00Regina Barber?

  I like that perspective. Ilya, thank you so much for talking with us today.

• 14:05Speaker 02

  Thank you very much for having me. It was a pleasure.

• 14:09Regina Barber?

  If you want to see some of the images I was looking at, you know, see the consequences of AI model collapse for yourself, we'll link to those in our show notes. Also, make sure you never miss a new episode by following us on whichever podcasting platform you're listening from. This episode was produced by Hannah Chin and edited by showrunner Rebecca Ramirez. Hannah and Tyler Jones checked the facts. Jimmy Keeley was the audio engineer. Beth Donovan is our senior director, and Colin Campbell is our senior vice president of podcasting strategy. I'm Regina Barber. Thank you for listening to Shorewave, the science podcast from NPR.