AI Fundamentals: Datasets 101

In April, we released our first AI Fundamentals episode: Benchmarks 101 https://www.latent.space/p/benchmarks-101#details. We covered the history of benchmarks, why they exist, how they are structured, and how they influence the development of artificial intelligence. Today we are (finally!) releasing Datasets 101! We’re really enjoying doing this series despite the work it takes - please let us know what else you want us to cover! Stop me if you’ve heard this before: “GPT3 was trained on the entire Internet”. Blatantly, demonstrably https://arxiv.org/abs/2005.14165 untrue: the GPT3 dataset is a little over 600GB, primarily on Wikipedia, Books corpuses, WebText and 2016-2019 CommonCrawl. The Macbook Air I am typing this on has more free disk space than that. In contrast, the “entire internet” is estimated to be 64 zetabytes https://healthit.com.au/how-big-is-the-internet-and-how-do-we-measure-it/#:~:text=In%202020%2C%20the%20amount%20of,and%20consumed%20on%20the%20web., or 64 trillion GB. So it’s more accurate to say that GPT3 is trained on 0.0000000001% of the Internet. Why spend $5m on GPU time training on $50 worth of data? Simple: Garbage in, garbage out. No matter how good your algorithms, no matter how much money/compute you have, your model quality is strongly determined by the data you train it on and research scientists think we just don’t need or have that much high quality data. We spend an enormous amount of effort throwing out data to keep the quality high, and recently Web 2.0-era UGC https://blog.hootsuite.com/user-generated-content-ugc/ platforms like StackOverflow, Reddit, and Twitter clamped down on APIs as they realize the goldmines they sit on. Data is the new new oil. Time for a primer! Show Notes * Our 2 months worth of podcast prep notes https://github.com/swyxio/ai-notes/blob/main/Resources/DATASETS.md! * The Token Crisis paper https://arxiv.org/abs/2305.13230 * Ilya Sutskever on datasets https://overcast.fm/+-yu8j7I5U/21:00 * OpenAI Tokenizer https://platform.openai.com/tokenizer * Kaplan Scaling Laws Lecture https://www.youtube.com/watch?v=sNfkZFVm_xs * Chinchilla Paper https://www.deepmind.com/publications/an-empirical-analysis-of-compute-optimal-large-language-model-training * Sasha Rush’s Tweet https://twitter.com/srush_nlp/status/1633509903611437058 * Karpathy’s Build Conference Presentation https://twitter.com/altryne/status/1661236778458832896 * LIMA Paper https://arxiv.org/abs/2305.11206 * Phi-1 by Microsoft https://the-decoder.com/microsofts-tiny-phi-1-language-model-shows-the-importance-of-data-quality-in-ai-training/ from MSR * Washington Post Article on datasets https://archive.is/YzguI * Our episode with Jonathan Frankle https://www.latent.space/p/mosaic-mpt-7b#details * Our episode with Mike Conover https://www.latent.space/p/mike-conover#details * BloombergGPT https://www.bloomberg.com/company/press/bloomberggpt-50-billion-parameter-llm-tuned-finance/ * Datasets * HuggingFace Hub https://huggingface.co/datasets * CommonCrawl https://commoncrawl.org/, Overview https://commoncrawl.org/2013/08/a-look-inside-common-crawls-210tb-2012-web-corpus/ * C4 https://huggingface.co/datasets/c4 * List of Dirty, Naughty, Obscene, and Otherwise Bad Words https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words * OpenWebText https://huggingface.co/datasets/openwebtext * books3 https://huggingface.co/datasets/the_pile_books3 * OpenAssistant https://huggingface.co/datasets/OpenAssistant/oasst1 * The Stack https://huggingface.co/datasets/bigcode/the-stack * The Pile https://pile.eleuther.ai/ * LAION https://laion.ai/blog/laion-5b/ * Audio: * LibriSpeech https://www.openslr.org/12: A dataset of audio recordings of audiobooks * CommonVoice https://commonvoice.mozilla.org/en/datasets: A dataset of audio recordings of people speaking different languages * Voxforge https://voxforge.org/: A dataset of audio recordings of people speaking different languages​ * Switchboard https://catalog.ldc.upenn.edu/LDC97S62: A dataset of audio recordings of telephone conversations​ * Fisher Corpus https://catalog.ldc.upenn.edu/LDC2004T19: A dataset of audio recordings of news broadcasts​ * Chinese: * CMRC (Chinese Machine Reading Comprehension 2018) https://github.com/ymcui/cmrc2018 * DuReader https://github.com/baidu/DuReader * ChID https://paperswithcode.com/dataset/chid#:~:text=for%20Cloze%20Test-,ChID%20is%20a%20large%2Dscale%20Chinese%20IDiom%20dataset%20for%20cloze,idiom%20are%20provided%20as%20choice. * Copyright & Privacy: * https://stablediffusionlitigation.com/ * https://haveibeentrained.com/ * https://githubcopilotlitigation.com/ * https://twitter.com/moyix/status/1662131770463072257 * OpenAI Opt Out Process https://help.openai.com/en/articles/7039943-data-usage-for-consumer-services-faq * Check if you’re in The Stack https://huggingface.co/spaces/bigcode/in-the-stack * Deduplication * Deduplicating Training Data Makes Language Models Better https://arxiv.org/abs/2107.06499 * Deduplicating Training Data Mitigates Privacy Risks in Language Models https://arxiv.org/abs/2202.06539 * Contamination * CodeForces example https://codeforces.com/blog/entry/113910 This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:09Alessio Fanelli?

  Hey everyone, welcome to the Latent Space Podcast. This is Alessio, Partner and CTO on Resident Admissible Partners. I'm joined by my co-host swyx, writer and editor of Latent Space.

• 0:20Speaker 01

  Today, we finally, finally, we have the second episode, our 101 track. This has been a long time coming. So last time we did Benchmarks 101, I think we got a lot of mileage out of that. We understood a lot about the benchmarks and we talked with a lot of our guests over the previous episodes about how they evaluate their models. And today, we wanted to dive into data sets, what they are, how they're constructed, and why they matter. I guess I should go into why we wanted to do this episode. It's a little bit weird to separate data sets and benchmarks. So we did benchmarks first, but a lot of the benchmarks were data sets. So pretty much they're the one and the same thing, right?

• 0:59Speaker 01

  Like, and I think where they start to diverge is a cause of significant interest. But mostly, actually, I wanted to focus on data sets for one primary reason, which is that many people say that GPT is trained on all the internet. So first of all, this is actually not true. And second of all, it actually causes some potential misperceptions. I say potential because there is some legitimate debate about this. There's a misperception about us running out of data. And we can discuss the pros and cons of whether or not we are running out of data. It's been named the token crisis by academics and quite a lot of commentators on AI. So in the show notes, we're going to link to a paper on "To Repeat or Not to Repeat:

• 1:44Speaker 01

  Insights from Scaling LLMs Under the Token Crisis." And then I'm also going to link to an opposite view from OpenAI with his... Ilya Sutskever, talking about how they're not anywhere close to running on the data that they want to train on. So whenever there's such an interesting divergence between practitioners and academics, I think it's a worthwhile thing to dive into. And just in general, I think there's a lot of foundational knowledge that people skip over when they assume that everyone knows what data sets we're talking about.

• 2:15Alessio Fanelli?

  Yeah, I was gonna say, I think also in terms of like the knowledge that the models have. If you say it's been trained on the whole internet, you would assume it knows everything on the internet, but it obviously doesn't. You can go in there and ask about people that have online presences and are not actually in the knowledge base of the model. And this also helps when thinking about what data to then use to fine tune. So if you understand what's in the model, if you're trying to build a verticalized model for a specific use case, you can better figure out what's actually gonna be meaningful versus what was already present in the first training run.

• 2:49Speaker 01

  Yeah, just for some comparison, let's say the total size of the internet, some people have estimated at around five billion gigabytes. Most of the data sets that we are going to talk about today are in the hundreds of gigabytes range. And it's growing every single day, you know, there's always new data being created every single day. And there's always new modalities to claim that data from. So, you know, a lot of the whisper behind open the eyes whisper is that they're actually transcribing YouTube, which is a, this is just source of extra tokens. And we'll have to explain what tokens are. The first thing is about the whether or not we're running out of data. The second issue is this divergence between data sets and benchmarks.

• 3:32Speaker 01

  And I wanted to dive into this specifically because they used to be essentially one and the same thing. In like a very standard machine learning tutorial, you would do something like the iris data sets. And then you would do train, test, validation splits. And you would basically evaluate data based on samples from the data itself. But more recently, we actually have decoupled benchmarking from the data sets they're trained on, except for the calculation of loss. I think in our Discord, in Lateshun's Discord, we've actually been doing a small paper club, where we've gone over some of the foundational papers. And we actually recently went over the BERT paper,

• 4:13Speaker 01

  which is the bidirectional transformers paper that was a predecessor to T5 and is a predecessor to all the large language models today. And in BERT, they actually invented this concept of masking, which meant that data sets could create their own training objectives, which I actually think is super interesting. So basically, what you have is, for example, a sentence. And out of the sentence, you mask one word, and you ask the model to predict that word, and you grade the model based on whether or not it's able to predict that word. This basically starts to go from supervised learning, where you have a data set that you're trying to train on, to sort of self-supervised learning, where you can just kind of let loose on unlimited set of data.

• 4:55Speaker 01

  And so this basically lets you scale as much as you want on the data side, as much data as you have, which I think is just really interesting and foundational. Like, you don't have deep learning without self-supervised learning. You don't have a good training objective until you have the concept of masking. And once you have really, really good masking, then you start to find algorithms for that. And it turns out that deep learning is the way to do it, to achieve loss unseen by any other algorithm. And then, once you have masking, you can predict. And then you can generate. And then, you know, everything kind of follows from there. But that is the data set and not a benchmark, right? Because you can have all of Wikipedia as a data set, but nobody bothers to benchmark on Wikipedia

• 5:40Speaker 01

  because that's not a reasonable benchmark to derive any score on. But it turns out that training on a data set leads to higher evaluation on benchmarks that we covered in the last episode, like common, uh, what was it? Helleswag.

• 5:53Alessio Fanelli?

  Like Helm.

• 5:54Speaker 01

  Big Bench. MMLU, Helm. Yeah, exactly. So I just think like, that's fascinating. That we've just summed up, you know, now it seems so retroactively obvious, but we've just summed up maybe about 10 years of progress on deep learning.

• 6:07Alessio Fanelli?

  Yeah, and finally, the most important thing about data sets is, one, understanding how big they are, and then using scaling loss to work yourself back into a size model you can train with them. So we talked about the Chinchilla scaling loss, and we'll cover that later. But if you want to train a model that is 100 billion parameters, you cannot just pick any amount of. Data, it needs to be a lot of it. So understanding come and crawl, how many tokens is that? C four, how many tokens is that? Helps you understand, okay, this is what I can get off the shelf. This is what I need to provide. And we'll dive into more of the data sets later. But first, I just wanted to do a quick explanation of what tokens actually are.

• 6:50Alessio Fanelli?

  So the thing you read on the OpenAI docs is like one token is like three quarters of a word. So when I first read it, I was like, oh, you're just doing character splitting. But that's not really how it works. So basically, one token is an integer that can be, you know, up to, actually don't know what the highest number will be. But it's an integer representation of words. And the same word can also have different representations based on where it is in a sentence. For example, one of the big things that transformers did to be more efficient is the space is included in the token. So if you're doing a long sentence, instead of having one token for each space in it, the space is inside the token of the word.

• 7:32Alessio Fanelli?

  So it really cuts down on how many you actually need to go through it. But the funny thing is that then you have different representation for the same word. So if you take the word red, like the color, there's one token for lowercase red with a space in front. That's token two, two, six, six. Then you have red with a space in front with a capital R. That's two, two, nine, seven. Then you have red with a capital R, no space. That's token seven, seven, three, eight. So you can see that when you say a trillion tokens, for example, it's not one trillion different English words. So that's also one of the main thing that you gotta be careful of.

• 8:14Alessio Fanelli?

  You can have a data set that is a lot of tokens, but has potentially a lot of repetition. That is not as helpful. And also when you're doing things like a logit bias in OpenAI, where you can deprioritize certain tokens, you have to find all the tokens for that. So the example that they use is if you want to do a recipe for a cake with no eggs, you have to set the logit bias of both egg and like space egg, egg space tokens. Not just one. So that's one thing to keep in mind. If you're coming from a non-ML background, that's probably one of the first gotchas that you have when talking about datasets.

• 8:50Speaker 01

  And just for more examples, OpenAI has a tokenizer tool that we're going to link in the show notes that you can use to see what any particular phrase translates to. So I just plugged in, for example, latent space. Latent space is three tokens. It's L A T, that's the first token, lat. And then second token is E N T, e- uh, ent. And then the last one is space and then S P A C E. And that's the last token. So latent space is three tokens and some combination of that will form other words as well. And it's an interesting tokenization scheme. As far as I understand, by the way, Alessio, I think the upper limit on tokenizers is between fifty thousand to eighty thousand tokens, which is amazing. It means that you can basically make up any sentence or language

• 9:35Speaker 01

  from these individual tokens. It's a small number. It's actually like, you know, five digits of numbers, not like, not millions and millions of tokens. And keep in mind that they speak multiple languages. So you have to actually tokenize other languages as well as numbers, as well as symbols, emojis. I actually am pretty amazed at the depth of tokenization. And honestly, there was actually a well-known flaw with GPT-3 where you could actually do a quick test to see if you're talking to a bot or not, right? One of the reasons that GPT-3 is just not good at math is because they don't tokenize numbers individually and they don't represent numbers the same way that humans do. Humans represent numbers digit by digit.

• 10:19Speaker 01

  But if I type in into this tokenizer, GPT-3, like I type in one to this, that's one token. If I type in one, two, which is twelve, that's also a different token. And that's a single token still. And if I type in one, two, three, so it's one hundred and twenty-three, that's also another single token. But if I type in one, two, three, four, there is now two tokens that is made up of twelve and thirty-four. So it's it's breaking up one, two, three, four, like an English word rather than a mathematical representation of, you know, one, one thousand, two hundred, three tens and four ones. And that's one of the reasons it doesn't do math very well, because it uh it looks as as though it's a word rather than uh numbers.

• 10:58Alessio Fanelli?

  Uh huh. Yeah. Yeah. And you mentioned the language thing. That's another good point. Some languages are actually less token efficient than others. For example, Spanish actually requires more tokens for the the same sentence. And they also have this issue where the lower the number of the token, like the more common the token is. And if you tokenize a Spanish phrase and you have things, syllables like man in it, the token value of the token is very low, because in the English language is used very often. So you can have predictions that are a little weird when you use different languages and we'll kind of go into language-specific data sets later as well.

• 11:36Speaker 01

  Yeah, there's a famous article about this uh called Why is GPT-3 fifteen point seven times more expensive for certain languages? And so the English bias is is real. Um. So I want a pizza. It's four tokens in English. And then in French, I don't speak French, but je veux une pizza. Uh. That's seven tokens. And then Chinese is fifteen tokens.

• 11:58Alessio Fanelli?

  我想要一个披萨.

• 12:02Speaker 01

  Don't actually know uh what what these characters are. But anyway, so it's interesting, right? Like all of this is represented within uh the token space that GPT has trained, and it's it's actually pre-trained. This is one of the first examples of pre-training being useful, because ultimately language models or the the transformers that power the language models are transforming a set of numbers to a different set of numbers or predicting the next token ID in a sequence of numbers. The transformers themselves don't actually know what word they're predicting. All they know is they have they're given some data sets of number after number after number, and in fact, like trillions of them. And then their job is to predict the next number given a sequence of numbers.

• 12:43Speaker 01

  And then we take the tokenizers to convert those numbers into words or images or audio. We referenced scaling laws a little bit. And basically, this comes out of research around what the optimal sizing of a model should be for a given dataset. I think in 2020, OpenAI published the first scaling law, which was the Kaplan paper. And that was estimated to be about 1.7 times tokens per parameter. So what that means is, the reason that GPT-3 is a 175 billion parameter model is because they had around about 300 billion tokens to train with. So they worked backwards. From the dataset size that they had of 300 billion,

• 13:25Speaker 01

  they said, okay, based on this, the largest possible model that we can train is 175 billion. Let's go for that. That was the state of the art at that time. And we, as far as OpenAI is concerned, larger models were always going to be better because of understanding around emergence and capabilities, and honestly, just research around what AGI could be. And just for reference, the GPT-2 was about 100 times smaller than that. So that's, that's super interesting. And then the year after that, Chinchilla came out of Google DeepMind, where they actually optimized for a different metric, which is compute optimal training. So a given compute budget, a given amount of days and hours running a certain number of GPUs. So that gives you a certain compute budget in terms of flops.

• 14:11Speaker 01

  If you hold that budget constant, so let's say, you know, that's roughly a few days or a few months of compute. That is actually very material for a research team or anyone because that translates directly into dollars, right? Like, how much time are you renting on the shared GPUs? So for a given compute budget, what is the best model that you can train, given some kind of a compute budget? And the number that it came up with was 20 times tokens per parameter. So that's actually 10x what the Kaplan laws were doing. What DeepMind did was, they had sort of replicas of GPT-3 that they called Gopher. And then they created another replica called Chinchilla and showed that despite being about 10x smaller, they were able to match or beat Gopher.

• 14:56Speaker 01

  And the assertion they would also beat GPT-3. Despite being 10x smaller, most people call this is basically that um GPT-3 was over parameterized. There was way too many parameters, 175 billion was way too much. And in fact, to train GPT-3 to a full 175 billion parameter model, you need 3.5 trillion tokens, not 300 billion. Which I think is just fascinating and a little bit depressing. It means like we just need a lot more data.

• 15:22Alessio Fanelli?

  Yeah, and it just shows you like how early this whole foundation model space is, you know, like these papers are not coming every 10 years, you know, they're coming like every 18, 24 months. And the other thing you mentioned there with the cost of compute, you know, if you think that 1.7x is good, you probably don't want to burn GPUs for like a much larger training scale, you know. And now the the next, the next thing is the Llama optimal, which is 200 times tokens per parameter. So now to train GPT-3, you need 35 trillion tokens, which is like, if you take all the books published each year, like all of Kindle Unlimited, all of like, all of that stuff,

• 16:04Alessio Fanelli?

  it's only 100 billion tokens. So to get to like 35 trillion, you need a lot of data. But again, is this the new optimal? We also don't know because it's not easy to find ways to train models at this size. Like it's not easy to find the computer, it's not easy to find the data. So once we record data sets, you know, 201 in a couple of years, we're probably going to say, it was crazy that we were using like 20x, you know, it's actually like this alteration now.

• 16:34Speaker 01

  One thing I want to, want to basically mention is that as far as I can tell, the researchers that I'm following, talking about this stuff, because we've asked this question on the podcast every opportunity that I've had. And essentially, we're going from compute optimal, which is like at training time, to inference optimal, which is at inference time, right? Llama was designed for an inference optimal situation where we start caring about the latency of the inferences. And basically that just means the size of the models have to be smaller. They cannot be hundreds of billions of parameters and they're all, you know, round about this like double digit parameter size now. Maybe even single digit with the, with the 7B and 3B type models from MPT that we talked about with Jonathan Frankel.

• 17:19Speaker 01

  This is a nice evolution. It basically balances practical requirements. You know, it's very funny coming from a software background studying AI. Because in software, performance means speed. Whereas in machine learning, performance does not mean speed. Performance means capabilities and evaluations on benchmarks. Where inference time doesn't matter. But now inference time does matter because AI is crossing over from research into software, into practical applications. So now we do care about things like inference costs and memory that you need to run all these systems. So I just think it's just fascinating that like Llama Optimal is like purposely overtraining Chinchilla, like DeepMind showed that Chinchilla is compute optimal. And now we're saying we just don't care.

• 18:02Speaker 01

  Like, we will purposely overtrain and be suboptimal there in order to be more optimal in inference, because inference is more important now.

• 18:08Alessio Fanelli?

  Yeah, exactly. The question is like, optimal for what? You know, if you're writing a paper, you're optimizing for like training and kind of building the proof. If you're training a model for like production use case, the training cost is actually just a small part of overall lifetime cost of the model. So the pendulum is kind of keep swinging depending on the the application.

• 18:30Speaker 01

  Maybe I'll go on to the next bit, which is LLMs as databases. Okay, this is interesting. So we have we have this concept of tokens. And then we have the concept of training in a compute budget. We say what the training process is, you can be abstractly viewed as a way to compress the data sets.

• 18:49Speaker 01

  For example, we have actually a nice conversion ratio between billions of parameters and the amount of data that they generate. So just as a rule of thumb, right? Let's say each parameter is eight bits. Usually it's 16, right? We always talk about FP16 in our podcast with George Tots. But let's just say eight bits is one parameter. Then 175 billion parameters is 175 gigabytes, right? 1 billion parameters is 1 gigabyte. That is actually the definition of a gigabyte, which is 1 billion bytes.

• 19:20Speaker 01

  So that's that's that's super intuitive. And so therefore, a full point, falling point precision sixteen bits means one seventy-five billion parameters uses three hundred and fifty gigabytes to store parameters. That's how much memory that you technically need to do that inference and and just to load the model itself. Uh, and you know, most, most uh graphics cards will not even have that. Even even the professional grade like A one hundred cards would be like eighty gigabytes for a single one of them. So to fit three hundred and fifty gigabytes, you need to network all these things together with very high bandwidth. But they're trained on three thousand gigabytes of data. So that is three thousand gigabytes of data being compressed into three fifty gigabytes of data. And that is a, that is a form of compression because from there you can sort of, it is lossy compression,

• 20:03Speaker 01

  but it's a lossy compression in a way that learns how to decompress itself, such that when you ask it to spit out some facts, it spits out something in the approximate neighborhood of what you, what you started with. So I think that that's an interesting analogy. Some people don't like this idea that L L M's are databases, but it's super cool. Another very prominent description of this, I remember from last year, which was stable diffusion, which compresses all the images that it creates and all the images that it trains on into, I think, something like two to four gigabytes. Like it has knowledge of flying saucers and horses and humans and beaches and, you know, computers in images, all in like a downloadable file that you can host on your laptop, which is crazy.

• 20:49Alessio Fanelli?

  Yeah, I think people get very surprised by how many things the model knows, you know? And if you were putting the raw, again, going back to how much memory you need, if you put the raw data in memory, it would be like impossible to actually run. Um, but if you take, especially if you think about sixteen bit versus eight versus four and like all the quantization work, it's like compressing the compression, you know? Now, all of a sudden, you can put data sets and like knowledge and like was impossible to like put on a certain device. Like I can run the red pajama like two billion model on my phone and my phone is like sixty-four gigabyte of storage. If I were to put all the data that got into the training of it would be like.

• 21:32Alessio Fanelli?

  Running my phone way out of surge instead it's only like a few gigabytes um of model. So that's another interesting thing, especially as we think about using these models at the edge and how much stuff we can get there on time, but that's for another episode, quantization one on one.

• 21:47Speaker 01

  And for those who are again coming from a George Hatz episode, um at the around the one hour mark, he talks about compressing humanity, a person's consciousness or knowledge or all your life experiences. How much information, how many bits of information is that? He thinks it's two gigabytes and there's probably too much compression. You'll probably be a really bad copy of yourself. But there is a point at which you should probably be able to replicate yourself in as a digital twin, which is the whole mind uploading uh phenomenon that we discussed. Um, cool. All right. I think we we sort of maybe knocked on the uh on that a little bit uh too much. In fact, I want to basically verbally go through this chart of tokens in scale because when we talk about, we say things like billions of tokens, trillions of tokens, billions of parameters.

• 22:32Speaker 01

  It's it's just very, very big numbers that we don't know how to estimate. And this chart that we had from uh this this person um S. Rush on Twitter actually was super helpful to me. I was actually planning to to make something like this, but this guy already made it. So I'm just gonna quote him. Um, so maybe we'll just kind of go through the the token chart. So this is just memorized tokens in terms of orders of magnitude. Being able to do token math, I think, is super important and order of magnitude math is super important when it comes to deep learning, right? Because everything is like so big that like the the significant numbers don't really matter. It's just the order of magnitude really matters. Okay, so um ten to the power of zero, which is one, that's on the order of like a hello world, like like individual word tokens, right?

• 23:16Speaker 01

  And then ten to the order of one, which is like ten tokens would be like, you know, one sentence, one phrase or whatever like that. Ten to the power of two, that's that's one hundred, uh there will be blank space chorus apparently, which

• 23:28Alessio Fanelli?

  By Taylor Swift.

• 23:29Speaker 01

  Yeah, this is this is definitely a Swiftie because he talks about Taylor Swift later. Ten to the power of three, that's one thousand tokens, uh that's the Wikipedia article on Fermi estimation. Ten to the power of four, again, that's the Taylor Swift article on Wikipedia. Ten to the power of five, that is the GPT three paper itself, including the appendices. Ten to the power of six is one year of The New Yorker. So that's a big jump, right? Ten to the power of five to ten to the power of six. That's a one order of magnitude jump, but we've gone from a single paper to one year's worth of magazine. Ten to the power of seven is the whole of Encyclopedia Britannica. Ten to the power of eight is number of Reddit posts per month. Ten to the power of nine is English Wikipedia, all of Wikipedia. Ten to the power of ten is the number of WhatsApp messages per hour.

• 24:12Speaker 01

  Ten to the power of eleven is the number of published books per year. Not the number, the amount of tokens inside of published books per year. And then finally, we get to ten to the power of twelve, which is the order of magnitude in large language model data sets operated. And so that's ten to the power of twelve is one trillion.

• 24:29Alessio Fanelli?

  I was mostly surprised by the fact that Taylor Swift's Wikipedia page is longer than the Fermi estimation one. But the amount of data that you need for these models is uh is great. And yeah, the I think the one trillion tokens is kind of like the MPT thirty B that was released yesterday. So when we when we publish this episode, it's gonna be pretty new. But that's the amount of tokens that they used for there.

• 24:54Speaker 01

  Yes, we're trying to keep this episode current, even though it's an evergreen episode.

• 24:58Alessio Fanelli?

  Yeah, that's how much was used there, but it it just gives you, again, a way to reason about this, you know? So when you read online next time, it's like a trillion in tokens. You understand this like a hundred times all of English Wikipedia, which is a lot of text to to collect.

• 25:14Speaker 01

  Yeah.

• 25:15Alessio Fanelli?

  Took us years to get that online.

• 25:17Speaker 01

  There's a question about whether we're running out, right? Like, is there more orders of magnitude to this? And arguably, there are, but you know, that that is one of the issues that we'll discuss at the end. Just to recap, because I'm I'm so excited about this, because this to me is an evolution in terms of smaller models, right? GPT three, breakthrough that that got a lot of us interested, was three hundred billion tokens for one seven five billion parameters, right? That's the one point seven ratio from Kaplan. But then Lama is one point two trillion tokens for a seven billion per model. So one order of magnitude higher tokens for one order of magnitude lower parameters. That's crazy.

• 25:51Alessio Fanelli?

  It really is. And I think like, again, we already made this point, but it's like, we're just really early in terms of like, how much data we need, what model size we need, that I wouldn't preclude use cases today, based on the, on the size of these models. And also for an enterprise, the other thing, the analogy that I like with databases is that when you get a database, like, you know, Postgres, MongoDB, there's nothing in it. There's kind of like a cold start problem. Like, you install the database, you need to start putting stuff in it. Now, machine learning in the past used to be similar, where before you even start using it, you need to collect a lot of proprietary data. You train your own model, and then you start to do the prediction.

• 26:33Alessio Fanelli?

  The way it works now is that the foundation model labs and researchers like OpenAI, Anthropic and the likes, they've already done all the work for you. So you already got these models and meta, of course, we cannot miss one of our open source paladins. Once you get that at a company level, what you need to understand is, okay, I have this model that knows a lot. How do I prompt it and how do I fine tune it to make it good for my use case? And then run my own inference on that fine tune plus prompted model. So the scale of data that you need as an enterprise is like so much lower. Like, you don't need to collect billions and billions of like examples and tokens. The model, for example, for code, the model is already really good at code.

• 27:18Alessio Fanelli?

  You just need to give it, you know, dozens of examples, you know, maybe a hundred if you want to be really thorough. And you're gonna get really good performance. And I know, Sean, you were a fan of Karpathy's presentation at Build Conference.

• 27:32Speaker 01

  Yeah, it was really insightful and authoritative, I guess, coming from Andre. And recently, at the Microsoft Build Conference, Karpathy had a State of GPT talk that I think was very well received. And he is just really good at. Outlining the important things in a lot of the mess that we're have to wave through in order to understand language models. And so he basically outlined this famous slide, which is the GPT assistant training pipeline that outlined essentially four kinds of data sets that go into making something like chat GPT. And obviously, he's extremely authoritative on that. And I just think it's useful to have this in your head, this slide in your head. Again, refer to the show notes if you want to see the image.

• 28:15Speaker 01

  But the four data sets are raw internet, which is just the raw data sets that we pull from Common Crawl and Wikipedia and books and all that. And the second one, which is the demonstrations data set, where you're demonstrating ideal assistant responses, so basically prompt and response pairs. Comparisons, so basically comparing outputs between, you know, output A, output B, and seeing which one's better and just sort of reward modeling that on the language model. And then finally, doing reinforcement learning with prompts. And so those are different kinds of data that has been collected in different ways. There's different orders of magnitude of them as well. So there's trillions of low quality, large quantity data from the raw internet.

• 28:56Speaker 01

  And then there's tens of thousands of demonstrations, hundreds of thousands of comparisons, because that's easier to just choose A and B, and then tens of thousands of prompts. So that's basically the kinds of data sets that they think is representative of the chat GPT, like goes into building something like a chat GPT.

• 29:15Alessio Fanelli?

  Do we want to, at last, get to the data sets part? I know we had a little bit also on on instruction tuning, but we have a bunch of things to get through, so maybe you want to start there.

• 29:27Speaker 01

  I'll just quickly mention that instruction tuning was another paper coming out of OpenAI, and obviously a very important part. That is under a subset of the demonstration stuff that we're talking about. There is some debate from the Lima paper, L I M A, that about how much data we actually need. And so we interviewed Databricks, and I know Mike Conover is a very good friend of the pod. And they collected like fifteen thousand pieces of data to instruction tune themselves. Open Assistant from Yannick Kouture also collected tens of thousands of pieces of responses to to instruction tune on. And there's always something on some research that indicates that maybe, you know, it's not so much as we need. So Lima is something that we'll call out as interesting there. But yeah, we can move on to the major data sets because this is data sets 101.

• 30:12Speaker 01

  It's it doesn't have to be today in data sets, which can be a whole different podcast. So first, we start with Common Crawl. That is the OG, that is the bread and butter of every single data set, including the image ones that we'll talk about later. So it was founded by this guy Gil Elbaz and actually did some research on him and then you you fact checked my research. So this guy Gil is actually, he has his own Wikipedia page, so you can look him up. He basically started the predecessor to AdSense and sold it to Google and worked at Google for a while. And obviously, Google, and this is like in the late 1990s, early 2000s. And he saw firsthand how important, like, obviously, that Google's crawling was important was for Google.

• 30:53Speaker 01

  And basically quit Google and started Common Crawl, like basically quit Google and started empowering competitors to Google, which is a kind of interesting and scandalous. You did some research on this, what did you find?

• 31:04Alessio Fanelli?

  It's kind of like the beginning of the open web. So mostly the data was used for like surfacing pages. And then the whole big data thing kind of came to be. And one of Gil's ideas was like, OK, this data is not only good for like Google search, like indexing. There's a lot more work that you can do with it. We're like, I think 20 years into it almost. Yeah, 2008 was like when the first published the first data set. And it's one of the the biggest ones out there. So there's 3.1 billion web pages, 400 terabytes of content, 43 million hosts. Only 46% of the content is in English. Going back to our discussion before, it's run as a nonprofit. So there's a lot of a lot of unique things about it.

• 31:50Alessio Fanelli?

  I don't know if today you will see a nonprofit getting started just to provide massive amounts of data to the public, especially in this world of AI where everybody's hiding the data that they have. So maybe we do need it, but I'm not sure if we're gonna get it.

• 32:05Speaker 01

  I think it's actually super interesting how it got started itself. Like, this is obviously a very significant effort that all research, all NLP research and all language modeling is downstream of. They were started in 2008, but I think they were stealth for four years. Because the earliest example I can find of them releasing any of the data was in 2013. Sorry, they called it 2012, but the press release is 2013. And it looked like they used it to crawl once a year, crawl as far as they know all the internet. And now they crawl once every two months, right? Like, and it's just an interesting example of a nonprofit-driven approach that people don't really question or look into, but it's actually secretly driving all of the LLMs that we had today.

• 32:48Speaker 01

  There's some issues that are very well-known in Common Crawl. So Common Crawl actually says that they only cover a fraction of the web. It's a nonprofit, works on nonprofit resources, doesn't cover all of the web. And all of the language models are trained on Common Crawl. Therefore, our language models are not trained on all of the web. It is also biased sampling. It's definitely biased towards the United States. There's a lot of data quality issues. I think we talked about in some of our other episodes where the labels for some of the languages that Common Crawl has might be completely off. Like, I think something, someone mentioned about the Arabic issues being tagged. If you actually look at all those pages that are tagged as Arabic, they're not Arabic at all.

• 33:32Speaker 01

  So just like really, really basic coding errors. Or, you know, nobody checks these trillions of pages that are being crawled. So it's just really, really difficult. If you have robots.txt that blocks Google, you will also block Common Crawl. If the page is too big because of just the amount, share amount of data that's on it or the images on it, the pages are deleted or if they're duplicated on multiple sites because of spammers. It's just really, really difficult. Or if your pages are written in SPAs as JavaScript, because Common Crawl doesn't render to JavaScript. It only executes limited JavaScript. So, for example, much of Facebook is not under Common Crawl, right? And this is increasingly a problem with the closed gardens or the walled gardens of the internet, right?

• 34:17Speaker 01

  As information migrates from the open web into apps like Discord and Slack and Facebook, it's just not available to the Common Crawl.

• 34:27Alessio Fanelli?

  No, no, then Common Crawl has its own clean subset, so to speak, which is Google's C4, which was created during the training of their T5 model. Jonathan Frank on our podcast called it weirdly good. I think that kind of explains a lot about data sets. Sometimes they're good and we don't know why. And C4 is made by using a few heuristic. So it's about 10%, I think, of Common Crawl. Like, it's much smaller and it tries to filter Common Crawl by different ways. So one, it's using this open source thing called List of Dirty, Naughty, Obscene, and Otherwise Bad Words, which is 402 terms written in English and one emoji,

• 35:11Alessio Fanelli?

  which you can guess which emoji it is. The list was created by Shutterstock, actually. They basically wanted to avoid bad words to be auto-filled in their search. So they created this blacklist of things that they wouldn't auto-fill for the user, which I think is a funny way to end up being one of the foundation pieces of modern large language models. The other thing is there's a lot of stuff that didn't get filtered out, like certain piece of 4chan, like Kiwi Farms, things like that. Again, the episode is not about giving our judgment on the data sets. It's just about what's actually in them. And there was a Washington Post article that went through the whole list that we're gonna link in the show notes.

• 35:54Alessio Fanelli?

  The other thing that I found fascinating is that if you look at what domains are in the C4 data set, the patents.google.com website is like twice as large as the second one, which is Wikipedia. And then Wikipedia is like three times as larger as the third one, which is Scribd.com. So it's obviously like less than half of a percentage point, but it's still interesting to see very formal, kind of like text as the largest represented one. But yeah, C4 is another, obviously, core data set. So if you're looking to train your foundational models, that's one you should check out.

• 36:35Speaker 01

  Yeah, and in fact, a lot of models will list both Common Crawl and C4 as part of their data sets. And it will be a very, very heavy weight. It will be something like thirty to sixty percent of the amount of the token budget that they that they have. Which is super important. I mean, this is the this is the starting point of all our all of our language models. It's it's really, really fascinating. Going off of the list, basically trying to reintroduce where GPT three gets its data sets from. So if you pull up the GPT three paper, we're basically going in order of explaining which of the data sets and and telling a little bit of the story behind each of the data sets.

• 37:14Speaker 01

  Right? Wikipedia is the next data set, and obviously it's a very high quality data set because a lot of people have spent a lot of hours editing them. But except for the fact that Wikipedia itself has its own bias, right? I always have this fun fact that I that I pulled from from Google here. Seventy seven percent of Wikipedia articles are written by one percent of Wikipedia's editors. Meaning there's just an extreme, extreme skew in terms of the representation of the kind of people that write Wikipedia articles. And the decisions that are being made, right? In particular, this one guy, Steven Pruitt, because he's constantly made the rounds as the highest ranking Wikipedia editor editor.

• 37:56Speaker 01

  He's made over five million edits and has made one edit to one third of all English Wikipedia articles. So if you want basically to seriously affect machine learning data sets or large language models, you should edit Wikipedia, is what I'm saying.

• 38:09Alessio Fanelli?

  That's funny. Oh, it's not just Wikipedia, you know, what?

• 38:13Speaker 01

  Yeah, Reddit.

• 38:13Alessio Fanelli?

  Yeah, exactly. Webtext is another major data set that was also used in GPT two. It's about forty five million links and the the text of those webpages. The way they collected is basically scrape every URL from every Reddit submission up to December twenty seventeen. They add at least three upvotes just to make it somewhat, I guess, like less spammy. And they've removed all of Wikipedia from it. So there's no Wikipedia in this. It's all Reddit links that are not Wikipedia between twenty seventeen. And I forget when the release date of this was. And then they did another round of heuristic based cleaning, which again is like, who knows what that means?

• 38:55Alessio Fanelli?

  Which makes it kind of complicated to then scale these model, these data sets, right? Because if we knew the cleaning process, then we could say, oh, let's take all submissions from like 2010 and clean them up, but we don't actually have the step by step rules for some of them.

• 39:10Speaker 01

  That's it. It's been replicated by the Luther organization, right? Luther is something is one of the organizations that I was consistently shouted out by our guests as doing really good work in L L Ms. And so they've replicated webtext from OpenAI. So what OpenAI, as far as I know, did not release webtext, did not really release the rules around them. But the Luther organization created Open Web Text, which is an open source reproduction of webtext. And so we're going to leave in the show notes the link to Open Web Text two, which is the latest reproduction of webtext. We also mentioned the issue with Reddit, which is also a very hot topic right now with them shutting down their APIs. But let's keep moving on in terms of data sets. Next, we'll go on to books.

• 39:53Speaker 01

  So this is just basically, as far as I can tell, open source books, quote unquote, open source books. It's a data set of one hundred and ninety six thousand books in plain text for training large language models such as G P T. And it's also included in the pile, which we'll talk about later. But one of the interesting factors in the books data set is that apparently the copyright for some of them is not clear. So for example, when Jonathan Frankel and his and Abi trained M P T seven B, they actually got attacked by this person who was basically saying, how can M P T seven B or M C thirty B be commercially available or commercially licensed if the data set he was trained on had some issues with copyright, right?

• 40:36Speaker 01

  Like, and they they went back and forth on it. We linked to that discussion on our episode page. But it's an interesting thing, like if any of these are compromised, like if they're not copyright free books, then that's an issue. So we need to actually make sure that our data sets are clear such that our models are clear.

• 40:53Alessio Fanelli?

  Yeah, and then copyright expires. So hopefully the more time goes by, the more data we get.

• 40:59Speaker 01

  Yes, yes, yes. But like, unless Disney says they want more time, and then Disney will extend the copyright. But as far as I understand, I think Sherlock Holmes copyright recently expired. So a lot of people are making like Sherlock Holmes fanfic now. Because you can you can start using that. And I think maybe like Winnie the Pooh, maybe. I don't know. I I forget what the recent expiry was. But every year, there's there's there's an expiry day, and you know, a bunch of stuff becomes public domain, which is useful for, I guess, training and children's book storytelling. Okay, then we go from, everything we've covered so far is natural language data sets. But then we go from there to code data sets. And there's a lot of different code data sets. Salesforce CodeGen, I think I would point to as a predecessor to what I'm going to talk about.

• 41:45Speaker 01

  The most significant one in my mind is the stack from Luther. It's basically a scrape of the GitHub archive, six terabytes of permissively licensed code data. So the beautiful thing about code data sets is that most of the code on GitHub will include a license file telling you what license they have. And so you just kind of pick the set of licenses that you think are permissively licensed, and you can kind of just get them out. So the raw data was one hundred and two terabytes from one hundred and fifty three million repos, and that's three hundred and twenty times larger than Wikipedia. Then you clean them for file extensions. So for example, you can take out the images and take out blob files for for whatever reason. And then you're left with sixty nine terabytes.

• 42:25Speaker 01

  Then you clean out the licenses, and that's ninety percent of code that's thrown away because they're not permissively licensed or they have no license at all. Please, please, please. By the way, if you open source any code on GitHub, please add a license file so that issues like this don't don't come up. And so we end up with six terabytes of permissive code data that anybody can train on. It's great.

• 42:44Alessio Fanelli?

  Yep, and they didn't stop there with data sets. There's also the pile, which it's like my favorite dataset name. Sort of like a pile of data. It's 825 gigabytes from 22, basically like 22 smaller data sets combined. So in there you'll find PubMed data, archive, papers, GitHub data, the Free Law Project, the Ubuntu IRC, channel, Hacker News, YouTube. There's a bunch of things in there. The thing that we know about this from Stella and Luther from one of her tweets is that only about one third of the contents are duplicated. And we'll chat a little bit later about deduplication and whatnot.

• 43:28Alessio Fanelli?

  But they actually deliberately upsampled the original dataset to include some of the duplicated data. So even without that, the dataset is quite large, but at 825 gigabyte of kind of mixed data is another one of the core data sets people use.

• 43:45Speaker 01

  Yeah, totally. So obviously, we can keep going and going and going. There's no end to these data sets. Alessio just mentioned a whole bunch of interesting data sets that we don't have time to go into, right? But you can always research them if you want to. We're just trying to give a one-on-one. But no one-on-one would be complete without mention of other modalities of data sets, right? So I'll just mention two of them and then hopefully we can just kind of go on to issues. Otherwise, this will be a 10-hour lecture. Lion is the Luther for images. Lion stands for Large-Scale Artificial Intelligence Open Network. That is the sister organization that came out of Luther that Stability and Iman Mostak worked with to create Stable Diffusion, right? So this is all the predecessors of Stable Diffusion. And actually, it came out of COVID, right?

• 44:30Speaker 01

  Everyone was kind of bored, sitting at home and looking at DALL-E and going, why don't we have an open source replication of DALL-E? And so the first thing that you do... When trying to replicate a model is you go collect data, right? And so Lion in 2021 collected 400 million images. From where? From Common Crawl, right? Like, Common Crawl keeps coming up. It's the OG. It's so goaded. And then in 2022, they released Lion 5B, which is 5 billion images for data sets. They've also released an aesthetic subset of the Lion 5B data set. And basically, there's a lot of filtering when it comes to images. Obviously, there's a lot of porn that you need to filter out and not say for work. And also copyrighted images, right? Like, you have to make some decisions around, do you want images of Spider-Man?

• 45:16Speaker 01

  And I was just at the Figma conference yesterday where Figma was very, very, where Adobe was very proudly showing off Adobe Firefly, which has no idea what Spider-Man is. And for them, it's a feature because for the kind of companies that are using Adobe to create images, they need to not run into trouble with Disney. So it's fantastic. It's very hard to get images of Spider-Man out of your image data set. And then Whisper is the other one. So instead of images, we need transcription, like ASR. So Whisper, if you look at the paper from OpenAI, again, is LibriSpeech, Common Voice, Vox Forge, Switchboard, and Fisher Corpus. That's all I know about them.

• 45:55Alessio Fanelli?

  There's a lot. Whisper is so good. I feel like nobody's saying, let's do another Whisper. You know, I think the NLP and text part is the most active one right now.

• 46:04Speaker 01

  Yeah, so for those who want to like research more data sets, I think the best place to go is probably Hugging Face Hub. A lot of people for training data sets, they also go to Kaggle. I maintain a list of useful big data sets on my repo of useful resources. So I'll link that in the show notes. But yeah, I think that's going to be the high-speed tour over all data sets. So again, we'll come back to like the key question, like, why are data sets important? First of all, like, we have to figure out, we have to know like the fundamentals about data sets to figure out whether or not we're running out of it or how we're using it. But also fundamentally, I'm a little bit concerned about the number of data set producers to the number, the ratio of data set consumers, right? Everyone wants the glory of training language models and saying like, I trained this model that does X.

• 46:49Speaker 01

  But not that many people are interested in cleaning data, right? Like, which is a common meme in the enterprise sort of data science world, the machine learning world, that everyone wants to be data scientists, no one wants to be a data janitor.

• 47:01Alessio Fanelli?

  Right.

• 47:02Speaker 01

  So I don't know if you've like run into any of these conversations in your line of work.

• 47:06Alessio Fanelli?

  Yeah, that that makes sense. I think like especially now, there's a lot of pressure on companies to use AI. And everybody wants to use AI. Nobody wants to do the work of getting the data ready to make AI useful for their company. So it definitely resonates.

• 47:20Speaker 01

  Yeah, and companies that are sitting on top of a lot of data are actually realizing that they are sitting a lot of gold. I call this data is the new oil part two, right? Bloomberg recently came out with Bloomberg GPT. Because guess what? They have proprietary license, they have the license on a few decades worth of Bloomberg financial news reporting. So if you ever need to generate or do any reasoning around financial data, Bloomberg has the best data set by far. And it's closed source and you have to pay Bloomberg to do it, right? Notion, if you think about it, Notion has twenty two million users and all their users use their Notion as a knowledge store, right? So Notion has a tricky issue because Notion doesn't own their customers' data, they just hold it for them.

• 48:05Speaker 01

  And so they don't actually have the right to train on them. And and so it's like this tricky dance between them. But if you enable, for example, customers to fine tune on their own company data within Notion, just with a single click, that becomes extremely valuable as well. So individual companies are realizing their modes. Just yesterday, the Stack Overflow CEO was saying like, you know, they are joining Reddit in in terms of shutting down their API access because they realize they have good data as well and they want to be paid. So and that's a a part of the issues that we'll go into. But finally, we should also cover the counterculture movement for open data sets, like open replication of data sets. And Yannick Kultur, I think, is is one of the leaders here, as well as Luther, for reproducing the instruction tuning data sets that people will need to train their own chat GPT.

• 48:50Speaker 01

  It's pretty interesting that YouTube influencers are coming to the rescue of open source. Because there's no other source of influence that is powerful enough to compete with OpenAI except for YouTube.

• 49:01Alessio Fanelli?

  Yeah, I think that was that was fair, Sean. Maybe we want to run through some of the issues and kind of things to keep in mind for the data sets.

• 49:09Speaker 01

  Yes. Okay, so I put this first because it's also fairly current. There's always this issue of data set quality, right? So when I ask researchers like, hey, why do you think we're running out of data? There's obviously hundreds of petabytes of data produced every single day. Why don't we just use that as data? And the typical answer to me is, well, that data is low quality, right? Which is true, and you need diverse sets of data as well. So for example, one of Stella's tweets about how we're not running out of data says, oh, there's like, you know, hundreds of terabytes of legal filings that are generated every single year. But all those legal filings have the same format. You don't learn very much by going over, you know, one thousand pages of parking fines or whatever, right?

• 49:52Speaker 01

  Like, they have to be unique and actually useful and and and high information. And so curating those data sets and making them useful is emerging, is becoming more and more important. And the way that we know this is actually from something that happened recently, which is Microsoft started training this small language model, phi one, that is one point three billion parameters. So not even a large model anymore, it's just one billion parameters. So this is the size of G P T two, on a data set size of seven billion tokens. So again, way smaller, like this is a seven to one ratio, not two hundred to one, way, way, way smaller. And it's basically comparable in terms of the benchmarks to state of the art models that are

• 50:37Speaker 01

  something like ten times bigger, right? Like, so it's comparable on human eval, because it's a code gen model. It's comparable on human eval and M B P P, which is their own benchmarks. And so it's just like, very interesting. I think that's something that's an emerging area of research, like how to improve the quality of data such that you train a smaller models with fewer tokens, right? That is the final step of this evolution. But also Falcon forty B, the model that came out of the U A, the U A E, that is now the top open source model. That was also on a proprietary new data set that the U A E government collected as well. And so that's just super interesting that you are not competing on size anymore, you're competing on quality.

• 51:16Alessio Fanelli?

  Yeah. The other issue that we mentioned obviously is copyright and privacy. We're not going to go over, you know, the same issues again, but there's a couple of interesting things going on. So there's the stable diffusion litigation, which is from the same council that is doing the GitHub Copilot litigation. Basically, the whole idea is that, hey, this is not really fair use. You cannot really use my data to do this. So they're basically suing the model trainers on whether or not they're, they should be able to use their work, even though they didn't specifically license it, you know, to not be used. There's kind of like an ethical question there. The interesting thing here is that now some of the AI providers are siding on the user's behalf.

• 51:58Alessio Fanelli?

  So for example, if you use GitHub Copilot and Copilot generates GPL code, which is licensed and in theory you should not. It's basically like a contamination license. If you use GPL code in your codebase, all of your codebase becomes GPL and you need to open source it. I actually wrote a very long post on the history of open source license, which I'll link in the, in the show notes. But basically GitHub is saying, we'll literally pay for your lawyers and like we'll send lawyers to fight on your behalf. So it's really interesting how the risk piece is not being clear by saying, hey, we definitely 100% not use copyright data. They're basically saying, maybe we do, and if we did, we mess up and we're going to pay for it.

• 52:42Alessio Fanelli?

  But ideally we're not doing it. And there's also different articles out there that are mentioned before by the Washington Post and companies like that, where a lot of the training data comes from newspapers, comes from magazines, and people have not always opted in to having that as a training of the model. So some of the work then comes out in the, in the inference. So yeah, that's kind of like another. Interesting piece of development. And again, if you're training your own model, you should be really careful. If you're using an off-the-shelf model, you should also be somewhat careful, but it seems like there's a lot more insurance on that.

• 53:19Speaker 01

  Yeah, licensing issues also come in the form of terms of use, which is not an official license, but it's something you agree to when you use services. So OpenAI has this very famous clause in their Terms of Service, which basically states that you can't use OpenAI output as input for your training models, which is exactly what the Alpaca and Vicunia students did in Stanford to train their models that now compete with ChatGPT. So this is why in our conversation with Mike Conover, he talked about, he was very excited about Red Pajama, which is an open source replication of Llama. Because Llama also has similar licensing issues, like Llama doesn't allow you to use it commercially, right?

• 54:02Speaker 01

  All these licensing issues, copyright issues, permissions issues are emerging areas that are being litigated. People are coming up with different ways to license this stuff. So for example, Hugging Face has this real license, responsible AI license that is different from MIT, different from Apache two. And that's the license that Stable Diffusion is under, but it has never been litigated in a court of law. It's not accepted by the OSI Institute as open source. So it's just unclear, like, can you use it? You have to consult your lawyer, quote unquote, which is a real cop out to basically say, nobody knows until some judge rules when a case is brought up. So that's, that's licensing thing.

• 54:41Alessio Fanelli?

  There's a lot of work there too, like Hugging Face has built like a PI removal pipeline for like their development. You can also go on Hugging Face and check if your information is in the stack and whatnot, but again, we're gonna put all this in the show notes. We're not gonna bore you live on all the details.

• 54:57Speaker 01

  Yeah, just, just for, so so people know, we have seventeen pages of show notes that we've been collecting for two months. So.

• 55:04Speaker 01

  It's just a little bit of a of a crazy thing to compress into one hour, but let's try to do that. All right, the next few issues are to do with quality as well. So we'll talk about duplication and filtering. And basically, like the amount of duplicates that uh there have been studies done on the amount of duplicates in open web text in C four. Uh and there's still quite a significant amount, like it's pretty interesting because every time you duplicate something, you're exposing the model to that set of raw data again without knowing it. And therefore, it's basically going to try to memorize that text way more because it's just been exposed to it a lot more. And if you're just basically wasting compute because that's not the kind of training that you want.

• 55:43Speaker 01

  So there's a bunch of research here that is uh reflective of studying these data sets and identifying those those duplicates and removing them. But just the impact of of this uh is is is really interesting. So we have this paper here that basically uh states that a sequence that is present ten times in the training data is on average generated one thousand times more often than a sequence that is present only once. So basically, there is a disproportionate amount of weight that is placed on repeated information, which makes sense, right? Like you're if you show a model repeated information, it's going to overfit to that set of information. But it's only order of ten of one thousand times more frequent. And that is a concern when trying to train general purpose models.

• 56:22Alessio Fanelli?

  The other thing is contamination, especially related to benchmarks. So for example, one of the things that G P T four showed is like, oh, they do very well on code forces, code puzzles. And but you'll see that the models, for example, does ten out of ten on the pre twenty twenty one problems. And then on the more recent ones, post-training code update, it does zero out of ten. It's basically, I think I looked at the scoring, it's like worse than a person doing it wrong on purpose. Which is, you know, it's actually pretty impressive. So when you're training a model, understanding what goes into it also helps you understand how to benchmark it. Like if you're benchmarking it against things that the model has learned, it's not super helpful.

• 57:06Alessio Fanelli?

  So that's another thing to keep in mind.

• 57:08Speaker 01

  Yeah, and this is why also, you know, releasing models and showing how you evaluate your models uh or releasing data sets, right? Because that's the topic of this episode is very important. So when Falcon forty B came out, they went right to the top of the Hugging Face leaderboard on the benchmarks that Hugging Face leaderboard provides. But we actually don't know if Falcon forty B's data sets were contaminated with the things that they evaluated on, right? If you just copy paste the The exact results of the test that you're going to be tested on, of course, you're going to do extremely well on it, right? There's actually a current thread of people now that the the model at least has been released. The dataset has not been released, but the model, as far as I can tell, has been released. People are replicating those evals and finding that it actually is falling shorter than than claimed.

• 57:53Speaker 01

  So I haven't done it myself, so I can't really claim to know one way or the other. But I think it's just one of those things where you have to have some healthy skepticism, because there's a lot of people trying to gain benchmarks, and the easiest way to gain benchmarks is to conveniently forget to remove the benchmarks from your datasets. Final point, because we have, we're running out of time, dataset imbalance. Obviously, we're all talking in English, the world is very English centric, but there's other languages in the world. And we already talked about the tokenization issues, which will cost more, right? Because all these APIs are charged based on the number of tokens generated. And if your tokenizer uses more tokens per language, then you will you will cost more to to generate those language. Some amount of that is honestly not the fault of any anybody, because the language is just more complex, right?

• 58:38Speaker 01

  As a Chinese speaker, I'm well aware that the average Chinese person is is required to learn ten thousand Chinese words to be considered literate, which is absurd.

• 58:48Alessio Fanelli?

  Right.

• 58:49Speaker 01

  But there are, I mean, obviously, China has is making a lot of progress on on language modeling as well. So there's actually a lot of papers coming out of China for Chinese datasets and English to Chinese conversion datasets, right? And which is a lot of the the original translation benchmarks. So there's some Chinese datasets that we've outlined here, the CMRC, DoReader and CHID, all of which we'll link in the show notes. Is there anything else that you wanted to comment on in particular?

• 59:12Alessio Fanelli?

  No, I think that's a lot of it. Actually, one of my friends and former co-founder, Andrea, he's working on an Italian language model. So I'm curious to see more of them come online. And I I think like in the episode we're gonna release with the Practical AI crossover one, we talked about how language is also used differently in different countries. So some are very oral driven, so like a language model that is only text is not as important. So the voice data is also crucial. So, well, again, dataset's two oh one, we'll get back to it, but I think we're already at one hour ten, so I think we covered a lot.

• 59:50Speaker 01

  Yeah, yeah. Hopefully, that was a good overview, especially for people who like keep hearing about things like common crawl, keep hearing things about contamination, and keep hearing things about tokens even. And this is a ground up reintroduction to these concepts. We we are recording these one on one episodes in order to be evergreen, right? That you can listen to this a year from now and hopefully it's still not be out of date.

• 1:00:12Speaker 01

  Who knows?

• 1:00:13Alessio Fanelli?

  Hopefully, we can keep going at this pace.

• 1:00:16Speaker 01

  But you know, I really want to like emphasize, yeah, data sets are great. Let's spend more time applauding data set creators because we are downstream of them.

• 1:00:22Alessio Fanelli?

  Yeah, if you want to train on the latest phase podcast, please go for it. We got all the transcriptions in the show notes, so we're doing our part.

• 1:00:31Speaker 01

  We're doing our part.

• 1:00:32Alessio Fanelli?

  All right, everyone. Thanks for listening.

• 1:00:54Speaker 01

  All right, goodbye.