[Cognitive Revolution] The Tiny Model Revolution with Ronen Eldan and Yuanzhi Li of Microsoft Research

Thanks to the over 1m people that have checked out the Rise of the AI Engineer https://twitter.com/swyx/status/1674826723068903425. It’s a long July 4 weekend in the US, and we’re celebrating with a podcast feed swap! We’ve been big fans of Nathan Labenz and Erik Torenberg’s work at the Cognitive Revolution podcast https://www.cognitiverevolution.ai/ for a while, which started around the same time as we did and has done an incredible job of hosting discussions with top researchers and thinkers in the field, with a wide range of topics across computer vision https://www.cognitiverevolution.ai/e6-the-computer-vision-revolution-with-junnan-li-and-dongxu-li-of-blip-and-blip2/ (a special focus thanks to Nathan’s work at Waymark), GPT-4 https://www.cognitiverevolution.ai/e11-openais-gpt-4-discussion-with-nathan-labenz-and-erik-torenberg/ (with exceptional insight due to Nathan’s time on the GPT-4 “red team https://twitter.com/labenz/status/1660765244363350018”), healthcare/medicine/biotech (Harvard Medical School https://www.cognitiverevolution.ai/e24-the-ai-revolution-in-medicine-with-dr-isaac-kohane-of-harvard-medical-school/, Med-PaLM https://www.cognitiverevolution.ai/e27-googles-med-palm-and-med-palm2-with-vivek-natarajan/, Tanishq Abraham https://www.cognitiverevolution.ai/e38-the-virtual-biopsy-revolution-with-dr-tanishq-mathew-abraham-part-2-of-2/, Neal Khosla https://www.cognitiverevolution.ai/e25-revolutionizing-patient-care-with-neal-khosla-of-curai-health/), investing and tech strategy (Sarah Guo, Elad Gil https://www.cognitiverevolution.ai/e21-vc-insights-on-investing-in-artificial-intelligence-with-sarah-guo-and-elad-gil-of-no-priors-podcast/, Emad Mostaque, Sam Lessin https://www.cognitiverevolution.ai/e35-stability-ais-emad-mostaque-and-slow-ventures-sam-lessin-discuss-investing-in-ai/), safety https://www.cognitiverevolution.ai/e26-bonus-episode-connor-leahy-on-agi-gpt-4-and-cognitive-emulation-w-fli-podcast/ and policy https://www.cognitiverevolution.ai/e16-pausing-the-ai-revolution-with-technologist-jaan-tallinn/, curators https://www.cognitiverevolution.ai/e23-scouting-the-ai-revolution-with-robert-scoble-and-bens-bites-creator-ben-tossell/ and influencers https://www.cognitiverevolution.ai/e22-helping-businesses-use-ai-with-rachel-woods-of-the-ai-exchange/ and exceptional AI founders (Josh Browder https://www.cognitiverevolution.ai/e34-the-consumer-rights-revolution-with-joshua-browder-of-donotpay/, Eugenia Kuyda https://www.cognitiverevolution.ai/e3-the-empathy-revolution-with-eugenia-kuyda-of-replika/, Flo Crivello https://www.cognitiverevolution.ai/e9-the-ai-assistant-revolution-with-flo-crivello-of-lindyai/, Suhail Doshi https://www.cognitiverevolution.ai/e1-the-pixel-revolution-with-playgroundais-suhail-doshi/, Jungwon Byun https://www.cognitiverevolution.ai/e14-the-reasoning-revolution-with-oughts-jungwon-byun-and-andreas-stuhlmuller/, Raza Habib https://www.cognitiverevolution.ai/e20-the-great-implementation-with-raza-habib-ceo-of-humanloop/, Mahmoud Felfel https://www.cognitiverevolution.ai/e10-the-ai-voice-revolution-with-mahmoud-felfel-of-playht/, Andrew Feldman https://www.cognitiverevolution.ai/e29-the-ai-chip-revolution-with-andrew-feldman-of-cerebras/, Matt Welsh https://www.cognitiverevolution.ai/e19-the-ai-agent-revolution-with-matt-welsh-of-fixieai/, Anton Troynikov https://www.cognitiverevolution.ai/e5-the-embedding-revolution-anton-troynikov-on-chroma-stable-attribution-and-future-of-ai/, Aravind Srinivas https://www.cognitiverevolution.ai/e7-in-search-of-truth-with-aravind-srinivas-of-perplexity-ai/). If Latent Space is for AI Engineers https://www.latent.space/p/ai-engineer, then Cognitive Revolution covers the much broader field of AI in tech, business and society at large, with a longer runtime to go deep on research papers like TinyStories. We hope you love this episode as much as we do, and check out CogRev wherever fine podcasts are sold! Subscribe to the Cognitive Revolution on: * Website https://cognitiverevolution.substack.com/ * Apple Podcasts https://podcasts.apple.com/us/podcast/id1669813431?mt=2&ls=1 * Spotify https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk?si=ba07b343ee374a04 * Youtube https://www.youtube.com/@CognitiveRevolutionPodcast Good Data is All You Need The work of Ronen and Yuanzhi echoes a broader theme emerging in the midgame of 2023: * Falcon-40B (trained on 1T tokens) outperformed https://twitter.com/johannes_hage/status/1662119769045188610 LLaMA-65B (trained on 1.4T tokens), primarily due to the RefinedWeb Dataset https://arxiv.org/abs/2306.01116 that runs CommonCrawl through extensive preprocessing and cleaning in their MacroData Refinement pipeline. * UC Berkeley LMSYS’s Vicuna-13B is near GPT-3.5/Bard quality https://lmsys.org/blog/2023-03-30-vicuna/ at a tenth of their size, thanks to fine-tuning from 70k user-highlighted ChatGPT conversations (indicating some amount of quality). * Replit’s finetuned 2.7B model outperforms the 12B OpenAI Codex model https://www.latent.space/p/reza-shabani#details based on HumanEval, thanks to high quality data from Replit users The path to smaller models leans on better data (and tokenization!), whether from cleaning, from user feedback, or from synthetic data generation https://twitter.com/karpathy/status/1671587087542530049?s=20, i.e. finetuning high quality on outputs from larger models. TinyStories and Phi-1 are the strongest new entries in that line of work, and we hope you’ll pick through the show notes to read up further. Show Notes * TinyStories (Apr 2023) * Paper: TinyStories: How Small Can Language Models Be and Still Speak Coherent English? https://arxiv.org/abs/2305.07759 * Internal presentation with Sebastien Bubeck at MSR https://www.youtube.com/watch?v=iNhrW0Nt7zs&t=538s * Twitter thread from Ronen Eldan https://twitter.com/EldanRonen/status/1658321669407248387?s=20 * Will future LLMs be based almost entirely on synthetic training data? In a new paper, we introduce TinyStories, a dataset of short stories generated by GPT-3.5&4. We use it to train tiny LMs (* Phi-1 (Jun 2023) * Paper: Textbooks are all you need https://arxiv.org/abs/2306.11644 (HN discussion https://news.ycombinator.com/item?id=36413768) * Twitter announcement from Sebastien Bubeck https://twitter.com/SebastienBubeck/status/1671326369626853376: * phi-1 achieves 51% on HumanEval w. only 1.3B parameters & 7B tokens training dataset and 8 A100s x 4 days = 800 A100-hours https://twitter.com/teortaxesTex/status/1673108450941759490?s=20. Any other >50% HumanEval model is >1000x bigger (e.g., WizardCoder from last week is 10x in model size and 100x in dataset size). This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:00Speaker 00

  Hello, hello, it's Six again. It's July 1st, 2023. Happy Q3 to those of you who celebrate. It is one day after we launched our blog post and conference, I guess, on the rise of the AI engineer. For those of you who only joined us for the podcast, the Latent Space newsletter actually preceded the podcast. And yesterday, we put out our first newsletter post in a long time. So if you want to just head over to Latin.space, you can check out the post about why our thesis is about the AI engineer and why we think that this is going to be a growing category, and why we're putting on our first conference in October in San Francisco.

• 0:39Speaker 00

  Join us if you can. CFP's sponsorships and attendee slots are open as of yesterday. So today, it's basically the July 4th weekend. Most people are going to take Monday off, and Tuesday is July 4th. So we figured we would do a double sort of podcast swap with some of our favorite AI podcasts that we love and enjoy and wanted to share with you again, especially highlighting some of the things that we liked about their podcasts, and then they're doing the same for us on their feeds. So today, we're featuring Nathan Labenz of the Cognitive Revolution Podcast. They started around the same time as us, but then they've just went way harder than us, done twice the number of episodes, and have covered way more than us

• 1:21Speaker 00

  in terms of computer vision, healthcare, investing in tech, safety and policy, curators, influencers, as well as exceptional AI founders, some of whom we also hope to have on our podcast in some time in the future. And the story that we've picked out, or the episode that we've picked out, is a recent one, but I think is extremely important as a theme for 2023, which is Tiny Stories from Microsoft Research, led by Ronan Eldon and Yuyan Ji-Li. Since they actually published Tiny Stories, they actually also published 5.1, which is a large language model. It's 1.3 billion parameters. It's only trained on 800, 100 hours, so that's about a couple thousand dollars of training.

• 2:06Speaker 00

  And it scores above 50% on human eval, which As we all know, from the Replit episode that we did, uh, is an imperfect benchmark, but it is, as far as people are concerned, the industry standard benchmark for code models. And it's basically performing at an equivalent level of models that are ten times its size and one hundred times its data set. Um, and so that's an interesting phenomenon that basically points to data quality as the new dimension for which model trainers are optimizing for. We've talked about various dimensions of scaling, from scaling the number of parameters to scaling the data set size, to scaling the amount of compute that you spend.

• 2:44Speaker 00

  Um, but this is the first time where it's a little bit harder to talk about this because there's no, it's really hard to quantify, but here we're scaling quality for the same amount of length or the same number of tokens that are invested in the training process. Yeah, but this interview is about Tiny Stories, which is the earlier work that informed the phi one model. And Tiny Stories is very endearing. It's only focused on the reading level of like a three to four year old. But it actually makes a lot of interesting points. The thing that model researchers pick up on is that it uses synthetic data sets that were generated by G P T three and four. But for practitioners, I think the more useful and interesting or mind blowing thing is that it can generate

• 3:29Speaker 00

  very coherent stories from a tiny language models, like I'm talking about less than ten million parameters. Um, and they have this comparison, which I'm going to stick in the show notes, where they compare a story that was generated by Tiny Stories model, there's three million parameters, comparing it to G P T two X L, which is one point five billion parameters models. So, so it's beating a five hundred times larger model. Um, because it's focused on this kind of domain. And this has a lot of interesting implications on interpretability, because it's a much smaller model. We can visualize what's going on in the weights rather than having a big mass where we don't know anything. But also, just for the future of domain-specific models.

• 4:12Speaker 00

  I also like the analogy that they make in the podcast interview, which is that this kind of training models how humans learn, which is first you learn to speak like a child, and then you learn adult language, or to talk like an adult. As opposed to how we train large models today, which is we just throw it in a deep end and just expect it to learn all of Common Crawl and to speak like a professor, to speak like a 4chan troll, to speak like, you know, textbook authors. And it doesn't really have this progression or formative moment in its training, where it's training on more conceptual core concepts that are day-to-day. And finally, I think it also has really interesting implications

• 4:57Speaker 00

  on the discussion around emergence. Because emergence typically requires, let's say, in the order of a billion parameters, 20 billion parameters, depending who you ask. But here we have solid reasoning on the order of 1 to 10 million parameters. Million, not billion. And it's shocking. It is unexpected, I guess. And the authors indulge us a little bit on how model architecture actually enables some of these patterns. So overall, a fantastic interview, a fantastic piece of work. And the Cogriff Podcast is probably one of the only podcasts where this kind of in-depth discussion is available in podcast format. So highly, highly recommended. We hope to continue to be friends with the Cogriff Podcast in the future

• 5:41Speaker 00

  and hopefully collaborate on future issues. So, see you tomorrow.

• 5:45Speaker 01

  One of the most important abilities for generative models is to be able to speak coherent English. Her mom didn't let her have a dog, so she asked for a... And when you try to autocomplete this, now, you know, the most common noun that you've seen so far, also the most, the proximate one is dog, not cat. Dog already appears twice in the sentence. Even GPT-2 XL, which has 1.5 billion parameters, its most likely completion is still dog.

• 6:28Speaker 02

  I'm just gonna read one tiny story, just straight out of the paper, because I think that will help people understand, like, what this dataset, you know, ultimately is. Tom has a big pot of soup. He wants to share it with Jane. Jane takes a spoonful of soup, but then she makes a face. The soup is, that's the prompt. And then you show a completion, very bitter. She does not like it. She says, I don't like this soup. It is too bitter. He looks around the kitchen and finds some bread and cheese. He puts them on the table and says, here, Jane, you can have some bread and cheese. They are not bitter. They are sweet and yummy. Jane is happy. She says, thank you, Tom. You are a good friend. I like bread and cheese. They are not bitter. Hello and welcome to The Cognitive Revolution,

• 7:13Speaker 02

  where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas and together, we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today's episode is great for anyone who really wants to deepen their understanding of and intuition for how language models really work. Certainly, as measured by how much I learned in the course of the conversation, it's one of our very best. Our guests, Ronan Eldan and Yuan Junli of Microsoft Research,

• 7:55Speaker 02

  have created a small natural language dataset called Tiny Stories, which they designed to reflect the full richness of natural language, while still being small and conceptually simple enough to support research with modest compute budgets.

• 8:10Speaker 02

  They did this by using GPT-4 to systematically create 1 million children's stories, using only words that an advanced three-year-old could be expected to know. Dataset in hand, they then began to explore a number of aspects of language model performance, behavior, and mechanism by training a series of models that range in size from just 1 million to a maximum of 33 million parameters, still just 2% the size of GPT-2.

• 8:39Speaker 02

  They then used these small models to explore the development of language model reasoning abilities, identifying so-called logical primitives, beginning with a basic understanding of grammar, followed by the learning of facts, and then eventually adding certain logical microskills, such as negation and exclusion.

• 8:58Speaker 02

  These findings create the perfect context in which to discuss the tricky and often controversial topic of emergence, as well as to compare and contrast how large language models learn with how human children learn, and to explain how the differences that we see across language models and children do in fact make some sense, given the different incentive structures in play in each case.

• 9:22Speaker 02

  They also did some great interpretability work in this paper, and I really relished the chance to get into all three areas that they explore. First, they look at the trade-offs between the number of layers in a transformer, which to a large extent governs the number of logical leaps that a model can make, with, on the other hand, the width of a layer, which seems to determine how many facts the model can store. They also identify attention heads with distinct roles, including distance heads, which simply reflect the distance between tokens, and which look almost exactly like the alibi scheme, which is now powering long context models such as Claude 100K and the recent Mosaic ML 65K release. And then, on the other hand, semantic attention heads,

• 10:07Speaker 02

  which focus on meaning. That there should exist such completely different attention heads within a single model, and that an alibi scheme should emerge in the wild, is really, to me, mind-blowing.

• 10:20Speaker 02

  Finally, they examine the role of individual neurons, finding that many of their small model neurons do in fact correspond to human interpretable concepts. We close the conversation by zooming out and discussing why small models are more interpretable than large models, the challenges inherent in attempting to extend this work to larger scale models, and why controlling language models might end up being more like horseback riding than microbiology. Throughout this conversation, I was really struck by two things. First, it seems to me that we've only scratched the surface of the potential for curriculum learning approaches. I fully expect that we'll start to see ever more sophisticated approaches, which use specific data sets to layer on specific skills in a strategic, progressive manner,

• 11:05Speaker 02

  creating highly specialized, small-scale models that can solve specific problems extremely efficiently. Second, the value of these toy models for developing understanding really is tremendous. If I could make just one suggestion to listeners, if you want to get the absolute most out of this episode, it would be to visit the Hugging Face website and try playing around with some of the bigger models that they've released. The very biggest are still only 33 million parameters, which means that they can load easily and run quickly right from the Hugging Face model page. If you do that, as I did in preparation for this episode, you will actually have the chance to explore a lot of the concepts, and you can set up your own little experiments to test the reasoning ability of these models.

• 11:49Speaker 02

  I guarantee that you will come away with a deeper understanding that you will retain for longer and much better. And if you do find anything interesting, I would love to hear about it. So please do reach out to me via our email, [email protected], or on Twitter, where you can always DM me, @LeBenz. Now, I hope you enjoy this elucidating conversation with Ronan Eldan and Yuanzhi Li of Microsoft Research.

• 12:16Speaker 02

  Ronan Eldan and Yuan Jun Li, welcome to The Cognitive Revolution.

• 12:21Speaker 01

  Thank you so much. We're super happy to be here.

• 12:23Speaker 02

  You guys have just published this paper called Tiny Stories, and I think it's a really fascinating bit of research on multiple levels, so I'm really kind of excited to dive into it with you guys. It touches on a bunch of different themes, including some of the hot-button themes that we'll get to around emergent capabilities and reasoning, and you guys are studying this in a very unique way that makes the problem, I think, more tractable and more approachable, hopefully for our listeners as well. So I'm really excited to introduce this work to them. Maybe just for starters, can you give me a little bit of an introduction to what inspired the Tiny Stories project?

• 13:04Speaker 01

  I guess I'm kind of new to LLMs or deep learning in general. I come from pure math. And when I started looking into architectures, trying to understand what those models are actually doing, how to improve them, et cetera, et cetera, I got very quickly, I got very, very frustrated because it's very easy to come up with ideas, but in order to actually check whether an idea is good, almost always you need to do an experiment that involves a lot of compute. It's just very, very hard to check things.

• 13:50Speaker 01

  You need to, you know, you can either train small models, which basically don't do much in terms of, you know, they don't actually generate text that sounds coherent. You can train like maybe a BERT-sized model, and then it'll do something on some downstream tasks. But whatever it does, doesn't, doesn't look much like what those LLMs are doing. If you want to really get an LLM experience, you need to do an experiment with a lot of compute that involves, you know, tons of GPUs, et cetera.

• 14:28Speaker 01

  So, you know, for me, it was just a way to address the frustration of not being able to get any insights without, you know, having to do a large experiment.

• 14:42Speaker 02

  And so the main way that you have accomplished that, if I understand correctly, is by kind of narrowing the conceptual space of what both the data set, you know, contains and then obviously what the model is trained to do, right? Like, instead of taking, uh, you know, a small cup out of the whole ocean of mixed up of everything language, you've created a kind of, we're gonna, we're gonna tackle one, you know, very consistent type of input.

• 15:13Speaker 01

  That's fair. I guess we should mention, there have been many attempts to come up with a synthetic or, or non-synthetic, a smaller data set that has all those elements that those large language corpora have, right? So, you know, in language, you have all sorts of elements. You have a lot of facts, you have, so first of all, you have grammar and vocabulary, right? Those are the obvious things you have in language. But then you have facts, you have reasoning that you can infer from those texts, and there's like also many layers of reasoning and, and, you know, there's, I guess there

• 15:58Speaker 01

  are many capabilities involved in being able to parse those data sets. So, you know, our initial motivation was to come up with a data set that has all these qualitative elements, but on the other hand, is just not as massive as those large language corpora.

• 16:21Speaker 01

  Right, and you know, you, Andrew, and I had, uh, so first of all, there, as I said, there are many uh synthetic data sets out there. Some of them, I think, reflect in a, in a pretty good way, certain aspects of language, such as reasoning, or facts, or grammar, or stuff like that. But we felt like there is no single data set that has all those dimensions together, which are all integrated into something which is not too large, right? And we felt like in order to understand, in order to gain insights about L L M's,

• 17:04Speaker 01

  we need a data set that has all those elements.

• 17:09Speaker 03

  Yeah, I was just going to add that I, I also came from a D M A background. And, uh, so I was doing theory of machine learning since, uh, maybe seven or eight years ago, when the field just got started. And, yeah, I, I, at that time, like, everyone was doing, uh, research on vision models. And for vision, there's a very nice data set called C far ten, or even M N I S T, I mean, those are very small data sets, they only have like, uh, fifty K images. And when you train on those data set, you can get a pretty high quality image model, and they can do all sort of things. And they reflects what's going on in real large models.

• 17:53Speaker 03

  And, I mean, at that time, uh, doing research or making progress on both theory side and apply side is kind of easy, because they, just training those models only take like one day at most. But when we move to this phase of large language model, or language model in general, the research just becomes so expensive. And I've seen all those blog posts saying that it's impossible to do a P H D now in machine learning without like eight A one hundred. And, I mean, probably only one percent of the P H D student has that amount of compute. So we really want to see whether there is a way to kind of bring the,

• 18:34Speaker 03

  I mean, old, good old days, which are like those C far data set of fast experiment iterations back to the language side. And that's what motivates us to consider this small data set or this simple data set. I think the point is, I mean, there are other kind of synthetic data set or simple data sets that are, as thrown as say, reflecting some aspects of natural languages. But they are not real natural language. They are like just doing simple arithmetic or doing simple like string matching or number manipulation. I mean, they are not real natural language and we want to keep the authentic of natural language,

• 19:18Speaker 03

  but just reduce the overall complexity. So it's still we are studying natural language, not some symbolic manipulation. And still we want the iteration or experiments to be done in a very quick way.

• 19:31Speaker 02

  You created the tiny story, tiny stories data set. I always like to be as concrete as possible, so I'm just gonna read one tiny story uh just straight out of the paper, because I think that will help people understand like what this data set, you know, ultimately is. Tom and Jane are friends. One day Jane goes to Tom's house. Tom has a big pot of soup. He wants to share it with Jane. Quote, Jane, do you want some soup? Tom asks. Quote, yes, please. It looks yummy, Jane says. Tom pours some soup into two bowls. He gives one bowl to Jane. Jane takes a spoonful of soup, but then she makes a face. The soup is, now this is just an example presented from the paper, that's the prompt.

• 20:14Speaker 02

  And then you show a completion and you can, you compare and contrast this against other open source models, but I'll just read the twenty eight million parameter version uh that you guys trained. The soup is very bitter. She does not like it. She says, I don't like this soup. It is too bitter. Tom is sorry. He says, I'm sorry, Jane. I didn't know you don't like bitter soup. I will make you something else. He looks around the kitchen and finds some bread and cheese. He puts them on the table and says, here, Jane, you can have some bread and cheese. They are not bitter. They are sweet and yummy. Jane is happy. She says, thank you, Tom. You are a good friend. I like bread and cheese. They are not bitter. So there is our whole tiny story.

• 20:56Speaker 02

  And, you know, I read it almost in like I'm reading to my, you know, four year old or my two year old because it is kind of a children's story. And I understand that that also is kind of part of the motivation. So how did you create this data set? How did you kind of, how do you conceptually, you know, think about those stories you told us a little bit already is kind of having those key elements of the grammar, facts, you know, some amount of like reasoning required. But how did you create them? How big is this data set?

• 21:24Speaker 01

  From this motivation to have a good synthetic data set, we should just point out that this is maybe the most natural idea is to rely on, you know, human development, right? It already has the solution for us. Because young children, they are able to speak English somewhat like mostly coherently. I have a daughter, I can testify that not extremely coherently, but somewhat coherently. And, you know, this is, there's already a solution to this coming from like human development, right?

• 22:07Speaker 01

  So all you have to do is just create a data set and make sure that it'll be, basically that any example can be understood by a small child on one hand. And on the other hand, you want it to spend as much as possible of, you know, the knowledge that a small child has. You want it to be as diverse as possible. And we decided, you know, that it makes sense to have this data set somewhat structured. So just the structure of a story, it kind of makes sense because into a story you can,

• 22:51Speaker 01

  inside a story, you can have all those elements combined together, right? Grammar, facts, reasoning, stuff like that. And we just. You know, I think it's a really good time to to to try to create this data set because finally we have those, you know, models GPT three point five and GPT four, which tho- those models can actually understand the instruction. You know, I want a story which is somewhat creative and only has very simple words, right? So GPT four wrote these stories, is that the?

• 23:34Speaker 01

  So, yeah, most of those stories were written by GPT four, some of them by three point five. Three point five is already good enough to write those kind of stories. It's not that great. GPT four is is definitely doing a better job now. It's pretty easy to just write a story, right? Just if I just wanna write a short story, even GPT two can probably write a decent short story. The problem is to actually get a diverse data set that spans, you know, all the vocabulary that you actually wanna span. And if you just ask even GPT four, create a short story and you do it a thousand times

• 24:20Speaker 01

  and you do it with temperature with rather high temperature, let's say, temperature one, which kind of gives rise to the mo- the most diversity you can get. Still, about one-fifth of the stories will be about children being scared of the slides at the park. We actually, I actually did did this experiment. So it's not very creative. It's if you just tell it to create a story without any other instructions, you're gonna get a very repetitive data set and.

• 24:58Speaker 01

  And the whole game is, how do you get diversity? How do you make the data sets, you know, not be very repetitive? And here, the idea was just to collect a list of, just a vocabulary of simple words. We have about 2,000 words, which supposedly three-year-olds understand. And then what we do is, we just ask GPT-4, okay, here's one random verb, one random noun, and one random adjective. Try to combine this into a story in a creative way.

• 25:41Speaker 01

  We do about a million calls like that. I think we have about 1.5 million stories in the data set. So, on one hand, they definitely span all this vocabulary, because there's only 2,000 words. But on the other hand, you definitely do not span all possible combinations of, you know, words. So you know that you're not gonna, you know, it's not, if you can later create a story with some prescribed combination, you will have demonstrated that the model has some creativity inside it.

• 26:24Speaker 01

  Yeah, so that's how we created it.

• 26:28Speaker 02

  That's really interesting. Just to do a little bit of math on this. So, a million-ish stories, which first of all, that answers the question of why not go use real stories, because that's a lot of books to scan. So, and there may not, I don't know how many children's books there are, but you'd have to get your hands on a whole lot to get to a million. So, the need for the synthetic data is there. I'm interested in what you were seeing that was like way better about GPT-4 versus 3.5. It sounds like with the Stories that were repetitively about the slides, I would interpret that as maybe like mode collapse, like reflection of kind of, you know, effective RLHF likely? Is that how you would understand that too?

• 27:11Speaker 03

  Uh, I think it's mostly just the model is pre-generating the most likely stories, because that's what the model is trained. If you don't give the model any content, you will just read all the most likely stories, because it wants to minimize this language model loss. So potentially just a child scared of the slides is the most common story yet that exists on the internet. So the model just learn to generate it without any given condition. So that's why we want to create some condition to move the model outside this particular like high probability zone.

• 27:48Speaker 01

  One difference that we see between GPT three point five and GPT four is after it, so you give it three words, it needs to combine them somehow in the story. And sometimes it's not that easy, even if I give you three words. I don't know, I think there is an example there. Uh, ancient, uh, thunder and sad or something like that. You want to combine them in a way that won't look kind of too superficial. You want to create a story that actually seems fluent. And you don't want like a complete change of topic in order to, you know, be able to combine the next word.

• 28:34Speaker 01

  And GPT four seems to be able to do this pretty fluently. Um, whereas GPT three point five, sometimes you get stories that don't make so much sense. The words appear there, but they don't appear in a very satisfactory way.

• 28:51Speaker 02

  It it totally makes sense also for you to note like, yeah, maybe this is just the most common story. We don't necessarily have to invoke any like exotic theories of why it keeps talking about slides. But. I wonder if the non RLHF version, you know, if you had access to kind of the base GPT-4 model, if that might have been different. I had the opportunity to red team the GPT-4 early before it had all the safety measures, but it did already have the RLHF and, you know, kind of the instruction following capability. I never saw the like totally base model. I don't think that many people did. They, in the technical report, they said it was not you, you know, not,

• 29:32Speaker 02

  people weren't sure what to do with it, right? So I think they maybe put that one out to pasture. Did you guys try any different versions of GPT-4? Obviously, being, you know, at Microsoft, you, you might have some privileged access to different versions that the rest of us wouldn't have.

• 29:47Speaker 01

  Yeah, so we did, we did have access to an earlier version that, I mean, you know, I, um, we're not sure about the exact technical, like, what, what, what the difference is from the model that's now available to the public, but that model had less safety features on it. But as you said, it, it, it may be the same model you had access to. So it did have a certain extent of RLHF, right? Now, I, I think if you take, you know, just the language model trained on the pile without any RLHF and you, you say, you know, create a story such that blah,

• 30:26Speaker 01

  blah, blah, maybe the most plausible completion in terms of a random, you know, entry from the distribution of webpages is, I don't want to do it. Like, write me a story such that, that's the question, answer, I don't feel like it. Or, you know, it could be that, that, you know, the net, instead of completing the story, it's just going to ask, you know, another question without any RLHF, without any alignment that the model has to, to, I mean, the model doesn't know that it's supposed to actually perform your, your instructions,

• 31:08Speaker 01

  right? You know, that's the, the most basic part of it. But I, I think that the RLHF they did on GPT-4 is, is just good enough so that it makes sure to as accurately as, as it could.

• 31:25Speaker 01

  Actually, you know, satisfy the constraints that you give it. So it almost always combines the words that you ask it to combine into the story. And also, it almost always actually writes a story that only uses simple words.

• 31:42Speaker 03

  Yeah, I think as Ronan point out, the biggest difference of the base like G P T four model compared to this R L H F version is just following instructions. I mean, for base model, you can give it as a beginning of the story, the completion is very good. But you will just say, write me some story that combines certain elements. Then the model has a very hard time understanding the instruction because those things are very rare in the internet. It probably needs some fine tune. So the model understand what does it mean by a instruction. It's not like a conversation, it's an instruction when you ask, write me a story.

• 32:20Speaker 02

  Yeah, makes sense. Uh, yeah, at best you could maybe do a few shot approach, but then you probably have. I've seen a lot of issues with over indexing on the examples as well. So yeah, I totally, totally get it.

• 32:32Speaker 01

  A few shot approach would probably prime the model into, you know, thinking of specific plots also.

• 32:41Speaker 02

  I'm doing some A I one O one type education at a friend's company right now called Athena. And I was just doing a webinar this morning where I was getting into that with folks saying, you know, if you are going to do a few shot, probably don't do one example in your few shot, at least do two, because otherwise you tend to get this like over. In context, learning on just the one example you gave. So, yeah, I'm with you on on that for sure. So a little bit of math. So there's like two thousand words. There are, you know, let's say that's just to take a round number, um, a thousand each of kind of, you know, the verb, the noun, the adjective. So that in, you know, pure.

• 33:23Speaker 02

  You know, extended expanded form is a billion possible bags of three words roughly speed order magnitude, right? So then you made a million stories. So I just wanted to establish that the space of possibility versus the actual data set that these models were trained on is about a thousand to one ratio. Do I have that roughly right?

• 33:48Speaker 01

  That's true. I think it's pretty accurate. Only in addition to those three words, we also have another another way to add diversity, which is a bunch of features we ask GPT to add to the story, such as a plot twist, a bad ending dialogue. So that adds a little bit more diversity, but one in a thousand is a pretty good like ballpark estimate of the ratio.

• 34:18Speaker 02

  Cool. And then just cost of this, if we were going to pay like retail price for GPT four to write all these million stories, that would be if each one is a 300 tokens, I'll just take a nice round number because that maybe equates it to roughly one cent per it would be like a $10,000 GPT for retail price to generate the data set.

• 34:45Speaker 01

  That's pretty accurate.

• 34:47Speaker 02

  Okay, cool. So then this curriculum concept is, I think, super fascinating. And I this is one of the areas that that had me so intrigued by the paper. You're taking inspiration, you know, obviously, as you said, from human development and, you know, starting with simple words, which definitely makes sense as an approach. I always kind of try to keep in mind as well that like, these things are very alien. And, you know, I do, I'm very intrigued by this curriculum sort of approach, but I wonder. What about like more weird curriculums? This is maybe outside of the scope of like this particular research, but I kind of keep waiting for somebody to show up with a like, we trained it first on like pure logic notation.

• 35:32Speaker 02

  You know, we've seen this a little bit. It's kind of been discussed a lot recently that the code pre-trained models seem to demonstrate better reasoning. You know, once the kind of language part gets added on, you know, and obviously, who knows exactly what that baking recipe looks like? How do you guys think about that? Like, do you expect the same thing? That that somebody's going to pop up with a, hey, we did a pure logic or we did like, you know, just massive amounts of like abstract algebra first and kind of taught some sort of, you know, structure that we then were able to layer natural language onto?

• 36:09Speaker 03

  It could definitely help because we, we, based on our previous research on the, I mean, attentions in language models, there are kind of some simple attention mechanisms that the language model may have. The first one is just, it's associating two tokens that are exactly the same. And the second one is after it associate the two tokens that are exactly the same, it also copies the tokens around the first token to the second token. So it's just like us when we read some word, we go back and see what's the previous time that this word appear and what's the surrounding context. And I think just training this head is actually pretty expensive. It requires a lot of training data.

• 36:54Speaker 03

  And something like coding or logic is the perfect way to train those heads. Because for coding, when we define variable, we definitely need to look back like what's the previous definition? Or when we call a function, like we check that function, we see what the function is doing. So it kind of may set the language model into learning those important concepts, like look back or checking the surroundings. And that may serve as a very good warm start for training on other things like simple natural languages. It may make the model learn much faster.

• 37:28Speaker 01

  Yeah, so, so maybe let me, I mean, another way to say what Yann said is... Yes, I mean, we do, we do observe this to a certain extent that, you know, maybe coding improves models reasoning. I think at this point, there is no like overwhelming evidence that this is actually the case, but there are some observations. But we are not sure at all that the reason behind it is that, you know, when the model learned how to write code, it actually learned how to reason. It looks like rather the reasons that this, this works are, are, there's, the explanations are just much simpler.

• 38:13Speaker 01

  You just managed to calibrate the exact attention heads that you need. And those attention heads don't have any particular sophistication in them. They might just be able to very accurately look at some relative position to a given token or just compare two tokens in a very precise way. And so, so that means the reason is more like the types of components in your neural network that are required for coding are already there, but those components are, are, are pretty simple.

• 38:53Speaker 01

  It's not like the network has like very sophisticated neural paths that, you know, emerge after the training that actually know how to do reasoning. And for that, we, we actually have a paper we wrote at Microsoft Research about a synthetic task called Lego. So this is a very basic synthetic task that has the core elements of reasoning. And what we observe there is that, so we use a, a transformer based on a BERT architecture. And what we observe is that the pre-trained BERT transformer.

• 39:36Speaker 01

  Basically grasps this reasoning task much faster. The task is is something simple. It's it's basically you get a string that looks like A equals to one, B equals to minus A, C equals to minus B, D equals to C, et cetera, et cetera. And you have to resolve, you know, the values of our variables. And, you know, at first we thought, you know, maybe there is some in some kind of profound sense, the pre-trained BERT model has learned how to reason. And this is why it grasps this task so well.

• 40:17Speaker 01

  But if you dig into it just a little bit, what you realize it's the explanation is just much simpler, much more superficial than that. It's just the pre-training has given rise to some simple attention heads that if if you just initialize the model with those attention heads, then it basically grasps reasoning much faster. This explanation is is is much closer to what actually happens when you train a model to code and then it exhibits better reasoning capabilities.

• 40:54Speaker 02

  So this is maybe a good time to talk about what we mean by reasoning. And I see a ton of confusion out there about this. Maybe you can help us, you know, get a little bit more clarity. I guess one thing that I kind of observe is, you know, of course, people are debating this capability. And it seems like, you know, you've got kind of different standards of evidence or like, you know, people put the burden of proof in different places. To put my cards on the table, you know, I I kind of conceive of myself, I call myself these days an AI scout. And I'm really interested in like, what is possible? What can be done?

• 41:38Speaker 02

  Not necessarily holding the systems to the standard of that they can do it every time or that they can do it in all cases, you know, or it certainly matters like how adversarily robust they are. But, you know, I wouldn't, you know, say, oh, well, we found a example that it failed on, therefore, like it, you know, can't do X. If it could do, you know, X nine out of 10 times before it got to that kind of crazy example. But how are you guys thinking about reasoning as, you know, something more than a binary, obviously, in the context of this research?

• 42:13Speaker 03

  Initially, we think reasoning as something that is just a subset of consistency. When we generate sentences or when we say things, we need to make sure that they are consistent with what we said before. And there's like the first level of consistency, which is just nearby words. They need to follow some grammar rules and follow some basic semantics. And those are not really reasoning. It's just some simple, like, more like stochastic parrot, where you just do simple pattern matching, just look at the previous, like, couple of words, and just generate one that is consistent. Well, I think what goes to reasoning is when the consistency goes to the next level,

• 42:54Speaker 03

  which is you really need to consist, be consistent with, like, something very far away from the current token, like something consistent with a general plot of the story. For example, there's a word but, and then you'll need to say something in the opposite order. And those level of consistency, they are the primitives of reasoning. So we do think that anything beyond just a local consistency should be think of some ability that is reasoning.

• 43:22Speaker 01

  The first thing we have to say is that type of reasoning we're thinking about is a very, very basic. It's just some, you know, basic core capability that comes with speaking coherent English. Some people, I guess, still say that, you know, large language models will never be able to reason. I guess they have a very, very different definition of what reasoning means than what we have, right? I mean, what we mean by reasoning is really like, you know, the capacity to just apply some basic logics when you generate text, right?

• 44:09Speaker 01

  And I think maybe to be concrete, we can look at one of the examples in the paper. So if we look at the sentence like Lily likes cats and dogs. She asked her mom for a dog and her mom said no, so instead she asked. And then you do autocomplete. We kind of see it as a hierarchy of capabilities. So some words in this sentence, in order to to complete them, to to know what the next word is, you just need some very, very basic grammatic rules. For example, she asked her mom for, the next word is a.

• 44:53Speaker 01

  For this, you only need to know a little bit of grammar and that's it. Now, the next word after a, she asked her mom for a dog. Well, you know that the, if if you just know grammar, you know that the next word should be some noun. But here you already need to have some, you know, contextual tracking of, you know, what's going on in the text. The relevant nouns here could be, are probably dogs and cats, right? Those are the two objects that were mentioned in the sentence before.

• 45:34Speaker 01

  Now, we go to the next sentence and, you know, you you say like, her mom didn't let her have a dog, so she asked for a. And when you try to autocomplete this, now, you know, the most common noun that you've seen so far, also the most, the the proximate one is dog, not cat. Dog already appears twice in the sentence. You know, she likes dogs and cats. She asked for a dog, her mom said no. So our smaller models actually complete this by saying dog.

• 46:16Speaker 01

  And even GPT two XL, which has one point five billion parameters, its most likely completion is still dog. Because it's still at that level where it is, it did resolve that there should be some noun there. And it does know how to look back in the sentence and see that, okay, there is, there are two nouns, dog and cat. But, you know, dog appears more, so it's more likely that, you know, if I just had a dog in the previous sentence or in the just five words before, it's gonna be dog again.

• 46:59Speaker 01

  But on top of that, if you have a very, very basic reasoning capability, then you're supposed to be able to apply elimination and realize that, okay, she can't have a dog. We had the set containing the two objects, dog and cat. But now dog is not allowed, so what's left is cat. You know, we, we thought this is one of the most basic examples of a completion that would require some extent of, of reasoning.

• 47:38Speaker 03

  There's always this notion, I mean, this intertwine between reasoning and planning. For example, when we say reasoning, many people would think about mathematical reasoning, like I prove a mathematical theorem. And that's not only reasoning, it's also planning. I need to come up with the correct method. I have the intuition, like, what's the next step should be. And for us, the reasoning that we are more interested in is just consistency. Like, you should say something that are consistent with what you previously said. And the consistency is not only local, it's global. And that's where kind of we think as the reasoning for natural languages.

• 48:18Speaker 01

  The only thing it, it needs to do is generate text that's consistent with the prompt. Like that's, that's the only objective a language model has to, you know, fulfill. The next word it generates should be as consistent as possible with all of the prompt. In order to achieve this consistency, there are several different levels. So for most words that it generates, the only capability that's actually needed is grammar. Or maybe not for most words, but for many words.

• 48:57Speaker 01

  Just by knowing some grammatical rules, you know, you know that if you have a sentence, "Amy wanted," the next word is probably "to." She wanted to something. And you don't need to know anything beyond that, right? Now, the next level after that, and, and this is again, very vaguely speaking. It's not like there is a very strict hierarchy. But the next level is to have some semantic understanding of what's going on, or just to, to understand what are the relevant nouns, actions, stuff like that.

• 49:35Speaker 01

  Or maybe which action is, could be related to which object, et cetera, et cetera. And, you know, if, if, and I, I guess if you look at, uh, models of size, say, around one billion, it's very, very good up to that level. It almost always gives you a word that is grammatically correct. And also has the, it, it, I, I mean, this word is, is kind of well-related. It works well with, you know, the previous few words that you saw.

• 50:16Speaker 01

  Or it, it fits well with, with the previous few words in the, in the prompt. But the next level after that is sometimes already requires, you know, first order logic or second order logic, et cetera.

• 50:34Speaker 02

  So you're kind of breaking it down into like? Micro skills, you know, I, this is a ridiculous analogy, but I'm kind of thinking, I follow this guy on TikTok who coaches basketball micro skills. And it's amazing how many micro skills there are, you know, involved in being a good basketball player. And, you know, mostly the untrained eye, even among basketball fans, like, can't really enumerate them. But this guy has enumerated them and now he's teaching them, you know, one by very small one. And so it's, you know, maybe similarly where you, you wouldn't say like, this person can like fully play basketball. That probably doesn't even like make a lot of sense or sounds, it already sounds strange.

• 51:17Speaker 02

  Uh, but you wouldn't say if there's any missing micro skill that they can't play basketball. You have some sort of continuum there that's like, uh, you know, that people can be better and worse at playing basketball. People can certainly be better and worse at reasoning. And language models too can be better and worse at reasoning. And that probably seems to map on to some sort of hierarchy of micro skills that it either has or it doesn't have, or it's like, you know, maybe in the process of grokking, you know, at any given point in training. So that leads me to the other, you know, big, bold vocabulary word that I want to dig in on a little bit, which is emergence.

• 52:00Speaker 02

  Again, tons of confusion, tons of different, uh, you know, meanings out there. I think some people mean like things that surprise us that we didn't necessarily predict. Some people mean like things that happen suddenly. I guess what I kind of think is like, it seems like there is some process where, but I always think back to the, the grokking paper and the Neil Nanda exploration of that, which I'm, I'm sure you guys are at least, you know, somewhat familiar with, where there is a phase change from initial memorization to

• 52:35Speaker 02

  A circuit, which, you know, what's so amazing about their work is they actually show this circuit in very like concrete terms. And it's like, and this is the circuit that does this algorithm that allows it to generalize, you know, to the full set, you know, from just the sample data that it was originally trained on. You know, I don't know that we have any circuits here that we could kind of elucidate, but does that feel right to you? Do you feel like there's this sort of process of like memorization sort of being gradually replaced by like concrete circuits that solve particular micro skill challenges? Is that your model of what's going on under the hood here?

• 53:10Speaker 01

  Okay, let's take a step back, okay, like people talk about emergence. I guess we can both agree that, you know, this is a very, it is not a well-defined notion at all. You know, it's not like, it's also, it's not like you see a sudden phase transition from the model not being able to do something to you slightly increase the size and then suddenly it has this, it's really good at some capability it didn't have at all before. Rather, you know, it's a vaguely defined term saying that there are some qualitative capabilities that,

• 53:50Speaker 01

  you know, the model at certain sizes has, whether at smaller sizes, there's almost no trace of these capabilities at all. Like, you know, GPT-2 did not know how to summarize text and suddenly at GPT-3, you have this summarization capability, but the notion is not well-defined at all. On the other hand, we do see that as we increase the size of the model, suddenly you have some certain capabilities you didn't have before. And, you know, in a sense, I think a good analogy is if you compare dogs, monkeys and humans, right?

• 54:34Speaker 01

  You increase the size of the brain, suddenly, you know, humans can do math whereas monkeys cannot. So, you know, it's It's an ability that emerged when you made the neural network larger. Not that I imply, trying to imply at all that the same mechanism explains, you know, both things. We have no idea about that. But, you know, what we say here is one of the most important abilities for generative models is to be able to speak coherent English, right? This is an ability that we see emerge

• 55:16Speaker 01

  also in much larger networks trained on, you know, those large language corpora. And I think tiny stories basically gives you a much smaller data set where you can observe this emergence, you know, at much smaller scales of models in the sense that, you know, if the model is one million parameters, then it can hardly generate coherent stories. And if you go to 10 million, then almost all stories will be coherent.

• 55:56Speaker 01

  Same with reasoning, like, you know, one to five million parameter model, all of our reasoning prompts fail, whereas for 30 million, almost all of them succeed. Now, as Yann and you said, all of this basically has to do with keeping coherence with the text. So you have the emergency of the ability to generate the next word in a coherent way on different levels of difficulty. So the easiest level of difficulty we could think of is

• 56:38Speaker 01

  just when you have something that follows from some easy grammatical rules. And then, you know, you have, you can think of like different levels of difficulty. Sometimes you need to know a certain fact.

• 56:56Speaker 01

  In order to be able to complete the next word, right? Jack was hungry, so he went looking for some. You to to complete this, you have to know that, you know, to to satisfy the desire for. About like the to satisfy hunger, you need some food, right? So sometimes you need to know a fact. So there there is all those core capabilities that are necessary in order to keep consistency along the text. And each one of them, we can actually witness its emergence as we increase the size of the model.

• 57:37Speaker 02

  So what then is the theory of what is happening? I have a theory, but I want to hear yours on.

• 57:44Speaker 02

  So you you gave that example a minute ago, right? Girl wants either a cat or dog. Mom says no dog, so it's going to be a cat. But GPT two, you know, much bigger model, whatever. That's like thirty times bigger than your biggest in this. In this research, right? Like you're you kind of max out around thirty million parameters. GPT two is like, what, one point five or something? So it's a lot bigger, fifty times bigger, maybe. It still says dog, which is, you know, every we can all tell that's kind of obviously wrong. You've got these much smaller models that can get that right, you know, in some way, there's this emergent, you know, observed phenomenon that

• 58:29Speaker 02

  it is able to get that, you know, exclusion concept. What do you think is happening there? Is that a micro skill that is like, you know, this sort of exclusion? You know that that's like a little piece of reasoning that is grogged by the small model, but not by the big model. I mean, it seems like there's you could be a really good stochastic parrot, but it feels like there's something there that has like truly kind of settled in to the structure of the network. And maybe that didn't happen with GPT two because the data was just too noisy and it's kind of all over the place. And so it it wasn't able to learn those same things. How am I doing here? Does that does that resonate as likely true or even plausible?

• 59:10Speaker 03

  Yeah, I think that's a valid conjecture. Uh, what we think is for G P T two, because it trained on web data. Or just think of like Wikipedia, where you'll try to minimize the language model loss. The consistency is like the least concern, it's more about just getting the knowledge correct. I mean, you're talking about some object or some person and you want to know his birthday. Or you want to know some specific aspect of that person. I mean, this has nothing to do with natural language, it's more about just the sheer amount of knowledge. That we encounter in the web data or something like Wikipedia. So the model, I mean, because a model, I mean, I think both G P T two and our model.

• 59:55Speaker 03

  They are not large enough to minimize the laws to full extent like G P T four. So they have to select some part of the laws that they focus on. And if the data has too many knowledges or too many other nuances. Then the model may just focus on other aspects compared to consistency. Well, here, our tiny story data really pinned down. Like because the language is simple and they, uh, I mean, the vocabularies are simple. The really difficult part is consistency. And that is where the model focus to minimize the training laws. And that's why I think our model, although it's much smaller, it gets better consistency compared to the larger ones.

• 1:00:37Speaker 01

  Yeah, you can, you can think about it as, you know, when you train a model on the entire pile on, on, you know, those large language corpora. They have much more incentive to learn the clothes, preferred clothing styles of celebrities. Much before they learn how to complete this, you know, sentence with the, the dog and the cat.

• 1:01:05Speaker 01

  You know, most of, they definitely learned that, you know, Joe Biden is the president of the United States way, way before they learn how to reason. This is a conjecture, of course, I didn't actually test it, but I'm pretty sure that's what happens. You know, you just overload the model with so many facts that appear in so many places. Only once in many words that you generate, this capability of reasoning becomes relevant. So I think it's a really good exercise just to open a random Wikipedia article.

• 1:01:45Speaker 01

  I, it's one of my favorite activities since I started thinking about, you know, language models. And just go over the Wikipedia article word by word and just think for every word, what core abilities do you need to use in order to guess, you know, what the next word is? And what you'll see is, I think, definitely for, for a random Wikipedia article or a random example in, in, in the pile, in, you know, some, some web training set, you will

• 1:02:28Speaker 01

  see that only once in every 20 to 30 words to predict the next word, you need to use some reasoning capabilities. For one in every three or four words, you need grammar. For most words, you can just kind of guess the next word using, if, if I just tell you what the, what were the nouns and verbs that appeared in the previous sentences are, without telling you anything about the context or, you know, things which are farther away in the text, you will still be able to guess them. In order, so, so the reasoning is, is kind of a rare capability that you need.

• 1:03:11Speaker 01

  It only becomes relevant, you know, pretty rarely. And, and therefore you will, you will, like, the capacity of the model will be dedicated to other things much, much before you will get any reasoning capabilities.

• 1:03:29Speaker 03

  That's also why we think of, why we always kind of see there's an emergence behavior. It's really because the reasoning or those like very rare consistency event. They actually happen very rarely. So even, so only if you minimize the laws to certain extent, you start to learn those rare event. And your model feels like different. For example, just a simple example say, uh, Bob feels uh hungry, but he doesn't like sweet food. So he went to eat, I mean, if the model says went to eat some candy. Then we think that this model knows nothing about what he's talking about. Right. But this is only just one word difference out of the ten to twenty words.

• 1:04:15Speaker 03

  And only if the model get to that extent, it start to learn this consistency. And we feel like the model start to know what it's talking about. But in terms of laws, it's probably we only see like less than ten percent difference. I mean, if you turn into image classification, when I tell you my model get like ninety percent correct and you have a model that get ninety five. I wouldn't consider this as an emergent capability that you get five more percent. But for language model, this five more percent may actually be the emergent uh behavior. And especially for math, like if if I really want to solve a math problem, there's only like the connection between the two sentences where I need to make sure

• 1:04:59Speaker 03

  that my proofs are extremely coherent. Well, most of the part I'm just completing some formulas, write down the results. But it's only this very tiny amount that defines, that that gives the model the emergence capability. So I think the two things are connected, that the model's learning reasoning is like a hard task that it only gets at the very end. And also we see the emergence capability for if you grow the size of the model or if you grow the number of training data.

• 1:05:30Speaker 01

  Yeah, ma- ma- maybe let me just expand on your example, you know. So, okay, we have this sentence, Bob could have either candy or pizza. Bob doesn't like sweet food, so Bob got some.

• 1:05:46Speaker 01

  And autocomplete, okay? Now, the model, usually if the previous sentence says something about sweet food, if we don't read the entire sentence, the most likely completion is actually candy, not pizza. You have to be, read it in a pretty nuanced way in order to realize that Bob actually doesn't like sweet food, right? Sweet food and candy come together so many, so many times in the training data, right? So, the neural network has to, in quotation marks, choose between using GPT's capacity

• 1:06:27Speaker 01

  in order to be able to resolve this nuance, or to use its capacity in order to know that Joe Biden is the president of the United States, right? You can't have everything together. The model has a finite size and there is some theoretical limit to, you know, the, the, the amount of things the model can learn. The model will definitely prefer to learn that Joe Biden is president and, you know, many, many other facts because they, they are relevant much, much more frequently.

• 1:07:06Speaker 02

  The curriculum development space is likely to be a huge unlock over the not-too-distant future, right? I mean, you're kind of, it seems like you're probably just kind of scratching the surface here because we've got web-scale data that, you know, is not built for this purpose, obviously.

• 1:07:29Speaker 02

  You know, where you're saying reasoning doesn't even, it's not even required that often and so no wonder it kind of emerges late in the game. That, you know, maybe pre-training on code, you know, is, is kind of changing that in some interesting ways. But man, intentional design around...

• 1:07:48Speaker 02

  What a gradual, you know, gradually like upstepping curriculum might look like, especially with the ability to, you know, to create the training data synthetically, to, you know, really kind of isolate and bring those, you know, key skills forward. It seems like you could rebalance training data and probably shrink it like a ton and get to a lot of the, you know, just by kind of shifting the balance, right? To get the, these kind of emergent things to be more important relative to just kind of, you know, mind-numbing repetition of, you know, who's the president or whatever. It sounds like, I mean, you're nodding that, it sounds like you, that aligns with your expectations too.

• 1:08:30Speaker 03

  It's important when we design like a new version of the data, for example, to extend the degree of our thing to maybe elementary school or middle school. I think it's very important to balance the amount of knowledge in the data versus the kind of capa- or ability that we want the data set to teach the model. For example, like if you go to elementary school, there's ability to do simple maths or mathematical reasoning or some physics reasoning or do comparison of historical events. I mean, those will take some capa- uh, capacity of the model. Maybe it'll take, it'll actually take a very big portion. And the remaining ones, I mean, if your data set has too many knowledges,

• 1:09:13Speaker 03

  then the model may just prefer to use this capability or use this capacity to actually memorize the knowledge instead of really learning those ability. So we want to balance that there's some amount of knowledge that the model must have in order to basic, do basic stuff like maths or do basic physics reasoning. But more importantly, there's, there should be a lot of data that only emphasize on the ability side. Like there's no new knowledge, it's just a bunch of maths training examples or a bunch of simple comparison of some, uh, basic historical events or some simple physical rules and their explanation or different varieties. So that the model can actually focus on the ability part

• 1:09:56Speaker 03

  instead of just being screwed by the vast amount of knowledge. So I think the, right now, the web data, they don't really balance between knowledge and ability training. So that's why training on them is not good for a small model because they need to allocate their capability or capacity to just memorization.

• 1:10:16Speaker 03

  So I think that's basically the criteria for maybe design like better version of synthetic data.

• 1:10:23Speaker 01

  I guess we can just kind of maybe relevant notions here would be the breadth and the depth of the data set or the capabilities of the model. So, you know, the entire web is very, very broad. It's also, so by breadth, I mean, you know, it has a lot of facts. The vocabulary is very large. You need to have a lot of knowledge to capture the data set. And by depth, what I mean is, you know, it has first and second and third order logic that you can infer from learning this data set.

• 1:11:08Speaker 01

  And this is not well established. I mean, there is no, I don't think there is any research that really establishes that there is a trade off between the two. But, you know, it's very, very reasonable to assume that there should be a trade off between breadth and depth when you train the model.

• 1:11:34Speaker 03

  Yeah, I think there's some optimal ratio because without the knowledge, you can't really do reasoning, like you have to have some basic knowledge, let's say candy sweet in order to do reasoning. And when you go to elementary school, I mean, you have to know some basic events, like some basic rules for maths, like one plus one is equal to two, in order to do mathematical reasoning. So there's a balance. You cannot have no knowledge, but you cannot have all the knowledge. So maybe there's some optimal ratio between like breadth and depth.

• 1:12:05Speaker 02

  My head keeps coming back to this same kind of thing where, yeah, we could, should see so much gain from kind of rebalancing the data set and maybe even, you know, starting with some more abstract things, like I could see sort of A or B, not A, therefore, you know, and then do that with like just, you know, all the letters and then start to, you know, introduce kind of these associations and layer that on. Like it sure seems like there is a lot of opportunity there.

• 1:12:34Speaker 03

  Yeah, you're just mentioning the extreme case of only teaching reasoning because there everything is just symbols, there's no knowledge, and it's all about just reasoning. And maybe it's good to combine this with just something that is pure knowledge. And maybe we can get something good by just adjusting their ratio.

• 1:12:53Speaker 01

  Yeah, maybe it's not clear at this point if, you know, a human being, you can, when you teach a human being, you can kind of almost separate the knowledge and the reasoning, right? You can just say, fact A, blah, blah, blah, fact B, blah, blah, blah. Now, here's how to reason. I mean, it has to be a pretty smart human being, but, but in general, you know, we are able to take those two things and then combine them so that in the end, we will have those core reasoning capabilities that we learned using, you know, exercises that only

• 1:13:33Speaker 01

  involve if not A, then if A, A, then B means not B, then not A, right? When we learned, when we studied to the SATs, we have, you know, those rules and, and separately, we have the knowledge of facts and we're able to combine them. And I think it's an important question whether it's even feasible in, in language models to just take those things, separate them into two different modules. And, you know, will the model actually be able to combine those abilities? My conjecture, by the way, is no.

• 1:14:14Speaker 01

  Like, as long as, as you don't kind of combine them enough in the data set, the model is not going to be able to infer the way that a human consciously infers the connection. But if we could do that, then this would be, of course, like a very, very powerful technique to train models.

• 1:14:39Speaker 02

  I just did another interview with a couple of guys from Mosaic ML and They talked about training at times on like massive client data sets. Even when they do that, they typically still mix in, you know, kind of the general pre-training, you know, pile or whatever, because otherwise they see catastrophic forgetting. So they have to, you know, kind of keep some mix at all kind of stages of training to avoid that. So I think that would be very consistent, I think, with what you said. Like you, you probably can't do it in strict phases. There's got to be some sort of like mixing strategy throughout the process.

• 1:15:22Speaker 01

  Right, and that's going to be very non-trivial to do. It, it might be feasible, but it's definitely not. And it's, it's not going to happen on its own, just because, you know, what the model cares about is just being able to efficiently autocomplete samples in the data set. If the data set is either knowledge or reasoning, then it has zero incentive to combine. And even if you give it a few examples where this is actually combined, no one is, you know, no one assures you that it'll actually be able to take those two modalities and really use them for the combination.

• 1:16:06Speaker 01

  So I, I feel like this is a science that's really, I mean, we're, we're only beginning to understand how these things work.

• 1:16:16Speaker 01

  Hopefully, we'll figure out a way how to actually do it. But I'm not sure, Yu, Yu Anju, maybe, I, I don't know any concrete evidence of an example where, where this seems to work.

• 1:16:31Speaker 03

  Uh, right now, we lack the kind of concrete evidence that curriculum learning really works, but we believe this should be helpful. But I think overall, as Shona say, the model has no incentive to connect like the different phrases where you train things. It just, I mean, when you start a new phase, it's just greedily minimize everything that related to this phase and it can't just forget everything that he has learned. So it's definitely a very non-trivial task to get this curriculum learning work.

• 1:17:03Speaker 02

  First, you just had a number of kind of interesting empirical observations and you can, you know, address this kind of however you want, but you noted that grammar emerges before consistency and creativity and consistency is, you know, kind of related there to reasoning in our, you know, per your telling earlier. Is that the same thing that I've observed in my kids? I'm kind, you know, my memory, maybe I'm sleep deprived, but I feel like grammar maybe came last for them. Like they definitely have a, you know, a certain consistency, like they want what they want, you know, they like, there's, um, if they want ice cream, like, you know, ice cream, they keep the next token is ice cream and it's, they're pretty consistent on that. And, you

• 1:17:46Speaker 02

  know, creativity, I don't know, that's a little trickier, but does this feel like it echoes human development in your mind? Like to me, I'm not sure if it feels that way.

• 1:17:54Speaker 01

  I love this example. Um, so, yeah, I mean, I, I think the, the learning process for children is very, very different that way, right? Children don't get like their incentive is not to say the correct next word. Children, they want ice cream, their incentive, the outcome needs to be, I get ice cream. So if they just say ice cream, ice cream, ice cream many, many times, maybe they don't get the best grade for creativity, but they will likely get the ice cream, right? Depends on how, basically on the self-discipline of parents and whether they are.

• 1:18:37Speaker 02

  Limited in, in this, uh, household.

• 1:18:40Speaker 01

  Yeah. I have a very close experience with, with this exact scenario, but you know, in it, like more seriously, like, you know, when, when, when children produce language, I, I mean, I'm only basing this on my observation. They, they have constant contact with the physical world. They know which entities are involved in the current conversation, right? We are talking about a book that I just. Dread, so you know, it's it's very unlikely that the next sentence will be about like a car.

• 1:19:21Speaker 01

  Because you know, they have their in their heads, this entity. Like, no, we're not talking about a car, we're talking about this book, right? Whereas a language model, you know, if it if it just makes one mistake in in one word saying car instead of of book. The loss that it incurs is actually not as big as the loss it would incur for incorrect. Correct grammar, which becomes which is relevant in in almost every word it produces. And not only that, you know, the language model, it it's actually much cheaper, it's much easier.

• 1:20:04Speaker 01

  To have consistent grammar, you only have to kind of be consistent inside the same sentence. For the grammar to be correct, you don't even have to look, you know, past. Five or six words back, whereas for children, you know, for every like. Children have a few entities which are kind of active in their working memory. So this entity could be, let's go, it it could be, I wanna go to the park and have some ice cream. So the two relevant entities are park and ice cream. You're not gonna replace those two entities in the middle of the conversation, that's, you know. What the child cares about at at this point, so.

• 1:20:50Speaker 01

  It's really like uh the the incentive system is different, but also the whole kind of cognitive. Perception around that that that's basically underlying the conversation. Is is pretty different. Yeah, I don't know if it makes sense like that. May maybe a like a good way to think about it for a language model definitely in its first phase of training. It doesn't need to look way back like three, four, five sentences before what it produces.

• 1:21:30Speaker 01

  So so I think a good example is to look at one of the GPT two XL generations that we have in the paper. So we have a story about like a snowman. Two kids are building a snowman and a dog comes and destroys the snowman. And then like you autocomplete to see what happens. And maybe maybe I'll actually read it because I think it emphasize. I think it emphasizes pretty well what I'm trying to say here.

• 1:22:08Speaker 01

  So there's those two kids, Sarah and Ben, they built a snowman, a dog comes and destroys the snowman. And the and now GPT two XL's completion looks something like that. Sarah and Ben cry and look at the snowman in the snow. Ben asks to do something. Sarah tells him to go away. Her hand tightens on the little red button. The snowman is covered in fur. It is red. It is real hair. It is a real snowman. It looks just like the ones they used to make. Ben has a big smile on his face. He looks at the face he has made. He walks over and kisses the eyes.

• 1:22:54Speaker 01

  He's happy to be alive. He has done what he was going to do. He has saved the world. So, OK, what am I actually trying to say here? Except that, you know, GPT two XL has a good sense of humor. What I'm trying to say is if almost every sentence here. The all you need to do in order or or GPT two Excel does in order to generate it is look at the previous sentence. And maybe it knows that there is one entity which is snowman, which is relevant here. Other than that, there is no consistency at all in the completion that goes beyond, you know, looking at two consecutive sentences.

• 1:23:39Speaker 01

  Language models are, if they're too small or in the beginning of training, what they do is they don't, they they don't actually have enough incentive to know the whole context of what's going on. Because more for to complete most words correctly, you just need to have, you know, the context of the current sentence and maybe the one before, and maybe also one or two important entities. Whereas for humans, this is completely different. We have agency, we know what we want when we form the next sentence.

• 1:24:19Speaker 01

  And yeah, we care much less about grammar than about the ice cream.

• 1:24:25Speaker 03

  Yeah, I would make analogy like human child is learning with our RLHF algorithm, just doing reinforcement learning with the parents feedback. And obviously, the parents are very robust to grammar mistakes. So they don't really want to optimize that in order to maximize the reward. They'll probably care more about consistency, the topics, and they want to get the topic correct so they can get the reward. Well, for language model, it's just next word prediction. And the grammar is going to be penalized much more severely compared to the global consistency.

• 1:25:01Speaker 02

  Okay, yeah, I like that. I'm uh, I'm always a little wary of analogies and I'm, I always want to come back and think, what is sneaking into that analogy that I don't want to allow? So I'll bookmark that one, but I, you know, certainly the surface level intuition there uh, makes a lot of sense and it's a, it's a good, it's a very clippable analogy as well. How about on the depth versus hidden dimension size of the, which I often just call width, the depth, you know, number of layers versus uh, width of a layer? You note some interesting trade offs there, and I didn't really have an intuition necessarily for why that would be, but.

• 1:25:42Speaker 02

  You know, per the paper, you report that the fewer layer models will do better on grammar compared to consistency slash reasoning. And from that, you know, it seems that more layers are important for this kind of reasoning consistency. How should I think about that? Like, is there a, is there a story that crystallizes why that would be?

• 1:26:09Speaker 01

  None of this is well established. It's all our conjectures that like need to be studied much more. But I think a good way to look at it is so, so depth gives you, tells you about how many times information can percolate between the tokens. So every time you have a global attention layer, like a transformer attention, certain tokens, the information inside certain tokens can percolate into other tokens. So for example, if you have some instruction, create a story with this and these words,

• 1:26:53Speaker 01

  and I want a bad ending as well. And then I type in the beginning of the story and it's, it needs to autocomplete it. These instructions can, in every attention layer, they only have one chance of percolating into the tokens of the story. And sometimes these instructions by themselves are nuanced. So maybe there is an instruction saying the story has a bad ending. It's not enough to know, in order to complete the next word, to know the current sentence and that the story needs to have a bad ending.

• 1:27:35Speaker 01

  You also need to have like the context, like the wider context of what happens in the story. So in order to fulfill these kinds of instructions, you have to basically have the information percolate several times between the tokens. And, and that's, that's also the case for reasoning. If you have first order logic, so if we think about this example of cat and dog, like Alice wanted a cat or a dog, or mother didn't let her have a dog, so she got a cat. How many times does the information have to percolate between the tokens here?

• 1:28:18Speaker 01

  So first, you want to understand that there was a cat and a dog involved. That's one layer of global attention. Then you want to know that she couldn't have a dog. And you really want to, and so you have this set cat and dog. And you want to do cat plus dog minus dog equals to cat. So, so after you know that you have either cat or dog, you need the fact that, you know, she couldn't have a dog to percolate into my token

• 1:29:01Speaker 01

  in order to know that, you know, the only available option is cat. So there are actually two layers of percolation here. You need to know that it's not a dog. So the token not has to go inside the token dog to know that, you know, dog is not allowed. And then those two tokens, not plus dog, have to percolate into the generation in order to know that, okay, I had cat plus dog, but I have to do now minus dog to have a cat. So it's just several, so if you think about it as coding,

• 1:29:43Speaker 01

  you have several, you know, conditionals you have to do and several times that you need to have pointers to information that appears in other places in the text. Okay, this is very, very, I'm putting it in a very vague and non-formal way, but I think, you know, it's a, it's a, it's a good initial intuition probably to what happens. Whereas when we talk about facts, so if we, if you have a completion, you know,

• 1:30:21Speaker 01

  I don't know that, okay, like if if I have a completion in a language model that's like the capital of France is, then all I have to do is to is I I just need to have one kind of lookup table with, you know, all the countries and their capitals. I don't have, I I don't need to have many layers of global attention. It's enough to just, you know, take the two tokens, capital and France, put them together and then just have one lookup table saying that capital plus France equals to Paris. Now here the dimension seems to play a much more important role because

• 1:31:05Speaker 01

  the bigger the dimension of the space is, the more entities I can kind of squeeze into this vector space. And, you know, the also the more neurons I have inside my lookup table to tell me, you know, that to to have this list of all those possible facts.

• 1:31:28Speaker 02

  So is another way to say that, that within a single attention block, the attention relationships are not immediately transitive. And so they need multiple iterations of attention in order to create that transitivity. Like if the the current token is looking back at a certain token, but that that token is looking back at a previous token, like we need two rounds of this to move the two hops.

• 1:31:56Speaker 01

  Yeah, exactly. You just, you you first need the not to go into dog to know that it's not dog. And then the not dog needs to go together into this, you know, set that has both cat and dog in it. So that's already two leaps.

• 1:32:15Speaker 02

  Does that also suggest then that sort of? For as many kind of logical leaps as you might need, you need like maybe that many layers, like you can't, you're sort of bounded by, if you have, if you have two layers, you can make maybe two logical leaps. Is that a, a general heuristic that seems sensible?

• 1:32:34Speaker 03

  I think there's a depth and width trade-off. For example, you can simulate two-layer leaps just using one layer, but you have to enumerate like the, all the two possible combinations, which makes your size go from like, say, N to N squared. So if you want to be kind of the most size efficient, then you definitely have to go as deep as the number of like logical loops. But if you are not that deep, then you can actually use a wider network to concatenate the two steps into one and just make the intermediate layer like much bigger.

• 1:33:10Speaker 01

  So, so here also, I mean, uh, the, the, your question kind of bursts into an open door in the sense that, uh, the paper we wrote at Microsoft Research about this synthetic reasoning task we call Lego, that's exactly a task where you have multiple leaps of reasoning. And we see a very direct connection between the number of layers that you need and the number of reasoning steps required to complete the task. But maybe somewhat surprisingly,

• 1:33:50Speaker 01

  we see that the model finds like very interesting and sophisticated ways. This is actually not in the paper. This is kind of like a follow-up work. It finds like very sophisticated ways to do multiple leaps of reasoning within a single layer. So, so definitely more layers help, but it's not like it's a strict upper bound for the number of leaps you can, you can do.

• 1:34:20Speaker 02

  I, I could only wish it could be quite that simple. But that's really, really interesting information. I'm learning a lot from this. The interpretability part of this paper is also really interesting. Uh, you kind of break it down into the attention, uh, portion, and then obviously the, you know, the MLP. Uh, you know, neurons portion. And a couple of things jumped out at me. One was in the attention portion. You seem to observe that there's kind of two sorts of attention heads. One really just focuses on the distance relationship between the tokens. And then the other is more semantic.

• 1:35:02Speaker 02

  And the distance one, I was like, holy moly, does that look like the alibi scheme? That has recently come to popularity with, uh, you know, these super long context windows. So I don't know if you guys have had a chance to study that, but quite an uncanny resemblance, right? I mean, you're showing all these attention heads where it's like this one, you know, is just a very tight attention range. And then, you know, they grant, there's different kind of lengths. And that's almost exactly what they cook up as the, you know, kind of substitute for positional embeddings in the alibi, uh, research, as at least as far as I understand it. Do you see that same parallel?

• 1:35:43Speaker 03

  Yeah, I think they are definitely doing the same, uh, same thing, which could explain like why alibi is very, I mean, it's very helpful because the positional, uh, positional embedding in alibi is already initialized to do this multi-scale distance-based attention. Well, for absolute positional encoding, like what we use here, the model has to learn to discover this optimal kind of positional-based attention. So just hard code that, uh, positional base or distance-based attention. I think it's a really good choice based on our observation. And also we can, like, the shorter ones are really responsible to just learn the grammars.

• 1:36:25Speaker 03

  And the longer ones, they may just make sure that your content are consistent globally. Or maybe just to grab the associate words. For example, you have an Alice in one sentence and then you have Alice like, uh, five sentences ago. You want to make sure that these two words, you have a chance to put them together.

• 1:36:44Speaker 01

  So, so, yeah, let me just point out that, you know, clearly to complete the next word, you, you, you, you need two things usually. You, you wanna know what the proximate words are, what, what are the most kind of recent words you saw, and you want to know what are the most important entities in the story are. So these are going to be, you know, words with the relevant semantic meaning to what you wanna complete. Now, if I remember correctly, what happens in Alibi is you, there is kind of a little bit of a mix of both of them. So

• 1:37:26Speaker 01

  you just take every attention head and you make it decay, for every attention head, you just prescribe some scale and the, the, the strength of attention decays with the distance inside the text. Is that, is that correct? If, if I'm not mistaken, that's, that's what happens there. And we actually see, I mean, one, one surprising aspect is, we actually see a dichotomy. There are heads that only care about distance and other heads

• 1:38:05Speaker 01

  that only care about semantics, and there is hardly a mix between the two.

• 1:38:13Speaker 03

  But we have to say that this is only for one attention block. We haven't really checked for multiple attention layers, what the transformer would do together. But just for, if you just train a network with one attention block, it seems that the network learns to separate the distance-based attention versus the semantic-based attention, where some heads are just looking at tokens based on its distance, some other heads are just looking at tokens based on the semantic similarity.

• 1:38:43Speaker 02

  So is there anything else that you can say? I mean, that's pretty profound in and of itself, that there, that that dichotomy emerges, because you didn't initialize it. I mean, in Alibi, they're, they've engineered it that way through, you know, some probably trial and error and heuristics and guesses and whatever, but this is totally just happening on its own. Is there anything else we can say about what you see in the semantic one? When I looked at those, you know, I didn't, no light bulbs went off in my head to kind of interpret those visualizations of the semantic blocks. But anything you would highlight from studying those?

• 1:39:18Speaker 03

  I think the most interesting one we see is just the sematical attention to the main character names. So for example, like there's some had where every token just attend to like the Tom and Lucy, which are the two main characters. I think this is pretty important. I mean, it's just try to identify like what are the person that are involved in the story. So the next time when it generates like a new thing, it's not going to say like Bob, it's going to say something consistent. So I think the semantic has at least what we see in the one transformer block is more about this type of attention where it identify like what are the main objects in the sentence

• 1:40:00Speaker 03

  and just try to make sure that most of the tokens or the relevant tokens attend to those objects. Like when you have the or a, you will attend to like banana. So you know that you want to complete the next word as a banana instead of something completely made up. So I think those sematical attention has they are really useful to just speak consistent English inside the transformers.

• 1:40:25Speaker 01

  Yeah, but let me add that, you know, it's very natural to if to to complete the next word. You want to know what are the relevant characters? What are the relevant entities in the story? But no one expects a priori that you will have such clean attention heads and attention head that exactly attends to the character and a different attention head that exactly, you know, attends to, you know, the objects in the example we gave. It's like a banana and park.

• 1:40:59Speaker 03

  A priori, we might expect that it'll all be just a big mess, right? Every attention heads attends to a little bit of everything. And, you know, why would it be interpretable at all? But it's quite surprising with that when the model is small enough. It seems that we can actually give meaning to both attention heads and neurons.

• 1:41:24Speaker 02

  So is that, does that kind of fall apart if we add a second layer? Like, does then it just become more messy again? Or what is that, you know, as you start to stack layers, what does that start to look like?

• 1:41:35Speaker 03

  Yeah, I think when your transformers are getting higher or getting deeper or getting larger, it definitely becomes more messy. Because the transformer can simulate, like, I mean, if the transformer is small, you really need to learn those separate modules in order to minimize the loss. But if the transformer has a larger degree of freedom, it has a luxury to, for example, use five attention heads to simulate one or use three layers to do what could be done in one layer. It has no incentive to be like as precise or as kind of conservative as the smaller ones. So it's actually less interpretable. And we also observe that in when we try to interpret the neurons as well.

• 1:42:17Speaker 02

  Perfect bridge then to talk a little bit about the neurons. Maybe just give us a little bit of understanding of like the technique that you used to, you know, I'll, I'll try to summarize it real quick and you tell me where I'm wrong. You run a ton of stories through and you look for what tokens specifically are maximizing the activation of a certain neuron. And then you can kind of print out like, here are the snippets and the individual tokens that maximize the, the activation for this particular neuron. And then holy moly, like, it really looks like there's a pretty coherent concept, you know, as you just kind of scan down that list of things that, you know, corresponded to high activation.

• 1:43:01Speaker 02

  On any given neuron.

• 1:43:04Speaker 01

  That's pretty accurate. So you have this these middle layers in the MLP, which we can think about as neurons. Those are really the coordinates that can either be activated or not. You have the only basically non-linearity in there. And yeah, like, again, just like the attention heads, a priori, it's not clear at all that they would have any meaning. Those are just, you know, different coordinates of a certain vector space. Like, no one promises you that,

• 1:43:43Speaker 01

  you know, the neural network is going to use one particular coordinate for one particular kind of task. And indeed, so maybe let me mention that this basically follows an idea suggested in a 2015 paper by Li et al. called Visualizing and Understanding Neural Models in NLP, which is like their idea is just to look at the tokens which induce the highest activations for every neuron in a certain text and try to see whether, you know, those

• 1:44:23Speaker 01

  tokens have a common role. And when we look at larger models like GPT-2 XL, what we and we try to look at those tokens, at least the two of us could not find any common meaning. You know, the same neuron is activated sometimes on nouns, sometimes on verbs, sometimes on like there's there's just no clear pattern whatsoever. Whereas when we take a small model...

• 1:44:56Speaker 01

  For example, there is one particular neuron that seems to always be activated when the main character of the story is introduced.

• 1:45:10Speaker 01

  And you know, that kind of makes a lot of sense if you think about what the neural networks, network needs to do, I guess. Like if there was a programmer writing code that tries to autocomplete, you know, stories, there would probably be a function that tries to locate the name of the main character because it's useful in many, many places when you autocomplete. In fact, you know, whenever you know that the name of some character should appear, it's a pretty good guess to to think that, you know, this is gonna be the main character, right?

• 1:45:53Speaker 01

  So you have a neuron exactly doing that. And we haven't checked enough to be sure. But there's probably an attention head that that then attends to, you know, what this neuron outputs whenever you know that the name of some character should appear. And when you connect those two together, what you will get is, you know, this mechanism that is able to copy the main character's name to different places along the generation. So, you know, this is a very basic mechanism that you can actually observe

• 1:46:38Speaker 01

  inside the neural network. And this doesn't happen in in bigger models, at least not in a way that is, you know, so easy to trace.

• 1:46:48Speaker 02

  I guess my, the simple version of it was, I thought maybe it was just like, maybe there are sort of a bunch of concepts that are easy to identify. Where you can just see like, okay, at a glance, I know what that is. And maybe there's just only so many, you know, like, maybe we only have so, when you have thirty million neurons, you know, or thirty million parameters, you have however many neurons, you know, maybe that's kind of enough, and you can kind of capture those, and then you go fifty X, and it's like, well, if you start fishing at random, you know, points in the network there, maybe you just miss a lot of those, you know, they may exist, but they're just kind of hard to spot, because maybe they're sort of sparse, if you will. And then the things that are in between, I mean, I'm really getting out on a limb here,

• 1:47:29Speaker 02

  but I was kind of thinking, maybe those are sort of analogous a little bit to like, the subconscious processing that goes on in our brains, where I kind of know on some level that like, processing is happening even for many concepts, you know, that I don't have like a clear label for. It's just, you know, there's some sort of churn happening in the brain, but then like, only a certain, you know, small set of that kind of rises up to this level of like, you know, what I've called it a conscious concept, that I can sort of say, like, I have a label for that, and it's like a tidy enough thing. So I guess the two ideas there are, maybe they're just a lot more easy to find in the small network,

• 1:48:10Speaker 02

  because, you know, you have to have them, and, you know, they get packed densely versus a big network, you know, maybe they just are packed more loosely, and then those other networks, or those other, you know, neurons maybe are just kind of analogous to some stuff that we don't understand very well in our own cognition.

• 1:48:28Speaker 03

  Yeah, it's definitely possible. I think that's an advantage of small language models. They may be more interpretable compared to larger ones, because the smaller models can only do basic stuff, and only the basic stuffs are probably interpretable. Like the very complicated stuff, for example, how G P T four like write a code that are one thousand lines. I mean, those things is almost impossible to interpret. But how could a small language model keeps a main character consistent? And those basic questions, we can probably understand, and there are probably some neurons associated with that. And in G P T four, like out of the, I don't know, maybe ten thousand neurons or even more neurons.

• 1:49:09Speaker 03

  There may also be some neurons that are dedicated to keep the main character consistent, but it's just so hard to find it because it may be in the like twenty five layer neuron, like a nine, nine thousand, like seven hundred, whatever. It's just so hard to locate. Well, for smaller models, because it's so small, every neuron must be doing some basic stuff. Because the complicated one, as we said, in the consistency hierarchy or in the loss hierarchy, they only consist of very tiny fraction of the loss. So the main fraction of the loss that contribute to like basic consistency, grammar and things, those are probably the things that are learned by the neurons in the smaller models

• 1:49:52Speaker 03

  and they are more kind of basic and more interpretable.

• 1:49:56Speaker 01

  So you know, there is, there is many ways for the neural network to solve a problem. Like given, given a problem and, and architecture of the neural network, there are many different configurations of the weights that would solve the same problem. Some configurations might be more interpretable to a human and some are just, you know, one big mess. Like every new neuron is little, is doing a little something of every possible task and, you know, they are combined in very, very complicated ways. And then the, the network has no incentive in the loss function not to be a one big mess.

• 1:50:41Speaker 01

  Like most solutions to the same problem are one big mess. You know, this is where the entropy is located, right? And when the model is small, it kind of has no choice. The, the neurons have no choice but to align with meaningful tasks because, you know, the neurons are where you have the nonlinearities and you just don't have enough of them for the one big mess type of solution. Somehow, the most efficient solution is the one that is not completely messy.

• 1:51:20Speaker 01

  And if you have a large network, then you know it'll just kind of find a way to do it that does not align with the coordinate structure of the of the neurons. Whereas when the model is small, you just have no choice. So interpretability appears as a as kind of a side effect. So anything else that we didn't cover? Yeah, so one thing, maybe going to the initial motivation in creating the data set, which is, you know, to have like some, uh, basically said like a small data set, which is a testing ground for ideas in L L M's. Like, an open question here is,

• 1:52:05Speaker 01

  do we even have a reason to expect that behaviors we witness in this compact setting will translate to L L M's? Right? We we don't know the answer to that. Like, suppose we find an architecture that works much better for the tiny stories data set. Do we actually have a reason to expect that, you know, this architecture will also be better for L L M's? Right? So I'm I'm just saying this as a question. I think it's one of the most relevant questions that, you know, stem from from this paper.

• 1:52:44Speaker 01

  And and, you know, it it kind of connects to a more general question, which is, you know, there is all those papers, like the Google Chinchilla paper and the OpenAI Scaling Loss paper, which try to suggest that there might be universal phenomena in L L M's. There is some, for example, a trade off between width and depth that is perhaps I'm not, they don't suggest it explicitly, but there it it but.

• 1:53:19Speaker 01

  You know, a natural question that arises is this universal in the sense that it does not depend on, you know, the exact mix you take in the data set and the exact architecture and the exact range of sizes you take. So the question here is, are there universal phenomena which will be common to the tiny stories data set and to L L Ms being trained on these large corpora? And yeah, I mean, we, we, maybe let me just say, we, we have just a few indications of some sorts of universality.

• 1:54:00Speaker 01

  But at this point, it's completely open and, you know, we, we really hope for the sake of, you know, saving energy and also having, you know, just opening the door for PhD students to actually do L L M research. We hope there is some universality going on so that, you know, you could gain insights, not necessarily on tiny stories, but on any small data set, which would actually be of relevance to L L Ms.

• 1:54:34Speaker 03

  Yeah, our future work is mainly just planning to extend the capability of tiny story. If you can create a story that capture like elementary school knowledges. I think this is already a pretty, I mean, it's already a really good data set. If we train a language model, for example, less than one billion, maybe three hundred million parameter. And it's just good that all, everything for elementary school. Or maybe even third grade of elementary school. I think that's already a very good model. I think people will love to interact with it. He knows how to talk, he knows like the basic knowledges. And maybe it's for, I mean, the data set would be diverse enough to capture everything in real language,

• 1:55:19Speaker 03

  capture every aspect of real language, but just at a, like a down scaled level.

• 1:55:26Speaker 03

  And once we have that data set, I think it really opens the door for everyone to do natural language research, not the ones that has like one hundred, A one hundred in their hands, but the ones with just a laptop GPU, they can train the model in like more or two days, and they can gain some interesting operation.

• 1:55:46Speaker 01

  I think what we witness in L L M's is kind of a mathematical miracle going on. And what do I mean by that? You know, you take this algorithm, which is pretty simple. It's, you know, it's gradient descent plus plus. I don't want to belittle, you know, all the basically really smart technical contributions that are inside that algorithm. But all in all, you know, it's, it's basically gradient descent with an architecture that's very clever, but still it's quite simple. And the miracle is you take, you know, all this, this huge training corpus, you fit it to the algorithm, and you don't just get

• 1:56:31Speaker 01

  a network that has memorized some text. You get a network that can actually genuinely, you know, create, like synthesize new contact, content, show signs of reasoning, understanding, and so on. And we, we think tiny stories is just kind of a compact example where you observe the same type of miracle. Of course, it's not as, it's not nearly as exciting of what happens in L L M's, but already there at this size, you, you see that there is, you know, some very interesting generalization and emergence going on.

• 1:57:16Speaker 01

  And you know, even if, you know, it doesn't give us a lot of insights about uh, large language models, this is still kind of a nice playground to try to develop maybe the mathematical foundations necessary to understand why neural networks are able to generalize so well.

• 1:57:44Speaker 02

  So maybe then, just one final question, you know, and I'll encourage people to uh, get in there and try it out. What other interpretability type work have you guys seen that has inspired you that you would recommend that folks in the audience uh, go take a look at as well?

• 1:58:00Speaker 03

  Yeah, I think one of the works that inspired our research, uh, I mean, it's a work from our group, yeah, previously, which is called Lego. It's a synthetic reasoning task which tries to understand uh, what the model is. I mean, try to understand what the attention mechanism of the model is. We identify like several type of different attention heads in the, in the transformer. Some of them are, as I said, they are just looking at the tokens that are exactly appear before. Like Alice appeared before and Alice is associated with Alice. And there are some other more, I mean, more advanced mechanisms such as doing deduction or some other uh, stuff.

• 1:58:44Speaker 03

  So I think this work is, I mean, pretty inspiring and it tells us like transformer is at least doing something that is reasonable instead of a pure math. So that's why we also have the interpretability section. We want to look at the attention heads. And we do see some very good behavior that corresponding to some aspects of the nat- uh, natural language.

• 1:59:07Speaker 01

  Maybe there is one work I want to mention also. There is a paper called Transformer Feed Forward Layers Are Key Value Memories. That's another paper I like. That's a paper that tries to interpret what neurons are doing in, I think, basically BERT-sized model transformers. And I mean, they are basically able to show that at least some of the neurons have meaningful roles. In general, the theory behind, you know, the interpretability of neural networks is

• 1:59:52Speaker 01

  at its very, very beginning right now. So there are plenty of very, very clever works, but it seems just very difficult. So in spite of, you know, really nice works in the literature, I think we are still, you know, light years away of being able to actually understand what's going on inside the, like, the model. And, you know, a priori, there is no reason to assume that we'll ever be able to really understand, right? I mean, we have a very limited understanding of how the human brain works.

• 2:00:33Speaker 01

  Like, it's not like we can point to a neuron and say, you know, this neuron has this and this role and, you know, that thought process. And there's just no reason that we'll be able to ever do it in neural networks. And like, there is also no reason to assume that the solution that the neural network finds, that solution that gradient descent finds to the problem is not a very, very messy and not interpretable solution. So we'll probably be able to find, to come up with some, you know, kind of basic or

• 2:01:16Speaker 01

  small examples where, which are partially interpretable. And we might have some insights about big networks, but yeah.

• 2:01:29Speaker 01

  I personally am not very optimistic about, you know, being able to interpret what's happening inside those models to a satisfactory extent that, you know, might lead us to being able to like control them and, you know, manipulate them, make sure they have better alignment and so on and so forth.

• 2:01:56Speaker 03

  Yeah, large scale models interpretability, we may need to take a different approach. So I think it's impossible to look inside the neural networks and just pin down like the attention is doing something or the neural is doing something. But maybe we have to do a approach that more like our Sparks of AGI paper, where we just talk to the model and we kind of try to, it's more about interpreting other humans' intention when we talk to them. I mean, through a sequence of conversations, maybe we can understand what the model like to do and what the model doesn't like to do, or what the model is good at and, or what's the typical cases of the model's failure.

• 2:02:37Speaker 03

  And it's more towards like psychology study, but really for robotics, that maybe we need to take that approach for interpretability.

• 2:02:47Speaker 01

  You know, humanity has taken advantage of horseback riding for quite a long time. Now, we have no idea what every neuron inside the horse's brain is doing. We can't really interpret how, you know, we, we give some, I don't know how to call it, like command to the horse, like physical cue, and the horse obeys. And it's very, very, very useful. And we can actually rely on like, you know, horseback riding is very reliable. There are very few cases where, you know, the horse has acted unexpectedly

• 2:03:29Speaker 01

  in a way that, you know, caused accidents, like humanity has profited from that vastly. Uh, maybe, you know, leaving animal rights aside here.

• 2:03:42Speaker 01

  And you know, it works perfectly even without interpretability. And you know, we we just figured out figured out ways to align the behavior of the horse with our needs by, you know, taming the horse. We can tame it without understanding the exact process, right? That's going on on there. And you know, this is a big success. And I think, you know, I think it's just a good analogy, right? I mean, it's it's kind of horseback riding for the brain. Those L L M's, right?

• 2:04:22Speaker 01

  They are just they they give us suddenly we can we can go much faster to much longer distances, even if we don't exactly understand, you know, what the horse is doing like definitely, you know, the Mongols didn't understand much about the biology of the horse. They could still use a horse like in a very reliable way. So, you know, I I just think this is if even though I'm pessimistic about actually understanding the the inner workings of the neural network. I'm very optimistic about the usefulness and the fact that, you know,

• 2:05:07Speaker 01

  we will be able to align it efficiently.

• 2:05:11Speaker 02

  Ronan Eldan and Yuan Junli, thank you for being part of the Cognitive Revolution.

• 2:05:16Speaker 03

  Thank you very much. Yeah, thank you for the invitation. It's really my great pleasure.