The Future of the Transformer Part 2 with Trey Kollmer

Trey Kollmer returns to discuss the latest AI research revelations with Nathan Labenz. They explore how new techniques will shave 10% off global compute needs, how analogical prompting beats few-shot prompting, and how compressive historical records can increase LLM memory and retention abilities. If you need an ERP platform, check out our sponsor NetSuite: http://netsuite.com/cognitive. SPONSORS: Shopify | NetSuite | Omneky Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you're looking for an ERP platform ✅ head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. RECOMMENDED PODCAST: Every week investor and writer of the popular newsletter The Diff, Byrne Hobart, and co-host Erik Torenberg discuss today’s major inflection points in technology, business, and markets – and help listeners build a diversified portfolio of trends and ideas for the future. Subscribe to “The Riff” with Byrne Hobart and Erik Torenberg: https://link.chtbl.com/theriff TIMESTAMPS: (00:00:00) - Episode Preview (00:01:11) - Paper: Think Before You Speak (00:03:13) - Multimodal models for combining vision and language (00:04:19) - Backspace Paper (00:06:25) - Chain of thought prompting for step-by-step reasoning (00:09:14) - Backspacing in language models to correct mistakes (00:12:05) - Attention sinks for expanding context length (0012:41) - Paper: Large Language Models as Analogical Reasoners (00:15:24) - Pause tokens for language models to "think" (00:18:23) - Analogical prompting to recall relevant examples (00:20:52) - Long context windows for language models (00:23:20) - Markdown works best for OpenAI (00:24:23) - Ring attention to break memory constraints (00:26:15) - Paper: StreamingLLMs (00:27:46) - Potential for superhuman performance with longer contexts (00:31:01) - Dynamic context window adjustment at runtime (00:33:53) - Retention and memory capabilities for transformers (00:37:12) - Planning algorithms combined with memory and scale (00:39:49) - Paper: Ring Attention (00:42:35) - Executive assistant prompting and critique (00:45:23) - Self-RAG for language models to find own examples (00:48:02) - Timelines and predictions for future capabilities (00:50:37) - Applications like analyzing long texts and scripts (00:53:15) - Local versus global attention in transformers (00:55:59) - Architectural changes versus just training adjustments (00:58:41) - Pre-training strategies like random start points This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  No less than Ahmad Mustak from Stability said brilliant researchers like this literally knock 10% off of global training compute needs with these improvements which are impossible to predict. 10 million tokens starts to give you the opportunity to put like whole bodies of literature into a single token, right? I mean, the Great Gatsby famously fits into Claude's 100K. Now you're talking perhaps about 100 books with full attention considered for the next token generation. If this allows the models to make those connections at such huge length, this could be where you could start to see tipping into superhuman performance

• 0:45Nathan Labenz?

  of like learning things that experts don't know. Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Le Bens, joined by my co-host Erik Torenberg. So then the next one is "Think Before You Speak" is the name of this paper and "Training Language Models with Pause Tokens." So "Think Before You Speak: Training Language Models with Pause Tokens." This comes out of Google and again Carnegie Mellon collab.

• 1:28Nathan Labenz?

  And this was, I guess, a student from Carnegie Mellon who's interning at Google. It's amazing how many of these papers are like couple-month processes and you know, you can do this kind of stuff in the context of a summer internship these days. Amazing. So what do they do here? They start off with an observation, which is a pretty simple one on some level. I'll quote this from the paper. "Language models generate responses by producing a series of tokens in immediate succession. The k+1th token is an outcome of manipulating k hidden vectors per layer, one vector per preceding token."

• 2:10Nathan Labenz?

  What if instead, we were to let the model manipulate, say, K plus 10 hidden vectors before it outputs the K plus 1 token? So basically, just like, especially in the early going, but you know, kind of anytime, you only have a certain amount of computational space to move information around. And maybe that's just not enough, or maybe more could be beneficial. First thing that comes to mind for me when I hear something like that is, I think that's a lot of what's happening in the chain of thought type prompting, you know, certainly that's been hypothesized that like, by giving the model, you know, time to think, you give

• 2:52Nathan Labenz?

  it time to kind of work its way through things and, you know, hopefully summon the right reasoning. And then you get better results. So certainly empirically, we see that you get better results. Well, this is now saying, okay, what if we just gave it extra space, but we didn't make it do anything with that extra space? I'm not asking for reasoning, I'm just giving it a pause token that it can just literally put a pause in when it feels like it needs to, you know, and how exactly that gets decided is a bit of a black box. And that's, you know, there's some trade-offs here, I think, for sure. But give it that opportunity to just take a pause when it needs to. Now it can just process information a little bit more.

• 3:33Nathan Labenz?

  It could, you know, potentially do a couple of pause tokens in a row if it needs to. They suggest, you know, what about 10, K plus 10? So 10 extra vectors that it can kind of manipulate and move information back and forth between. Does that give us the opportunity for better performance? And obviously, you know, this makes the research roundup because indeed, they show that they are able to improve performance on this. So.

• 4:02Nathan Labenz?

  First thing on this, I was like, boy, I saw that one coming. There was the first thing I, you know, we covered a little bit of the backspace paper on a couple earlier episodes. That was a Stanford one where they had shown that, and this kind of combined some concepts from the last one, too. In the backspace one, when they got out of distribution, as measured by like not having high confidence on any next token, then they started to train the model to use the backspace to go back and be like, well, we kind of, we must have gotten off the rails here a little bit because now we're not confident of what to do next. So let's instead go back, try that one again, maybe make a different prediction this time, and then maybe that will, you know, lead us towards something where we can feel more confident.

• 4:45Nathan Labenz?

  When I saw that, I was like, well, it seems like you could probably have a, you know, if you can go back one, you could probably just add a padding one, too, and just kind of, you know, quietly think to yourself, sure enough, you know, 60 days or so later, you know, here's the publication. Was this inspired by that? I'm not sure. It was kind of on the, on the border where like it, there was just enough time for them to have done it, you know, in response to that, but I would guess honestly, they probably had the idea before. So this is probably independent, you know, kind of parallel lines of thinking. So it was like, okay, cool. You know, that's good confirmation, you know, that I'm starting to build some intuition about this kind of stuff. But then, as I think about it more, I'm like, boy, do I like this? Do I not like this?

• 5:27Nathan Labenz?

  I like chain of thought in that I can at least read what it's outputting, you know? And when I can read what it's outputting, then I can be like, well, if the reasoning is wrong, then no wonder the answer is wrong, you know? So I can kind of maybe go back and coach it on the reasoning a little bit more. Here, if you're just using these pause tokens, which is what they're kind of doing in the experiments, they show that, you know, it works, but you've kind of lost a step in terms of interpretability, because now you don't have this reasoning that you can examine. Instead, you just have this pause and like, yeah, it improves performance, but what's happening there? We don't really know.

• 6:12Nathan Labenz?

  We just, you know, again, we're back to, well, we can try to look at the activations and figure it out that way. But, you know, it certainly is nice to, as a practitioner, definitely, to see this kind of... Reasoning that you can audit and you know, kind of get comfortable with seems to be approaching the problem in the right way. Now, like many things, of course, not mutually exclusive. They do show that this, it works. It works better if you incorporate it into pre-training as well. So there's kind of a, you know, going back and like synthesizing some data, adding pauses

• 6:51Nathan Labenz?

  into kind of show, you know, at some scale, when pauses are appropriate, that helps it even more. And then, I don't see any reason you can't use both of these techniques together. You know, you could have the pause and then the chain of thought, right? So, one thing that I do sometimes worry about a little bit with chain of thought, there's actually multiple worries with chain of thought. One is that there has been some research that shows that it's not always super faithful, you know, that the, which is to say like the answer that you ultimately get is not always as determined by the reasoning as it may seem or as you may wish. So that can be a little bit complicated.

• 7:32Nathan Labenz?

  You can't just totally naively trust the reasoning output. So maybe here you could do something like first pause and then reason? Because I've often kind of thought, well geez, if my, if I force it to give an answer immediately, and that's subpar, and then I, you know, can get better performance by reasoning, well don't I still have a little bit of a problem where I'm like, it's immediately reasoning? You know, what if it's not reasoning right, you know? And again, I don't want to analogize too much between human and AI, but in this case, I do feel similarly, right? If I'm, bam, forced to answer some question, you know, okay, wait, it'd certainly be advantaged if I could think it out.

• 8:15Nathan Labenz?

  But the same kind of thing, if you're like, you must begin reasoning immediately, you know, I'd be like, can I just think quietly for a second, then explain my reasoning and then get to an answer? So now we've kind of got the ability for the AI to do something similar. I don't see any reason you couldn't train for first, the thinking pause. Then the chain of thought, hopefully now your reasoning becomes better, hopefully becomes more relevant, hopefully becomes more faithful. And then, you know, maybe you get the best of both worlds with kind of combined tactics. You get the, you know, the best possible accuracy.

• 8:47Speaker 01

  How do they train for the pause tokens to output pause tokens? Do they pick situations like with the backspace paper where it's not that confident? And then they say that's a situation you should output a pause token? Or do they, they have some other method? Cause that seems like a hard thing to do, to put in the data. It's not like automatically in the data.

• 9:09Nathan Labenz?

  Yeah, it's definitely not automatically in the data. So they do it with pre-training and also with fine tuning. It works best with the, with both pre-training and fine tuning, although you do get some lift. Well, it varies, I guess, across data sets. When you do both, it's like clearly the best. In some cases, it seems like, depending on the data set, it seems like in some cases just fine tuning makes it better. And in some cases it makes it worse. Cause it makes it better more than it makes it worse. But there are a couple of data sets where just doing it at the fine tuning stage does make it worse. I'm not immediately seeing how they're preparing the data.

• 9:48Speaker 01

  Is there, I guess they could just train it to always start with 10 pause tokens?

• 9:53Nathan Labenz?

  Yeah, it looks like it might actually be as simple as that because in the inference, it says during inference on the downstream task, we append M pause tokens to the prefix. And as always, we ignore the output of the model until the last pause token is seen. We term this pause inference. So at least when they're testing it, it looks like they're presenting these benchmark questions and they've got, you know, like grade school math and common sense and web QA and you know, there's like eight here, just in the, in the one graph of results. It seems like, you know, for those, these are just like straight up questions you're supposed to answer. And so they just.

• 10:36Nathan Labenz?

  Say, okay, you're gonna definitely pause and use these extra tokens. And then it looks like it can add more pauses. And it's still not quite clear how it's learning when to do that or not. You could easily imagine kind of setting up, certainly at the fine tuning scale, a data set that just like shows when to, you know, when you would want to pause. There's more work to do, I would say, to generalize this beyond the, you know, the current set of benchmarks that they're, they're running it on. Because here they're basically just saying, right, we're, we're in benchmark land. We're in question answering land. We're just going to give you some pause tokens at the start of, of your answer. And, you know, see how you do and, okay, cool.

• 11:18Nathan Labenz?

  It helps, but when to pause doesn't seem like it is something that the model has fully learned here. And so, yeah, but you could imagine, you know, also scaling this up. You honestly could imagine probably G P four helping you scale it up, you know, annotate these examples, you know, with where thinking breaks would be inserted. Like I bet it would do that perfectly well. And then, you know, next thing you know, you're, it can kind of learn, you know, when to do the, the pause breaks. I would, you know, I bet pretty well. Again, we've seen that with the backspace. So I would expect that something similar would be possible here too.

• 12:03Nathan Labenz?

  Yeah, it does. It is, there is one section on appending versus pre-pending the pause tokens. So it does seem like this is still kind of in the manual manipulation realm as opposed to a fully learned tool or, you know, technique that, that the model can use at its own discretion. Also, again, this was a presumably a PhD candidate, uh, working as a student researcher at Google, uh, who's the first author on this paper. So, you know, plenty of, uh, additional muscle there to take the ball forward a little further. Okay. So that's think before you speak. Next, analogical prompting.

• 12:44Nathan Labenz?

  This one kind of surprised me, honestly, but the more I thought about it, the more it started to make sense. So analogical prompting is presented as yet another improvement in how to just get the most out of the current language models that we have. They compare this explicitly to few-shot chain of thought and find that it can beat few-shot chain of thought. So this might be the new best prompting technique, and it's also easier to do than some of these other prompting techniques. What they do this time is they ask the model, first they present a problem, then they ask

• 13:28Nathan Labenz?

  the model to recall relevant examples and then solve the initial problem. And that's in contrast to, you know, few-shot chain of thought. Few-shot would be, here's a couple of examples. Chain of thought would be, you know, think step by step, et cetera. Few-shot chain of thought would combine both of those, where you have examples and the examples show the reasoning that you want. Here, you're able to just say, here's the question. It's on the model itself to recall the relevant examples and to then cycle back to solving the original problem. So the fact that this gives better performance, initially, I was like, wow, that seems weird because it seems like you're relying on the language model kind of a lot, right?

• 14:09Nathan Labenz?

  Like, when I give it few shots and I show the kind of reasoning I want, then I'm, you know, I feel like I'm doing my part, you know? I'm like guiding it to where I want it to go and showing it how I want it to behave. Here, it's responsible for generating the examples. This is not a tool use, there's no database here. It's just generating the examples, you know, with the same kind of generation as always. And yet, at least, you know, again, we'll see as this gets into the wild, but this is somehow better than chain of thought? How would I understand that?

• 14:45Nathan Labenz?

  So what I came up with was, maybe the right way to think about this is heuristic recall. Maybe what it's doing is...

• 14:59Nathan Labenz?

  It's sort of, and you know, we have these, got all these results, right? That have this kind of high level conceptual, middle layer sort of understanding. Maybe what it's able to do, when you say find relevant examples, is it's able to kind of load this problem into some high level of representation. It's maybe able to do a better job than you are, at least for this particular problem of, you know, based on these high level, quite decoupled conceptual representations. Based on that, it seems to be able to then zero in on the, a really relevant canonical example. And it's, you know, seen examples of so many things, obviously, right? That it has a lot there to draw on.

• 15:43Nathan Labenz?

  So it seems like it's maybe able, it's better able to pick the most useful example than your few shot example, you know, especially as you then take that to the diversity of whatever you're trying to do. So if it can locate a better example, and it's kind of memorized or, you know, learned the heuristic that solves that canonical example that it's able to load into place, then it essentially could get better few shot loaded from its own memory for this, this particular challenge than the hardcoded few shot that you try to cook up for it, you know, try as you might, right?

• 16:25Nathan Labenz?

  So I almost think of this as kind of self-rag, rag being retrieval augmented generation. And I think I'm going to do another episode on kind of the state of rag, maybe in the next week or two as well. But, you know, the, the classic rag setup would be, you have a database, the query, the question, whatever gets, you know, sent into the database to find relevant stuff, that gets loaded into context, and then, you know, you proceed from there with the benefit of whatever was retrieved out of the database. This is like treating the model itself as the database and saying, you know, you could embed that and go, you know, hit some vector database that has like a ton of examples.

• 17:06Nathan Labenz?

  But the model itself kind of represents all those relevant concepts and can generate relevant examples. And then once it has the, you know, most familiar, kind of happy example, then it can, you know, apply the same heuristic to the question at hand. And

• 17:27Nathan Labenz?

  It works better. You know, it was like, wow, okay, this is across a number of different models. So yeah, basically, it doesn't, it seems like up to the frontier models, you know, this still helps. It's not like a dramatic change relative to other prompting techniques. It is a pretty dramatic change relative to like just zero shot, you know, nothing. And then I wonder, this is probably most competitive with like a good RAG implementation. Cause you think, you know, I've been building some of these types of systems. I built a first version of a prompt coach for executive assistants recently. And I just did a, you know, few shot chain of thought implementation there where this is kind of a meta prompt.

• 18:08Nathan Labenz?

  Where the idea is like, you know, the executive assistants don't necessarily have a ton of experience prompting. They may do wrong things, you know, so can we identify things that they're doing that are suboptimal? And provide feedback on how to better prompt the AI for better results. It's a little bit like the, you know, improver, except we're trying to improve the humans prompts to the AI. So my few shot chain of thought is just like, okay, here's a number of prompts, whatever. I kind of just grabbed some random ones. I wrote critiques of these prompts that had these various shortcomings and, you know, showed what kind of feedback I wanted. And the system, you know, kind of works. And then I, you know, written out like, what are the next things I would do if I wanted to improve this further?

• 18:50Nathan Labenz?

  And I was like, I think the best performance that we could probably get to would be with a RAG setup where I would, instead of using the same examples for every prompt, I would go get specific examples that are most relevant to this. And then we could show, you know, and how big does that database have to be? I mean, that's where it starts to get a little bit scary, you know, because I don't want to have to write a thousand of those by hand. And that might not even be enough. I don't know. My few shot examples, just, you know, four or five, that was manageable enough, but a thousand, you know, that's a different scale of project for sure. Of course, I can have GPT-4 help me, but it's, um, we'll actually use Claude Instant for that, interestingly enough, because it's quite a few tokens.

• 19:35Nathan Labenz?

  We need the long context window and the instant is really good for the responsiveness. And seem to be performing almost as well as Claude 2. It was like at 10% of the cost and a lot faster. I decided to go with the cheap one. But anyway, this now starts to suggest. Maybe a different approach where like, maybe I don't have to do this whole rag setup and have a thousand examples and go find the most relevant examples. Maybe I can just have the language model recall the most relevant examples and then have it go from there. If there's one thing that that's probably not going to work on quite yet, it might be meta prompting.

• 20:12Nathan Labenz?

  Cause like, you know, it certainly has seen the types of questions that are being asked in this study, which are, you know, grade school math again, and that kind of thing. When you flip over to improving prompts, like depending on the training data cutoff or whatever, you know, has it seen huge numbers of like prompt coaching in its training data? Maybe it has, maybe it hasn't. If it has, it's probably because they've consciously baked that in as opposed to it like being out there on the internet. You know, as of the training data cutoff, that does seem like something that Anthropic might be a little head on because they do have this kind of constitutional approach where there is this like more, you know, more kind of iterative internal sculpting.

• 20:56Nathan Labenz?

  Obviously, OpenAI has a lot of that too. But anyway, I'm, that's my next test on my prompt coach is to see, hey, maybe I can skip this whole complexity of the rag setup. By just having the AI itself recall the most relevant example heuristic recall is kind of my, my own name for this. Because once it has that example and it knows how to solve that problem, then, you know, we kind of move pretty smoothly into applying that same heuristic to my particular challenge. It was, you know, one of the examples that they show in the paper is finding like the area of a square or something like that. And you think about this, it's fascinating. It's weird, but again, it has a certain logic. You can kind of see how this would make sense for yourself, right?

• 21:42Nathan Labenz?

  If you're like, okay, I'm a, I'm a middle school math student. I've been presented with a find the area of this square. And what am I going to do? You know? Am I going to look at four other math problems with like reasoning and be like? Now I understand what to do. Not exactly, not if none of them are finding the area of a square. Right. What I'd really do is be like, okay, how did I find the area of a square? Let me recall that, you know, let me recall the simple example. Now I'm going to apply that to, you know, the particular numbers and, and details of this particular problem. And so it does seem like a little bit more human like way, you know, human like behavior. Because that is, that is more like what I would imagine myself doing internally.

• 22:26Nathan Labenz?

  If I was trying to solve, you know, a similar problem. And I'm going to, I'm going to look for the one that is most analogous. Recall how I did it and then, you know, proceed in the same way. That's available for free everywhere you want to prompt right now, by the way.

• 22:40Speaker 01

  That's crazy. It is. I mean, it's interesting that performs better than giving it a few shot examples. It just seems like another data point of just because you don't find something in the model with an AI prompt, doesn't mean it's not in there somewhere. And that there's some work to coaxing out the full, yeah, full capabilities of the models.

• 23:02Nathan Labenz?

  Yeah. Another little tiny detail of this is they seem to be using markdown for the instructions. And I just saw some chatter online the other day from a couple of people that are definitely super knowledgeable. Who said that markdown works best for OpenAI. Cause that's kind of how they tend to train stuff, you know, in their own processes. And XML tags seem to work best for Claude. And that, that is something that Anthropic kind of officially recommends is like to use XML tags. So just another little footnote, markdown for OpenAI.

• 23:43Nathan Labenz?

  XML for Claude, your mileage may vary on that, but this is showing the markdown format. And even for GPT-4, you're squeezing out a couple more percentage points on these math benchmarks. So pretty incredible. It works almost across the board. There, I think they show one.

• 24:05Nathan Labenz?

  One thing, basically, where it's not the very best. And in that case, it's like 0.4 percentage points lower than the chain of thought. But in every other thing that they show, it's a couple of percentage points, as much as like five percentage points higher. Yeah, go improve your apps.

• 24:25Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 24:29Nathan Labenz?

  All of these things, right? I said kind of earlier, like we're kind of climbing a level of kind of impact, you know, conceptual importance, perhaps, whatever. Now we're getting to a couple of things toward the end that I think are definitely pretty interesting, pretty notable. So there's two papers that specifically speak to the possibility of much longer context windows, and they do it in quite different ways. So first one is called Streaming LLM. And this is a paper out of Meta. The supervisor, Mike Lewis, you'll see his name on quite a few of the big Meta papers.

• 25:11Nathan Labenz?

  Basically, what they do here is they manage to extend the, not the context window exactly, but the length of text that the language models can handle dramatically with just a relatively superficial change in a way that applies to all the existing open source models. So they're able to show that this technique can be quickly retrofit to Llama and all the other, I don't know all, but like a lot. They've shown multiple different major open source models that they apply this technique to. And hey, it works across these already existing trained models out there in the wild today. So what are they doing?

• 25:57Nathan Labenz?

  It starts with this observation that

• 26:01Nathan Labenz?

  It's another one of these things where you're like, wow, that is a pretty simple observation. Kind of surprised nobody noticed that before, but you made a lot of, you made a lot of hay with it. So the observation is that there's this, they call it attention sinks, but let's just start with the purely observational. What they find is that as language models are making their predictions, the late tokens often, you know, they, they attend to tokens that are very close to them, often, you know, just one of the, the words immediately prior, which, you know, makes total sense, right? Cause if you're just, you're thinking, okay, I need to predict the next word. Well, the most important words are going to be the ones that just came immediately before, just for even simple things like

• 26:43Nathan Labenz?

  part of speech, you know, and just general continuity. So you're going to see this intense attending to the immediately preceding tokens. Then you see like, kind of random, but not that much attending to things that are farther back. Because, you know, most of the time, and again, just kind of sanity checking yourself, if you were going to try to predict the next word in a paragraph, you know, you'd be looking at those last few, you'd be, you know, maybe read the whole thing, but like, and a lot of the details in the middle paragraphs, you know, they're probably not super hinge-y, you know, for what the, what that next word is going to be. And so there's like, not a lot of attention into these middle things, although, you know, key things do matter. And then there's a lot of attention, for many

• 27:24Nathan Labenz?

  tokens, there's a lot of attention to just the very first tokens in the sample. And so why is that? It doesn't seem like it's super influential, but you know, what's going on? So the hypothesis that they come up with is that because of the mechanism where the sum of the attention for a given token across all earlier tokens has to sum to one, that's a constraint of just the way everything is calculated, you know, in the, in the computation.

• 28:03Nathan Labenz?

  The attention has to go somewhere.

• 28:07Nathan Labenz?

  So if there's nothing for a given token that is like super relevant, still the attention has to go somewhere because it's required to sum to one. They hypothesize that these intensive attentions to the very first tokens are what they call attention sinks. In other words, you've got to have this thing sum to one, but there's not really much here that like seems super relevant. So just kind of put it at the beginning. And, you know, then we can kind of somehow some other part of the network can like not, you know, not pay too much attention to that. Not to use attention in different ways, but I am. The, the prop, we're talking here about the proper attention mechanism.

• 28:51Nathan Labenz?

  And I'm just saying, you know, downstream, it seems like that high level of attention paid to the first few tokens is like somehow accommodated for in such a way where, you know, it's kind of expecting that and it's, it's kind of fine, you know, downstream performance wise. So what they then do is say, well, what if we,

• 29:14Nathan Labenz?

  and you know, there's been a lot of different techniques over time that have tried to figure out, well, how can I have an ongoing? How can I have a long running conversation with a language model? Especially when it was, you know, 2,000 tokens limit, you'd hit that limit pretty quick. Now with GPT four, 32K with call it 100K, you don't reach it super quick, but it's still nowhere near enough to have like a super long running dialogue. And so, you know, people tried various things. One thing that folks have tried is kind of a sliding attention window where you basically just only look back and tokens and everything before that you just kind of forget about. And that doesn't work for some reason.

• 29:58Nathan Labenz?

  And I don't think it was necessarily clear before this, like why that wasn't working. I'm sure it's entirely clear even still with this. But as an empirical matter, what they find is that.

• 30:10Nathan Labenz?

  If they keep the first however many tokens, which they now call the attention sync tokens, and they experiment with some different things where like they even just put some padding there, put some empty stuff so that there is this kind of designated place. And again, this is another thing where if you do it with a pre, you know, you can just do it. But if you do it with pre-training, then it will work even better. You know, I don't think we've seen the end of this. But keep those early tokens, those attention sync tokens there at the beginning, and then do the sliding window. Now it works. And now you can basically keep essentially the same level of model performance way beyond the actual context window that you have.

• 30:55Nathan Labenz?

  Somehow, the ability to put this extra attention that doesn't really have any other natural place to go, back to these early starting attention sync tokens that don't change, allows the models to kind of stay coherent. Where otherwise, what had been observed is as soon as you kind of get past and start to start to drop some of those early tokens, the thing kind of blows up and doesn't work anymore. Now it kind of stays coherent. You're still dropping, as you get to a long enough thing, you're still dropping the middle stuff. So if you have a 32K model and you've got, you know, 40K worth of text, you've got the initial attention tokens, then you're skipping 8,000

• 31:38Nathan Labenz?

  and you're just looking at the last 32,000. But the ability to not have to force everything into that window and continue to put some stuff toward the beginning into the attention sync allows it to basically stay consistent perplexity score for super, super long transcripts. And we're not talking about like a small difference in transcripts here. We're talking like going into millions of tokens. Like this goes a long way. So it is a little confusing. I mean, I think I sort of want to understand this a little bit better. Notably, it seems like the

• 32:16Nathan Labenz?

  Perplexity is staying basically flat. It does not seem like it's getting better. And I think that's consistent with the general understanding that, okay, for any given time, we are only looking back at our 32K or our 8K or whatever our attention window is. So anything that is being dropped before that, we're not able to take advantage of. So we're still kind of operating at a, whatever, a 32K or an 8K capacity. But we're able to slide that window out into the future. This could create some odd experiences for users. If you're like, well, you know, I had some interaction about this particular topic. Is it still in window or is it now out of window? That could be kind of weird.

• 33:00Nathan Labenz?

  You know, you could have some situations where it's like remembering, remembering, remembering, forgetting. The perplexity score as measured on these, you know, super long text things like doesn't show a pain point there. You know, it's just kind of humming along. It consists of whatever perplexity it can achieve with its 8K or its 32K look back like that's what it can do. And it can just now continue to do it on a rolling basis. But from a user standpoint, that might be somewhat weird. If like all of a sudden, you know, something that you did just know about, now you like no longer know about. So I don't think this is exactly the form in which this is going to hit production. But I do think it could be a

• 33:44Nathan Labenz?

  a pretty powerful aspect of something that I think could be kind of, you know, start to look more like a next generation system. So.

• 33:53Speaker 01

  Why do you think the beginning tokens are such more effective attention sinks than just whatever tokens are at the beginning of the sliding window?

• 34:04Nathan Labenz?

  I don't know. It's odd. Maybe they just don't look like a beginning. You know, that's kind of the best thing I can come up with. I mean, I do think you could say their hypothesis was pretty minimalist. It was just, they were phrasing questions slightly differently. Like, why would the attention sinks be at the beginning? And the answer was, well, the beginning is the stuff that all downstream tokens can see. So that's a natural place for that to go. So if you start with the notion that you need an attention sync, then it seems reasonably intuitive that it would be at the beginning. Why the attention sync behavior doesn't roll with you as you roll the context window through a long text?

• 34:48Nathan Labenz?

  I don't have a great intuition for that. And I was trying to come up with something. I think that's the best I came up with, just that maybe it doesn't look like a beginning. And maybe things that look like beginnings are kind of what it's looking for when it's doing that. And if you're kind of right in the middle, maybe it just gets straight up confused. This may be a reflection of how things are pre-trained. It does seem consistent across these models. So they've published, I believe they've already published the code. They have published that, yeah, the code is out there on GitHub. So this is a framework that you can apply to existing language models.

• 35:31Nathan Labenz?

  And they show that they are indeed applying it to Llama 2, Long Chat, a few different ones in here. Seems like it's a lot of Llama 2. This is out of Facebook. But there were others that were not like, yeah, Falcon, MPT. So quite a diverse set of different existing open source models that they've done this for. And it seems to work comparably well, basically, across all of them. Like, perplexity looks pretty similar. It looks pretty flat. You know, I was thinking, well, maybe it has something to do with how they're trained, like, the way that the text is run through them. If it's always kind of chunked in ways where it's like, starting with something that kind

• 36:11Nathan Labenz?

  of looks like a beginning, then you could kind of understand this behavior where it needs the thing to look like a beginning in order to use it that way.

• 36:22Nathan Labenz?

  I would be, you know, I'd be curious if there's a model, and I don't know, you know, people don't really disclose their full pre-training mix super often. But if we could find a model where they specifically like start in the middle in pre-training, you know, and just kind of take random starting points that could be, you know, mid-paragraph, mid-sentence, whatever, and just try to chop it up very noisily like that. You know, maybe that could be a way in which the rolling window could work better. And then maybe you wouldn't even need the attention sync dedication tokens. Maybe the, you know, maybe that attention sync behavior could roll. But I don't know of any models like that. And I don't even know that we would know, you know, even for these models that they study.

• 37:06Nathan Labenz?

  Does that often that level of detail is just not disclosed? That's my best guess as to what is going on there. And again, this works on existing models, right? So like now we're not far at all from people being like, oh, hey, I'll take my llama to whatever and just apply this framework. And now you can have running chat long-lived as long as you want. You don't hit that hard. The kind of user experience trade-off would be today you hit a hard end and it's the end. You gotta start over. Next generation, you know, with, with just applying this, now you can run forever and it'll at least stay coherent, but you're going to have amnesia for like, you know, stuff that has now rolled out of the window.

• 37:51Nathan Labenz?

  But it's not like catastrophic amnesia where you get totally insane right away. But you may have these kind of weirdnesses where you're like, when is that thing in or out? And, you know, how's that affecting me? And it definitely could create some, some strangeness. But again, I think there's a synthesis of kind of almost all of these things coming. Again, keeping in mind that you can, you know, train a GPT-4 model in a week. If you're a, if you're an inflection, uh, once you get all your stuff online, combining a lot of these techniques into one is, I think, into one model is like definitely not too far into the, the future. And it certainly doesn't feel like science fiction at this point. So one more paper and then I'll kind of sketch that out. So the last paper, Ring Attention.

• 38:33Nathan Labenz?

  This one is like the deepest tech. It's really at the intersection of hardware and algorithm. And no less than Ahmad Mustak from Stability said that the researchers, he's like, you know, brilliant researchers like this literally knock 10% off of global training compute needs with these improvements, which are impossible to predict. And I honestly don't even think that's necessarily an exaggeration. But, you know, for one technique to potentially chop off 10% of global compute needs means that probably just a lot more is going to happen, right? It's not, I don't think we're going to make any fewer H100s instead, you know, there's just more is going to get done with them.

• 39:17Nathan Labenz?

  So how does this one work? I caveat this one by saying I'm definitely not a big expert here, but one thing that is interesting to know about transformers particularly and, you know, more generally like models in general, is you can represent them in different ways. And, you know, that could be like, obviously you could represent them in code, which is how they're ultimately kind of represented. You can represent them in like linear algebra notation, you can represent them in diagrams. And different representations have very different trade offs in terms of the intuition that they help you to develop on the one hand, and then the actual like compute efficiency on the other hand.

• 40:01Nathan Labenz?

  So I think Anthropic has made great use of this in some of their research work where they're like, nobody would compute with this representation, but we find this representation to be the most intuitive for how we want to think about what is actually happening in the transformer. So we're trying to kind of decoupling like, how does the machination actually happen subject to the hardware and, you know, the RAM and the layout and all that stuff from the way we want to organize our thinking about it in our heads for kind of intuition building purposes. So just separating those for one thing is like a pretty important conceptual move. And what these guys have done is they have restructured the computation, seemingly identifying that

• 40:47Nathan Labenz?

  The current bottleneck can be worked around. Now, there's going to be, of course, some next bottleneck. But the next bottleneck seems to be like a much more appealing overall bottleneck. So they are restructuring the compute. They're notably, this is not a shortcut. It's not an approximation. This is still fully literal attention computed, you know, the same, to the same level of precision with no shortcuts. It's just a more efficient way of structuring that compute. And basically, it amounts to passing different things back and forth between devices instead of other things that used to be, you know, passed back and forth between devices as all this kind of information is flowing, right?

• 41:32Nathan Labenz?

  Because you've got the parameters of the model itself. You've got the data that's being represented and flowing through. You've got to, if you're doing training, you've got to keep in mind like all the gradients as well. Like if I tweak these things, how is that going to change? So you can do the backpropagation. So you've got like huge memory requirements and also huge data passing requirements between devices. So by optimizing this in a new structure, basically, here's a couple of key quotes. Ring attention lets you scale context length linearly with device count, breaking free from memory constraints. Scaling context doesn't mean a quadratic increase in flops per dataset.

• 42:17Nathan Labenz?

  For the GPU rich, you can go from 4,000 token context to 10 million token context on 175 billion parameter model for 150 times the training compute. What is that? That's a 250 increase in the context length, which, you know, we've all been kind of taught, well, attention scales quadratically. But with this reorienting of how things get passed around, it costs you 150 times more compute. Which is not even the 250x that you're getting in terms of the expansion of the window.

• 43:01Nathan Labenz?

  And certainly, nothing like 250 squared. So it does cost more to train a 10 million token thing versus a 4,000 token thing. But only two orders of magnitude more. And it's not like running away with, you know, a quadratic function. Then they also say for the GPU pore. If you have just eight GPUs, then you can expand your context by eight at just two times the cost. So if you were going to train a 4,000 token context window on eight GPUs. Now you could train 32 and it would only take you twice as long.

• 43:45Nathan Labenz?

  That's a pretty huge difference in terms of utility between a 4x and a 32 for just 2x the compute. And again, not an approximation, right? This is the full attention, every token to every token. So what I think is really incredible about this is.

• 44:07Nathan Labenz?

  You know, you can only put so much into the current token limits that we have today. And that's important at runtime, certainly, because you can, you know, you can only pack so much in there. Beyond that, it just can't handle it. But I think it's also probably pretty important at the training layer. And I recall Ilya from OpenAI saying, you know, giving this example of like, I think maybe you even mentioned this in one of our earlier discussions about the mystery novel where it's like, you know, you have maybe hundreds of pages, thousands, you know, it could be, it could be an epic, right? You can have thousands of pages all leading up to, you know, and the person who did it is blank.

• 44:47Nathan Labenz?

  And, you know, if you could only had read the last chapter, you're going to struggle, you know? If you could, if you'd read the whole book, you can do a lot better. So the longer the window is, the more opportunity there is to, yes, infer effectively, but also to just learn connections between things that are potentially quite remote from each other.

• 45:08Nathan Labenz?

  10 million tokens starts to give you the opportunity to put like whole bodies of literature into into a single token, right? I mean, you're the Great Gatsby famously fits into Claude's 100K. Now you're talking perhaps about 100 books, 100 books, you know, all being with full attention considered

• 45:37Nathan Labenz?

  for the next token generation. Now, would you run that at runtime for inference? Like, probably not. I think it's probably more impactful for training. If I really want to get, you know, into science, if I really want to start thinking about what, how can I use language models for like understanding DNA interactions, you know, things that are really kind of data heavy or where there's just, you know, so many possibilities for connections that are just not obvious. Then the 32K or the 100K is still just not enough to do that. You need more than like one Great Gatsby worth of stuff to, you know, start to draw these really far flung, non-obvious connections. But at 10 million, you know, and

• 46:21Nathan Labenz?

  there's no rule here that says it stops at 10 million. It's just that, you know, that's the one example that they gave. At 10 million, you can start to load up like really pretty serious amounts of data. And my guess is that of all the things we've talked about today, this would be the thing that would start to drive overall lower perplexities, right? When we talk about the sliding one, even with these attention sinks, like the perplexity is basically flat. You know, that just means like model confidence, the performance is basically flat. But that's rolling. Now you really allow it to learn from 10 million tokens at a time.

• 47:06Nathan Labenz?

  There's just so much more input there that it can learn from. So much more opportunity to make huge, different, uh, you know, long distance connections between things. It seems to me like This could be a huge unlock. And again, to put that at two orders of magnitude. Let's say, you know, you've got your GPT four today. And we know that, again, I keep coming back to that. Maybe that takes you a week, two orders of magnitude up from that would be 100 weeks. That would be two years. You know, we're not that there are a lot of H100s getting shipped. We're not necessarily that far from being able to train a GPT four. You know, kind of, I mean, it starts to get to be a different class of thing.

• 47:51Nathan Labenz?

  I think at 10 million tokens per thing for it to learn from. Uh, that's just like so ridiculous that, um, this feels like the thing where maybe you could see your way into like superhuman performance just because, you know, I can handle the whole Great Gatsby. I can kind of remember what I read, you know? I have a decent, I would like put myself up against the language model for, you know, making that guess as to who done it at the end of the mystery murder. But you now give me 10 million tokens. I can't hold that, you know? I can't do 100. I cannot do anything, you know, analogous to like full attention for a 10 million token thing.

• 48:34Nathan Labenz?

  And so if this, if this allows the models to make those connections at such huge length, it seems like this could be where you could start to see tipping into superhuman performance of like learning things that experts don't know. And if you're learning things that experts don't know, you know, now you're really into a whole new era. I mean, that, that's like probably the biggest, you know, threshold that everybody's kind of wondering if and when we might cross. And I would say there's a decent chance in my, you know, very subjective estimate that this could be the thing that. Could unlock, you know, the ability to learn things that experts don't know.

• 49:16Speaker 01

  Yeah, the paper seemed really cool. And I mean, I know for me personally, it'd be amazing to be able to load in an entire script and ask it, you know, are there any logical inconsistencies? What holes can you see? What should I work to improve? Which is impossible right now with this current size of the context windows. I also wondered, I just glanced at the paper last night after you sent it. It seems like the key, you know, breaks the context up into these blocks. And then the key value pairs get passed around the full ring of all the different blocks. So you still calculate the full all to all attention. No shortcuts is my understanding, and I think that's what you just said.

• 50:00Nathan Labenz?

  Yeah, no, that's my understanding, too. It's just unless memory, I think memory has been the current bottleneck with previous approaches. And now, not actually sure what the next bottleneck becomes, but you know, they're getting around this memory bottleneck.

• 50:16Speaker 01

  But I also do wonder if you really need to pass the key value around the entire ring. If each block could only see its neighbor, then each layer, each block, its field of view of the input grows exponentially. So I do wonder how much you really do need to pass it all around the full, the full ring? And how much you could just do attention somewhat locally in blocks? And then each layer, each block would see more. Cause like when I read a novel, I'm not thinking, how much does this word the compare to like this word? You know, the name just in 18 chapters earlier.

• 50:57Speaker 01

  But you build up almost hierarchically each section of the, of the context.

• 51:04Nathan Labenz?

  And again, there might be a difference here between training and inference. If you're trying to get a model to learn things that experts don't know, then kind of by definition, you have to like crunch a lot of shit. Right? Because you don't even really know what you're looking for. But at runtime.

• 51:21Nathan Labenz?

  Yeah, you don't, you know, there's probably a lot of shortcuts that you could take. So, you know, here's kind of my sketch for where this might be going, you know, just based on all of these results. Like, what does a language model look like? And by the way, we're, you know, we touched briefly on multimodality and vision toward the beginning. But this is not even to say like everything, but just kind of the stuff that we've, you know, marched through today. How does that end up looking if it's all kind of combined into, you know, a single system that, you know, that has all this stuff? I mean, for one thing, it might be just way more capable if this, you know, 10 million or whatever

• 52:01Nathan Labenz?

  type learning does allow it to learn, you know, more, a more nuanced representation of the world, then you might just have straight up higher capabilities. My guess is, though, you probably end up running it with a smaller kind of, you know, more manageable inference window most of the time. From the paper, the attention sinks, it seems like you can vary attention kind of on the fly, you know, they're like, pen and few more tokens here or whatever. It seems like we may be looking at something in the not too distant future where there's a ability to adjust the context window depending on exactly what you're doing. And sometimes you may need it to be longer and sometimes you may, you know, be fine with it kind of rolling because I'm not,

• 52:44Nathan Labenz?

  you know, if I'm having a single long running dialogue with my AI, you know, assistant, most of the time I don't need the way old stuff, but occasionally I do. And, you know, that might be something that could be kind of used dynamically on the fly. The thing that I think is kind of most interesting, though, that isn't quite here yet, but really is, I think, strongly suggested is some sort of retention built into the transformer. You know, trying to combine these ideas. You've got these attention sinks at the beginning, then you can kind of skip a bunch of stuff. We've seen that you can like have a pause token that allows you to think more and kind of just, just represents like more ability to kind of process and store information.

• 53:29Nathan Labenz?

  Then there's the backspace, we could, you know, we get out of distribution, we're not happy about something, we can back up.

• 53:36Nathan Labenz?

  It seems like the rolling window probably is the way, but that there's some sort of like highly compressed historical record that kind of comes to represent and allow you to retain information from stuff that is no longer in the context window. In a way, kind of like what you're describing where it's not necessarily every word, but it's like a higher level of representation that you can carry forward with you. It doesn't take that many tokens, but is like beyond the, you know, in the way that the pause is like beyond a token and just kind of represents space to store stuff or to process stuff. A place to store stuff that is kind of the historical record.

• 54:19Nathan Labenz?

  That seems to be the the next big thing that I'm really looking for. That all these other kind of tools and techniques would start to be able to take advantage of. And you could even imagine the real self rag would be like, okay, I've, you know, I've got these attention thing tokens at the beginning. I've got like these sort of historical compressive representations of like history. Now I've got like the conversation we're currently having loaded into memory. This is starting to sound like a lot more, you know, like what I feel like I experienced on a day to day basis. And then I kind of imagine like, oh, I'm attending now to this like slice of history, right? There might just be one or a couple of tokens in terms of its

• 55:02Nathan Labenz?

  size of representation, but which then corresponds to, you know, the real self rag would be. And here's the whole history that underlies that. So that, you know, we see the pause, we obviously seen many tool uses. But can you imagine something where as this history gets kind of compressed and added in on a padding basis toward the beginning? You could also start to identify, hey, I need that section of history in more detail for this particular thing. And I can see that because I can see the meaning, not necessarily at the token granularity level, but then I can call up the token granularity level, load that into my current context and, you know, get the granularity that I want

• 55:44Nathan Labenz?

  based on recognizing that some portion of previous history, which has been compressed into a few tokens worth of space, is what I need to, you know, to to draw on to do the current thing. Honestly, that does not feel far off. It feels like, you know, if we're sitting here six months from now and we haven't seen something like that, I would be quite surprised. You know, a version of it sort of was the RetNet paper, but it notably was like, there's a change to the architecture there and we'll see how well that generalizes. But what I'm kind of describing here seems like something that you could have even without, you know, any kind of additional fundamental architectural changes, but just like a training

• 56:29Nathan Labenz?

  for this kind of compression for future recall that, you know, you could still kind of attend to in the normal way, pluck it out as needed, not even necessarily have to go to the database all the time to have like the conceptual backing, but the ability to go to the database sometimes when, you know, that seems like it's super relevant.

• 56:50Speaker 01

  Kent, so you're saying like, like a memory just in language that you could then treat as like a database that you can recall certain slices from it. You don't, you're not even saying like some hidden state in latent space, like, you know, the old LSTMs that it's also carrying forward.

• 57:08Nathan Labenz?

  Yeah, no, I sort of like that. So like by analogy to the pause and by analogy to the backspace, every so often, I think you could train the model to represent some token that would be like a memory token that would represent the high level conceptual content of that memory that then could just be included as context, but super efficiently and not have to take up a ton of tokens, not have to have token level resolution, but which would kind of, it would be in still the same latent space as the model is operating, but not necessarily accessible via direct

• 57:50Nathan Labenz?

  tokens, but only sort of the kind of thing, kind of like the pause where, you know, this gives you an extra space to like put, manipulate, do stuff with data. Here, the, the purpose of it would be to say, I want to kind of summarize this, not in text, but in some representation in the space that I can come back to later, uh, super efficiently at like high conceptual, uh, meaning, but like low resolution in terms of the actual exact language that was used.

• 58:23Speaker 01

  Very cool.

• 58:23Nathan Labenz?

  We're not at the end of history. And that's kind of, if I had to guess what would be the next big thing that would unlock a ton of stuff, it would be, it would be that kind of, you know, it seems like we're so close to all those pieces being there. With the rolling and, you know, with the, with the pauses, with the backspaces, with the like, kind of version of self rag that we're already seeing. Imagine that self rag thing had these memories that it could go back to, you know, as well, right? So now it's not just looking at its own pre-training, but like also the conversation that you had. And able to say like, it looks like, you know, on this slice that represents this whole exchange, you know, it looks like that is where this was kind of, you know, covered. And so maybe that's enough. Maybe I have to go deeper into the actual transcript.

• 59:08Nathan Labenz?

  But either way, you know, to be able to, to compress and kind of, you know, in a way that I think would start to represent, would start to look more like what we're doing, right? Because as you said, you're not keeping every word of the novel in mind, but you have these kind of vague associative, high level, more representational, not token by token sort of things that are like much more easy to recall. And of course, you don't have the actual, you know, token by token at your command. The, you know, that's another reason I think superhuman performance definitely cannot be ruled out because if we can figure out the high level representational thing that's more like our kind of loose, but like really relevant memories,

• 59:51Nathan Labenz?

  then it'll be way easier to connect the computer's version of that to the actual raw transcript than it is for us, you know, to go back to our, you know, raw transcripts, which, you know, obviously we just basically usually don't have. Yeah, that's where I think this is going. Not super confident, but I'd, I'd bet we see some of this.

• 1:00:09Speaker 01

  It makes sense to me. And it feels like, I mean, if we figure out that recall element and then add in some planning algorithm and a little bit of scale, more scale, major breakthroughs.

• 1:00:23Nathan Labenz?

  Yeah, the timelines are, are not necessarily very long. I, I think it's, um, more I think about it, the more it does seem like the next couple of years could get really, really crazy. It's been fun. Any concluding thoughts on your end?

• 1:00:37Speaker 01

  No, this is very fun and, you know, that I'm working all day. This was a great way to catch up on, on what's been happening.

• 1:00:44Nathan Labenz?

  Well, that's the goal. It's, uh, it's too much for any person, you know, and this is by no means everything, but trying to give the, you know, that kind of middle depth where hopefully people walk away with some real understanding and, you know, some, you know, some real food for thought, but can hopefully do it, you know, in a compressed way that, um, that allows people to get a survey of a increasingly crowded and noisy landscape. With that, I'll say Trey Kollmer, thank you for being part of the cognitive revolution.

• 1:01:13Speaker 01

  All right. Have a great day, man.

• 1:01:14Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.