Titans: Neural Long-Term Memory for LLMs, with author Ali Behrouz

In this episode of The Cognitive Revolution, Ali Behrouz, a PhD student at Cornell University, delves into his research on enhancing memory mechanisms in large language models through his latest paper titled Titans. Behrouz discusses the limitations of current models in maintaining long-term coherence and introduces the concept of a neural network as a memory module. Highlighting architectures such as memory as context and memory as gate, he explains how these innovative approaches can significantly improve long-term memory retention in AI systems. The discussion also touches upon challenges such as catastrophic forgetting and the need for more effective models in reinforcement learning and decision-making tasks. This insightful conversation sheds light on the future directions and potential applications of advanced memory mechanisms in AI. SPONSORS: ElevenLabs: ElevenLabs gives your app a natural voice. Pick from 5,000+ voices in 31 languages, or clone your own, and launch lifelike agents for support, scheduling, learning, and games. Full server and client SDKs, dynamic tools, and monitoring keep you in control. Start free at https://elevenlabs.io/cognitive-revolution Oracle Cloud Infrastructure (OCI): Oracle Cloud Infrastructure offers next-generation cloud solutions that cut costs and boost performance. With OCI, you can run AI projects and applications faster and more securely for less. New U. S. customers can save 50% on compute, 70% on storage, and 80% on networking by switching to OCI before May 31, 2024. See if you qualify at https://oracle.com/cognitive The AGNTCY: The AGNTCY is an open-source collective dedicated to building the Internet of Agents, enabling AI agents to communicate and collaborate seamlessly across frameworks. Join a community of engineers focused on high-quality multi-agent software and support the initiative at https://agntcy.org/?utm_campaign=fy25q4_agntcy_amer_paid-media_agntcy-cognitiverevolution_podcast&utm_channel=podcast&utm_source=podcast Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive NetSuite: Over 41,000 businesses trust NetSuite by Oracle, the #1 cloud ERP, to future-proof their operations. With a unified platform for accounting, financial management, inventory, and HR, NetSuite provides real-time insights and forecasting to help you make quick, informed decisions. Whether you're earning millions or hundreds of millions, NetSuite empowers you to tackle challenges and seize opportunities. Download the free CFO's guide to AI and machine learning at https://netsuite.com/cognitive PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, I am thrilled to share my conversation with Ali Behrouz, PhD student at Cornell and lead author of the fascinating paper on integrated large language model memory, Titans: Learning to Memorize at Test Time. This paper represents another significant step forward in addressing what I've often called "the missing middle in memory for large language models." We've got world knowledge baked deeply into the model weights, and we've got attention-based working memory in the context window. But that missing middle layer, the integrated, persistent, and ever-evolving longer-term memory that humans use to maintain coherence and identity over time, still seems like

• 0:44Nathan Labenz?

  a necessary piece for success in high-context environments and has been a frontier to watch in AI for some time now. It was once famously said of computers that you could see them everywhere but the productivity statistics. And to a lesser degree, that's been true of AI as well, at least so far. In part, this is because the technology itself is really only recently useful, and people take time to adjust. Tyler Cowen recently went reverse Oprah, pointing out to an influential audience and saying, "You are a bottleneck. You are a bottleneck." But I think often, the bigger barrier is best summarized by another famous Tyler quote: "Context is that which is scarce." To be honest, I've never been entirely sure what Tyler meant by that in the context of

• 1:27Nathan Labenz?

  humans, but in the context of AIs, it makes perfect sense. Today's AIs know a literally superhuman amount about the world at large, but out of the box, they know little to nothing about the individuals and businesses that they're meant to serve. Meanwhile, assembling and maintaining relevant context for them, especially since it's often spread out across Slack, email, documents, GitHub, meeting transcripts, task management systems, you name it, that is tedious work. And the AIs have really only recently, I would say truly with Gemini 2.5 Pro, started to properly reward it. But for the sake of argument, imagine a world in which context is not scarce for AIs.

• 2:08Nathan Labenz?

  A world in which an LLM trained specifically for a major company, say GE or 3M, for example, knows as much about that company, including its products, its history, its team, its internal processes and debates, its finances. It knows as much about the company as it does about the world at large. Obviously, such a model could be created. No company has more than 1% of the data on which the AIs are already trained. And if it were created, it would immediately know more about the company than any single person at the company. It might still be trained to search official records to ground its analysis and work, but unlike today's model, it would know, in a way very similar to humans do, when it's actually

• 2:49Nathan Labenz?

  found what it's looking for. And it would almost certainly pick up many of the subtle patterns that constitute what I call "how we do things around here," which could make it relatively easy to manage, especially as compared to the overall process of hiring, onboarding, and retaining human knowledge workers.

• 3:08Nathan Labenz?

  Startup costs for a model like this could reach into the millions or perhaps even tens of millions of dollars, which is notably roughly where OpenAI has publicly priced their custom models offering. But for a 100-year-old enterprise, that would still be a bargain for an AI that you can drop in and have do a significant portion of the work at the company, particularly since you'll be able to amortize those costs across as many copies as you need. For smaller businesses, meanwhile, which have much less data, using today's fine-tuning prices as an anchor, I would expect costs to be more like tens of thousands, maybe into the low hundreds of thousands of dollars. Obviously, still affordable. We're starting to get a glimpse of this future as individual ChatGPT users with their new memory features these days, but I'm not aware of a productized version of this that

• 3:53Nathan Labenz?

  works well at scale. And I think that's ultimately because there's still the gap in the foundation models themselves, which scaffolding isn't quite enough to fix.

• 4:03Nathan Labenz?

  The bottom line then is that it seems plausible to me that the main thing between where we are today and a future full of drop-in knowledge workers that begin to very quickly and dramatically disrupt the labor market is simply a breakthrough in long-term memory. And that's why I think today's topic, Titans, is such a big deal. Unlike RAG-type systems, which store data explicitly and then make it searchable via a mix of traditional or embedding similarity or graph searches, or even more integrated strategies like Mamba and other state space models, which encode memory as a matrix of numbers that gets updated as part of each forward pass. The Titans architecture that Ali and his co-authors propose uses a neural network,

• 4:45Nathan Labenz?

  which itself is updated via gradient descent at runtime as the LLM's memory module. This is a qualitatively different approach and, in my humble opinion, represents a significant conceptual advance. So with all that in mind as motivation, in this conversation, Ali and I explore the technical details of Titans, including how he conceptualizes and takes inspiration from human memory systems, how the associative memory loss function works, the role of surprise and momentum in updating the neural memory module, and the various approaches they experimented with for integrating this long-term memory module with the standard attention mechanism. Beyond the technical details, this conversation also offers a fascinating glimpse into how

• 5:28Nathan Labenz?

  a highly original and obviously quite brilliant researcher thinks about pushing forward in a relatively new architectural direction. I was really struck by how many times in this conversation I wanted to dig in and understand the reasons behind the decisions that Ali and co-authors had made. But his response was basically that they see all of this as very early foundational work, and so they just did the simplest thing possible for now, trusting and expecting that others will come along to improve and refine it later. Overall, I really enjoyed this conversation, and while it is pretty technical at times, I think Titans is one of those relatively few papers that is worth taking some time to grok. The intuitions behind it are elegant, and this line of work might just produce the

• 6:11Nathan Labenz?

  last major technical unlock needed for AI to hit an inflection point in economic value and impact. As always, if you're finding value in the show, I'd appreciate it if you'd take a moment to share it with friends, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we welcome your feedback either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. And finally, a quick reminder, I'll be speaking at Imagine AI Live, May 28 through 30 in Las Vegas, the Adaptive Summit, August 12 and 13 in Sao Paulo, Brazil, and the Enterprise Tech Leadership Summit, September 23 through 25, again, in Las Vegas. If you'll be at any of those events, please send me a message and let's meet up in person. For now, I hope you

• 6:56Nathan Labenz?

  enjoy this technical deep dive into the frontier of large language model memory with Ali Behrouz, PhD student at Cornell and author of Titans.

• 7:09Nathan Labenz?

  Ali Behrouz, PhD student at Cornell and lead author of the fascinating new paper on large language model memory Titans. Welcome to the Cognitive Revolution.

• 7:19Ali Behrouz?

  Thank you very much. Thanks for having me.

• 7:22Nathan Labenz?

  I am really excited about this. I think every once in a while, one of these papers comes along that just has me thinking and thinking and thinking, and this is one of them. So I am really excited to get into it. I guess just for a little background, I have kind of had a little obsession with the memory aspect of large language models for a while now. So when I saw the Mamba paper drop roughly a year before the Titans paper, I thought, boy, this is really a big deal because I think everybody kind of knows that large language models have this incredible world knowledge that's super vast, but it's like fixed, right? And, you know, the knowledge cutoff date can be somewhat recent, or it can be more in the

• 8:06Nathan Labenz?

  past. And whatever it is, that's sort of a crystallized thing that doesn't really change too much between versions of a model. Then, of course, we've got like the runtime memory that is the context window. And the attention mechanism, you know, is great for like connecting all the different tokens to each other token. And making sense of what's going on there. But I've often called the sort of gap between those like the missing middle in memory. And, you know, you can fill the context window pretty quick, but it's really hard. It's proven really hard for the field to create a sort of sustained coherence where a model can like kind of know who it is, like know what it's done, you know, know where it's trying to go.

• 8:48Nathan Labenz?

  And I think that's going to be a really important piece of the overall puzzle as we think about like trying to create effective agents. You know, that might be the long running in the real world. So Mamba was a key moment where I was like, oh, it seems like that's a major step forward. And this is, you know, with a different approach, another one of these things that I think really kind of shines a lot of light on the path. With that preface, tell us just a little bit about your background, how you got interested in this work. What angle are you coming to it from? I think the paper is really interesting in that it's got some elegant math. It's also got some clear analysis of just kind of the recent history of people's attempts to bridge this gap. And it's even got a little bit of touching on just kind of human memory and takes inspiration in some ways from that.

• 9:33Nathan Labenz?

  So I'd love to hear your kind of inspiration for this work, maybe for starters.

• 9:38Ali Behrouz?

  And so I'm listening. One thing that I need to say is like my past this direction is very unusual. I started doing research during my masters mainly. And so I started working on some graph algorithms and then these kinds of things completely like very, very far from this area. So like when I did some research, I realized that some of the tasks that I'm trying to do are much easier. When we use some graph neural networks and these kinds of things. So I started learning more about graph neural networks and do these kinds of things for those kinds of tasks. And so.

• 10:17Ali Behrouz?

  I mainly focus on some like questions about anomaly detection and these uh kind of tasks in graph learning. And uh like at the end of my masters, I realized that there are some interesting connections between something like anomaly detection in graph algorithms, in graph data sets uh and also uh some neuroscience tasks. For example, let's say want to detect some disease or for example, want to understand some of the disorders, brain disorders, or for example, yeah, generally these kind of tasks.

• 10:55Ali Behrouz?

  Uh and so I started learning more about some uh neuroscience concepts and also understand how how like different part of the brain works. And so that was a part make actually made me to be more interested to like these be be more interested in these kind of approaches, like neuroscience inspired methods for deep learning. And yeah, generally, my intuition is that, you know, humans are are very effective. Or I mean, the learning process in human are uh is very effective, is very efficient. We can just learn with with like a small number of samples. And actually, that's that's the results of millions of years of evolution.

• 11:40Ali Behrouz?

  So basically, it's it's really hard to believe that we are smart enough that we could like beat millions years of, you know, evolution and then come up with one architecture that is even more efficient and more effective than humans without like passing those steps. You know, we we can we can come up with one architecture that is very similar to humans and then improve it to to, for example, be the super intelligence, wha- whatever we want to call it. But I I really believe that before that, I mean, at this time at least, we need to like get inspired from how our brain works.

• 12:19Ali Behrouz?

  So actually, the the start of my PhD was was somehow like coincide with with so many great papers about like Modelings and then some alternative architectures. I think at that time there were like S five paper, which is like the first paper that that like introduced the S can algorithm for city space models and also there were Brits and all these kind of great models at that time. And so all of them were somehow like some motivations for me to try to understand what we can do for some alternative architectures and then transformers. But personally, I think one thing that is different.

• 13:01Ali Behrouz?

  I mean, you know, there are several people who are working on alternative architectures. And actually, each of them has has their own perspective. And then basically, all of them are, I I believe that all of them are really good and great. But you know, there are some people who doesn't believe in attention and they they want to like fully replace attention with like RNN. And there are some other people who believe in hybrid models. And I'm I'm one of one of the people who believe in hybrid model. Because I really think that the attention part is really necessary for some accurate modeling of like independence of like dependencies between different tokens in the contexts that we have.

• 13:43Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 13:49Nathan Labenz?

  Yeah, I'm with you on the instinct that the hybrid models are ultimately the way to go. So quick kind of review of just what is the problem with the attention mechanism? You know, it works great, but the memory requirements as you keep extending the sequence just get longer and longer because you are doing this all token to all token calculation. So the size of that memory footprint grows as the square of the sequence. Like, I think everybody watching this probably already knows that pretty well, just in case there's the kind of core problem that we're trying to solve. So that can't go on forever. Now, I guess you could have, and we have seen also some schemes where it's like

• 14:30Nathan Labenz?

  not necessarily part of the model itself, but you can have sort of maybe arbitrary disk space. But arbitrary disk space is quite different from arbitrary like in memory, you know, computation. So it feels like intuitively we need some. Thing that is more like the human brain, which is like it's finite size, right? It's not, our brains are not growing with every time step through our lives. And we need some like elegant updating mechanism for the memory that allows us to keep what's important, but knowing that it is a finite state, that means we also kind of have to let go of some stuff over time as well.

• 15:08Nathan Labenz?

  And intuitively, we all know that our memories do this, but we haven't quite cracked the perfect way to do that in the context of an AI system. I kind of wanted to take just a couple of steps for a little bit more foundation. One would just be to get your sense of kind of how you conceptualize the, the human system. Maybe get a few thoughts on your, like how you understand the linear approximations of attention. And then maybe a little bit on kind of your, how you understand the state space, you know, or kind of maybe slightly more generally like

• 15:43Nathan Labenz?

  the prior attempts to create these sort of finite, you know, constant size, constant update time architectures like Mamba and RetNet and others that have, have come before. So if you want to take those one by one, give us any, it doesn't need to be a full lecture, obviously, but just kind of, you know, if there's anything that you feel is sort of distinctive about your perspective on those three topics, I'd love to hear how you conceive of them and how they kind of motivate this work.

• 16:10Ali Behrouz?

  So let me start with like some explanation about like traditional RNNs. So let's look at what RNNs are doing in, in very high level. Basically, the data comes, we project the input or the data into the hidden space. And also we project our memory, which is like a vector. Usually people in the literature call it like hidden state. And we project that and we just add these two together and update the memory and we repeat this process.

• 16:44Ali Behrouz?

  Anytime that we want to like extract something from the memory, we just multiply the hidden state or our memory by a matrix like C. Usually H times C or C times S, depending on how you define the dimensionality, it equals to the output, which is like Yt. So that's how linear RNN or generally RNN with non-identity, we can simply apply that here works. Now, let's go to the next model. The next model is like linear attention. Linear attention in the causal setting can be, again, uh, be written as like recurrent neural networks. Again, everything is similar.

• 17:28Ali Behrouz?

  Data comes, we project the data, there is a hidden state, and we update the hidden state and so on and so forth. But now, this hidden state is a matrix value memory. So if we want to somehow make some, like, say that there are some differences between linear RNN, traditional RNN, and also, on the other hand, the linear transformer. One thing that we can see is that traditional RNN and also modern linear RNN, like for example Mamba, are using vector value memory. But on the other hand, linear attention are using matrix value memory. Now, let's say that we know all these things.

• 18:13Ali Behrouz?

  The question is, what is memory in our brain? Is it something like a set of neurons that we define? Or it's something like a neural network? Larger sets of neurons that are interconnected? And actually, I think this is the second one. We we cannot say that there are like these three different neurons that are encoding all the memories and all this stuff and everything. Uh, so, you know, usually I, when I, when I talk about Titan, I start my talk like with some explanation about different perspectives. So I think here it might be useful to like, again, start from that one.

• 18:56Ali Behrouz?

  Uh, you know, there are some different perspectives that over time actually help us to design new architectures. When we are like thinking about L S T M and that kind of model, they're inspired from brain. And actually, you know, that's, that's long-term, short-term memory, you know. Uh, we wanted to again, design something very similar to our brain. But, you know, the tools that we have at that time, our understanding of the brain and all these things are, are, are getting changed over time. So basically, that's the main difference from the traditional perspective on the human memory and like current perspective.

• 19:36Ali Behrouz?

  Another thing, when the time passes, we have like the transformer era in which most of the focus is on like efficiency, matrix multiplication. We have like a lot of models that are designed based on, for example, efficient matrix multiplication algorithm, like M L P mixer and then its variants, like a structured matrix and all these things. So basically, that's another perspective that we have. And you know, each of these perspectives gives us some new intuition, some new paths, uh, passes that, that we can like go and find new architectures. Probably starting from two thousand twenty, we have the perspective of dynamical systems.

• 20:20Ali Behrouz?

  I mean, treating the sequence as a dynamical system and use status-based models for modeling sequences. So again, we have like some new intuition, some new insights about how we can design efficient and effective sequence models. And I really think that now we need to go farther and go back to the Human memory perspective, but with some novel understanding of how human memory works and how we can incorporate some novel techniques that we are learning. So, you know, in this perspective, the question is, what are we missing in in previous architectures? That we need to like address in this perspective. And the first thing is that our memory is not something that

• 21:06Ali Behrouz?

  we can like break into pre-training and post-training. It's constantly learning. And basically, that's a very important part. That's very good, for example, R N N, because now from the T T T paper, we know that most of these R N N, or all of them, are like test time training, are doing test time training. So basically, it seems that we are like good in that shape with modern R N N, but probably we might face some I mean, we need to do additional things to make them more effective in test time training. Another thing that we might

• 21:48Ali Behrouz?

  miss in this like perspective is, as I mentioned, the shape of the memory, is it like a vector, is it a matrix? Or is it neural network that might have different architectures, different like designs and all these things? So I I believe in the last one, I I don't think that, for example, it's suitable to say we could design architectures with with so much data and like scale them. And all of them are done with only one vector in the, you know, as as a memory of of your model. So that's somehow, I think, oversimplification of of this design. So probably we need to take more complicated architectures as our memory.

• 22:32Ali Behrouz?

  Another thing that we need to consider in this perspective is let's say that we have some recurrent neural networks. These recurrent neural networks actually have a memory that is fading over time.

• 22:48Ali Behrouz?

  On the other hand, we have this attention which consider all the pairwise interactions between the tokens within the context window. And actually, that's very similar to like our short term and long term memory. Because it's really hard or even impossible to hallucinate about the information that we just got in the past thirty seconds or twenty seconds. Because we really have all the information into our memory. We don't like hallucinate about any details like about a piece of information that we just got. But it's very likely that we hallucinate about information or one event from twenty years ago. There are so many details that we don't remember.

• 23:33Ali Behrouz?

  There are so many details that are even changed. For example, we remember something that is not true at that time, you know? We are just hallucinating about that. So if we think about RNN as a fading memory, we can see that this decay mechanism, this inherent decay mechanism, not before you get gate blocked. Because you know, in any design, RNNs have this decay mechanism. We are always adding new data into the RNN and we might ignore some of the data that we have. So this RNN is very similar to our long term memory. It's fading, it has the ability to manage the information and all these things.

• 24:17Ali Behrouz?

  On the other hand, we have this attention part, which is very accurate and it's very similar to our short term memory. I think we need to use the combination of these two to design more powerful architectures. But the question here is, how can we do that? Because even in the neuroscience literature, there are some like controversy that how we are passing memories from long term, sorry, from short term to long term. And for example, how we are doing these things. So basically, that's a part we do not have anything to get inspired from. So that's a challenging part and probably.

• 24:59Ali Behrouz?

  An important research pass in the future because you know, uh, we might like design some models that are more effective in passing memories from short term to long term and all these things, you know? So basically that's the architecture that I believe in. And so generally, this kind of thinking led us to like the design of Titan.

• 25:22Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 25:27Nathan Labenz?

  Cool. So I guess one thing just to really emphasize is in some sense, the most like fundamental change here with this work as it compares to everything previously that I've seen is that the memory module in Titans is itself a neural network. So we're kind of moving from a, and you said this, but just to really hammer the point home. The earlier recurrent architectures had either like a vector or, you know, a 2D matrix or whatever, but basically just a bunch of numbers that information would be projected into and then also sort of projected out from at each inference time step.

• 26:11Nathan Labenz?

  But within that, you just have these numbers and they don't interact with each other. There's not really much going on there, right? It's, it really is just sort of a place to store the results of these kind of projections in, projections out. In moving to a neural network as the memory module and you use an MLP. And, and I definitely noted the part in the paper, he said, this opens up a whole new research direction in terms of like, what, you know, is the best possible architecture for that sub module because an MLP, you know, maybe hard to beat, but like probably beatable. We've certainly seen it beaten, you know, many other, for many other purposes. But moving to this architecture creates a dynamism within the memory module

• 26:53Nathan Labenz?

  and creates the potential for sort of information to interact within that piece that is qualitatively different and seems like, as you said, like much closer to the way that our own memory systems work. One thing I don't have a great sense for, and I'm sometimes surprised by is How much of this is all sort of reformulatable, if you're smart enough about the math, such that things can become more equivalent? So this is not intuitively obvious to me, but the linear approximations of attention, you noted, can be sort of reframed as a recurrent neural network.

• 27:37Nathan Labenz?

  And one of the big things that we've seen through this whole progression of the linear approximations, and then to Retnet, and then to Mamba, and Mamba 2, and many others besides along the way, is a much more granular management of that internal state. And also, with Mamba in particular, a jump to input-dependent ways of managing that internal state. So like, early on, you would just have kind of, okay, here's this date. We have like a fixed kernel that's gonna do this projection. And no matter what information comes in, we always kind of compress it in the same way. And that is what it is. With Mamba in particular, you had an input-dependent

• 28:22Nathan Labenz?

  and highly granular way of updating. So if I recall correctly, each number in that matrix could be sort of updated with a different strength. So you weren't just doing sort of one overarching single decision for how much to decay and how much to emphasize the new information. But you're kind of doing it in a much more granular way. And again, that depended on the inputs. So that was quite interesting. Another element of this that also always comes up is just like, how much of this is dictated by first principles? And how much of it is dictated by the hardware that we have available to run it? Mamba versus Mamba 2, I thought was a really interesting illustration of that,

• 29:05Nathan Labenz?

  where Mamba 2 was actually a less granular method for managing that internal state. It was a coarser update function. And you'd think, well, geez, like, how do they get a better model out of a less granular or more coarse? way of managing the memory. And the answer, as I understand it, is basically in making that sacrifice of that super granular, you know, Mamba one structure, they were able to make the whole thing run a lot faster. And so for a certain number of GPUs, they were able to just train a lot more.

• 29:44Nathan Labenz?

  And so you get a better ending model, although it does have in some sense this like coarser internal structure. I guess to, you know, put that in the form of a question, how much of all of this do you see as being like very dictated by the hardware that's available? Is there a fundamental break here when we move from a matrix shaped memory module to a network as the memory module? Or is there some fundamental equivalence perhaps as well, where this could all be kind of with enough insightful math that could be sort of understood as being in some sense the same thing?

• 30:23Ali Behrouz?

  So let me use the last part, I mean, last page of the Titan paper that we did some like comparison between, for example, Python and some recent modern architectures like Deltonet or for example, TTT Longhorn and all these models. So what we did at that part is to like saying how we can connect, for example, Deltonet to Titan and what is like changing in that sense. And now let's say we have the perspective of TTT, what is changing in that sense and so on and so forth. So one thing that we need to consider is, you know, all these models are connected, you know?

• 31:08Ali Behrouz?

  There are some mathematical formulation that we can use. I mean, at least in the sense of these test time training framework that, for example, unify all these things. And so basically all of them are connected and there are some small changes in each of them. But I don't see. The contribution from that side, you know, I really believe that the value of the contribution is to show how you want to do some future stuff. Because probably new architecture is like endgame. Or nothing is endgame, we are progressing over time, you know, it's not good even for science to say that

• 31:52Ali Behrouz?

  this model is coming and it's the like end of the world, we are not making any progress over time. Even transformers are really great. Even when we have the transformers, we have better implementation of transformers like flash attention, all these things. We have better additional components to transformers, make them transform cross plus and all these things, you know. We always are making progress. So the important part, I really believe in that, the important part is not just to deliver a model.

• 32:25Ali Behrouz?

  Because probably there are a lot of things that we can do to make a model better. But to deliver a new perspective, deliver something that leads to future studies. So personally, if I want to like say, what is the impact of one work, for example, from like very old time, I would say that the impact is something that's like shows a future work and and those future studies led us to this state that we have right now. So I think that's a really important part. So yes, we can look at all these models that are very connected, but most of them are from different perspectives.

• 33:10Ali Behrouz?

  And each perspective can lead us to different type of thinking, how we want to improve current architectures and all these things. And the question is, we don't know actually the answer to this question, but the question is, which of these directions will, for example, lead to a good architectures that is like better than other methods that we know at that time. So for example, let's say that we use a matrix value memory in Titan. And for example, let's say we remove some layer normalization. That we are doing in inside the memory, which make the memory like linear.

• 33:52Ali Behrouz?

  Uh but let's say that we we don't do all those things. So basically, in that case, our approach can be very similar to, for example, gated delta nets or delta nets. So, you know, if you look at these things, you can say that's like. There are all small differences between these models. For example, as I mentioned, for example, you can remove some of the components that we have. And also like consider the special case of our approach, which we use like simple matrix format for the memory. Then, you know, there are some connections to these models, similarly for T T T, similarly for Longhorn and all these models. But, you know, even a small change in these kind of approaches.

• 34:37Ali Behrouz?

  Can lead to completely different architectures. I mean, regarding the perspective that it can bring and all these things. So it's really important to see what is the connection between these models. And also which type of approaches can help us to design the future architectures. And in this. Direction, I can say that the focus, you know, we have a fixed size memory. And this fixed size is something like lifelong memory for the model, you know, we we are we are adding data into the memory. And so the question here is, which type of approaches do we want to take?

• 35:23Ali Behrouz?

  One approach is to make memory management better over time. And what does the memory management here means? It's just a recurrent formulation that we have. So different recurrent formulation help us to better manage the memory and better understand each information. Like is worse, like memorizing or which information we just need to ignore. So that's the important parts, in in my opinion, but also there is another.

• 35:54Ali Behrouz?

  Perspective here that says we we need to do some stuff to have better memory management. But on the other hand, we need to see how we can use different architectures for our memory to make it more powerful. So we started from a vector, then goes to a linear layer or a matrix. And now we can have MLP with two, three or four layers. So what's next? Do we want to use convolutional neural networks inside the memory? Do we want to make more deep architectures in the memory and all these things? So that's another way that we can approach this problem.

• 36:40Ali Behrouz?

  And I I really think that these two different approaches are are very separate. So it's very hard to connect these two because they are answering different questions. And we need to see which one is more promising. And definitely, we can use both in future architectures. We can have better memory management with better memory architectures. But, you know, one thing that I I want to emphasize is that these two passes are separated. And they are they are trying to answer different questions in this domain. So that's that's, I think, very important.

• 37:26Nathan Labenz?

  Cool. Well, let's get a little bit deeper then into just the nature of the architecture itself. And there's a lot of little detailed choices that you've made that I understand are, you know, not the end of history. I always say transformers are not the end of history. And I totally understand that this paper, you know, represents more of the beginning, probably, of a new direction, certainly not the end. If if I could try to summarize the overall architecture simply, it's. akin to a transformer in the sense that you still have like attention as a pretty core mechanism. But now, instead of being, you know, truly all tokens that the model has ever seen in the episode, we now have essentially a sliding attention window, which that's something

• 38:10Nathan Labenz?

  that we've seen, you know, in the past with various schemes. Then there's also this bit that I didn't have a great sense for, which is the persistent memory is, I think, how it's called in the paper. And it's also called learnable data independent weights. What I see in the diagrams is that there's this sort of persistent memory layer that kind of gets seemingly like always put at the beginning of the sequence at every time step. So you sort of have the persistent memory followed by the output from the long-term memory module, which we'll describe in more detail in a minute, and then followed by like the current sequence that is the sliding attention window max.

• 38:55Nathan Labenz?

  And then all that kind of goes into the attention mechanism and gets processed through the larger super network, you know, in more or less a transformers like way. Tell me if I am missing anything important there. I didn't quite have any great intuition for why you thought that the persistent memory, like where did that idea come from? I have seen some things in the past. I remember a paper called Attention Sinks or something along those lines where it seemed like they had found that there was basically just an over dependence or sort of over like the initial tokens were sort of too important in some ways, like having these extra kind of early tokens that could sort of not be very important, but like not also allow the

• 39:39Nathan Labenz?

  model to not overweight the actual runtime first tokens seemed to be a performance benefit. But yeah, I don't know, de-confuse me on that. Like where does that persistent memory bit come from? What role should we understand it as playing?

• 39:54Ali Behrouz?

  So the persistent memory part is the part that might not

• 40:00Ali Behrouz?

  Be very like necessarily. But that's a part that complete our architectures, you know. Because if I want to like. Explain why we are using that persistent memory is there are different reasons for that. So the first. Reason is basically we are motivating the paper as how we can design something similar to human memory. Basically, there are some long term memories, there are some short term memories. And also we have something like persistent memory that encode the knowledge about the task that we are doing somehow. Like it's not related about the data, it's about general knowledge of the task that we are.

• 40:43Ali Behrouz?

  And so that's something that can somehow. Make this design complete. And so we just put it there. And actually, in practice, there are some improvements. You know, when we use that persistent memory, we we get very, very slight improvement in the architecture. So there is no harm in that. So that's the main reason in the from the technical perspective that why we are like doing these things. Uh but actually, on the other hand, one uh reason that we are. Doing like persistent memory is because we are. Having the concatenation of long term memory and the current context.

• 41:26Ali Behrouz?

  And actually, as you mentioned, attention can focus more on the initial. The causal attention can focus more on the initial tokens. Uh so that's challenging, that might result in like dropping the performance and all these things. So having some data independent parameters. That's our learnable. Beginning of the token can help us to improve like the performance. So uh but actually, one thing that I I need to say is that. This idea of adding additional persistent memory to the sequence. Uh comes from an earlier paper than Attention Sync. Uh which we we already discussed that in the paper.

• 42:09Ali Behrouz?

  Basically, they they what they are saying is that we have this attention in the transformers. And after that, we have this MLP. So one way to see this MLP is to see it as the attention mechanism. So let's say that we have like two-layer MLP. And so formulation would be something like W1 times W2 times X, which here we have a non-linearity between the W2 and W1. Now, if you look at that, you can see that W times X is very similar to, for example, Q times K that we have in transformers or in attention.

• 42:51Ali Behrouz?

  And then another W is basically our value matrix that we have. So that's very similar to the attention mechanism. But what is the main difference here? The main difference comes from the non-linearity part. We usually use, for example, ReLU in the MLP or, for example, other nonlinearities, but in attention it's softmax. So what they say is that we can use softmax here. In that case, basically, what we are doing is just concatenating some additional learnable parameters to the beginning of the sequence. And basically, when we apply the attention on top of that,

• 43:35Ali Behrouz?

  it seems that, you know, we are already applying that MLP part. So basically, we don't need that part anymore. So that's the main intuition from that paper. So here again, as I mentioned, we can see that there are different perspectives. There is one concept, mathematical formulation, but there are different perspectives that all of them like lead us to this mathematical formulation. And yeah, that's generally the main motivation for the persistent number. I understand that, you know, some people might not use that persistent number when they want to use Titan. And that's understandable because it can save us some like parameters and also it's simpler in design.

• 44:19Ali Behrouz?

  But on the other hand, we can gain like slight improvement in the performance.

• 44:27Nathan Labenz?

  Gotcha. Yeah, that's really interesting. That is, in a way, that is also why I've been just so fascinated with studying AI broadly, because there's just so many different angles on it, so many different perspectives. And it feels like, in some ways, you know, we're all trying to get at the same ground truth, but in some ways, like, the different perspectives do have quite different value to bring, even if there is some underlying mathematical equivalence. If only, I take like a big part of what you're saying is like, the different perspectives sort of get people thinking about different future directions in ways that underlying mathematical equivalence may like not actually be so useful for. That's definitely true

• 45:08Nathan Labenz?

  for me. So, okay, that's interesting. Let's talk about then the long-term memory module. Obviously, this is again, the core thing, right? And we are now moving into the world of having a network that is the memory. I don't know if you want to take a minute and talk about the read or the sort of write and read paradigm. We kind of already covered that, but is there more to say about the sort of modern paradigm of writing to and reading from memory before we get into the like, specific details of what you've implemented here?

• 45:40Ali Behrouz?

  Basically, the intuition here is very similar to what I mentioned about like, a simple paradigm of RNN and linear attention and these kinds of things. So, basically, as I mentioned, like data comms, we project the data into hidden space and project the memory and then just consider their summation and like update the memory and so on and so forth, repeat this process. So generally, what we are doing here is just writing to our memory. The data comes, how we want to write this data into our memory, the way we do in like linear RNN is to like, project that and like add it to the memory. There

• 46:24Ali Behrouz?

  is another part, let's say that we have like these memory.

• 46:29Ali Behrouz?

  And how do we want to like extract information from that?

• 46:34Ali Behrouz?

  The way we can think about that is to like sending a query to the memory and asking to like, give me like corresponding information. So how we can model that is to multiply your query by the inputs or like pass it to the input. If if your memory is is a simple like vector, you can simply just multiply that. Is a simple linear layer, you can just simply multiply that, multiply your query to to the memory. But on the other hand, if your memory is like a neural architecture,

• 47:13Ali Behrouz?

  this like read from memory becomes something like forward pass. You need to pass your data into your memory. And the output of your memory is the corresponding information about your input. So that's how this write and read like intuition works.

• 47:35Nathan Labenz?

  Cool, so let's get into the details of this. I found this really super interesting and I have to give a shout out to Grok 3 for helping me work through some of the math and and develop my intuition for it. I would say Grok 3 definitely has been shown to have a number of interesting properties and some interesting issues, but it does perform quite well when given a paper like this and ask just conceptual questions about it. One fundamental decision that you made here is to create memory that you describe as associative memory. So basically, the memory module, it's an MLP, right? So it can take some input and then it's going to give you some output.

• 48:14Nathan Labenz?

  And the loss that you're minimizing there is the difference between what the memory outputs for a given inputs key vector and then the value that same input actually ends up with on the other side of the attention mechanism.

• 48:36Nathan Labenz?

  So I kind of interpreted this as being like an approximation of the attention mechanism, where what we're doing is saying, okay, we want this finite MLP to, given the key value for some input, be able to predict what the value output would be for that same input. And we'll do our runtime updates of the MLP. And again, it's probably, hopefully obvious to people at this point, but like, one major change here from previous architectures is, this MLP is actually undergoing a gradient descent process at runtime, such that the

• 49:18Nathan Labenz?

  weights of the MLP are what is changing, right? So in previous things, you had this, like in Mamba, you have this matrix of numbers that is changing with every time step. Here we have an MLP that is changing with every time step, and it's changing with gradient descent, as it usually does. And the loss function is this, that is defining those gradients, is this prediction of, we want the memory to, given a new input and the keys for that input, we want it to be able to predict the values for that input. Maybe you could even just take a second and talk about like, how you think about the queries, keys, and values structure of attention? I mean, people have encountered that in the

• 50:00Nathan Labenz?

  past. The simple shorthand, you know, that I can kind of recite procedurally, is the query portion is like, what a given token is sort of looking for. The keys are what information it has, and then the values are sort of the payload of like, what information, you know, then gets fed into the rest of the network. I've always kind of held that understanding like relatively loosely, because these, you know, architectures are weird, like interpretability is like a immature science and shows us a lot of weird stuff. And like, even if that was sort of the idea that people had in their heads when they set up this architecture, does that mean it's really working that way? But you seem

• 50:41Nathan Labenz?

  to really embrace that intuition or that understanding of what's going on, and reuse that understanding in the design of this memory module. So.

• 50:52Nathan Labenz?

  Yeah, tell me how you understand the queries, keys and values and tell me if I'm right about how you uh applied that to the design of this.

• 51:00Ali Behrouz?

  I think one way that we can like describe the attention mechanism, as you mentioned, is we can see attention as an associative memory. And basically, we have like keys and values, they're connected. So we want to like pass keys into the memory. And the memory is responsible to like find value corresponding to that key and like pass it as the output. So that's how the memory should work. Now, the main difference here is that we might not have the exact values of the key. When we are doing some inference and these kind of things.

• 51:45Ali Behrouz?

  So that's where we use like query. So we have some stored pairs of keys and values into our memory. So then we want to send the query to the memory. And memory, as I mentioned, is responsible to like find relevant information to the query and like pass it as the output. So how should we do that, you know, because let's say our our memory has like K one to K as our keys. And we have another queries, which is like so much, not so much, but but generally different from K one to K L.

• 52:26Ali Behrouz?

  How how we can like find the relevant information from this memory. And one way to do that is to see how similar this query is to the keys that we already have in the memory. And how we can like describe this query as the like combination of the keys that we have in the memory. So how we can do that, the way we we can do that is to like using matrix multiplication, you know, because the dot product can help us to understand the similarities between each two vectors. So we have a query. And we consider the dot product of this query to keys

• 53:11Ali Behrouz?

  to understand how we can describe this query.

• 53:17Ali Behrouz?

  Using the keys that we already have in the memory. And based on that similarity, we can like extract information from the memory. Because, you know, we already know the connection between keys and values. When we can describe this query as the combination of some keys, we can find the output of this query, the corresponding output to this query, as the combination of those values that we already have in the memory. That's exactly what we are doing in attention. So we consider the multiplication of Q and K to find the similarities and how these things are similar.

• 54:01Ali Behrouz?

  And then like multiply that one with the value to extract the information from the memory. So that's one way to think about attention. But if we write attention as it's done in the TTT paper, if we write attention as a test time training, one thing that we can see is that attention is the non-parametric solution to this like loss function that we have. Basically, these keys and queries are interconnected, assigned to each other.

• 54:40Ali Behrouz?

  The attention is the non-parametric solution to this assignment. But when we are talking about RNN, we are talking about something like applying optimization algorithm, gradient descent, gradient descent with momentum on top of this one. So basically, probably RNN might result in like weaker performance. Because, you know, the attention part is the non-parametric solution to that one. But on the other hand, we have some efficiency gain that we can use.

• 55:22Nathan Labenz?

  So let's just dwell on this for a little bit longer because I feel like this is maybe the most important thing for people to develop an intuition for. And if they can grok this, a lot of the things I think will make, you know, intuitively, relatively intuitive sense like pretty naturally. So.

• 55:43Nathan Labenz?

  We want to have a memory module that says, okay, for this given input, feed it into the memory module and get the most relevant historical information through all of time. And to do that, we say, okay, we've got the sort of query aspect of a given input that is like understood to be, or was maybe like originally conceptualized as what that token is looking for. And we can feed that in. And we want it to come out with all the right information. So how do we update that memory module over time so that it contains that information?

• 56:26Nathan Labenz?

  So that the query of the latest input at runtime actually gets the stuff that we need? We at every previous step need to have encoded it. How do we encode it? We say, well, let's take the keys of all those previous steps and train this memory module to output the values associated with those same steps. So and this builds on the fact that in the attention mechanism, it's the similarity, the dot product of the new token query versus all the previous token keys that determines like

• 57:10Nathan Labenz?

  what portions of the values will actually be used in the downstream calculations. So at every runtime step, we want to make sure that, given something similar to the keys of this input, we can output the value so that we have that payload information to pass into the rest of the network. We'll update so as to be able to do that in anticipation of actually getting the query that says like, this is what I'm looking for. So we want to store, essentially, we want to get the network to be able to predict what each token has. That's how we're updating it as we go, so that we can then come later and say, when a new token is looking for what those previous tokens had,

• 57:54Nathan Labenz?

  we can return the something similar to at least the values that they, that would have been produced had we done the full explicit attention mechanism. Anything wrong about that? I I'm uh.

• 58:09Ali Behrouz?

  No, I think that that's the correct way of like thinking about these mechanisms.

• 58:16Nathan Labenz?

  Okay, cool. So.

• 58:20Nathan Labenz?

  Yeah, I think that's really super interesting. I mean, it took me a little while to get there, but it was worth the work. And again, appreciate Grok for helping me work through some of the notation to make sure I was understanding it correctly. I think this does probably also will help a lot of people understand a little bit better the attention mechanism itself. And it does give me a little bit more confidence. The fact that this all works gives me a little bit more confidence in the idea that like the attention mechanism sort of actually is maybe working the way people have sort of described it as working, because it's really easy, I feel like, to fool yourself into thinking that, you know, what's going on in these massive machinations and and number crunching processes is like what you intuitively think it should be.

• 59:04Nathan Labenz?

  But if you can actually make predictions based on that understanding and design new things and they work, that, you know, certainly gives you a reason to be more confident. So just to say it one more time, at each step, we have this memory module. We want to make sure that we can, given a new token in the future, pull out the payload that we would have got if we were doing a full attention process. And we do that with an approximation that is facilitated by this MLP that says, I'm going to make sure that I can return, given the key of the current token, the value of the current token, so that, because that's like what this, what these tokens have, leading to a certain payload.

• 59:47Nathan Labenz?

  Now, I can at runtime take the query, what the new token is looking for. Those are definitionally similar, right? The key of the previous tokens and the query of the new token. So it's that similarity that is then ensuring that you get the right value at the retrieval from the retrieval process and can carry on. As if, you know, approximately in a fuzzy sort of way, in a more human memory sort of way, carry on from there with not exactly what you would have had with full attention, but at least with some close approximation of it. So that is, again, super, super interesting. I think definitely worth just kind of, for folks who want to develop their intuition

• 1:00:32Nathan Labenz?

  on how these things all work and what's really going on inside, this has been one of the best things to really take time to understand for me in quite a while.

• 1:00:42Nathan Labenz?

  So let's talk about just some details of that. It sounds like these things aren't super big at this point. What do we know about kind of how much information you can compress into an MLP? And like, how do you think about sort of how big this memory module should be versus, you know, how long you want episodes to be able to run versus like how long the sliding attention window should be? Just so far, how have you been thinking about kind of how to size these things relative to each other?

• 1:01:14Ali Behrouz?

  So honestly, I think that, you know, the number of tokens that we are using as this segment, attention part, or for example, the number of tokens that we are using for the memory and all these things are actually are not something that's really, I mean, the way that we want to tune these hyper parameters are not very challenging because that's directly about how much do we want to spend on these things, you know? For example, do we want to train a model with hundreds of million parameters? Or do we want to like, train a model with billions of parameters?

• 1:01:57Ali Behrouz?

  One thing that is important here is there are some like equivalent version that we can use to understand how we can set these parameters for our design. So basically, let's say that we have a MAC or MAC architecture, basically memory as a context or memory as a gate architecture. In that case, the only thing that, because you know, the memory part is something like, it's running in parallel to the attention in some sense. So basically, it's very similar to Ahead, like when we have a multi-head design,

• 1:02:37Ali Behrouz?

  we can see that branch as as a different head than the attention part. So one thing that we can do is to just use what resources do we have? Like, how many parameters do we want to use and all these things? When when we know the answer to all these questions, one thing that we can do is to just use half of the number of heads as the memory part and other half as the attention part. That's one way that we can do. Or, for example, in the memory as a layer architecture, we can simply use whatever configuration that we are using, but make like the number of layers.

• 1:03:20Ali Behrouz?

  I mean, one, the thing that we can do is to like use half of the layers as memory or other halves as the like transformer or something like that. As you mentioned, it's not very clear that, for example, if we change these things, what would be the performance of the model? There might be a good point that, for example, using three memory heads and seven attention heads works much better than five memory heads and five attention heads and all these things. So basically, I think there are a lot of space to explore all these combinations and see which one is better. But generally, if if you're talking about just to use titans

• 1:04:03Ali Behrouz?

  instead of like whatever model that they are using that is based on attention, the configuration would be very similar. We can just use half of the heads as the memory, half of the other half of the heads as as the attention. Or if if they're using something like memory as a layer, we can just simply use half of the layers as the memory. I mean, like one memory layer, one attention, one memory layer, one attention, so on and so forth. So that's another way that you can do.

• 1:04:29Nathan Labenz?

  Do you want to just take a little more time and just describe the three different ways that this works? I guess I was gonna definitely make sure we talk about the momentum piece or the surprise slash momentum component to the update.

• 1:04:44Ali Behrouz?

  Yeah, let's just talk about like the moment part. I think that's the part before the how we want to incorporate the memory into the architecture and combine that with attention. Yeah, so regarding the momentum part, I think we tried to like explain the intuition in the paper. Basically, the intuition is everything that is like surprising probably is worse memorizing. But here, one thing that we need to consider is, for example, there is, there is a specific token that is very surprising for us. And like the consecutive tokens are like describing what was in the like that surprising token.

• 1:05:28Ali Behrouz?

  So basically, all of them are are important. So we cannot just ignore that one. But those tokens might not be surprising to us, you know. For example, let's say that I'm, I'm saying something like I'm leaving. And, you know, this sentence can have a lot of meanings, you know, depending on the context. So without like hearing other parts, I cannot say whether this one is is like surprising or not. Maybe I'm talking to a colleague and now it's the end of the like chief and I, I am saying that, why, I'm leaving. It's not surprising at all, you know.

• 1:06:10Ali Behrouz?

  I, I really need to understand what's going on around that specific sentence, around that specific token and so on and so forth. But for example, we as- assume that I, I want to like leave the company. That might be surprising, you know, when, when I'm saying that I'm, I'm leaving, depending on the context. There are some situations that is surprising, there are some other situation that is not. And basically, that's, that's one way to think about that. And another thing that we need to consider is that also other con- other part of the context that we need to like

• 1:06:46Ali Behrouz?

  pay attention to that part is somehow important because those tokens might not be like surprising. They are just simple description of what we are saying about that specific surprising tokens. So how we can Model this process, how we can say something like, you know, this momentum part, how we can say that it's very effective to not only model the momentary surprise of the token, it's also effective to understand how surprising the past tokens are.

• 1:07:30Ali Behrouz?

  So that's a part we introduce the momentum. So the way that we can think about that is to break the surprise matrix. The surprise matrix is just a metric that says what information should be remembered and what information should be just ignored. And so one simple way to define this surprise metric is to base, is to define that based on gradient descent. But another way that we can do to make it more powerful is to break it into two parts. The first part is the momentary surprise. How surprising is the, this exact token? So this is the momentary part.

• 1:08:14Ali Behrouz?

  There is another part that somehow describes how surprising other tokens are. So that's the like past part. And actually, that's the king, you know? Let's say that there's a surprising token. For example, sixty-four tokens ago, there is a surprising token. And actually, we need to like, forget about that token over time, you know? And that's different from the forgetting part that we have in the memory module. That's the forgetting part for the surprise matrix. A surprise metric also needs to decay over time. And the reason is the context might change.

• 1:08:56Ali Behrouz?

  Or for example, when time passes, these tokens might not be like relevant that much. So basically, we need to also have this decay mechanism. But again, that's one way to motivate these kind of approaches. If you want to just technically discuss that, if you want to somehow like, you know, focus on the mathematical formulation of that part. One thing that we can say is to, we have this design of test time training.

• 1:09:28Ali Behrouz?

  And now, instead of gradient descent, we can use more powerful optimizer. For example, we can use gradient descent with momentum.

• 1:09:39Ali Behrouz?

  Basically, this more powerful optimizer can lead to more powerful architecture. And now, another way, for example, one can say, let's use Adam. We use Adam optimizer as like the way that we can do that, that one will lead to a new architecture. And potentially more powerful than what we have, for example. So that's another way to like, think about these, this specific formulation. But one thing that I found like,

• 1:10:15Ali Behrouz?

  you know, that's, you know, that's a trade off between whether you want to focus more on the mass side, or do you want to like motivate everything in, in the sense that like, everything is like compatible and all these things, you know? So basically, I, I, I think the momentum part was a good spot that we can like focus on to say that everything just makes sense about how, how human works and how, for example, we can mathematically get better results. But definitely, yes, you can use, for example, use other optimizers to make it work. And again, that's, that's different perspective to, to this matter.

• 1:10:58Nathan Labenz?

  Okay, so let me try to summarize this back and tell me if I get anything wrong. So again, we have this memory module, the purpose of which is to allow at runtime, a new query, a new, new token to come in, take the query vector from that token, put it into the module and output the relevant payloads that we had from earlier tokens. That is finite in size, so it can't just be more and more information in there forever. So like, how do we update and, and manage this?

• 1:11:35Nathan Labenz?

  This is looking at equations 13 and 14 in the paper. It's actually relatively simple. It's basically just saying we first decay the current memory state by a bit. And then we add an update term. And that update term is the is determined by the loss from the current token plus a momentum term from the previous token. It's the loss of the current token that constitutes the surprise. The bigger that loss is, the more we realize we need to update, right? Because that that token we did not predict well. So we realize, okay, geez, we really need to update significantly to be able to do a better job on this particular thing.

• 1:12:18Nathan Labenz?

  And then the momentum term says we need to sort of maintain a sort of a significant update for some tokens to come. We want to not just update on this one token, but this highly surprising token signals the beginning of a partial episode where we want to make sure we gather the information from like that entire period of time, that upcoming sequence. We want to make sure we pay extra attention to all of that. Attention not in the technical sense there, but in the weight of updating of the memory sense. That intuitively feels quite right. And it is interesting that it also looks so similar to other optimization algorithms generally.

• 1:12:58Nathan Labenz?

  But it just on kind of an introspective basis, it does feel like I can think back on moments in life where I've been very surprised. And then there's this sort of modified mental state where I'm a little bit sort of dizzy, off center. But you often come away from those moments with very clear memories of what, you know, I remember where I was when a certain thing happened, right? And I remember that morning that the classic one, of course, is at least for people my age is like, I was in high school on nine 11. I remember who told me that it happened. I remember that, you know, where I was standing. I remember the class I went to next. I remember like what we talked about in the next hour. And so it wasn't just that one token, but they did sort of create. You know, I remember when I talked to my now wife, then girlfriend about later in that day.

• 1:13:41Nathan Labenz?

  I remember my dad came home from work like that whole episode, that whole day is like way more salient in memory than the day before and a few days after. And it seems like you're essentially capturing a similar or creating a similar process here with these update rules. One question I did have about the parameters, there's sort of the rate at which the current memory state decays, and then there is the like weights, the sort of strength of update. There's like a free parameter, you know, you have the loss and then you have a free parameter that you multiply by that to determine like how much to update. And then there's also

• 1:14:22Nathan Labenz?

  another free parameter on the momentum term. Are those all learned but fixed? Am I understanding that right? Or did you just like pick them?

• 1:14:32Ali Behrouz?

  So just to make sure that I understand it correctly, do you mean the parameters like alpha t and for example, theta and?

• 1:14:40Nathan Labenz?

  Yeah, alpha new and theta, yes.

• 1:14:43Ali Behrouz?

  Actually, they are learnable parameters and actually they are inputs dependent. So basically we project the input and so the model is learning how to project the input to parameters like alpha or theta. And yeah, so basically based on the token, we will decide which part of the past information is important. And based on the token, we will decide which, I mean, whether we want to like use the surprise from the past or not. And again, based on the token, we will decide to whether we want to, for example, consider these a specific

• 1:15:25Ali Behrouz?

  momentary surprise or not. So all of them are inputs dependent.

• 1:15:30Nathan Labenz?

  Yeah, gotcha. I should have got that from the sub t in the notation, but I'm not always as great with notation as I'd like to be. So there really is like a lot of, and this is kind of like Mamba callback too, where it's not even something as basic as the decay of the previous state of the memory is dependent on the input at that particular time step. So you really have like a lot of ways for the input to influence how the ultimate state of memory is being updated at each time step. So it's a super expressive, I guess you might say, setup in that sense.

• 1:16:14Nathan Labenz?

  Cool. I think that is, again, really interesting, hopefully, pretty intuitive. How hard is it to like, make all this stuff work in, you know, a sort of scalable, efficient way, like on the given hardware? Because you are doing something that hasn't been done much, right? Where you're like, doing a gradient descent at each time step, making these updates. I really don't have a deep sense of how hard that would be to manage. And I suppose you were using, I don't know if you're using GPUs or TPUs. And I'm also not even entirely sure how much difference that would make in terms of how hard it might be.

• 1:16:56Nathan Labenz?

  But how much wrangling at a low level of manipulating all the relevant vectors and matrices, did it take to make this scalable? And like, how scalable did you ultimately achieve in this portion of the work?

• 1:17:12Ali Behrouz?

  Yes, so actually, we have some BT experiments in the paper. One thing that we can see is that if we implement these kind of approaches, like updating the gradient descent at each step and so on and so forth. If you want to do all these things, then the model can become like recurrence model. And so these recurrence can be like very slow in practice when we want to train the model. But the good thing is that this process of like calculating the gradient and also like updates the model and so on and so forth, can can be reformulated as a matrix multiplication notation.

• 1:17:57Ali Behrouz?

  So basically, this matrix multiplication can be very fast using GPUs, TPUs. And so the good part is we can we can make this process faster and parallelizable.

• 1:18:12Ali Behrouz?

  The TTT paper discussed the dual form of how we can, for example, write this recurrent formula for gradient descent in the matrix multiplication format. But we have additional modules that we need to consider. The first one is the weight decay part. When we add the weight decay, we need to multiply the memory by a constant number, by a scalar or, for example, a vector if we want to do the channel-wise decaying. So the interesting part is even using this decaying formulation,

• 1:18:55Ali Behrouz?

  we can again have a matrix multiplication format, but we need to just add additional Hadamard multiplication or, for example, build or construct a diagonal matrix and do the matrix multiplication for that one. So that's generally the way that we can make the process parallelizable when we use decaying part. But we have also that momentum part, which is much more challenging if we want to like handle that. But the interesting part, if, let's look at the formulation in the paper about the momentum. As you mentioned, the momentum is something like S T equal, which is the surprise matrix,

• 1:19:39Ali Behrouz?

  equals to decaying parameter times the previous state minus the gradient part. And interestingly, this one is again a recurrent neural networks, you know, linear recurrent neural networks. So there are different ways that we can like make these processes parallelizable. The first one is using a scan algorithm, parallel scan algorithm, because, you know, this formulation, it can be very similar to, for example, Mamba. It's a linear recurrent models and we can use a parallel scan to like calculate all the surprise metrics in one chunk. That's one way that we can do. Another way is to like you use the matrix multiplication

• 1:20:24Ali Behrouz?

  format for this linear recurrent as well, because let's just expand the surprise metric part, the recurrence of the surprise metric. Again, we can see that the gradient part can can simply be reformulated as the matrix multiplication and also the coefficient of like theta can can again be formulated as the diagonal matrix multiplication. And you know that that's again another formulation that we can have to make the process faster. So there are different ways that we can incorporate these techniques to the training part. And actually, you know, these techniques are very effective because in the

• 1:21:06Ali Behrouz?

  experiment part of year nine that we have, we compared like the training time of Titans, we did some modern recurrent knowledge first and also attention. And we can see that, you know, when we increase the context length, definitely attention will drop. And so that's that's one motivation for using these linear models. And comparing to other models, we can see that, for example, Titan is is like faster than Mamba. But for example, there are some other modern linear models that are somehow

• 1:21:47Ali Behrouz?

  faster. But one thing that we need to like somehow emphasize here is in Titan paper, we focus on like delivering a new perspective, delivering a new architecture, but we didn't like spend so much time to optimize the implementation, for example, use some kernels to make the process very fast and all these things. Basically, our focus was on the architecture side, was on the designing new memory modules and also delivering a new perspective in that sense. Definitely in the future, that would be very interesting to see how we can design different kernels to make the training process faster.

• 1:22:29Ali Behrouz?

  In that case, I really believe that we can achieve comparable efficiency even to simple or linear RNNs that are very fast. So yeah, that's generally about the efficiency part.

• 1:22:42Nathan Labenz?

  Cool. Yeah, I mean, it's off to a great start, I would say. And yeah, no doubt there's still room for further optimization. Let's talk for a minute, I guess, with the three strategies for integration. You've kind of alluded to this a little bit. There's memory as context, memory as gate, memory as layer. I think memory as context and memory as layer are probably the two most intuitive. Context basically being like do the retrieval, get the information out of the memory module, put that into context, do essentially like normal attention. And we've seen like many things like this, you know, even including like multimodal, right? Where you, if you have like a fusion of a sort of vision language,

• 1:23:25Nathan Labenz?

  a vision model and a language model into a vision language model, sometimes, you know, there's like separate encoding of the image into its space and then the text can go into its space and then later they can sort of have a attention that makes sense of them together. That's kind of how I read the memory as context. The memory as layer also is pretty interesting, a little less intuitive in the sense that I sort of feel like, especially as you were talking about kind of interweaving the layers, it's a little odd to think about accessing memory multiple times through the forward pass. And especially as you get to the very late stages of the forward pass, like still going back to the memory, it's like not super intuitive to me how that would work, or why it would work that way.

• 1:24:10Nathan Labenz?

  And then there's also the memory as gate, and I probably have least intuition for that. But, you know, with that prompt for you, you can help me develop intuition for all three.

• 1:24:20Ali Behrouz?

  So let's say that We have this long-term memory. And we are using attention as the short-term memory. Let's just ignore the persistent one, because you know, the way that we are treating the persistent memory is always the same. So let's just ignore that part. So we have this short-term memory and also we have this long-term memory. As I mentioned, it's not very clear that how we can connect these short-term or long-term memory, even from the like neuroscience part. So there are different ideas in the literature. And so, for example, the memory as a layer is the most common way that people do these hybrid models of R N Ns and attention.

• 1:25:01Ali Behrouz?

  So what they are doing is just simply like using each of memory or the R N N and also attention as as one layer and so on and so forth. And I think one of the important messages that we have in the paper is that, you know, most of the literature is focusing on one configuration that is not great. You know, we have two different configurations or two different type of architecture that we can use and they are actually more powerful than the architecture people are using in the literature. So that's an important message here. And actually, there might be some configuration that that outperform, you know.

• 1:25:43Ali Behrouz?

  So generally, one thing that I can say is that these kind of approaches like were are worth exploring. So that's, I think, an important message here. But what is the intuition behind each of these designs? I can say that, for example, let's say we have something like short-term and long-term memory. The data comes. One way to like model this process to say that our short-term and long-term memory are interconnected. So what does that mean? It means that short-term memory will help the long-term memory and long-term memory will help the short-term memory. So how they can do that? The data comes, goes to our long-term memory,

• 1:26:27Ali Behrouz?

  extract the relevant information from the long-term memory and then use that as the context.

• 1:26:37Ali Behrouz?

  So when we use that as a context of the memory, one thing that we have here is that the attention part will decide whether it wants to focus on the current context, or whether it wants to focus on the past information from the long-term memory. And then the output would be something that is somehow like combined or compressed by the attention. It's a piece of information that is like go through the attention. And basically, we know what information is important and what information is not important. So the output of attention goes to the long-term memory.

• 1:27:21Ali Behrouz?

  And like, let it know that what information should be stored in the long-term memory. So that's how we can like describe this MAC architecture. So as I mentioned, at the beginning of the process, the long-term memory is helping the short-term memory by extracting the past information that can help the long-term memory to decide about how to encode, I mean, the attention part, the short-term memory, to decide about how to combine the information or, for example, learn from the data. And on the other hand, the output of the short-term memory will like help the long-term memory to understand what information should be stored

• 1:28:05Ali Behrouz?

  and what information should not be stored. And so that's the MAC architecture. Another way that we can think about short-term and long-term memory is to treat them as two different modules, two different branches of memory. So basically, in this design, the data comes and then goes to long-term and short-term memory at the same time. We have the information from the long-term memory that is related to this specific input. And also, we have the information that comes from the short-term memory, how to understand the pervasive interactions of all these things.

• 1:28:46Ali Behrouz?

  In the context, and then we just concatenate these two or multiply these two at the end. So we are using this long-term information and short-term information. And then just concatenate them or multiply them to use both types of the memory that we have. And that's basically the main intuition for how do we want to do these things. And finally, the memory as a layer. The main reason that we put memory as a layer is as a unified way to combine the memory and attention part. Because most of the people in the research are using this configuration.

• 1:29:27Ali Behrouz?

  And can help us to understand which of these type of configurations are more helpful. And way that we can think about that is, let's say that data comes. And what we are saying is that short-term memory and long-term memory are basically some modules that are sequentially connected. So the data comes goes to, for example, our our short-term memory or the previous state of the long-term memory. Depending on how you want to order these layers. But for example, the data comes goes to your short-term memory. And basically, the short-term memory will decide which information should goes to the long-term memory.

• 1:30:10Ali Behrouz?

  And we repeat this process, the output of the long-term memory again can help the next layer of the short-term memory and so on, so forth. So in this case, each layer of our short-term and long-term memory is helping the next layer of long-term and short-term memory. So that's the main intuition, we can think about this memory as a layer.

• 1:30:34Nathan Labenz?

  Yeah, so it is interesting to observe, just looking at all the results in the paper, that. As you said, for the three sort of hybrid approaches, the memory as a layer is winning very few categories. And that is the, I guess, again, it also kind of speaks to the sort of motivation or like the sort of higher level of thinking being an important way to approach this because the sort of memory as a layer is like very similar to a lot of the hybrids that we've seen between like attention and Mamba state space in the past. But I think the way that that sort of came about was like, well, the original Mamba was

• 1:31:16Nathan Labenz?

  just all Mamba, you know, there was no attention in it at all. And then people said, well, geez, what if we can get the best of both worlds by just interleaving these things together? And that also, you know, seemed to improve on the base, but it wasn't necessarily a super principal thing. It was just like, well, I see that I can stack a bunch of attention layers and I can also stack a bunch of Mamba layers and now I can, you know, let's shuffles them in together. And lo and behold, it's, you know, gives me in some sense like the best of both worlds. But it's interesting to hear like the memory as context and memory as gate both kind of have a more principled, like higher order rationale motivating them. And indeed, they do outperform the the layer approach almost across the board, not maybe

• 1:32:00Nathan Labenz?

  entirely, I'm just roughly counting, it looks like nine different categories and three different scales at which these experiments will run. The memory as layer wins like maybe two of, you know, roughly 30 of those different categories. And then it's between context and gate, it's like maybe 50 50 across all the rest. So one big question I have, and one of the things that got me kind of most interested in the Mamba architecture and the sort of hybrids in general, is that it seems like there are different, what I have started to call micro skills across these different architectures.

• 1:32:40Nathan Labenz?

  So transformers, for example, you know, are much better than Mamba was when it comes to retrieving, you know, like repeating past patterns, right? Like it can see the past, which is not intuitive because it can see the past pattern explicitly. And so it can like, repeat that past pattern as needed. Mamba has some other, though, micro skills that attention kind of struggles with, including learning, like really sparse signals or signals in really noisy environments. Sometimes the transformer can like, kind of struggle to learn those, but the Mamba architecture seemed to do better. Do you see any like, micro skills? I don't know if you've had any, any ability to study this, but are there sort of

• 1:33:23Nathan Labenz?

  things that this new mechanism can do like qualitatively differently or better? Or perhaps things that it can't do as, as well as the, I mean, I guess we probably would know that if there's anything it can't do as well, we'd probably have a pretty good sense for that being related to just not having the full thing in explicit context anymore. But are there, have you seen any sort of micro skills that are like, new, different, exciting, or even just, you know, informative?

• 1:33:53Ali Behrouz?

  So I understand one thing that's probably, if you want to like, answer that question, if I want to answer that question, one thing that we need to consider is that do we want, I mean, are we talking about the neural memory module? Part of the Titan or the like, the entire architecture of Titan? Because if we just focus on the like neural module, memory module, one thing that we need to consider is that it's similar to linear models. It's very similar to RNNs. It's like just, I mean, it probably has the properties of

• 1:34:31Ali Behrouz?

  What we know about R N Ns and these kind of approaches. But it's generally more expressive than the other R N Ns that we have, because you know, the the memory is is more expressive. The architecture that we are using for the memory is more expressive and also the recurrent formula is is more expressive. So basically, that's that's the main advantage of that's the raw memory. But the entire architecture of Titan, let's say, for example, Mac architecture or Mag architecture. These two are something like hybrid approaches. So they somehow has the ability of like having the best of both worlds and they they can use, you know, the advantage of transformers and R N Ns.

• 1:35:15Ali Behrouz?

  At least on the paper, you know, because we can just ignore the output of the attention when task is like R N N specific or there is like vice versa. So that's one thing that we need to consider. You know, so on the paper, as I mentioned, or theoretically, these hybrid approaches can have the best of both worlds. But you know, there are some cases that we cannot have like bitter

• 1:35:44Ali Behrouz?

  results than a pure R N N architecture. And there might be similarly for attention, there might be some cases that we cannot achieve better results than a pure transformer architecture. And so the question here is that is there any specific tasks or something that hybrid approaches are better than transformers and R N Ns? So we have another paper and actually it calls the best of both worlds. So we like see this process through the lens of graph algorithms. So we want to see whether, for example, hybrid approach can have better results in some specific tasks.

• 1:36:26Ali Behrouz?

  And actually, it turns out that there are some specific tasks that using hybrid approach is better than a pure R N N and a pure transformer based model. So, you know, one thing that we can say is that start on a style architecture can be connected to the RNN models. And actually, theoretically, they are more powerful and expressive than existing models. For example, one thing that we can say is, like, models like Mamba, or, for example, RetNet, S4, S5, and all these models, are limited to TC0 class of problems.

• 1:37:10Ali Behrouz?

  And there is a paper called The Illusion of State Space Models, I think. So, basically, they have shown that state space models, or, more accurately, diagonal state space models, and also transformers are limited to TC0 class of problems. On the other hand, for example, if we go beyond diagonal, like, linear RNNs, similar to, for example, Mamba, we can see that we have more expressive architecture and we can go beyond a TC0 class of problems. And so, for example, I think that- To say more about what it is, what a TC0 class of problem is?

• 1:37:52Ali Behrouz?

  That I can say is the state track problem. Let's say that we have, we are in the, like, 0-0 point. And so, there is a string of actions, like, go left, go right, and something like that. And at the end, the question is, what are you right now? And the model is expect to, like, understand these things. But, for example, diagonal RNNs, including state space models, and also transformers, are limited and cannot do that task.

• 1:38:32Ali Behrouz?

  But for example, nonlinear RNN can do that with only one layer. So that's a very simple task for them. But on the other hand, linear RNN and transformers cannot do that. Some models like Titan, I mean, just a neural memory without the full architecture, are also on the nonlinear RNN part, and they can do this state tracking problem.

• 1:38:56Ali Behrouz?

  So basically, in these kind of tasks, they are more expressive and powerful than the other approaches that we know. So basically, one thing that we need to specifically say here is that there are some other RNNs that also capable of doing like a state tracking. For example, DelphiNet is also capable of doing that because it's not diagonal. But some models like Mamba, Mamba 2, or for example, Retinet, Spive S4, these kind of approaches are limited. And they cannot do this state tracking problem. But this nonlinear RNN approaches similar to what we have in Titan. Basically, they can do that.

• 1:39:38Ali Behrouz?

  And also linear RNN with non diagonal transition matrix, similar to DelphiNet, they also can do that.

• 1:39:46Nathan Labenz?

  Quick interjection, is that analysis limited to like a single forward pass? Because I would assume that like I could give O1 or similar like reasoning models a problem like that, and I would be pretty surprised if they couldn't like talk their way through it.

• 1:40:06Ali Behrouz?

  So I think one thing that we need to somehow clarify is that some of these approaches might work practically good. And when we want to understand them in the theoretical side and with some theoretical perspective, it's really hard to somehow model everything that we have in real world. And basically, we need to have some. Assumptions that make the process simple. So we could use this theoretical understanding that we have and use that theoretical frameworks to develop some results. So, yes, I think in practice, definitely, large models are capable of doing some of the tasks that I mentioned.

• 1:40:49Ali Behrouz?

  But, you know, when we are talking about some theoretical results, definitely, it's not like completely about how the models are working in practice. Actually, the paper that I mentioned, the illusion of a city space models that I mentioned has some also experimental results that support the claim. Yeah, I think that's the main thing that we need to consider. Definitely, there are some other combination of models or, for example, additional techniques, reasoning models and all these things to make the model more powerful to do a lot of stuff. But, you know, when in these theoretical frameworks, we are just talking about like simple model that we have.

• 1:41:33Ali Behrouz?

  And, yeah, so basically, that's about the RNN side, the recurrent neural network side, which is our neural memory. But the attention part is also well studied in the literature, so we don't have any specific contribution in that side, you know, the attention is attention. We also have, as I mentioned in the best of post work paper, I mentioned, we have also some results that shows hybrid models are better at doing some tasks that are hard for both RNN and attention, and their combination can be more effective. So that can also be another motivation to consider these kind of hybrid approaches.

• 1:42:18Nathan Labenz?

  Cool, yeah, I'll need to go check that one out a little bit more deeply. It sounds like, if I'm understanding correctly, it's like there's a

• 1:42:28Nathan Labenz?

  In some of these theoretical contexts, and maybe this is like a good thing for people to be more aware of in general, if I'm understanding it right, it sounds like some of these papers that sort of say transformer architecture can't do X are maybe better understood as saying like can't provably reliably do X under certain conditions all the time, but maybe still in practice, you know, can do them like much of the time, but maybe without a guarantee, but you know, maybe with enough like inference tokens to burn, like you don't necessarily need a guarantee, you just need like, you know, a consistent success, even if it's not like fully theoretically proven that you will always get that success. So

• 1:43:13Nathan Labenz?

  yeah, those different frames are maybe why there's like a decent amount of people talking past each other online about what exactly can and can't happen. Cause it is often strange when you see these things where it's like, we've proven that a transformer can't do X and then somebody goes to chat GPT and does it. And you're like, wait a second, these two things can't both be right. And it seems like it's probably a question often of different assumptions and different levels of what they are considering to be proof, like the the sort of robust guarantee level versus the most of the time it works level can be yeah, quite different. I guess just to follow up real quick on the sort of micro skills

• 1:43:55Nathan Labenz?

  concept, like when you look at just all the different things that you tested the new Titans architectures on, what jumps out to you as like the most exciting thing that it is doing better than other architectures?

• 1:44:11Ali Behrouz?

  I think the most exciting part is the long context because you know, for example,

• 1:44:18Ali Behrouz?

  Actually, I need to like say that the long context part is some synthetic data set. So we might not be as effective as what we are showing in the paper. And when we are talking about general tasks and a general model. But basically, what we are doing in the paper is just comparing the model with some other counterpart approaches and then to see how they're performing in specific tasks. So we could compare them. Uh, but the interesting part, I think, is that Titans can outperform uh, other models in in long context. And this is is very like uh impressive, in my opinion.

• 1:45:01Ali Behrouz?

  When we can see that they can scale to like two millions of tokens. Uh, and that's something that some models like G P T four uh cannot do that. No, the the the performance uh in the benchmark for G P T four can drop very fastly. But for example, Titans with a very small number of parameters can uh, for example, scale to millions of tokens and even go to ten millions of tokens with the accuracy about like seventy percent. So I think that's the most impressive part, in my opinion. And actually, that part is, I mean, for that long context modeling, the M A C architecture is very important.

• 1:45:44Ali Behrouz?

  You know, it's it's very important to have two different branches of long term and short term memories and then like these two help each other to understand the context. When we are using some other approaches like M A C or, for example, memory as layer, we have like much, much less context lengths. I mean, with with higher accuracy. Uh, so basically, the M A C architecture for long context modeling is very important. I think that's that's the most interesting part. I think.

• 1:46:21Ali Behrouz?

  When we are talking about some, some language modeling tasks.

• 1:46:26Ali Behrouz?

  You know, the behavior of all these models can, can be different when we use different size of models. For example, there is a very great plot in the, a paper called Mixture of Mamba, I think. If I recall it correctly. Uh, so there's a very great plot that we can see is something like, the plot is the perplexity of transformer Mamba. And like the mixture of Mamba. And one thing that we can see is that the R N N approach, or for example, in this case, Mamba. Skills better than, than we, we, and we have like less data. But when we increase the number of tokens that we are using for, for training.

• 1:47:11Ali Behrouz?

  We can see that transformers scales better and then just outperform when we have so much data, uh, in that sense. Uh, so that's a very important part to, to, to consider. And, uh, so one thing that, uh, we can see about like titans is that. At least until once we experimented. Uh, we couldn't see that pattern. They actually like, uh, consistently have a good results compared to like transformers and also other linear R N Ns and like. You know, that's definitely a good thing to like converge faster, but when we are talking about like.

• 1:47:52Ali Behrouz?

  Using additional data for, for training the model, we might see some like. Up and down in the, uh, order of models, uh, with respect to the perplexity. And so basically I think that's also very important. Uh, which might not be very clear from like academic papers, because you know, the, the, uh. We need to like have a fixed number of tokens that we are using to train our model. And basically it's generally feasible to, you know. Train the model for la- uh, larger and larger, uh, number of tokens.

• 1:48:33Ali Behrouz?

  Uh, and yeah, I think, uh, that's another, uh, good results that we can see in Titan. They are very consistent to outperform the other models. When we like increase the number of tokens that we are using in the training part.

• 1:48:48Nathan Labenz?

  So as we look ahead, how much do you think like data is going to become the limiting factor on long-term memory? I mean, when you show the, you know, still reasonable, like very few of the architectures even can attempt to get out to the 10 million token context. And of those that can, the titans is like blowing away the other two that you show in the the experiments. It strikes me that just like, we don't really have many sources of data that are really such long episodes. And so I guess I'm wondering, like, because the internet is, you know, full

• 1:49:33Nathan Labenz?

  of short and medium and even long blog posts, but you know, they're not reaching into the millions of tokens very often at all.

• 1:49:42Nathan Labenz?

  If we think about wanting to have long-running agents, it seems like an architecture like this could be really key to getting a lot of the sort of behavior that we might want out of a longer-running agent. But we still don't necessarily have the like, long horizon data sets to be able to train on. What's your take on whether we have the data that we need or we don't and have to go create it if we want to really take full advantage of these sorts of architectures at scale?

• 1:50:16Ali Behrouz?

  I think related to two concepts in the community that usually is important about RNNs, and that's length generalization and length extrapolation. The question is whether our model can somehow works better, or at least does not show any performance drop when we increase the sequence lengths. As you mentioned, most of the data that we have might not be very long context. So it might be challenging to train the model on millions of tokens. But we actually don't need to do that. So the generalizability in some models can be somehow

• 1:50:56Ali Behrouz?

  important for like effective training. Let's say we train our model on a data set that might not have a very long context, but we expect our model to understand how to generalize to longer context.

• 1:51:15Ali Behrouz?

  Uh. So when we have a single model like R N N. Or for example, our neural memory.

• 1:51:27Ali Behrouz?

  We can see that these models are struggling with length generalization. Uh. When we increase the sequence lengths, actually, we can see that they perform as drop. And uh that part is is like very soon, like sixteen K. Or something like that. It's it's not very like, you know, one one millions of tokens. Uh. Again, with transformers, we can see some similar results. Without doing any trick about how how we can like make them better at length generalization. Um. But, you know, there there are some different architectures that can somehow like mitigate these approaches.

• 1:52:09Ali Behrouz?

  And make the model more generalizable with respect to like sequence lengths. And also, there are some other tricks that we can use. For example, the positional encoding, different positional encoding that we can use. Or for example, uh, you know, there are different normalization techniques that people use for like length generalization. And all these things, you know, for example, architectural side, I mean, regarding the architecture design. We can see that this MAC architecture is is like very good for uh the length generalization. Based on my my personal experience with with like with that. So my understanding is that when we have something like MAC architecture.

• 1:52:49Ali Behrouz?

  The memory module is only responsible to uh like summarize the data for the attention into a fixed size. Uh, you know, uh, number of tokens. So it doesn't have a very hard time to compress the information and and learn things. Uh. On the other hand, attention doesn't need to like attend to so many tokens. We can just use a fixed size a local attention. So uh again, it doesn't have a very hard time to learn from that part. And understand how it can use the information from the long term memory. So in general, uh, one thing that I can say is having more data definitely is is very good.

• 1:53:32Ali Behrouz?

  And uh so it can help to have a better model. Uh. But not the end end of the story. We we need to have better architectures. And better techniques that we can use in the training of the model. So we could like make them more generalizable to, for example, make them more generalizable to different sequence lengths and make them like better at longer sequence and so on and so forth. So I think architecture and the model itself is very important.

• 1:54:07Nathan Labenz?

  I guess one question I have is how sort of retrofittable do you think today's open source models are with this technique? If I wanted to take a off the shelf Llama model or R1 or what have you and integrate a long-term memory module of the sort that you have, it feels I could probably do that without, you know, I mean, I don't want to make it sound simple, but based on what you just said, it seems like I could probably, you know, hack the Llama architecture a bit, bring in the memory module as you've designed it, and then like do some amount of kind

• 1:54:48Nathan Labenz?

  of continued training to get the memory, to get the thing to like actually effectively pay attention to the memory as context. And I kind of think that that will work. Are you expecting that to happen?

• 1:55:04Ali Behrouz?

  I think potentially it's possible. You know, the main thing that I like about the MAC architecture is, you know, no matter how large you want to choose your context lengths for training your attention, you can always go farther with the long-term memory part. So you can like use attention with like thirty-two context lengths, you can use one hundred twenty-eight, five hundred twelve, and then go beyond that, like.

• 1:55:39Ali Behrouz?

  Eight thousand one hundred ninety two context links and so on and so forth. You know, no matter how large you want to choose the context lengths for training the transformer part, you can always go farther with using additional memory. So uh uh with with the memory as a context uh design, the Titanist side model. So I really hope that like these these kind of approaches can be used in the future and and basically become something that we can use because they are actually working and and they can they can help with the performance in long context without, for example, performance

• 1:56:19Ali Behrouz?

  drop or without any like damaging the performance of of pure attention. So I I really think that these kind of uh future work are really promising.

• 1:56:30Nathan Labenz?

  What what other kind of conceptual question I had, this goes back to sort of how the memory module is predicting the keys. It's it's sort of struck me that like if I wanted, you know, one of the promises of long term memory, at least in humans, right, is like we can learn whole new domains of stuff that was just like we've never encountered before. If we, you know, spend time on it, it can sort of become part of our like background world knowledge. Here it seems like we still have kind of a the the memory module outputs something that is kind of by definition in the latent space of the model as it was kind of originally pre-trained.

• 1:57:13Nathan Labenz?

  And because those like main weights don't get updated at runtime, the memory module is always outputting something that's in the space of that pre-trained model. So that got me thinking like, would this ultimately be something that people might want to combine with sort of a continued pre-training? If, for example, you're like an enterprise and you're like, you know, let's say I'm 3M or I'm GE, I'm some company with like long history and, you know, millions of products and, you know, just some super deep history that I want to get my um my models to learn.

• 1:57:48Nathan Labenz?

  It seems like I might want to do like continued pre-training so that the model itself like has more representations of the things that I care about. And then that might be needed to get that long-term memory to be able to like output things in that sort of modified space.

• 1:58:06Nathan Labenz?

  Does I guess the key question there is like, does that disconnect between sort of the model's fixed latent space and that and like the the fact that the memory module is outputting into that space?

• 1:58:19Nathan Labenz?

  Does that suggest sort of a frontier for like future elaboration or, you know, perhaps that that continued pre-training approach would solve it, but I'm interested in your your thoughts on, you know, just how far could you push this? Could you get the model itself to like learn whole new domains of knowledge this way? Or would you expect that you'd kind of bump into some limits and have to do something to overcome that?

• 1:58:44Ali Behrouz?

  I think that if we want to like train the model for like long period of time, and by training, I mean, like even test time training, you know, just updating the parameters of the model over time and use that for different tasks or like different type of things. We can face a problem which is very well studied in the literature, which which is called like catastrophic forgetting. So basically, one thing that we can see is that the model we learn a specific task or for example, a specific context.

• 1:59:24Ali Behrouz?

  And when we want to like go to the next task and like learn the next next task, one thing that we can see is that

• 1:59:34Ali Behrouz?

  the model might completely forget about the previous task to like get adopted to these current like configuration and task that we have. So.

• 1:59:46Ali Behrouz?

  I think the part that you mentioned is very promising for future study. But actually, there are some challenges for doing that. And these catastrophic forgetting is one of them. Because if you want to like have a very long, long, long time in the test time training part. In that case, the model might forget about like initial tokens. Or for example, the previous task that's trained on and all these things. And that's really challenging.

• 2:00:19Ali Behrouz?

  If we could solve that, actually, we will solve all the like robotic stuff and all the reinforcement learning stuff and all these things. So that's a very challenging question. But you know, I think we might face this issue and it needs to be like addressed.

• 2:00:38Nathan Labenz?

  Well, I know we're just about out of time. Any closing thoughts about like where you are most excited to see all this stuff go next? To me, this feels like a pretty notable step on the path to really figuring out that kind of continuity of memory. And I do think that's going to unlock some of the most exciting applications that people are interested in. You know, whether it's the drop in knowledge worker that has that full context of your enterprise's full history and everything that it's ever done. Or the long running agents, you know, that can go out and kind of make mistakes and maybe learn from those mistakes. And you know, not make the same mistakes twice.

• 2:01:18Nathan Labenz?

  I don't know that this gets us all the way there, but it feels like it does take us a meaningful step forward. So what to you is like the most kind of exciting or promising directions that you or you hope others will go going forward from here?

• 2:01:34Ali Behrouz?

  Honestly, one thing that I really think that I think that it's really interesting is really using these kind of approaches for like other tasks and modalities. Because you know, there We are trying to like do these long contexts for text. But sometimes there are other modalities or other tasks that are more suitable for long contexts. For example, as you mentioned, when we are talking about like agents or for example reinforcement learning, decision making, all these things uh require some long term memory to understand some uh different types of patterns uh in the data.

• 2:02:16Ali Behrouz?

  I think that's that's really promising to see how these long term memory style architecture like Titan can can work in those domains. As I mentioned, decision making, reinforcement learning and for example other modalities, data modalities, I think that's a very promising future direction uh to see whether these designs that that like we just talked about are are actually effective outside of the language modeling. And uh whether they are just some good architectures for language modeling

• 2:02:58Ali Behrouz?

  or they are just good architectures in terms of all areas of deep learning. You know, that's the important part because I think one of the main reasons that transformers were really successful because, you know, they were successful in different domains. And actually, I think that that's really important to explore and see whether, for example, these additional modules that we are adding and designing new architectures like Titan can be also effective in uh other domains, other data modalities and all these things. And definitely, there there are so many rooms to like modify them for different tasks.

• 2:03:38Ali Behrouz?

  Because, for example, are the design of like Mac mod, Mac architecture might be very suitable for for language modeling. But, you know, for example, Mac architecture using like gated short term and long term memory might be very effective, for example, for decision making. Yeah, yeah, we don't know that.

• 2:03:59Ali Behrouz?

  All of them are really interesting for future work to see like which one works better and which one is effective.

• 2:04:07Nathan Labenz?

  Well, it never ends. There's always plenty more to do, but at least until the AGI takes over and starts doing it all for us. But for now, this is a really fascinating development. I think great combination of high-level, principled, intuitive motivation for the work and obviously some really deep work in terms of making it work on the mathematical and even the computational level. So it's a pretty impressive piece of work. I have really enjoyed studying it and I'll certainly be looking forward to what you guys come up with next. For now, I will just say, Alibiruz, thank you for being part of the cognitive revolution.

• 2:04:48Nathan Labenz?

  Thank you very much for having me. Thank you. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.