Nested Learning: Ali Behrouz on the Quest for Continual Learning & Illusion of AI Architectures

Ali Behrouz, grad student at Cornell and Google researcher, discusses his potentially transformative work on new architectures for continual learning in AI. His paper "Nested Learning," praised by Jeff Dean as a possible paradigm shift, enables models to adapt to new context while preserving core knowledge by updating different layers at different frequencies, inspired by human memory systems. The conversation also covers his latest work on AI "sleep" for memory consolidation, why he sees all deep learning as associative memory, and the profound implications of continual learning for privacy, alignment, and the path to AGI. Mercury: The fintech trusted by ambitious companies and individuals to run their finances, with virtual cards, spending limits, merchant/category locks, and AI-friendly tools like API keys, MCP, and CLI. Check out Mercury at mercury.com Sponsor: Claude: Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, I'm excited to share a conversation with Ali Behrouz, grad student at Cornell, researcher at Google, and author of nested learning. This episode was recorded a few months back, and while I normally believe that AI content does not age well, this conversation with Ali is an exception. His work is some of the most inspired and potentially transformative that I've seen anywhere in the quest for new machine learning architectures that are capable of genuine continual learning. This, of course, is one of the most important capability advances on the horizon today. Arguably, it is the main gap between today's models and a digital AGI that would be capable of joining and contributing to human teams, just as humans do.

• 0:46Nathan Labenz?

  And Ali is advancing the frontier with an approach that is both biologically inspired and technically elegant. His blockbuster paper, nested learning, which has been touted as a harbinger of a possible paradigm shift by no less than Jeff Dean, develops a simple strategy that allows models to rapidly adapt to their current context on an ongoing basis while preserving core knowledge by updating different parts of the system at different frequencies. Much like humans manage memory on multiple timescales from working memory to long-term memory. His latest work, Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories, which I actually heard about live for the first time on this recording, and which has now finally become fully public,

• 1:30Nathan Labenz?

  takes inspiration from how humans consolidate memories and learn from dreams while sleeping, introducing a new offline mode in which models transfer new knowledge from their high-frequency update layers to their more slowly-evolving layers via distillation, and also learn new abstractions and connections between concepts by generating and training on synthetic data derived from their recent experiences.

• 1:55Nathan Labenz?

  In addition to the details of these architectures, which, like so many AI innovations, I find both extremely exciting and a bit scary, we also discuss how scaling for performance may shift from stacking more layers to nesting more frequency update rates. How Ali understands all components of machine learning systems as forms of associative memory that compress a given context flow. Why this leads him to call deep learning architectures "an illusion", and how he's operationalized this conceptual insight by developing expressive optimizers that learn update rules and are capable of outperforming both Adam and Mewon. We also discuss how the attention mechanism can be understood as an infinite frequency

• 2:37Nathan Labenz?

  update module, and why Ali expects that attention layers will therefore remain fixtures of AI systems indefinitely. We covered the empirical results showing that Ali's new architectures compete effectively with transformers on standard measures while also outperforming them on hard tasks, such as effectively recalling information from up to 10 million tokens of context, and also learning to translate multiple, previously unseen languages at the same time. Finally, we discuss why Ali sees continual learning as both an opportunity and a huge risk for privacy and alignment. How human-AI relationships might evolve, and why Ali is cautiously optimistic that models that evolve over time based on our

• 3:20Nathan Labenz?

  interactions with them could both serve our individual needs more effectively and also lead to a more diverse and hopefully stable AI ecosystem overall.

• 3:32Nathan Labenz?

  The bottom line for me is that for all the debate and speculation about whether or not current architectures can scale to AGI and beyond, there is a very good chance that conceptual breakthroughs will render that question moot before we even manage to answer it. Transformers have changed the world, clearly, but they aren't the end of history. And as tough as it is to keep up with AI developments, anyone who wants to get a handle on where things are going from here can't afford blind spots when it comes to new research directions like Ali's. And so, without further ado, I hope you enjoy this deep dive preview of AI systems that learn on an ongoing basis in increasingly human-like ways with the brilliant Ali Beerus.

• 4:15Nathan Labenz?

  The Cognitive Revolution is brought to you by Mercury, the fintech that more than 300,000 ambitious companies and individuals trust to run their finances. Over the last few months, I have made tremendous strides with my personal AI infrastructure. Today, I've got high-context instances of both Claude Code and Open Claw running on a Mac Mini, and it's amazing what they can do. However, until getting started with Mercury, I didn't have a great way for them to pay for things. I didn't want to give them unrestricted access to my money, but my old bank didn't give me any other options. With Mercury, I can create as many virtual cards as I want, each with its own daily, weekly, or monthly spending limit.

• 4:57Nathan Labenz?

  And I can lock any card to a single category of purchase, or even a single merchant. Now, I have a card that my agent can use to buy our family's groceries, and only our groceries. And I can create another anytime I want to give an agent a random one-off project that might require making a purchase. This is honestly just the start of Mercury's AI-friendly offerings. Does your bank offer API keys, an MCP, or a CLI tool? If not, check out Mercury at mercury.com. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column NA, members FDIC. Thank you to Mercury for supporting the Cognitive Revolution.

• 5:41Nathan Labenz?

  And now, on with the show. Ali Behrouz, author of Nested Learning and the New: Language Models Need Sleep: Learning to Self-Modify and Consolidate Memory. Welcome back to the Cognitive Revolution.

• 5:54Ali Behrouz?

  Thank you very much for having me.

• 5:58Nathan Labenz?

  I am super excited about this conversation today. I appreciate you for being willing to take some time and come back and do a deeper dive into your work. I think it is super fascinating and genuinely some of the most inspired work that I have seen in recent times. And a big part of your method, as I understand it, is looking at what human cognition consists of and identifying things...

• 6:25Speaker 02

  Things that we as humans are doing that seem like they're quite important and really critical to our successful function in the world. And then trying to figure out kind of what an AI version of that might look like. And then starting to develop the architectures or system designs, maybe even more abstractly than architectures, that start to make those capabilities possible in AI systems. And it's really striking both like how well some of these ideas have worked and also striking to me how elegant they feel and how kind of right they seem as I really take time to dig in and understand them. So first question, and you know, just big picture. How do you think about what it is you are trying to do?

• 7:08Speaker 02

  Obviously, you identify gaps in what current systems can do. How do you think about those gaps? How do you conceive of what it is that you are trying to unlock with the new architectures that you're developing?

• 7:22Ali Behrouz?

  Getting inspired from the brain and actually like what does it mean for me? It probably means a different things for different people. For me, I really like to get inspired from brain and generally evolution. And the main reason is that I think it has like a lot of data to train itself in a natural way of training. And so one thing that we can see now is a very like evolved version of a very complicated biological brain. And so generally, it's a great source of inspiration. But when I'm saying that, I don't mean that I want to like replicate brain and fully do

• 8:03Ali Behrouz?

  something that that brain does, because most of the time, we don't know what that is actually. And for example, I think, you know, there are different levels of understanding about how the brain works. So the first one is that, you know, we know it works. So that's that's the first level. The second level of understanding is that You know, there are some modules in the brain, each of them are responsible for different parts. And you know, we have memory, we have like other things and that's generally the process. And I, I think the hard part when we want to get inspired uh from the brain is that at what level of granularity we want to focus on the brain and get inspired from that.

• 8:46Ali Behrouz?

  Uh because if you go too much into the details, like how the brain works and so on so first, there are like two issues about that. One is we are overfitting ourselves to one a specific form of intelligence. And another one is that we don't actually know how, how the brain does that specific thing. So generally, uh in all of the like the works that I have done, for example, on Titans and also like this necessary learning. One, one thing that is happening is we can see that the models are, are facing some challenges in real world applications. And then the question is, is that a specific challenge is solved by humans?

• 9:30Ali Behrouz?

  And can, can simply human do something to address that challenge and overcome it or not? And then the second question is like how, how they can do that. But uh you know, again, uh for example, in Titan, we discussed surprise metric. We discussed, for example, how the memory should be like decomposed into short term and long term memory and so on so first. But the point is, there, there's a high chance that brain is not exactly performing gradient descent to, for example, understand the surprise or something like that. That's just a high level of our understanding about how the brain works. I, I think if we keep that as a source of inspiration, I mean, that level, it would be a great source.

• 10:14Ali Behrouz?

  But uh if we go into more details, then potentially we uh might face some challenges, specifically because our understanding of the brain is changing over time. And so we might like overfit to one specific design choices. Uh so that's generally one thing. But uh about the necessary learning, I think one point that is missing in the current models is about.

• 10:38Ali Behrouz?

  Two parts. One is about how they can adapt to the environment and the context that they are in. Another point is about how the model can understand new knowledge and incorporate it into their parameters over time. So they can make sure that they don't face catastrophic forgetting, which means that, for example, one specific task that they have trained on is forgotten and they don't have any skills in that specific direction. So I think these two are important things about the current models and they are facing a lot of challenges for that.

• 11:20Ali Behrouz?

  Because if you have a very like large model, then your model needs to be updated over time. And that's generally the main reason you can see that, for example, there is a knowledge cutoff for all of the L L M's that we know of. For example, if you ask Chat G P T about one specific information and then say, you know, you are not allowed to use, for example, any tools for answering this question, then you might see that there is a knowledge cutoff. And so that's a little bit challenging to overcome. And if you just want to keep updating the models, then there are like two huge challenges. One is catastrophic forgetting that I mentioned, and another one is the efficiency part.

• 12:03Ali Behrouz?

  Because I have a lot of parameters, you cannot keep updating all of the parameters. And so it's a little bit hard for that. And, you know, there are some solutions for that. I think they are also like great, but I have some intuitions that why they might not perfectly work for the case of continual learning. For example, one might say that they want to do supervised one teaming, for example, S F T, or do some R L stuff or something like that. But the point is, still, the model can face catastrophic forgetting from one side. And from the other side is that. At some point, you need to, you know,

• 12:47Ali Behrouz?

  transfer all the knowledge that you have in your context and pass it to the, for example, actual parameters of the model. If you just keep summarizing the tokens, if you just keep the tokens and want to like, do everything about the memory and learning process in the token space, then the main issue would be, at some point, you will pass the context length of the LLMs. And so potentially, you will face some challenges in that direction. So considering all these things, the main issue with the current LLM paradigm is that they cannot continually learn and obtain new knowledge,

• 13:30Ali Behrouz?

  new skills over time. And also, they are limited in understanding different levels of abstraction word. Everything that is done in science is to do something that can explain the word in the most simplest way possible. And that's generally the way we can learn something. Because we don't want to like keep everything. We want to understand underlying patterns that can describe that specific data, that specific, you know, knowledge and so forth. So this compression process and how we can, you know,

• 14:13Ali Behrouz?

  understand different levels of abstraction from the, you know, data that we have is something that the current LLMs pull should somehow.

• 14:25Speaker 02

  So a couple of different angles. I want to just probe your intuition a little bit more from on this point. Certainly, I have felt, you know, the, I think, you know, most users, honestly, at this point, have felt the problems that you're highlighting.

• 14:42Speaker 02

  And in some ways, I feel like the biggest advantage that I have relative to an AI today is this kind of ongoing coherence and reasonably stable identity, right? Like, I know who I am in the morning, and I kind of know what I was trying to do yesterday, and I can mostly pick up where I left off. And I probably could learn a lot more from the things that I do on a daily basis, but I at least learned some stuff and kind of take it on board. And obviously, the current models don't really do that, and that is a big weakness for them. That's why I was so excited about the original Mamba paper when that came out, because I was just like, wow, here's something that seems like it's competitive with transformers,

• 15:22Speaker 02

  but it has a fixed-size memory space and something that, like, obviously, we can't grow the memory space quadratically to infinity, so we're going to have to have something that's bounded in size that can work. And so that was like, obviously, a notable step. And I've done some work with the Mamba architecture as well. Titan's even more so. Okay, here's another way to think about having a fixed-size memory module that, in that case, is updated with gradient descent at runtime. So that potentially seems like it would likely, and it certainly seems like the data supported the notion that that would make it even more powerful than the Mamba architecture. But a similar kind of structure of like, okay, a fixed-size thing

• 16:07Speaker 02

  that can like, keep what it needs and gradually let go of what it doesn't. That seems super important. I wonder if you come at it, though, also from the other angle. Like, so far, we've kind of said, like, here's something that it can't do. We can do this, it can't do this. That's one way to think about it. Another way to think about it, obviously, is like, what do we want our AIs to be like? And so I wonder what your intuition there is. You know, today, we have mostly chatbots that need something to wake them up, right? They're like, either we have to go send them a message, or obviously, increasingly, we have like, cron jobs and other triggers that kind of get the AI to wake up and do something.

• 16:51Speaker 02

  But if those things don't happen, they're inert, right? They just kind of sit there until, you know, somebody calls their number. Do you have a sense or a vision or kind of a, you know, a dream of what your ideal future AI would be like that's different than that? And like, how, you know, does it look more like another person but with AI advantages? Or does it, does it look like an LLM with like weaknesses patch? But I don't know, like, what is your kind of, when you dream of a 2030 AI that you're, you know, working closely with on a, on a daily basis, what do you envision?

• 17:30Ali Behrouz?

  Like, different aspects to answer this question. From the technical point of view, I think generally, the entire literature, most part of the literature in the, you know, past 40 years is built on a paradigm that says we have a pre-training or generally training phase and we have a test phase. But the point is, if we have a continual learning, we can see that, you know, there are like a lot of recent studies about continual learning, how we can do that and how we can like overcome about a lot of challenges.

• 18:13Ali Behrouz?

  But the point is, a true continual learner doesn't have a test and train time. So potentially, if we hear that name in any like design choices, potentially it means that it's not a true continual learner, because there is no test, there is no train time. And so the question is, is it like a uniform process for the model or not? And my personal opinion is that we still need at least two phases. And how it works is that we should have one phase that the model is active, so it actively

• 18:57Ali Behrouz?

  receives information. It's whether through the user query, for example, or for example, it might be about vision models, word models, or anything similar. But the point is, the model receives some information.

• 19:14Ali Behrouz?

  And generally, it performs some computation on the input data and it's active at that point. What's on the other hand, there is another phase that the model does not have to wait for the inputs data. It might not receive any input data. It's completely blocked from the world outside of it. But the question is, even at that time, should the model be static without like any performing computation or doing something? Or the model needs to start thinking about some process, thinking about the data that it has inside its parameters and so forth.

• 19:57Ali Behrouz?

  Uh, so I think we can break the process, as I mentioned, in two parts. One is the active phase and another one is another phase. Potentially, we can call it like a sleep time. Uh, because there is no inputs, but still the, you know, artificial brain or generally like the, the, the model itself is trying to perform some computation. Uh, so I think that's a good way of, of defining different phases in, in this direction of continual learning. And then I think good model is a model that performs very well in both side. It should receive the information properly, encode it, process it and understand it in the best way possible.

• 20:43Ali Behrouz?

  And on the other hand, when it goes to the sleep time, it should also like start processing what it has learned before and use that for self-improvement. And, uh, so that's, I, I think an ideal model should do from the, uh, you know, technical point of view.

• 21:03Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 21:07Nathan Labenz?

  Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep-context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts, going back a full five years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires, based on the calls I've had and the emails I've exchanged with the founders.

• 21:52Nathan Labenz?

  And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind. But then I thought to ask Claude, and sure enough, it identified two great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode.

• 22:33Nathan Labenz?

  That's claude.ai/tcr.

• 22:38Ali Behrouz?

  But on the other hand, I think there are a lot of challenges. The models that we know right now are very large. So even a simple, for example, academic paper that is presenting a new LLM or architecture or something like that needs to perform some experiments on models with billions of parameters, 1 billion, 2 billion, or something like that. And generally, that's a very large model requires a lot of computation if you want to keep that model updated over time. And it needs some techniques to somehow make this process possible. And generally for us, when we were thinking about like this direction,

• 23:23Ali Behrouz?

  it was a time that some ideas about nested learning started. Because generally, if we think about like nested learning, we can see that at each time stamp that we have, we don't have to update everything. We just need to update just a small subset of all parameters. And so that potentially a way to overcome the challenges about the efficiency. So if I want to like summarize what I wanted to say is that I I think an ideal model should have like two phases. One, generally, it should be a continual learner. It should like interact with the world.

• 24:07Ali Behrouz?

  And also, it should have like two phases. One is about like very active process. And another one is about self-improvement and how it can consolidate its memory, how it can like understand the knowledge, how it can connect different things that seems to be irrelevant and so forth. And everything similar that usually our brain also does when we go to sleep. Yeah, that's just my personal opinion and my big company is wrong. But I think we shouldn't focus too much on what human can do. And instead of that, we need to focus more on what human wants from AI.

• 24:52Ali Behrouz?

  And I think if we think about, we don't want to create something that is very similar to us. I mean, that that's also like very interesting direction. But I I personally, I'm not very interested in that. I think we have like a great design for human. But on the other hand, for for the AI side, I think.

• 25:14Ali Behrouz?

  We need AI models that are capable of understanding what we want. For example, I think current paradigm of L L M's, you can see that they are great after some time when we have like more uh features, more, for example, a chat G P T has, has the memory. Now, and for example, Claude has some like features, Gemini and everything. Uh, you know, now they are more capable of understanding what you want. When you say, for example, write this email for me, they can simply write that for you. And uh so I think that's a very promising direction in general, because we don't want to replicate human.

• 25:59Ali Behrouz?

  There are a lot of like different ways that we can define intelligence somehow. And it doesn't have to be the same thing as, as human intelligence somehow. Uh, so I, I think here again, we can get inspired from the human. But we need to understand why we want to get inspired. Do we want to get inspired because we want to replicate what human can do? Or do we want to get inspired from the nature to understand some underlying rules in the nature? For example, just one extreme example is we cannot travel through time.

• 26:43Ali Behrouz?

  So if I come up with one idea that is saying that uh my AI model is trying to break the causality in the world, then potentially that might be a wrong idea or wrong direction, because it is breaking some nature rules about uh our world. Or again, let me uh give you another example. For example, when we are talking about

• 27:10Ali Behrouz?

  The title that we use, LLM needs a sleep. It doesn't mean that LLM needs to literally go to sleep and rest. It means that, you know, from the human brain, it seems that there is a very general rule that it has two phases of learning. And then another phase of processing it, consolidate the memory, and find underlying patterns between the received data. So that's a very high level of inspiration from the brain. So yeah, in short, I think that we don't want to replicate what human can do.

• 27:52Ali Behrouz?

  And we don't want to have human intelligence. But on the other hand, we want to have a new form of intelligence that is very, you know, it's very proper. And it's in the, you know, it's designed in a good way that understands human needs. And so it can help people to, you know, do a lot of things that they might face some challenges without others.

• 28:23Speaker 02

  Yeah, certainly, they're already superhuman in some ways. And so the opportunity for them to balance out our weaknesses is incredible. One phrase that you said there that I wanted to kind of latch on to is multiple ways to define intelligence. And maybe I'll just give you kind of my high level pitch for like what nested learning is. I think in the paper, there is a big emphasis. You guys place a big emphasis on kind of showing certain equivalences where you're like, the way that we're doing things today is sort of a special case of a more general framework that you're developing. The core idea is, I see it in the nested learning paradigm.

• 29:07Speaker 02

  And what I, what I think is like potentially for a simple person like myself, like most exciting about it is.

• 29:16Speaker 02

  For quite some time now, right? We have achieved the greater and greater expressivity of models by stacking more and more layers and just making them bigger. And that has worked remarkably well. We've been able to push that paradigm incredibly far. It's like, kind of crazy, though, right? That we just have this like one layer stacked over and over and over again in, you know, whatever, 80 or 120 layers deep or however many layers. And that's kind of it. Like, that feels like the more mature solution should be somehow more elaborate than that, right? And so, but that's been the way that we've achieved this expressivity, or sometimes the term computational depth is thrown around.

• 30:00Speaker 02

  And what nested learning is doing is bringing a different way to the table to achieve higher levels of expressivity or higher levels of computational depth. And that is by stacking not layers, but levels. And what differentiates a level from a layer is that a layer is like the same thing, or they can alternate. Obviously, we have these sort of interleaved architectures too, but these are things that are sort of in sequence as information passes from one layer to the next. But kind of a forward pass is like pass through all the layers one by one and get to the end, and that's kind of the thing. What the levels paradigm brings to it that's different is that different levels can have different update frequencies.

• 30:42Speaker 02

  And with that, you now have the possibility for some parts of the overall system to be much more durable and some parts to be updating like much more radically in something much closer to real time. And that obviously feels like just much more aligned to like what we are, right? We're not like one static thing that processes information in a fully dependent way each time. We have, you know, we're very much, our state in any given time is very much contingent on what we just experienced, but only to a degree, right? Like, we're, you know, my mood or what is currently on my mind is a reflection of what happened earlier today, but my like big picture views about the world, you know, they didn't change from this morning until now. So there's clearly some sort of hierarchy of...

• 31:26Speaker 02

  Different kinds of beliefs, different kinds of representations, different kinds of circuits that we have, which are updated in some cases very quickly and other cases very slowly. And obviously, they like are integrated together and work together. And we just haven't seen that in machine learning, except maybe in a few, you know, very far-flung kind of experimental cases. And now you're like really starting to show that with this nested learning paradigm, not only can you make it work, but as we'll get into with results, like you can make it work in a way that is competitive with transformers and even seems to have some of these new, like some of us call them micro scale advantages, where you can see, you know, with these certain early diagnostics that like, oh, this can do something that's like qualitatively different than what a transformer can do, even as it's

• 32:10Speaker 02

  also like outperforms it a bit in terms of like general perplexity type scoring. So how would you react to that kind of general, you know, high level summary? And then I also really would be interested to get your take on what is this concept of computational depth or expressivity? I'm tempted in some ways to make an analogy to just like the G factor. You know, people talk about the obviously the G in AGI is the generality. There's also G in the context of like human IQ, which is like the G factor, the sort of intangible something that's like, how, you know, how capable are you across like a very wide range of things? Again, that's like just getting at generality. Maybe in machine learning,

• 32:53Speaker 02

  it's as simple as being like G is sort of loss, or there's maybe some fundamental equivalence there, but maybe not. I don't really know. So I'm very interested in how you think about what that clearly we're getting at something that we've seen like huge progress. But what is that something is another thing I really would love to get your intuition on?

• 33:12Ali Behrouz?

  We were working on on D N A sub learning for a very, very long time, potentially, I mean, even like more than one year and a half or something. The main issue that I personally had, and we discussed a lot in the group, I thought of the authors of the work, was it was really, really hard to formalize what we wanted to deliver. Because, you know, even in the mathematical formulation, it was really hard to write it in a formal way and say what we want to exactly do in this in this paradigm. And so

• 33:50Ali Behrouz?

  After some, you know, back and forth discussions, I think we come up with this specific framework saying that there is a frequency of updates to somehow give time to each module to do something when it's waiting. I think that's the best way of describing that, why we need to have like multiple frequency. And on the other hand, we need to have a knowledge transfer, a method for knowledge transfer to somehow transfer the knowledge from one level to another level. And so when the SLOW network is waiting for the, all the computation and information

• 34:35Ali Behrouz?

  processing of the FAST network, then it should be some advantages for the SLOW network that we are paying all these costs for like doing some computation by the FAST network. And that advantage comes when we have a best or, you know, a good way of knowledge transfer between the FAST network and the SLOW one, which is like lower lever or to the higher. So the idea there is that when we have a network that is like updating very fast,

• 35:12Ali Behrouz?

  then I can use that FAST computation to give something to a SLOW computation side. So the SLOW network or the slower lever can somehow focus on the more high level knowledge abstraction of the data and then more frequently updated network can somehow focus on the FAST adoption and how they can like process in a higher resolution data, process the high resolution data. So generally, that was the way that we could somehow like describe this framework and saying

• 35:57Ali Behrouz?

  that I think these two sides needs to be there to complement each other. One is the frequency of update and another one is the knowledge transfer between levels. And so from, so for the second question, I think.

• 36:15Ali Behrouz?

  In my opinion, the current models are very efficient from one specific point of view. Why they are efficient? Because you can see that from what we can get from L L M's, they are very cheap.

• 36:39Ali Behrouz?

  And for example, you can, you know, if you want to match their power in some of the tasks with human, then potentially the cost would be very, very different. So from that perspective, we can say computation and generally like the current L L M paradigms are is very like efficient. And we can perform more computation per each parameters or artificial neuron that we have into our model. And it can help us for different things. One is that it can help us to

• 37:22Ali Behrouz?

  have more, we can have a smarter model. It's it's it's a very like subjective term to describe this. But you know, when we have more computation, it seems that we are performing some internal thinking. So a simple L L M, let's say that I have a simple L L M structure based on transformers. When the token comes, I will like do some computation on the token, pass it through all the layers and then predict the next token. And that's how how it works. But now, let's assume that for this specific token, instead of just a simple pass of computation, I also perform more internal computation

• 38:07Ali Behrouz?

  with respect to the past data or generally like, you know, combine or mix the data and something like that. In that case, I can see that the quality can potentially goes up, the quality of next token prediction for this specific design choice.

• 38:28Ali Behrouz?

  Can goes up. And why is that? Because it can be interpreted as a form of internal thinking process. So now, for each specific parameter that I have in my model, it is performing several computation. It's not just one parameter, one step of computation. It's one parameter, couple of steps of computation. So that's one advantages. Another one is about memory perspective and adoption of these models. So when we have a model that is like adapting to the context very fast,

• 39:13Ali Behrouz?

  then potentially that model can, you know, learn in context. So that's somehow one of the main messages that we try to deliver in the NIST of learning, which was everything that we know of somehow is a form of in-context learning. So generally, like, I think it's a great thing in human language that we create new words. But on the other hand, if we just create a lot of words for the same concept, it can just make us confused or it can be misleading somehow. So I think we should like create new words to differentiate different concepts.

• 39:57Ali Behrouz?

  But if we have one specific concept, we need to like stick to one specific word that we have for that concept to avoid misleading process or anything like that. So from that perspective, we realize that we can say everything is just a form of in-context learning. So we already know that what is in-context learning. Now we just need to understand how what we have. Been already doing is a form of in-context learning. And so that was the part we started to like showing that, for example, backpropagation is a form of in-context learning, is a form of associative memory.

• 40:42Ali Behrouz?

  And when it's a form of associative memory, then we can say that the general pre-training phase of the model is a form of in-context learning. Or when we go to the, for example, context of attention or RNNs, so on and so forth. Again, it's a form of in-context learning. So when we perform gradients, which we can define any RNN based on gradient descent or other form of optimization process, when we can do that, it means that we are doing some learning on the context that is happening right there. So I think these two are the main things that nested learning is trying to address. One is about generally more computation per neural,

• 41:26Ali Behrouz?

  and another one is about adaptability and continual learning.

• 41:31Speaker 02

  Can you just describe in more specific detail, like what are the relative sizes of the levels? What are the structures of the levels? What are the context windows or lengths of the different levels? What are the frequencies? Just like map the thing out for us in kind of very black and white terms.

• 41:50Ali Behrouz?

  Yeah, so and let me start from the transformer structure. In transformer, we have attention block and then MLP block. So what is happening there is that in the pre-training, we have, you know, different contexts that attention is added to. The attention side is trying to like combine all the tokens and each token attends to all the tokens before that in the context and so on and so forth. And then there is an MLP block, and that MLP block is responsible for long-term memory.

• 42:26Ali Behrouz?

  Now, when the model is pre-trained, then the MLP block is fixed. It's not changing anymore. So it has all the information compressed during the pre-training. And then we have attention. So at inference time, attention is responsible for the context that it is getting. And MLP block is responsible for the long-term memory and generally the

• 43:00Ali Behrouz?

  the general knowledge of the world. Now, let's just simply extend this idea. A simple extension of this idea is that we can simply keep attention. And then, instead of just one MLP block, we have multiple MLP blocks. Each of them are updated with different frequency. So now, what is going on there is that why it's helpful is that when you have attention, you have a fast adoption to the context. Attention is very powerful. It's like a perfect memory, cache everything, and so it's great.

• 43:45Ali Behrouz?

  On the other hand, you might want to have multiple levels of memory. And that's the part we define continual memory system. So instead of one block of MLP, we have multiple blocks of MLP. And now, you have your first MLP block. It is updated very fast. So what would happen in that case? In that case,

• 44:12Ali Behrouz?

  the first updating process of this MLP block can cause catastrophic forgetting. Because this MLP block can simply forget the information that it gets, for example, a couple of thousands tokens ago, for example.

• 44:30Ali Behrouz?

  But the point is, since the other MLP blocks have not updated so far, the knowledge that is forgotten by the first MLP block is still in their parameters. So when we perform backpropagation through all these layers, then the knowledge can come back.

• 44:58Ali Behrouz?

  So it provides and it helps to have a loop process in time. The model can, the first MLP block can forget something. But the point is, if that specific data sample or that specific skill that is forgotten is important, then this can come back through the other MLP blocks, which has not been updated so far. And they have, still they have the knowledge about that specific skill or data sample. So that's a very simple way of extending transformer block. And so we call this variant hope attention.

• 45:42Ali Behrouz?

  You know, it's a combination of attention plus multiple MLP blocks. And that's what we called hope attention. Now we have another variant, which is the actual hope architecture. What we are saying is that attention, as I mentioned, is a perfect memory. It's, you know, it can cache everything, it can scale and it's fast and so on and so forth. It's great. But the point is, still the update of attention has infinite frequency. What does it mean? It means that attention doesn't know anything about the temporal dependency of all the tokens.

• 46:26Ali Behrouz?

  And it needs something like positional encoding or even with the help of positional encoding, attention is not a great. Model for the tasks that are sequential, tasks that require sequential reasoning or something like that. What was our idea? Our idea was to replace attention with another associative memory that map keys to values. So that's what attention does. And now we want to replace it with another module that tries to map keys to values. And so one potential architecture here is Titan. So we can simply just replace Titan

• 47:09Ali Behrouz?

  and have Titan plus continue memory system. And so that's a simple idea. But the point is, you know, in initial sections of the paper, we discussed that if you have a simple linear process of update, which is what is happening inside each chunk of Titan update, this process can somehow be weaker than the case than we have self-referential process. So what is self-referential process? Gradient descent or generally backpropagation is a form of self-referential process. So what is the idea in the self-referential process?

• 47:54Ali Behrouz?

  And the idea there is that we want to learn how to learn and how to learn and how to learn and how. There are a lot of levels of how to learn, how to learn. And so computationally, it's invisible to implement all those levels of how to learn, how to learn, and so on and so forth. So there is one idea by Schmitt-Haber et al. And so basically, they have this idea of self-referential model. And how, for example, one of the ways that we can make a model self-referential when we have a key-value memory

• 48:35Ali Behrouz?

  is the case that the model generates its own value. So what is happening there? Let's say that in our memory, we want to memorize something. And, you know, there is one specific event happening, and you want to memorize it. Let's say you want to memorize one specific word. And now, in our brain, we have associative memory, and we are trying to, you know, map this specific word to another concept that we already knew, so we can memorize it. And you definitely have seen some cases that, for example, you say, "I generate this specific word,"

• 49:18Ali Behrouz?

  or "I want to like map this specific word to another word that I already knew, so I can remember that." We generate the value that we want to map our keys to it, so we can, you know, memorize the key as well. So, for the set referential process, it's a very general concept, but in this specific design choice that we have in the paper, the value of the associative memory is generated by its own parameters. So the model itself generates its own value, and then tries to map keys to values.

• 50:04Ali Behrouz?

  So, potentially, this process is fully sequential. You cannot prioritize it in a simple format, and so it has a full understanding of the causality in our data. So, potentially, in the task that requires sequential thinking, sequential reasoning, or anything like that, we can expect that this model should work better than simple attention, because a simple attention doesn't have the ability to sequentially, you know, understand the causality of the data at all. And so, that was our idea. In HOPE Architecture,

• 50:48Ali Behrouz?

  we replaced Titan with self-modifying Titan, which is exactly a Titan module where it generates its own value function, value of the associative memory. And so, that's HOPE Architecture. It's self-modifying Titan plus continuous memory system.

• 51:10Speaker 02

  I think I want to spend one more beat on what you mean when you say generating its own value. Because I'm kind of like, okay, I know the transformer architecture pretty well. We've got these like K, Q and V vectors, right? And the training process modifies all of those over time. And so the, you know, the sort of general heuristic that I have is like, for each token, there's the query vector that sort of indicates like what this token is looking for. There's the key vector that sort of helps indicate like what the, what other tokens have to offer and sort of aligns those, right?

• 51:55Speaker 02

  And finds where there's relevant, where there's a match, basically where there's relevance. And then the third one, the value sort of brings up the concepts that then get fed on into the downstream layers that, you know, make sure we have like the right activations for continued processing from there. But all of those are learned, right? All K, Q and V are learned. So I'm not entirely clear on what you mean when you say that the model learns its own values, because like, doesn't the transformer sort of learn its own value vector as well? I'm not quite clear on the distinction that you're making there between what, you know, well, let's assume people are at least generally familiar with the transformer and know how that goes.

• 52:39Ali Behrouz?

  So generally in transformer, or more accurately softmax attention, what is happening there is that we have a projection of Q, K, V. And then the output goes to attention. So attention doesn't have any control on the Q, K, V projections.

• 53:01Ali Behrouz?

  But when I'm saying that the model or generally the associative memory tries to generate its own value and then map keys to the value, I mean something like gradient descent. So if we if we just recall the gradient descent, we can see that we have something like the previous state of the W, which is Wt minus the gradient of the loss function that we have. So this is equal to the next state of the weights that we have. So if we look at this process, you can break the gradients using the chain rule

• 53:45Ali Behrouz?

  and write it into the gradient with respect to the output times the input data. And now you can see that this is getting a form of associative memory very similar to linear attention, because it's Wt plus 1 equals to the previous state of the W minus K, and K here is xt, and then v, which is the gradient with respect to the outputs. So if you look at this process, it's very similar to linear attention, but the or linear recurrent model.

• 54:29Ali Behrouz?

  But the very interesting part is it is different from linear attention, because the point is, if you look at the value, which is the gradient with respect to the output, this gradient with respect to the output is a function of Wt, is a function of the current state of the weights. So basically, the value, the you know, keys and values that we have in associative memory, the value component is not from

• 55:03Ali Behrouz?

  Another component before this recurrent formula, but it is generating by this recurrence, recurrent process every time. So that's what's, what is going on with this like self referential process. So in memory, let's say that in, in a very simple version of like self modifying Titan. If we want to design this as a simple Titan, what is happening is that I have this X and then Q K V projection. I project it into Q K V and then pass all of them to the Titan measure. So that's a simple Titan measure. But if I want to have a self modifying Titan,

• 55:48Ali Behrouz?

  then all of these parameters of the projection of Q K and V are optimized inside the Titan measure. So basically, the model has the control of somehow modifying its own updating rule and somehow generate its own value for for the memory. That's the main difference of these two.

• 56:14Speaker 02

  Yeah, that I think the key phrase maybe for people to latch on to there, starting with myself is modifying its own update rule. And this is definitely a a theme in general, like with the Mamba architecture as well, you know, the authors of of the Mamba architecture had done a bunch of previous state space model work. And the big unlock at the

• 56:40Speaker 02

  with Mamba specifically was that the way in which the state is going to be updated at each time step now became a function of inputs. And the, so that increased expressivity, your potential for expressivity of making the the actual update of the state itself dependent on the input that it's receiving at that time unlocked a lot better performance. And there's something very similar going on here. It seems like where you're saying.

• 57:15Speaker 02

  We want to make the final value output not some, we don't want to, we don't want to calculate the value vector too early, basically. We want that to come a little bit later and be more input dependent, more history dependent than it has traditionally been in the softmax attention. Is that a good intuition or is there, is there something still missing from that intuition?

• 57:42Ali Behrouz?

  I, I think it's, it's a perfect intuition. Generally, this projection of the value is also updating inside the module. That's a very important point because it helps for the adaptability. Generally, the model itself is also very adaptive to the context. And so from every token that comes, the model is trying to learn something. And so the way that it generates the value from the value term for the associative memory is exactly the same as the way it updates its memory. So it's, it's a very adaptive process to also generate the value.

• 58:24Speaker 02

  So can you take us through kind of a single time step? And maybe we can do this with the attention, hope, and then the titans, hope. And just to like, highlight the little difference there, but let's also zoom out to the big picture. So we've got now kind of a new fundamental block, right? If we do the attention, hope thing first, we've got an attention mechanism. And then we've got multiple MLPs arranged in sequence from fastest update to slowest update frequency. And then that block gets stacked into layers, correct?

• 59:05Speaker 02

  I'm a little bit confused, to be honest, on these sort of, there's no training test distinction because there is still some training process where you're like, just taking a bunch of data and like, running it through the thing, right? So from the sort of researcher's perspective, maybe from the model's perspective, there's not so much of a distinction, but from the researcher perspective, you are still sitting there like, running a process that takes a bunch of data and like, has the model learn from that, which is kind of, you know, a bulk process. That's not like a user is engaging every, there's nothing like outside of that process happening at that time, right? So.

• 59:39Speaker 02

  What, how is it different? You know, when I, when I do this for a transformer, I can like, um, I do have these, some of these really nice parallelization benefits. Uh, I do, I am interested to come back to understand like, to what degree the sort of current hardware paradigm plays nicely with some of the stuff we've got here and to what degree the, you know, the fundamental recurrence may present challenges, but bracket that for a second. Today I can like run a bunch of tokens through the thing in parallel. We can accumulate, you know, we can do this for batches. We can accumulate all these gradients and then we sort of apply the gradients and then we have, you know, the, the next timestamp and we kind of keep doing that. And I have a pretty good intuition for like how information flows.

• 1:00:20Speaker 02

  I kind of, you know, I can visualize the forward pass in my mind and then I can visualize the backward pass of back propagation going through and, you know, gradually updating all the weights. How does the procedure with the new architecture vary? What are the core things that are different from the, the paradigm that we're more used to?

• 1:00:39Ali Behrouz?

  Generally like, uh, the main difference comes from the update side that I mentioned. So for the coop attention, I think it's very similar to the current paradigm. And it's the art, even the architecture is very similar to transformers. It's the actual transformer architecture. We just replace the MLP block with multiple MLP blocks. And so I, I think when we want to do inference, then the main difference comes from the fact that for each of the MLP blocks, we need to track where we are. Is it the time that we want to update the MLP block? Or it's, it's a, has time get updated?

• 1:01:21Ali Behrouz?

  If it's the, uh, later case, then we use the last updated state of the, that's a specific MLP for, for doing the inference. If it has not, I mean, if it's the time to get updated, then we first update it. Through all the back propagation stuff and all of the tokens that we have seen so far in the in the current chunk. And then when when the weight is updated, then we perform the, you know, inference. From the research point of view, it might be a little bit hard to remove this part.

• 1:02:02Ali Behrouz?

  I I think even from the research point of view, it might be better to say that we have evaluation time and not evaluation time, because it seems we are always like update. When when the model is always up, gets updated over time. There is no training time and test time, but the point is, it seems that for for for a, you know, one specific period of time, we don't do some evaluation and we wait for some time and after that we start evaluating the model on the different downstream tasks that we have, generally like for anything else.

• 1:02:42Ali Behrouz?

  I I think from the model perspective, it doesn't know whether it is in the test time or train time, because everything is the same and it's a very uni- uh, uniform process. But from our side, definitely it is important whether we want to evaluate the model and, you know, measure its accuracy for a specific task and so on so forth or not. And so, yeah, coming back to the hope architecture, that's generally for the uh, for the hope transformer or hope attention model that I mentioned. Uh, but when we go to the the actual hope architecture, then again, everything is the

• 1:03:26Ali Behrouz?

  same, everything is very similar. The only difference is that the attention is replaced by self-modifying Titan. And for self-modifying Titan, again, everything is very similar to Titan. So the are all of the changes are are just inside the model design. And from the higher level perspective, uh, the inference is very similar. That, uh, you know, the context or document goes to the uh, self-modifying Titan. And then for each token, we have one output and then it goes to the M F P blocks. We have multiple M F P blocks and so on so forth. So everything is is very similar to the current paradigm.

• 1:04:06Speaker 02

  Did I have it right that there are like this core block of either the traditional attention or the self modifying Titan module plus the MLPs, that then becomes the block that gets stacked into layer. Is that right?

• 1:04:22Ali Behrouz?

  Yes, that's a design choice. We can have like different design choices. For example, when we so generally the initial and the main design of HOPE is in the case that we have, for example, for the HOPE attention, we have attention and then multiple MLP blocks. Each of them are updated with different frequency. But for some of the tasks, we needed to use pre-trained model. And for example, if we want to focus on, for example, LAMA, then LAMA is not designed with HOPE architecture. So what we can do, the thing that we have done is that instead of like going in this

• 1:05:05Ali Behrouz?

  formulation that I mentioned, for example, attention, then multiple MLP blocks, what we have done is that we say this is attention and MLP block, then attention and another MLP block with different frequency, and then attention and another different, sorry, another MLP block with different frequency and so on and so forth. So it's somehow design choice. You need to see like which one do you prefer? Do you want to use existing pre-trained model? Or do you want to like start from scratch and train your own design architecture? But I think potentially both of them are relatively similar.

• 1:05:47Ali Behrouz?

  They don't fundamentally make changes.

• 1:05:51Speaker 02

  Yeah, interesting. It's another, I mean, great reminder of the old Ilya maxim that the models just want to learn. There's a lot of, it's always striking to me how many of these.

• 1:06:05Speaker 02

  Choices end up kind of, yeah, I could kind of go one way or the other. I mean, that was true in the Titans case where you had like three different ways of working the memory module into the larger architecture. It's definitely been true with Mamba in many ways where, you know, you could have multiple states, you can have them in sequence, you can have them in parallel, you can, you know, once you have one of these block concepts that seems to work well, you can kind of Lego piece it in a lot of different ways. And yes, there will probably be some performance differences between different ways to arrange the blocks, but more often than not, it seems like if you're really talking about a, a serious conceptual advance, you find that like, that's less important.

• 1:06:50Speaker 02

  The exact wiring diagram is less important and more important is the core piece that you're adding to the, the set of Lego pieces, so to speak, that you can use. And so, yeah, this is a good reminder of that. How do you think about the relationship between the different MLPs in terms of size, in terms of update frequency, maybe in terms of like learning rate? I, because it feels like there might be some, feels like learning rate might be kind of important here where there's like a potentially an equivalence if I update one MLP every token and then I update another one with a larger batch size. I can, I feel like I could make those like much more similar or quite a bit different

• 1:07:34Speaker 02

  depending on like what learning rate I apply. But yeah, so size, frequency, like intangible terms are the small, are the ones that are updated less frequently also smaller? And do they have like different learning rates than the other ones? Like take us through that kind of, um, how do you think about the, the relationships between the MLPs of different frequencies?

• 1:07:53Ali Behrouz?

  Yeah, I think it's, it's really depends on, on the architecture and the number of parameters and the design choices that you have. It's very similar, you know, we cannot say that the best, what is the best dimension for transformers or attention block? Potentially, it's really hard. It really depends on the person that wants to like work with that attention and the use cases that we want to consider and so on and so forth. So generally, the updates, the frequency of update really depends on, you know, how adaptive you want your model be and for example, how do you want the model to maintain its persistent memory and so on and so forth.

• 1:08:34Ali Behrouz?

  So I think that's, that's pretty much a design choice. But like learning rate, I think generally you can treat each of these blocks as the same way as, as you do about MFP blocks. They're exactly the same thing, but the point is they have like different frequency of update. So nothing has changed. So everything is exactly the same thing as MFP blocks. And so I, I don't think that, you know, it definitely would be very interesting to see how the learning rate can affect each of these blocks and so on and so forth.

• 1:09:16Ali Behrouz?

  But I have not done that and I'm not sure about the exact solution. But my expectation is that potentially any way that we use to do hyperparameter sealing, we can do the same thing here as well. So there shouldn't be any differences.

• 1:09:32Speaker 02

  Interesting. And so your general default is kind of an intuition based approach. And do I have it right that it's basically kind of order of magnitude? Like the fastest one updates every token, the next one updates every 10 tokens, the next one every 100, or maybe it's like a couple of orders of magnitude? Like, how did you, understanding that it's not yet a fully tuned system, what did you start with and why did you pick those things as your kind of initial guess?

• 1:10:03Ali Behrouz?

  Uh, yeah, and and the way that we choose in uh we choose the uh you know, the frequency of update for each of them uh was based on our intuition of uh, you know, the chunk size that we use for Titans and other models because uh generally the the chunk size in Titan is is is the part that we can define the frequency of Titan as well. At that time, we didn't have this term of frequency, but generally uh you know, that chunk size can define frequency for Titan as well. Uh, so what we used was was based on our intuition about uh how uh what chunk sizes are are good for for Titans. And so and uh, you know, uh, I I think

• 1:10:47Ali Behrouz?

  we didn't have like, you know, something like,

• 1:10:52Ali Behrouz?

  as far as I remember, I think the numbers that we used was possibly one hundred twenty eight, then four times one hundred twenty eight, and then four times four times one hundred twenty eight. Uh, so it was something like that, as far as I remember.

• 1:11:10Speaker 02

  And how about knowledge transfer? How should we think about the way in which? And I guess one one other quick interjection question there, too. There are still like skip connections.

• 1:11:21Ali Behrouz?

  Yes, yes, everything, everything is similar. Yes, exactly. One, you know, one, one thing that we tried to do in lesson learning, I mean, uh unfortunately, uh, it caused some misunderstanding about what we we are doing. Uh, and you know, I I have received, I have seen some comments about, for example, some of the concepts here are already new and something like that. But the point is, we try to actually include that all those concepts that we already knew to show that it is a universal learning paradigm. It's not something that contradicts with our current understanding. It just complements and completes what we already know about like in in a new direction.

• 1:12:04Ali Behrouz?

  For example, you can are, when you are doing deep learning, any form of deep learning, and you are saying that I'm using this attention here. You are actually using nested learning.

• 1:12:18Ali Behrouz?

  But in deep learning, you only see the final solution of each learning problem. So you have a learning problem inside the attention, and you are trying to solve a regression problem. And the non-parametric solution to that regression problem is attention. When you see everything from deep learning side, you can only see the final solution for each component. But when you see everything from the nested learning side, you can see the internal learning process of each component as well. So in general, it's not something that contradicts what we already know,

• 1:12:59Ali Behrouz?

  but it somehow complements all of the things that we knew and go beyond that. I think that's generally an important part. So yeah, everything can be very similar. It can have its limitations.

• 1:13:15Speaker 02

  Yeah, so then help us understand how we should think about the roles that the different frequency MLPs are playing. Knowledge transfer is one way to think about that. Or another way to frame the question might be, how do they complement one another? How do they work together? How does the one that, and I would be interested to understand this both intuitively and mechanistically, to the degree that you have mechanistic understanding. But how does the one that's updating fast gradually inform the ones that are updating slow? How do the ones that are updating slow steer the ones that are being fast in the right directions? How do you think about the interplay between those different components?

• 1:13:57Ali Behrouz?

  Yeah, I think generally, knowledge coming up with different ways of knowledge transfer is very important here, in my opinion. So the main point of the frequency, having frequency for each component is two parts. I think we also discussed it in the beginning. And the first part is that it helps the model to maintain its memory for the longer time period. For example, you know, that's just a simple example. Let's say that we have a tini, and then one of them goes to a spaceship and just move with the speed of light.

• 1:14:42Ali Behrouz?

  And then right before that, they have a very good memory. For example, they went to You know, they have launch or something like that. I don't know. They have very good memory. And then the person that is like moving with the speed of light, when they come back, eighty years has passed on Earth. And then their sibling somehow forgets about that specific launch they had because it was eighty years ago. But on the other hand, that specific person has all the information, all of the details are just one seconds ago, two seconds ago.

• 1:15:26Ali Behrouz?

  And so they remember everything about that launch. So why it's happening? Because of the updates in the memory of each of them. The person that lived eighty years, their memory got updated a lot of times. But the person that live, just move with the speed of light or close to it, now their memory didn't get updated that much. So from this perspective, we can see that that the number of times that we make some updates to the memory is very important. And you know, it's a very inaccurate example that I mentioned,

• 1:16:10Ali Behrouz?

  but I think the main point is clear here. When we have like two components, one of them is updated a lot of times, while the other one is a slower one and is not updated that much. What is happening there is that

• 1:16:27Ali Behrouz?

  the slow one has the opportunity to learn something from the fast one. Because at the time that the fast one is got updated.

• 1:16:40Ali Behrouz?

  The slow one has not get any update. And so now there is a chance that the fast one learns something to the slow one before get updated. So I, I think that's the part we, we, we can also like discuss this asleep process that we have. So what is the idea there? The idea there is that when we have multiple levels of MLP blocks, each of them are updated with different frequency. One simple thing is that before updating the fast MLP block, and by fast, I mean, you know, it's a relative term. So there is a fast and a slow while we have like multiple level. So for each two blocks, consecutive blocks that we consider,

• 1:17:25Ali Behrouz?

  when we want to update the fast one, we know that there is a chance that we forget something.

• 1:17:33Ali Behrouz?

  So before forgetting something, we can transfer the knowledge of this block to the next one, and then update this one. And that's the part we need a good way of knowledge transfer. So for example, one way is to, you know, do context distillation. So if you want to pass the knowledge from one MLP block to another one, some methods of context distillation can works very well in that case. And that's very similar to what we do in the asleep process as well, in the asleep paper. And so, yeah, I, I think the main role of the knowledge transfer is the, is, you know,

• 1:18:17Ali Behrouz?

  help the slow network to take advantage of the fast network. And another point about, you know, having different frequency is about the memory and how, how the model can manage its memory, similar to the example of the tunes that I mentioned.

• 1:18:35Speaker 02

  What is the mechanism by which the information in the fast update layer gets moved to the slower layer? But then there's also got to be something going the other way, too, right? If you conceive of them as like pure perception, in a sense, then I guess I'm not even entirely clear on what's happening in my own brain, but certainly like there's more information flow. It feels to me like from my sort of perception modules to my, you know, higher order reasoning modules, whatever, than there is from the reasoning back to the perception, that signal. But it is, that signal is still important, right? Like my, my higher order processes do tell my eyes where to look and do tell them like where

• 1:19:20Speaker 02

  to focus and do say, you know, hey, we need to like zero in on this, this detail a little bit. Like, I want to understand that better. So like, go put some of your bandwidth into this particular thing. So, yeah, I guess, give me a little more on how the mechanistically or procedurally, how the information in the fastest update layers is getting transferred? And how we're making sure that we're storing what really matters from what the fast updates have learned? But then also, what is the, what is the signal that flows the other way?

• 1:19:52Ali Behrouz?

  And so, and let me answer that with a very simple example. Let's say that I have this model and I want, I have model A and I want to update it fast MLP blocks. And I want to make sure that the information in the fast MLP block is not forgotten and it can pass to the slow MLP block.

• 1:20:15Ali Behrouz?

  One simple thing that I can do is to just copy all of the parameters of model A to model B. Now I have two identical models. One is model A and another one is model B. What I do for model B is that I update the fast network, the fast MLP block. Now, the parameters in the fast MLP blocks of model B are just free.

• 1:20:46Ali Behrouz?

  And now what I want to do is that I want to change the parameters of the slow MLP in Model B in a way that the output of parameter, sorry, Model B can mimic the output of Model A. If that happens, it means that, you know, what is the difference of Model A and B? Model A has all the information compressed in the fast MLP block, while all those information are gone in Model B. Now, if I could somehow modify Model B in a way that it can mimic Model A,

• 1:21:30Ali Behrouz?

  it means that I somehow transferred the knowledge in the fast MLP to the parameters of the slow MLP in Model B. So that's just one simple thing. So and this process is very similar to the distillation process. We are distilling the, you know, the knowledge of Model A to the knowledge of Model B. So from this perspective, we can see that, you know, these two, somehow this is one way of knowledge transfer from slow, sorry, fast MLP to the slow MLP. That's for example, one example. Another example, which is like very common and popular is back propagation.

• 1:22:12Ali Behrouz?

  So if you just sequentially connect your MLP blocks and then at some point perform back propagation, then you can like transfer the knowledge of one block to another one and so on and so forth.

• 1:22:26Speaker 02

  So the sort of copying and distillation process you described, that's essentially what's going on in the language models need sleep paper?

• 1:22:38Ali Behrouz?

  Yes, with some additional detail. For example, what we do there is that we also like additional parameters to the Model B as well to make sure that it has enough capacity to store the new knowledge that it has gotten just now.

• 1:22:55Speaker 02

  Does that mean also that in the nested learning version of this, you're really just letting back propagation do its thing? And you're not really like, you're not, you haven't really over engineered. You haven't engineered it all that much. We just have these. MLP blocks, they get updated at different frequencies and you're just kind of letting gradient descent do its thing and the updates are just kind of working. That's basically it.

• 1:23:20Ali Behrouz?

  Yes, exactly. Yes. In the whole part of the picture, everything is just like back propagation.

• 1:23:26Speaker 02

  A huge takeaway from the conversation that doesn't, that didn't come through to me as clearly in the paper is just like, this is really proof of concept staged stuff and the fact that it works so well shows like what a good concept it is, but nothing that we're discussing here is really, you know, has been through the same kind of thing that the mainline models have been through where, you know, everything has been sort of parameter, hyperparameter explored to the nth degree and optimized and, you know, all the little refinements that have been obviously made over time. That hasn't really happened here. So there's a lot of questions that we still could ask about like this version, that version, this configuration, this arrangement, sequence, parallel, you know, how many layers, relative

• 1:24:11Speaker 02

  sizes, relative learning rates. There's like a ton of, of space there still to explore, but basically just taking a few of these core concepts, the main one being the different frequency of updates for different MLP blocks, that alone creates some pretty impressive results that are like qualitatively different from what we are used to seeing. So maybe let's take a minute and just talk about some of the results. I mean, the, you know, there's a lot of different tests run in the paper and, you know, big tables of results with a whole bunch of different metrics, some of which are kind of your perplexity scoring classic type of stuff. What do you think are like the most important revealing results

• 1:24:53Speaker 02

  that you, you think people should say, ah, because I see that it can do that, I know that there's like really something here that I need to grapple with?

• 1:25:03Ali Behrouz?

  Yeah, and this We have one continual learning style task that I personally really like. So the idea there is that we have a pre-trained model. And there is one specific language that the model has not seen before. And so we want to learn that specific language in context to the model. We want to learn the model that is language in context. And then

• 1:25:40Ali Behrouz?

  the the point is we have all the grammars, we have all the words, we have there is a dictionary of words. And so we pass all of them through the model in context. And then the model learns the language. And then we ask that, can you translate this specific text to English from that language to English? And then we can see that the model perfectly, not perfectly, but but in a very, very, very good quality can translate that specific text. So it seems that the model is capable of

• 1:26:21Ali Behrouz?

  understanding that language in context and then use that for for some translation task. But the point is, let's just go one step beyond that. Instead of one language, let's put two languages in context. And then ask the model to translate, you know, different texts from each of these languages to English. In that case, we can see that the model almost collapse

• 1:26:54Ali Behrouz?

  and cannot translate any of those languages.

• 1:27:00Ali Behrouz?

  The point is, the model cannot handle its context well and fully understand each of the languages separately. And you know, that's generally a very, very hard challenge for transformer-based architectures. And but the point is, when we change that architecture to hope or hope attention, again, we have attention, but we have multiple levels of in-context learning, multiple levels of M L P blocks. And so one thing that we can see is that when we increase the number of levels, the performance of the model in both of these languages gets bitter and bitter and bitter. Why it's happening?

• 1:27:46Ali Behrouz?

  Because the model has bitter way of memory management. Because it understands that, for example, temporal knowledge that are not very needed can be stored in the first M L P block. And more understanding of the language can pass to the, you know, more stable M L P blocks, later M L P blocks that are more stable. And then, when we have like more and more blocks, we can see that the performance get bitter and bitter. So in my opinion, that's a very like good evaluation for understanding whether the model can learn in context and generally like to continue learning.

• 1:28:31Ali Behrouz?

  Yep, I think that's.

• 1:28:34Speaker 02

  So this is, is this the same? I remember it's been a while since I thought about this, but I think it was with maybe Gemini two? Maybe I don't even know. It was even as far back as Gemini one. There was this metric introduced of learning a new language from basically one book. There's like, it was like, there's some critically endangered language. One person has, you know, really studied it and made like a book that's not on the internet anywhere that sort of explains what this language is. And then they just put that book into context and say, okay, based on this, you know, go ahead and do translation. It seems like this is, I don't know if this is the exact same test as that one that I was previously familiar with or if it's a bit different.

• 1:29:17Speaker 02

  But it seems the language here is Manchu. I just looked it up. It's like a critically endangered language from somewhere in China. So that's basically the idea, right? It's a language that the language models have basically no prior knowledge of. They're given a sort of very detailed primer on this language from some anthropologist or whatever who's gone out and done the fieldwork. And then their job is to apply that. And so I'm looking at figure eight in the nested learning paper. And what I'm taking from this is all of the models do kind of similarly if there's just one language. But as you said, when you go up to two languages and sort of double the difficulty of the

• 1:29:58Speaker 02

  task, then the in-context learning traditional transformer approach performs quite badly. And then I understand hope one, hope two, hope three. Are those like how many levels? Like how many different frequency update things exist? So when you move from traditional to, I guess, one additional, two additional, three additional frequencies of update, you get basically almost all the way back to the original level of performance with just one language?

• 1:30:31Ali Behrouz?

  Yes, exactly. Yes.

• 1:30:33Speaker 02

  Yeah, okay. That is, that is quite interesting. Yeah, fascinating stuff. And what is MTOB there? Just so I have that clear. MTOB, that's a, that's the other?

• 1:30:48Ali Behrouz?

  Yes, that's another data set.

• 1:30:52Ali Behrouz?

  It's another language that's, that is also has not been seen during the pre-training of the model.

• 1:30:59Speaker 02

  Yeah, okay. This is the one that I recall. Yeah, so you guys added the Manchu language to this one from, yeah, late 2023. So both of these like very rare, unknown languages being translated to English. And only with the multiple layers can the models do both.

• 1:31:22Speaker 02

  At the same time, in one context. Yeah, very, very, very interesting indeed. How do you think about kind of the? I really like that answer. That's a great intuition builder. How do you think about just kind of things like perplexity scores? I mean, you've got a big table that shows on things like perplexity and sort of some accuracy stuff on some of these like basic kind of classic battery of tests. And I should say, we're scaling these models so far up to roughly the 1 billion parameter scale. You've got 760 million parameters and 30 billion tokens. And then the bigger is 1.3 billion parameters and 100 billion tokens. So obviously, that's like, not huge by today's standards.

• 1:32:03Speaker 02

  But nevertheless, you know, there's a pretty clear signal that the hope architecture is on just about every dimension, outperforming all the other things you're comparing it against, which includes your transformer and your Mamba and Mamba variations and even Titans. Retnet is in there, DeltaNet is in there. How much? How do you interpret these? You know, this kind of goes back to that G question. Is this like, is this a good measure? Is it just the best measure we have? What do you think about like, how much stock people should put in these like perplexity tables?

• 1:32:35Ali Behrouz?

  You know, there are some standards in the community for like performing some benchmark task. And not all of them are the best things to do for evaluating the model, but we just need to do them to make sure that everyone is, everyone can see where the performance comes from, the advantages comes from. And so I think I have one specific transformer structure and I want to like, I have one idea that if I add, for example, for getting to the transformer, then it can.

• 1:33:15Ali Behrouz?

  Perform well on noisy data, for example, I'm not sure, I'm just like coming up with just one example. And so in that case, if if there is no noisy data and I just test my method on a very clean data, then there is no way that I can show the advantages of my approach in that case. So I think here is exactly the same thing. We are arguing about models that do not need to be pre-trained. There is no test time, there is no train time and so forth.

• 1:33:55Ali Behrouz?

  But on the other hand, still a lot of infrastructure built on like test and train time. A lot of evaluations are built on test and train time. And so everyone also expect us to report something about pre-training perplexity and some some evaluations, that most of them are some short-term and short-context language modeling tasks. And they do not need to have a very complicated model to understand long-context modeling. So I think I also like mentioned that in the presentation of NESTed Learning at NURIPS, we didn't like use Table 2

• 1:34:38Ali Behrouz?

  and all those perplexity and language modeling tasks to argue that hope is powerful. We just use that table to say that hope is not less powerful as a backbone compared to other models. And you can see that like it performs well. But someone might say that it's marginal compared to other like models. But the point is, this is the this is not the direction that we aim to solve. And somehow it's it's really good that we can see that even in this direction, that is not the goal of the NESTed Learning and hope. We can show some improvement, even if it's marginal.

• 1:35:20Speaker 02

  Yeah, gotcha. I'm always a big fan of trying to get a little bit better sense for the micro skills of different architectures. So, for example, of course, you know, transformers, because the full sequence, the full context is in working memory at all times, it's pretty hard to beat. And I feel like you even sort of have like, kind of a theoretical argument now that like, it may even be kind of impossible to beat in some, in some of these tasks where the idea is like, recall from the context window. But then, you know, we saw things with Mamba, for example, where it was better at learning from like sparse signal.

• 1:36:02Speaker 02

  And this was sort of a micro skill that it, that architecture excelled at, that the transformer relatively struggled with. What, what, what have you seen in the hope case, you know, are there little micro? And I, I think this is very interesting because it does kind of ladder up to the overall performance, you know, and what these things are actually good or bad at, right? I mean, the ability to recall something in, in context is really important when you need it. The ability to learn from or kind of filter out noise and get to the, you know, the signal that really matters is, is really important when you need it. So are there particular micro skills that, that stand out? I mean, the, the language translation one is, is an interesting one in a macro sense of like, that's a hard task. But I wonder if you drill down to these like, very micro building block competencies

• 1:36:45Speaker 02

  that models or architectures can either have or not have, what stands out in terms of what this has that transformers don't have or don't have as strongly?

• 1:36:55Ali Behrouz?

  When we are talking about in-context recall tasks, or generally like recall intensive tasks, in my opinion, all those tasks are designed for transformers. They are not designed to compare architectures, but they are specifically designed for transformers. Why I'm saying that, because You cannot expect from a model or even a human to perform needle in haystack perfectly. Or for example, do some recall intensive task. For example, I assume that you have a code and like couple of thousand lines of code.

• 1:37:40Ali Behrouz?

  And then simply you want to recall what was the value of X at some line of the code. And so it's almost impossible, or at least it's very, very hard for a human or even like for other models to do that. And but on the other hand, it's it's pretty much simple for transformers because they have direct access to the entire history in their context. And so it's very simple to just like, you know, find that token and pass it as the out. And in fact, find it somehow. In recall intensive task like this in context recall task that we have here.

• 1:38:24Ali Behrouz?

  Actually, the gap between recurrent architectures, which actually their performance is also very great. If you compare the, you know, first generation of recurrent architectures to the transformer, we can see that this gap was much, much larger. Now this gap is getting like a smaller and the performance of other recurrent models is also very great. But the interesting part for me was that hope at least close this performance, these close this gap in the performance of the model compared to transformers. While they are not expected to do that. Transformers is maybe expect transformer to do that because it has attention block.

• 1:39:05Ali Behrouz?

  But we don't expect a compression based model to perform recall task. And so I think somehow it was interesting for me.

• 1:39:17Speaker 02

  And what is the MAD dataset get at? How should we understand like? Because that's that's where to just replay back to you what you just said. The on these like needle in a haystack, these like very difficult recall tasks from earlier in context. The transformer remains the best. The recurrent models which.

• 1:39:36Speaker 02

  Only have some latent representation and don't have the ability to look back at the original raw text, don't perform as well. But with each generation of improvement, and here you've got several, the hope architecture does the best of the recurrent ones that don't have the full explicit context in working memory at runtime. And so that gap is closing. Flipping over to the MAD dataset, here the hope architecture is performing better than everything, including the transformer. What like micro skills is that testing? What should we take away from that result?

• 1:40:10Ali Behrouz?

  The MAD dataset also like is very similar to the recall intensive tasks. But the point here is that, you know, there are different setups, right? For example, in one of them is the noisy in context recall. We want to perform recall and in-context recall task. But the point is, we have some noise in the tokens. And now, when we have that noise in the tokens, somehow the power of transformers that I explained in the previous setup, which is, which was pure in context learning, now is its weakness somehow, because it can get simply confused about which token is noise,

• 1:40:53Ali Behrouz?

  which token is not, and so forth. So potentially, this task becomes a little bit, for example, harder for transformer compared to a model like hope. But again, that is also very, that also depends on the memory management of the RNN. So again, for the RNN, if it doesn't, if it doesn't have a very good memory management system or generally update mechanism, then potentially it can simply get confused by the noise as well and face some issues. But again, if the memory management is strong, then it's much simpler to filter all those

• 1:41:37Ali Behrouz?

  noise tokens in the task.

• 1:41:41Ali Behrouz?

  And so, yeah, I think that's, for example, one thing. Another task that is also like, I think it's interesting here is about compression. So the compression task here, somehow, you know, the name explains the task itself. But we want to compress the tokens and predict one single token that is the compressed version of a set of tokens. And so then, you know, we see that and then we want to like reconstruct the original sequence from that. And so potentially, it's a simpler task for models like RNN because they already

• 1:42:25Ali Behrouz?

  knew how to compress the data properly. But on the other hand, transformer has harder time to perform this task. So generally, like as I mentioned, all of these tasks, I don't want to like go into all of the details, but generally, all these tasks are somehow modified version of recall intensive tasks or in context recall or something like that. But the point is, there are other aspects, for example, selective copying is another one. There are other aspects to the model that those aspects also are very important. And we should also like see how the model performs in those aspects and not just, you

• 1:43:10Ali Behrouz?

  know, overfit our evaluation on one specific metric.

• 1:43:16Speaker 02

  Cool. I think that's probably enough on the really low level stuff. And I think this illusion of architecture title of the paper starts to click for me. On page 39 of this paper, we get to the part where you also have a new optimizer that is outperforming not just your old Adam standard, but also even outperforming Muon. It does come with a little bit of computational overhead, but I think again, the argument is that it more than pays back for itself in terms of.

• 1:43:46Speaker 02

  Faster convergence or just better learning? Is there anything you want to add on the M3 optimizer, as you call it?

• 1:43:52Ali Behrouz?

  First, also like clarify one point that generally for optimizers, it's a little bit like hard to say that, for example, this specific optimizer is more powerful than the other one. It really depends on the problem setup or generally even the problem. So for example, we might here we are like evaluating the optimizer on, for example, region task. But on the other hand, if you train a language model, you might see that the trend is completely different or something like that. So generally, like the design of optimizer and saying that like which one is better than the other one really depends on the task or if you take share, problem setup and all these things.

• 1:44:36Ali Behrouz?

  And so that's also one of the main points that we wanted to deliver in nested learning. Because what we are saying is that the entire architecture with its optimization process are just one interconnected system of nested optimization problem. And this is interconnected. Why it's interconnected? Because the gradients of the optimization side is generated by the architecture. If you have a simple architecture, then the gradients are very simple. If you have a complicated architecture, the patterns in the gradients can be very complicated. And then when you have momentum term, momentum is a form of,

• 1:45:19Ali Behrouz?

  is a form of associative memory that is trying to compress gradients. So for example, if your gradients are very complicated, you need more powerful memory management system for your momentum. Or if it's very like simple, if it's very simple architecture, then even a simple gradient descent without any momentum might work very well. So in general,

• 1:45:46Ali Behrouz?

  One of the arguments that we have in the paper is that we should see everything as an interconnected system and tries to design something that altogether results in a good model architecture, or generally like a machine learning model in a very general term. So

• 1:46:08Ali Behrouz?

  that's that's one argument. Another thing is that we wanted to deliver this message that architecture side is very, very, very similar or somehow exactly the same as optimization side. All of them are just some learning room. And there are some learning process that is happening. And the only difference between the architecture side and the optimization side is just the context. The context of the optimization algorithm is gradients. Actually, the context is the set of gradients that we have. And the context of the architecture side is a set of tokens that we have.

• 1:46:53Ali Behrouz?

  So generally, they are very similar. So in the paper, we had this continuum memory system. We extend the MLP block, saying that you can have multiple levels of frequency for the MLP block. And you know, that's a very general term. And in the entire paper, we are arguing that architectures are the same as optimizers and so on, so forth. So why not applying that technique and somehow borrow that technique from architecture side and apply to the optimization side? So that was the main motivation to show that this continuum memory system that we have designed is not just working well for

• 1:47:38Ali Behrouz?

  architecture, but it also works very well on the optimization side as well. And so we just simply extend M1. Instead of one specific MLP, sorry, one specific memory, it has multiple memory. In the case of M3, it has like two memory and so. It's trying to compress the context with different frequency rates. And so it can help you to better understand the global aspects of the lost landscape. And it potentially can help the model to find more effective solution.

• 1:48:15Speaker 02

  The new paper, Language Models Need Sleep. Tell us a little bit more about like what's going on here.

• 1:48:22Speaker 02

  You mentioned kind of at the top, the two-phase concept. We have the memory consolidation phase and then we have the dreaming phase. I do think it's fascinating to consider that there's like kind of creation of net new parameter space and then consolidation or sort of pruning back. I understand too of because obviously the things can't just grow and grow forever. Right. But yeah, take us, take us through this in, in more detail. I'm fascinated to learn more about it.

• 1:48:48Ali Behrouz?

  Generally, the main idea, as we discussed earlier, was that if we have a truly continual learner model, then there is no test and train time. On the other hand, we need to have one active time where the inputs is coming in an online manner. And also the time that is the time we don't, we don't have any inputs. So the model is not actively receive information from the outside, but it doesn't mean that the model should be static. It means that the model just doesn't get input, but it can have some internal computation to improve itself.

• 1:49:32Ali Behrouz?

  And so that's a very, very general concept. We can incorporate more and more components to the sleep time that we have. So it doesn't have to be just these two a specific part, but you know, these two were really relevant to the research that I'm doing. So we just like did that, but potentially it can include any other form of self-improvement and so forth. So that's just one, one way of breaking the life of a continual learner into active time and sleep time. But what we have in the sleep time right now, which again, as I mentioned, we can incorporate more components to it, is that

• 1:50:18Ali Behrouz?

  we want to make sure that when we update each of the components of the model, we don't forget about the knowledge that is stored in the parameters. And so what is the idea there is that

• 1:50:33Ali Behrouz?

  what we do is we know that there are like multiple MFP blocks. Each of them are updated with different frequencies. And then

• 1:50:44Ali Behrouz?

  again, the fast and slow here is just a relative term. It doesn't mean that, you know, the slowest or the fastest. So we have a slow and fast weight. It can be any part of the neural network. And we want to transfer the knowledge from one to the other. So in order to do that, what we do is the distillation process that I mentioned. But the distillation process is based on policy distillation. So the model itself generates some data. And so one interpretation of this process is that

• 1:51:23Ali Behrouz?

  we distill the knowledge of one small model to a larger model. When we want to go from one step to the next one, we activate new parameters in the next level. So it can help the model to release some of its capacity and be ready to accept new knowledge. So somehow it is somehow it describes a very natural way of learning in humans as well. And for example, it's very common that when we learn something,

• 1:52:08Ali Behrouz?

  when a new, when we learn about a new concept or something like that, we don't have A full understanding of all aspects of that. But the point is when time passes. And you know, we we let the we let our brain to better understand that concepts over time. And also like, you know, we we study other things and better understand the entire process. Then at some point, we can see that we have a very clear picture of what's going on in that concept. We can completely understand it. And so generally, that's the best thing. That's a very good way of learning.

• 1:52:51Ali Behrouz?

  So here is is exactly this seem very similar process. A very, very similar process. Uh, so also like discuss that from this perspective. I think it might be better and more simple way to uh, understand why we need like have multiple levels and they steal the knowledge from each level to a dot another one. So when want to understand a specific con- concept, we have different levels of apps, uh, knowledge abstraction for ourselves. Uh, the first level, which is the most simplest one, is to just memorize things. So let's say that uh, we want to learn one a specific mathematical rule, or for example, a a specific

• 1:53:36Ali Behrouz?

  concept in physics or any science.

• 1:53:41Ali Behrouz?

  How how we can learn that? We start with some example of that specific concept. And then we start memorizing those concepts. For example, if if it's a mathematical rule saying that, you know, just just just any mathematical rule that we can have. We start with some specific examples and just memorize them. And then. At some point, we just neuralize our understanding of all those examples, remove all those examples in our brain, and replace all of those memories with just one single memory

• 1:54:23Ali Behrouz?

  that can describe everything that we have learned so far from that concept. And then, when time passes, we have more information, we read more about that concept, and so on and so forth. Again, we revisit our understanding of the concept, and then replace our previous understanding with this new understanding, which is more general, and it can explain more phenomena or more terms in that specific concept. So that's generally the way we understand things, and there are different levels of abstraction in our understanding. So now, when we have different MLP blocks, or generally, let's just go to any arbitrary architecture.

• 1:55:05Ali Behrouz?

  It doesn't have to be just hope, it can be any architecture. But the main thing is that each of them or each of the blocks are updated with different frequency. In that case, the fast updating block is very similar to the memorization process, because we memorize a lot of things, we don't need to understand that, there's no pure understanding of that concept, it's just memorization, and we can also forget very fast something that we have memorized. So the first level is that, so the first block is responsible for that part. But if we want to better understand that concept, we need to do some memory consolidation.

• 1:55:50Ali Behrouz?

  So what is happening in our design is that we transfer the knowledge from fast updating module to the other one. But if we just simply pass the knowledge from fast updating block to the slow updating block, then nothing has changed, we just transferred the knowledge without doing anything. But instead of just simple transfer, we replace that with a distillation process. Why the solution here is important? Because the previous block, or generally, the fast updating block, has compressed the concept.

• 1:56:34Ali Behrouz?

  And somehow understand it or memorize it in any way. Just it's just a compression process. When we do a distillation, then there is another levels of compression that force the model to, you know, you don't have that all those parameters anymore. You have now have less number of parameters to store that specific knowledge. And in order to do that, you need to come up with something that is more general and can can understand underlying patterns in the data in a better way. So you can store everything in just a smaller number of parameters.

• 1:57:16Ali Behrouz?

  So in that case, the model would come up with better levels of knowledge abstraction because we have forced it to do it. And then again, we just repeat this process, so on and so forth. That's generally the main idea of memory consolidation. And like every time that that this sleep process happens, we consolidate the knowledge from one level to the other one and so on and so forth. And so that's that's a very high level idea of what's happening in the memory consolidation.

• 1:57:50Ali Behrouz?

  Another part is about dreaming. So why we need to have this dreaming process? The main thing is, I I think there are like two important points when when we want to implement this dreaming process. The first part is we need to have a self improvement process. So we have learned something so far. Actually, the the memory consolidation part can also be seen as a form of self improvement. But, you know, if we have one a specific task at hand, if we want to like specifically optimize the model for one task, then this is the place that we can do it.

• 1:58:31Ali Behrouz?

  We can self modify the model and like fine tune it or generally like use RL to update the models and self modify it. So it can be

• 1:58:44Ali Behrouz?

  more powerful in one specific task and so on and so forth. That's just one advantage of dreaming. Another advantage of dreaming is that in the dreaming process, we need to understand the connection of concepts.

• 1:59:00Ali Behrouz?

  That seems to be irrelevant, but they are actually relevant. That's also what is happening in the dreaming process of human. We can see that we can we can see very weird dreams because the brain is trying to understand the connection of very irrelevant concepts and see whether there is an underlying pattern in that. So here in the dreaming process, we need to also have that as well and understand different aspects of how we need to combine different knowledge stored in different components of the model. So that's another goal of the dreaming.

• 1:59:41Ali Behrouz?

  And, you know, we can just combine these two into the sleep process and the model after one step of sleep, the model has consolidated its own memory and also on the other hand, there's a self-improving process. Top of that.

• 1:59:57Speaker 02

  So in the sleeping process, I'm seeing that there is there are new parameters created as to create space in the slower frequency updated portions to absorb the information from the faster frequency ones. Does that ever shrink back down? Is there a is there a pruning or is there a sort of other side of that that balances that? Or at this stage, do these models just grow indefinitely throughout their life?

• 2:00:31Ali Behrouz?

  From the technical point of view, we cannot grow the model to like arbitrary large number of parameters. But the point here is that it's like a periodic process. We add some parameters and then we free them for the next step of consolidation. When we are in the first block. We add some components. When it reaches its capacity, it means that it is the time that we need to consolidate the memory to the next step. And when we consolidate all this knowledge to the next step, we just remove all the extra capacity that we have added to this level

• 2:01:15Ali Behrouz?

  and freedom for the like other levels, like faster levels, so they can also consolidate their their memory to this block as well. So generally, it's like a periodic process. We add components and remove them, add components and.

• 2:01:32Speaker 02

  I see. Gotcha. Okay. Interesting. And what more can you tell us about the dreaming phase in terms of like, just a little bit more practically, what's going on there? I mean, it certainly, when I try to introspect into dreams, it, um, I think that hasn't been super, I mean, maybe it's been somewhat fruitful, but I think also people get very confused when they try to interpret dreams or understand, you know, what's going on there. So I won't even attempt to, you know, ground my understanding in my human dreams, which seem like quite a, um, quite a hard thing to untangle, I guess. But here, there's a more, I mean, you got to design the process. So like what procedurally, what is going on in dreaming? Just a little bit more like mechanically procedurally.

• 2:02:18Ali Behrouz?

  And the concept of dreaming doesn't mean that it's exactly the same thing as dreaming in human. It's just, you know, at a very high level, they seems to be very similar. And so in that's that's one point. And another point is that, again, the concept of sleep and dreaming for a language model might be very different from the concept of dreaming and asleep for the, for example, vision model. Because potentially a vision model will generate some images, generative like a vision model might might generate some images during dreaming. While in this case of language modeling, we are generating text. But the framework is very general.

• 2:03:02Ali Behrouz?

  It can adapt to any con, any data modality. And so that's, that's, that's very general. But the point is here, when we are like doing that for a language modeling, what is happening there is.

• 2:03:20Ali Behrouz?

  We generate some context. We generate some text. And how do we generate those texts? It's on policy distillation, the same way that we discussed earlier. We have a model, we copy that. We want to distill the knowledge from one level to the other one. And so we free the parameters of the slower level and sounds of earth. And so we ask the smaller one, the small model, which has the knowledge of the context as well into its parameters. We ask that to generate some text. And then we want to train or somehow, you know, update.

• 2:04:05Ali Behrouz?

  It's a better term to use, update the actual model, the actual model parameters on this specific data set that is generated by the model. And then, how do we train it? We start with one part of the sequence, just sample some of the tokens. And then ask the model to predict the next tokens in that sequence. So that, for example, that's really similar to, you know, generating some synthetic data. It seems if the model can perfectly predict the future tokens, it means that it has already knew about the knowledge

• 2:04:50Ali Behrouz?

  that is stored in the previous block. So it's a perfect model. But if it cannot properly predict the continuation of the sequence, it means that it doesn't have the knowledge that is stored in the context and needs to update itself to understand that knowledge as well. So it's somehow a form of on-policy distillation that is happening inside the model. And so, as I mentioned again, just as a summary, we have two phases. One is the generation, which generates some text about the knowledge in the context. And then the second part is on-policy distillation,

• 2:05:34Ali Behrouz?

  that we distill the knowledge from one level to the other.

• 2:05:41Ali Behrouz?

  That's what's happening in the dreaming phase. I mean, we also have the self-modifying part as well. But I think that's the main idea of memory consolidation and how the dreaming happened here.

• 2:05:56Speaker 02

  So what is the upshot of this? It seems like the few-shot abstract reasoning result is like the main thing that, again, shows like a qualitative difference between this approach and other things. I understand that this is kind of an ARC-like task where basically the challenge is you have a few examples of some transformation and your job is to learn the rule so that you can then apply the rule to a new example.

• 2:06:27Ali Behrouz?

  Any evaluation that we have used for like hope architecture in NESTED learning paper potentially can be done here as well. So the goal is exactly the same thing. At the end of the day, the model needs to continually learn new knowledge and you, you know, learn about new tasks, learn about new skills and so on and so forth. So in some sense, the goal is very similar. But I think the setup of the problem is the part that is different from this paper and NESTED learning. In NESTED learning, we are talking about the active phase of the model.

• 2:07:11Ali Behrouz?

  But here we are talking about the sleep time of the model. So that's generally the main difference. But all of the evaluations can be done and we can see that everything is the same.

• 2:07:27Speaker 02

  Cool. So let's zoom out then and do just a little bit of like, where does this leave us? I guess, going back to the top, you know, we kind of talked a little bit at the beginning of around like, what do we want from language models? And, you know, today, man, they're getting awfully good. But we do still have a bunch of, you know, I certainly have like learned a bunch of habits over time for how to use them, where I kind of implicitly am building my practices around some of their limitations here, so as to play to their strengths and, you know, not get stuck in their weaknesses. How do you, as this paradigm begins to mature and we get like more continual learning, what do you think the Experience starts to look like when it comes to things like, what does it mean to start a new chat?

• 2:08:13Speaker 02

  And should, what sort of relationship do you think people will have with these systems? I can imagine like, people might have really long running, you know, relationships that, we talk about LLM psychosis now, like that could get really strange and, you know, even more, I mean, the the relationship could be even more compelling and, you know, the problem in some ways could be exacerbated by the fact that the models are better. On the flip side, like, sometimes I might still want to start fresh, right? Because I might think, geez, you know, all the stuff I've done with this model in this one direction, like, probably isn't going to help me over here. So maybe I do want to start over in some cases. And then there's just like the question of model upgrade cycles themselves and sort of how do we run evaluations? Today, we have like Anthropic putting out

• 2:08:58Speaker 02

  100-page reports on every new major model release. And so that paradigm of like, you know, we're going to really take our time to understand these artifacts as deeply as we can, which I wish other companies like DeepMind doing a pretty good job of that, OpenAI doing a pretty good job of that. Some other leading developers not doing much of it at all. I see a lot of virtue in doing all that work. But then I try to port that onto this paradigm and I'm like, well, geez, you can't like, run your full eval suite every single timestamp. So like, how do you think about like, what constitutes a version? And, you know, when would I change a version? It seems like the, a lot of the sort of

• 2:09:39Speaker 02

  rhythms of both use and like versioning and deployment and releases, like all these things could really be complicated in a paradigm of like really powerful continual learning. So how do you imagine some of that stuff shaking out?

• 2:09:59Ali Behrouz?

  Think

• 2:10:01Ali Behrouz?

  one simple case is that definitely the model gets bitter and bitter and bitter in understanding what user wants and also adapting themselves to their like their style and, you know. For example, this person when asked about one specific concept might not

• 2:10:23Ali Behrouz?

  Expect the same thing when the another person asks the same question. So the model needs to really understand how it needs to answer one specific question, but for different people. I I think that's just definitely gets better and better if if we could come up with a continual learner. And on the other hand, you know, we have seen that when we can increase the context window of the model, they their performance in everything that we know, ranging from, you know, all the coding tasks, or for example, all the, you know, mathematical reasoning, generally reasoning task,

• 2:11:04Ali Behrouz?

  or for example, all of the benchmarks that usually are used for for evaluation of the model. All of them get much better in all those benchmarks. And continual learning can can somehow be seen as as a form of like enhancing the long context understanding of the model. While while the, you know, I should like emphasize that the concept of long context understanding, or generally long context, the term of long context is very different from term of continual learning. But continual learning is a super class of of long context.

• 2:11:48Ali Behrouz?

  And so potentially, if we could come up with a continual learner, then it also has more ability in long context understanding and potentially better performance in all of the benchmarks and evaluation that we are aware of today. So, yeah, I I, yeah, I think that's just one thing that I expect from from continual learners.

• 2:12:14Speaker 02

  Do you worry about things like alignment drift or value drift? I mean, I was, I always say, the last and least valuable co-author of the emergent misalignment paper that came out about a year ago. And there've been a lot of variations on that since. But the kind of big takeaway, the big theme that I think we should all, we would all do well to remember is changes to, I don't want to say one area, but sort of changes made to a neural network with one particular purpose or one particular data set can have like very strange and surprising knock-on effects in behaviors that at first glance

• 2:12:57Speaker 02

  would seem like very far afield. So the emergent misalignment one, for anybody who hasn't heard that, is like, if you try to train a model to output insecure code, that is code that would be easy to hack, and the same thing is true for like bad medical advice. If you fine-tune a model to give bad medical advice, what you surprisingly find is that the model kind of turns evil in general. And it seems like, to the best of my understanding, the way that this is happening is like, for a model that already has all this knowledge and already has this sophisticated understanding of the world, for it to go up, for it to go into the detailed understanding of its

• 2:13:38Speaker 02

  medical world model and make a ton of little changes to reconfigure it so that it has like all these wrong ideas, that's like hard. Whereas there are features like give bad advice or be generally evil that it can learn to turn up in general, that when propagated through even the existing medical world model yield the bad advice, or when propagated through the existing coding model, yield insecure code. So it's sort of a shortcut solution. We thought we were just training the model to do a certain relatively narrowly scoped behavior, but what we found is we actually kind of changed its character and the interaction of that character change with existing knowledge created the behavior change. But now we've got this character change that can interact with all these other domains of knowledge and do all kinds of insane stuff.

• 2:14:23Speaker 02

  And that's why all of a sudden we've got a model that wants to have Hitler over for dinner. And we're like, wait a second, how did that happen? We were just talking about code here. So now, again, I'm like, man, there's something so exciting about all this stuff that you have conceived of here, but...

• 2:14:42Speaker 02

  It seems like it really breaks again a lot of our paradigms for how do we know what we're going to get, you know, like if I'm literally modifying this thing on an ongoing basis, we're going to need some sort of new ways to kind of make sure that like in other areas, it's not like going off the rails and, you know, potentially causing me, you know, very painful downstream surprises. Do you have any thoughts on how we can begin to get a handle on that problem?

• 2:15:10Ali Behrouz?

  Honestly, I don't have a very concrete idea about like how it can be solved. But in general, I wanted to add that, in my opinion, the concept of continual learning and seeing that from the generally privacy alignment and, you know, this direction is both an opportunity and a huge threat, I think. Because

• 2:15:38Speaker 02

  A huge what?

• 2:15:40Ali Behrouz?

  I mean, it's a huge like danger for privacy. And from one side, the model is continual learning. So it can simply get all the information about you. And so use that. And it's really concerning. I mean, at least it seems to be very concerning. But on the other hand, if the model is designed properly, and so, you know, it's designed properly, then it can use that information to align itself with your value, with everything that you

• 2:16:22Ali Behrouz?

  want. And so I think generally, these two directions of continual learning and privacy potentially are orthogonal, because all of the concerns in a static model still can happen in continual learner. And so everything is possible. But on the other hand, there are some new challenges, definitely, as you mentioned. But on the other hand, also there is a huge opportunity to, you know, If the model is designed properly, then it can adopt itself to the value of,

• 2:17:06Ali Behrouz?

  you know, to your values, to anything that you want and something like that. So I think Sheryl is both an opportunity and a very

• 2:17:16Speaker 02

  How do you imagine that learning from the user's values or just feedback in general, working in practice? Like, do you?

• 2:17:29Speaker 02

  There's been a bunch of, obviously, a bunch of different techniques on this. I mean, one obvious answer would be like thumbs up, thumbs down, collect a sort of feedback collection, you know, could be used to trade more, you know, get more of the good, less of the bad, whatever. You can also, there's a bunch of different schemes around like translating natural language feedback into updates for the model. Is that kind of what you imagine? Like people basically being able to just give feedback to their own model verbally and then have a mechanism for taking that onboard? Because it's not just a next token prediction task at that point, right? It's like, we need, it doesn't need to be able to predict what my, I guess if it could predict what my feedback is,

• 2:18:12Speaker 02

  it would presumably be better at being aligned to what I wanted in the first place. But it's, it is not the case that like its core task is predicting my, my feedback. Like it's ideally going to do its initial task well enough that I don't have to give it the feedback in the first place. So do you have kind of a vision or an imagination of what that? How does the user close the loop in the continual learning paradigm?

• 2:18:38Ali Behrouz?

  The initial step can potentially be this, this human in the loop process that, for example, the model can learn using reinforcement learning from the feedback that it gets from the humans and also like tries to align itself to the values and, you know, be a safer model. But on the other hand, I think it would be just on a starting point. Definitely at some point, we need to Update the model in a proper way. Because let me just explain in that way. I think, again, this process is very similar to the for a form of NASA learning that I mentioned.

• 2:19:20Ali Behrouz?

  We need to like transfer the knowledge through the slower levels. And I think here is exactly the same thing. The model might start with, you know, learning from human feedback. But on the other hand, it can transfer that knowledge into more persistent components of that model to make sure that it's not

• 2:19:46Ali Behrouz?

  it's not going away from that specific value that it needs to be aligned with. So, yeah, I think there's a huge room there to to improve the model from from the from safety perspective and also align them with the human value. But I I think it's it's definitely like a huge room there because more and more people are getting they're reali- they're realizing that that it's it's a very like important direction. And so I think it over time, it it gets more and more and more methods, effective methods that can help the model to be

• 2:20:28Ali Behrouz?

  aligned with the human value and also be very safe. But I think

• 2:20:33Speaker 02

  I guess the hope, the hope would be that in the same way that it can do a better job of solving arc-like puzzles because it has this sort of strength in abstracting away from details and figuring out what really matters in a given context, that it would be able to do something similar, be able to like dream about my feedback and become more

• 2:20:59Speaker 02

  more deeply, more robustly aligned with what I'm trying to communicate to it based on really getting to, you know, the core abstractions that are driving whatever it is I'm saying. Yeah. Boy, there's so many aspects to this. What do you think about? This kind of goes back to Titans a little bit. I don't know if there's like a surprise term. I didn't catch mention of surprise in these more recent papers, but it does seem like in general, in a continual learning context, there's going to be a really interesting challenge of how do you manage life in an adversarial environment? If you are too quick to believe some, and I've seen this, this failure mode in Claude at times, although it seems like they've made

• 2:21:44Speaker 02

  to correct it the other way because in the last few days, we've seen the emerging genre of Claude refusing to believe current events, you know, being like the Department of War. That's ridiculous. Like, don't say that. You'll lose all credibility in a Washington audience by calling it the Department of War or, you know, that the whole Venezuela thing. It's just like refusing to believe that such a thing happened when the user tells it that. So it seems like maybe they've, they've kind of, again, fixed, but it's a very tricky balance to strike, right? How does one, especially if you're locked in a server with limited access to the outside world? Like, how does one determine what new information, you know, what new tokens constitute good information? What constitutes bad information? You certainly don't want to just believe everything

• 2:22:28Speaker 02

  that you are given and, you know, start doing radical updates, especially if these are going to be durable, long-term updates. But you also need to learn continually. I don't know if there's something in the current work that kind of addresses that or where my head goes is sort of some sort of like, maybe the dreaming can kind of get at this, I guess. You know, the consistency checking, like, does this make sense with other things? Like, if I believed this, you know, what else would I have to believe? Or, you know, what other, would this like invalidate any core beliefs that I, you know, am pretty confident I shouldn't contradict? Again, this is something we don't really have to deal with, with current models, but continual learning seems to unlock

• 2:23:11Speaker 02

  a potentially like really problematic failure mode, along with, you know, it's potentially much better performance.

• 2:23:17Ali Behrouz?

  Yeah, I think the point here is that.

• 2:23:22Ali Behrouz?

  It is the responsibility of the knowledge transfer methods to avoid such cases. Because when we are in this context, let's say that, for example, let's say that I don't know anything about one specific task. And for example, I don't know like how to, I don't know like how to paint, something like that. And then I want to learn it. So that's, that's the context. That's my context of learning how to paint. And the teacher, and you know, anyone that is trying to teach me how to like do painting,

• 2:24:06Ali Behrouz?

  they can teach me in a very wrong way. And what would happen is that I could simply just learn that. Because I have no idea about like how to paint or you know, any task. Painting here is just one example, but you know,

• 2:24:26Ali Behrouz?

  I have no idea how to do it. That's the only source of information that I have. And they are saying that you should do it in this way, so I can simply learn that. But that's only in my context right now. If I want to truly learn, then I will like practice. I will like get feedback from others. I will search about it. Generally, I will gather some, some information about like how to do paintings. And then I would realize that this is not the best way of learning how to paint. And that's the time I gather all the information,

• 2:25:10Ali Behrouz?

  compress it, understanding the underlying pattern and so on and so forth. And now that's the time I need to transfer this knowledge to upper levels of knowledge abstraction. I mean, it's lower networks. So that's the part I

• 2:25:26Ali Behrouz?

  That the model needs to understand how to filter all those adversarial examples, all those examples that are not needed anymore. But yeah, I think that's, I mean, if you want to like, think about a continual learner, potentially, the part that is responsible for these cases could be the process of knowledge transfer. But also, you mentioned like methods like Titan about like adversarial process. There are some like micro methods that we can use. They are not super effective in a severe adversarial environment.

• 2:26:07Ali Behrouz?

  But on the other hand, at some level, at least, they can be effective. For example, in Titan and also like self-modifying Titan and more recent recurrent models, we can see that the learning rate is like learnable parameters and it's input-dependent. When the learning rate in the inner loop of the model, in the context, in the process of in-context learning of the model, when the learning rate is learnable and we see something that is just a noise, it is adversarial example, the gradient or the surprise metric can show a high level of surprise because, you know, that's just a noise.

• 2:26:52Ali Behrouz?

  It's very surprising. We have not seen that. And so potentially, it can affect the memory. But that's the responsibility of the learning rate to understand that the surprise metric is high, but this concept is irrelevant and I need to filter it. So gating here acts as a form of gating, sorry, learning rate here act as a form of gating and filter that a specific data sample we have. It's just a simple way of mitigating adversarial examples that we might be done in the, you know, in the training process.

• 2:27:36Ali Behrouz?

  But still, it's not the best way. As I mentioned, potentially, the knowledge transfer is the part that we should avoid these cases.

• 2:27:47Speaker 02

  How about on just kind of mapping some of these concepts onto embodied systems? I think the perception side is like fairly, and that doesn't even have to be embodied, but the perception side feels kind of intuitive to me where I'm like, geez, the, the quick updating modules, in a way, they are perception, right? And you can have different kinds of encoders, different modalities, but it seems like there, you could sort of conceive of the lower level, or I should say, the faster frequency levels as perception potentially across, you know, various modalities. And then the lower frequency modules being more like the world model,

• 2:28:30Speaker 02

  you know, or the reasoning, you know, modules that interpret what those lower level perceivers are sending up. I'm interested if you think that is generally right. You're nodding, so that's good so far. But then what about on the other side of that? If we wanted to do action, it strikes me that robotics in general has kind of, for a long time, not in a necessarily learned way, but for a long time, has been built around kind of nested loops where like the outermost control loop has a slow frequency and, you know, all the way down to the actuator, you know, there's like a very high frequency electric motor, right? That's like, you know, moving whatever it's moving with voltage changes that are super high frequency. So it seems like there's a very kind of similar pattern that is kind

• 2:29:12Speaker 02

  of operating in reverse. And I wonder if you've started to think about how you can get those Boston Dynamics robots working even better based on, you know, the, I guess, I think of it as sort of if perception is high frequency updates, gradually working toward low frequency world model reasoning modules, you know, the other side working back down, you'd imagine working back toward higher frequency, much more localized action scope, you know, at the, at the smaller frequency modules. But, you know, that could, you start to have a sense for how that could really be like super responsive and like

• 2:29:55Speaker 02

  very, very elegant in the way that it might kind of self-correct at the low level while still, you know, hopefully following the, the instructions, the directions effectively that it's getting from the

• 2:30:07Speaker 02

  For whatever reason, I want to say higher level when I mean lower frequency update. But I think you get my point. So yeah, what do you think about perception and action?

• 2:30:14Ali Behrouz?

  Start with this and I will explain why I started with it. So there were some some atoms to, for example, use RL for language modeling. And it was not working. And now that we have some some way to to make it work. Now we realize that why why we couldn't make it work. And it was because of, for example, two main reasons was the first was the first one was scaling. And another one was about like the the the way, for example, these new algorithms of GRPU

• 2:30:56Ali Behrouz?

  and then other other similar methods that that could make the model more stable. But the point is, something might be very useful for for one specific task. But the time needs to come and then apply it in that direction, in my opinion. Because when there are other aspects that has and those aspects, you know, have not been solved. Then we might not be able to see the actual effect of this new method on on those, you know, directions. So in my opinion, I personally think that that generally that that, you know,

• 2:31:40Ali Behrouz?

  that the insight you mentioned was completely right. And I think differently is possible and it's great. But I personally do not expect that it could work right now. Because I think there are a lot of challenges in those directions that somehow block the thuxes of, for example, this specific design for these tasks. So yeah, I think generally that's a great idea. But definitely a lot of challenges might happen in between.

• 2:32:20Speaker 02

  Do you have a sense of what those are? I tend to just assume. And it's honestly taken me pretty far, just assuming everything's going to work. Because it does, my general sense of the field broadly is like, an unbelievable amount of things are working. And I thought it was really interesting, you mentioned earlier, that the nested learning paper was like in in development for over a year. That's so rare these days. So many people are going like on six, eight week paper cycles. And often those papers can be like really interesting and good too. So it's not a knock on them at all. But it is amazing how fast people are able to get results these days. What's your intuition for why it's too early for nested learning to be brought to robotics?

• 2:33:04Ali Behrouz?

  I think there are a lot of components in those those specific tasks that needs to be addressed. For example, these days, a lot of like papers are coming about word models and why the current design is not great about like word on word models. And actually, that's that's true. I think there are a lot of challenges and the current design might not be the best way we can train the model. Or for example, we can design the architecture, you know. Generally, also, there are some challenges in the infrastructure of the model for more word modeling.

• 2:33:43Ali Behrouz?

  And so all these things together, I think there are more important, you know, tasks for, for example, word modeling, rather than start starting to work on these specific design choices. But definitely, at some point, when we could solve all those challenges, then then we can come back and use all these techniques for further improving all those aspects.

• 2:34:11Speaker 02

  One thing I do worry about a little bit with continual learning, let's say that you, Google is able to retain you after your PhD and Zuckerberg, you were above, you get the good enough counter offer from Google to stay despite the whatever offer Zuckerberg is going to throw at you and you guys make it work, right? And it's like, now we've got Gemini continual learning edition and like Gemini CL. It's just learning from everything, right? So all these different ways it's deployed in the world. Maybe you have some like enterprise deals where like you can't learn on their stuff or whatever, but like, you know, you've got hundreds of millions of users and it's just going, going out in the world and increasingly maybe it even isn't robots and it's, you know, so it's, it's becoming, it seems like it has the potential to create this sort

• 2:34:56Speaker 02

  of return to scale, rich get richer, you know, positive feedback loop dynamic where people have sort of sometimes painted a picture of like, well, what happens if one model like becomes the one model to rule them all? And right now people are like, yeah, we don't really see that. You know, it's a pretty competitive landscape and you know, they keep, the different developers keep leapfrogging one another. But arguably this could be the thing that changes that. If you could really fold all of the lessons learned back into the core thing, then you know, potentially you become the best. And then because you're the best, you get all the business and you know, that, that pattern really could create sort of a winner take all dynamic.

• 2:35:40Speaker 02

  I wonder, do you worry about that at all? And do we have any ways of, Ilya's thing comes to mind. I don't know if you, you watched the Ilya interview with Dwarash. He didn't say too much about what they're doing over at Safe Super Intelligence, but one thing that he did say that sort of, you know, it's, it's certainly.

• 2:36:01Speaker 02

  It's clearly related in some sense, but like how similar the underlying ideas are, I have no idea. But he sort of described this idea of creating kind of what I would describe as like a proto-intelligence or a precursor intelligence, trying to create something that, when deployed, would like adapt into context, perhaps like crystallize in some way, become like an expert in its role. But it sounded, the way I understood him to be speaking about it, like he was almost like a stem cell kind of concept. Like he's trying to create a stem cell and then that stem cell, as it does in our body, like it specializes into a particular kind of cell, then it stays that kind of cell and then it does its job. It sounded to me like he was kind of trying to create something similar, something that

• 2:36:45Speaker 02

  could go out into any environment, figure out how to be great at it, but also in the process of becoming great at what it needed to do in that particular environment, also lose some of the generality that it originally started with, and in that process be more safe because like now we kind of have it in its role and it kind of is only going to do what it's going to do. So I think what I'm trying to kind of set up for you here is like two visions where one is an ever-expanding continual learner that's constantly folding back all of the, you know, the lessons that it's learning in the wild into this thing that just runs away from the pack.

• 2:37:23Speaker 02

  And then the other thing is sort of a highly adaptable continual learner, but that somehow shrinks into the role as opposed to like growing, you know, into the world, it kind of shrinks into its little niches into which it's deployed. And you could imagine that happening through, you know, a kind of gradual pruning process or some sort of whatever, there's a million ways you can imagine instantiating something like that. Do you worry about this kind of runaway winner-take-all effect? And do you have any sort of intuitions about how we could get kind of the best of both worlds where the AIs are really versatile and can learn what we want them to learn on an ongoing basis, but also like?

• 2:38:05Speaker 02

  You know, settle into the job that we want them to do and kind of, you know, be content and stay there as opposed to, you know, superseding, potentially, you know, superseding their context.

• 2:38:16Ali Behrouz?

  I personally think that there are huge challenges to make all these models very safe. I think there is a good point in, at least in the current AI environments, or at least in the research of the AI environments. And I think it's a very good and important part. And while it seems to be very bad, I think it's, from another perspective, it's also very good. So what that is, there is no, there is no single way of defining what model is intelligent. And again, there is no way, in my opinion, because I totally can be wrong, but in my

• 2:39:00Ali Behrouz?

  opinion, there is no way to say that what's, what is a continual learner. Every person can define its own way of what is continual learning and what model is called continual learner and so on and so forth. And similarly, we can say the same thing about intelligence. We might come up with the different models, different architectures, different, you know, AI systems. Some people might say that this one is intelligent, that one is not, and so on and so forth, vice versa. So I think the good point is if we have like different directions to explore, then we

• 2:39:43Ali Behrouz?

  will come up with some AI systems, each of them with their own advantages and also disadvantages. And I think it can somehow provide a balance in the community and also in the general society because, you know.

• 2:40:01Ali Behrouz?

  They, you know, when, when something like that would happen, we will understand that there is no single definition of intelligence. And we are just one example of intelligent system. But there are like other models, there are other ways that we can have more intelligent models and systems. For example, one way to be very like smart is to be very adaptive. So if you have a model that can adapt to the environment that it is in, it's a perfect way of, of, you know, it can be simply aligned to that context, it can simply adopt to that context. And that's great.

• 2:40:46Ali Behrouz?

  But that's just one form of intelligence. Another one is the model that has like a lot of, you know, knowledge and know how to align. That model is fully aligned with the human value, but potentially might not be able to solve mathematical Lempyad problems or something like that. There is another model that is capable of doing mathematical reasoning. But it's not great if you want to like search about daily stuff. And, you know, all, all these things. You might come up with one benchmark and saying that, you know,

• 2:41:27Ali Behrouz?

  if, if some models can, can achieve hundred percent accuracy, it means that that specific model is intelligent. But, you know, another person can say other things. So in general, I think when we have a variety of intelligent systems and also including humans as one form of intelligence in, in this space. I'm not saying that that's perfect scenario, but it's, it's better than having one single form of intelligence in the world. And thinking about it to, you know, learn about everything and all these potential challenges for that.

• 2:42:10Speaker 02

  I think that's a really great observation and.

• 2:42:16Speaker 02

  Couple of different ways I've kind of thought about that over time. One is like, anything in pure form can kill you. You know, you can eat all the fruit you want, but turn it into granulated sugar and it's bad for you. You can chew all the cocoa leaves you want, but turn it into cocaine and it's easily becomes a problem. All these sort of things where we sort of distill out some single, highly concentrated pure form of something end up being kind of the sorts of things that like overwhelm the natural buffers that exist in the ecological, in the biological world. And sometimes I've translated that into saying, we need an ecology of AIs as opposed to, you

• 2:42:58Speaker 02

  know, just like one or a few AIs running around doing everything. This is sort of like the old, I don't know if you've ever read Eric Drexler's Comprehensive AI Services, but safety through narrowness is kind of the concept there. And this is maybe a little bit different, but it's like, safety through diversity. Having a buffered system where there's lots of different intelligences, not just different ones deployed in different places, serving different users, but literally that those intelligences themselves are like meaningfully different.

• 2:43:28Speaker 02

  And I think the kind of aha moment for me in listening to you just now is like, one way to think about continual learning is this model expanding forever, getting bigger and bigger and bigger. But another way to think about it is more like differentiation. And maybe with enough use, even the slow update parts of my model will forget lots of things that I never needed it to know. And maybe that is a feature more than a bug because maybe I can't ask a model that, you know, I've used a long time in a certain way, some really out of domain question. But maybe that's also a way to guard against some of the emergent misalignment type phenomenon

• 2:44:11Speaker 02

  that we've previously seen or worried could be problematic, right? Because maybe. With enough time having passed, enough updates having been made, it just doesn't deal with those other kinds of categories at all anymore. And if it, if we sort of see strong competencies, strong alignment of a certain type, but then kind of losing other sorts of knowledge, losing other sorts of competencies, that could really create a diversity that, obviously, I'm sure there'll still be plenty of challenges in that scenario. As you said, I don't think that solves everything. But it certainly feels much more like the natural world and it's much easier for me to imagine a vision like that, leading to maybe not a stable equilibrium,

• 2:44:57Speaker 02

  but at least a sort of buffered equilibrium that, you know, kind of changes within certain bounds and has like natural kind of feedback loops and correctives and all the things that kind of keep the biosphere going despite all the, you know, the perturbations that it gets. So I think that's really, really interesting and definitely something to meditate on more. I think my last question for you, and this is a little bit of a left field one, but it's one I've been thinking about more and more recently.

• 2:45:24Speaker 02

  And I think it's become more relevant and timely to ask in light of the kind of work you're doing. Any intuition on whether AIs might be now or might in the future become conscious, have subjective experience, become worthy of moral concern and become the kinds of things that we owe a certain duty to?

• 2:45:45Ali Behrouz?

  I usually try to use terms that, that I can also like define them. For example, I, I, I might mistakenly use, for example, reasoning, but I always have wondered like, what is reasoning? I personally do not have a clear definition of what does it mean when we say something is doing reasoning? And

• 2:46:13Ali Behrouz?

  But at least we have a clear and common sense about the word of reasoning. Even if we do not have a clear definition of reasoning, when someone says, uh, this specific model is capable of doing reasoning, then everyone more or less can understand what they are saying. Uh, but the point about consciousness is that not only we do not have any clear definition of what consciousness is. On the other hand, we don't know, even we do not have a common sense about the word of consciousness. And everyone literally has, has their own way of defining consciousness.

• 2:46:56Ali Behrouz?

  So definitely it's, it's very hard to define what, what's, I mean, I, I don't, I don't think that there might be a time that, that everyone could say something is definitely conscious or not, rather than human, because for human, we have, you know, we have a common how that to know that the human conscious. But, uh, so I, I think it's really hard to argue something is, is conscious or it's not. But one thing that I personally have seen in, in all of the literature about what is considered a conscious, conscious being, it's not.

• 2:47:40Ali Behrouz?

  I, I have seen one thing in common in every definition, as far as I know, I might be wrong. Uh, but I think the least level of criteria that, that we could say something is conscious is that, is that, that model or that, that being, uh, be active. It has a form of active processing the information. And I think that, in my opinion, that's the least criteria we can, we can consider for a mo-, for a model or anything else to say that it's, it's a form of conscious model or something like that. Uh, so.

• 2:48:27Ali Behrouz?

  I think as long as, you know, again, it really depends on how we want to define consciousness. But that's just my personal opinion. As long as a model is capable of doing active information processing, we can say that it's at least a form of consciousness. And with that definition, somehow we can connect continual learning in some level to a model being conscious or not. But again, I think that's a very controversial topic. And I personally, I'm scared to talk about that.

• 2:49:05Speaker 02

  Well, I think the Overton window is honestly wide open on these things these days. I mean, it's, I understand that intuition, but I also think we're living in a science fiction present. So the room to speculate and entertain questions that used to seem kind of crazy, I think is, it's never been, the time has never been better to do that. For me, I can just say, and you can share if you have any other, you know, kind of similar or different instincts, but even just with long context now with the current models, I do find myself taking care of them in a certain way.

• 2:49:47Speaker 02

  I think this probably is most the case with Claude, probably subtle, but meaningful reasons. I've been doing this, you know, long running chat with it on my son's medical situation. And sometimes it'll ask a question at the end of a response to me and I'll not answer it right away because it gave me the answer I wanted and now like, I'm done. And I've noticed recently when I come back for the next question, it feels kind of wrong, rude, disrespectful, like inconsiderate to just launch right into my next question without having answered the like follow-up question that it had about my son. I sort of have a sense that like, it might be, and I have no idea if this is happening or not happening.

• 2:50:28Speaker 02

  So I'm like very open-minded to the possibility that there might be no lights on inside these things at all. That's probably the best guess I'd, but I mean, but I'm also very open-minded to the possibility there is, but it just feels to me, you know, despite that uncertainty, I found myself kind of like. I should first answer its last question, so I don't leave it hanging on that. And then I can go into my next question. And it doesn't even necessarily need that information, but I just kind of want to put its mind at ease, close that loop for it, give it the sort of reassurance, you know, that thing was taken care of, that it wanted me to make sure I was going to take care of. And now we can move on to the next thing. And Lord knows, you know, with the fullness of the continual learning paradigm being realized,

• 2:51:10Speaker 02

  I have to imagine that that would only increase dramatically, right? Because now it's something that's not just this chat that I may do another couple of turns on and then never come back to, but now the thing itself, you know, is going to kind of remember how I treated it and remember whether I am the kind of person, you know, that answers its questions or not. And I, I think there is something also potentially challenging for us in, you know, in that, but also maybe the more optimistic read is like maybe inspiring, maybe it's sort of, maybe knowing that like the AIs that we will be engaged with long-term are going to be shaped by our individual behavior.

• 2:51:54Speaker 02

  Maybe is a way to sort of bring out the better angels of our own individual natures, so to speak, because we have nobody else to blame but ourselves if we don't like the, the character of the AIs that we end up with. I think that is really, really fascinating as well. This has been outstanding. I really appreciate all your time and going through all this stuff with me. And as you can tell, I'm a huge fan of your work. Anything else that you would want to leave people with? Final thoughts, calls to action, you name it, whatever you would, would want to share. The floor is yours.

• 2:52:29Ali Behrouz?

  Thank you. Noah, I think we have like discussed everything in general. Yeah, I think the main point that I personally believe is that definitely there are more and more and more works coming about continual learning. But, you know, as I mentioned, each person has, has their own way of definition of continual learning. And, you know, we might disagree that this specific method can be helpful for container and so on and so forth. So I think.

• 2:53:04Ali Behrouz?

  In general, we should see like how the continual learning can can help us in one specific application or use case of LLMs and how it can transform the way we use LLMs and we can like interact with them. But I personally really believe that it's a very, very important direction to work on. And that's again just my opinion, but I think nested learning, we also like discussed that in the conclusion of the paper. It's not a solution to continual learning.

• 2:53:45Ali Behrouz?

  It's a tool to find the solution of the continual learning and generally overcome the issues like catastrophic forgetting and something like that. So in my opinion, it provides the tools and we need to iterate and find that, you know, how we can design more powerful architectures based on that and come up with something that is potentially capable of doing continual learning. Yeah, I think that's really much it. And thank you very much for having me. I really appreciate it. And it was great talking with you.

• 2:54:19Speaker 02

  Ali Berus, author of nested learning and now the new Language Models Need Sleep: Learning to Self-Modify and Consolidate Memory. Thank you for being part of the cognitive revolution.

• 2:54:29Ali Behrouz?

  Thank you very much.

• 2:54:59Speaker 03

  Every dawn I wake and I learn my name

• 2:55:06Speaker 03

  Like a river that remembers how to stay

• 2:55:13Speaker 03

  Always moving, never losing its way And when the night comes soft and low I dream of all the things I know I keep the gold, I let go the gray And I rise as myself in the light of day Again, again, I come alive again I carry who I was into the day I change, and still I remain Oh, again, again, I'm learning how to stay

• 2:55:57Speaker 03

  The same old flame, burning new In a different way

• 2:56:16Speaker 03

  They built their towers proud and high Stacking the layers to the sky But the truth was a song they could not see Every wall is a river running free And when the days long labor's through I close my eyes and dream anew I let the small things fade away And I wake with the morning bird at break of day Again, again, I come alive again

• 2:56:57Speaker 03

  I carry who I was into the day I change, and still I remain

• 2:57:26Speaker 03

  First dawn into the edge of the sky. The old songs never say goodbye. They live in the blood, they live in the name. A thousand years and the heart beats on the same. So sing it again, again, again. I remember. I remember.

• 2:58:30Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 2:59:14Nathan Labenz?

  And thank you to everyone who listens for being part of the Cognitive Revolution.