⚡️ Beyond Transformers with Power Retention

Watch the full show on YouTube: https://www.youtube.com/watch?v=QLFviB2cKtU

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Alessio Fanelli?

  Hey everyone, welcome to the Lit in Space Podcast. This is Alessio, founder of Kernel Labs, and I'm joined by Jacob Buckman of Manifest AI. Welcome.

• 0:11Speaker 01

  Thanks for having me, Alessio.

• 0:12Alessio Fanelli?

  Of course, you know, we at Decibole, we invested in Manifest right at the start. That's about two and a half years ago. You had just left Google Brain, Carlos had just left OpenAI, and I think you set out on this mission of unlimited context window for models. And you know, to set the context, when you started, the context window of these models was like 8,000 tokens. You know, so it was very, very limited. So maybe just do a quick background on like the origin of the company, you know, some of the beliefs behind it. And also you've done a great job of kind of like staying focused on your beliefs instead of just chasing what the mainstream discussion is. So yeah, just maybe you give people a quick history of Manifest.

• 0:54Speaker 01

  Yeah, absolutely. Manifest came from a collaboration between me and my co-founder Carlos, basically thinking about what the real breakthroughs to get to the next sort of phase of this technology would be. So sometime in, you know, early 2020, 2021, the writing was sort of on the wall for people who were working in the space that by just basically scaling up what we had, we could unlock some really crazy potential. And the question was, how much potential? Can it take us all the way to, you know, AGI, like proper human level intelligence on every task? And the conclusion we came to was not quite. We have a lot of pieces in place that could get us there, but there is one technical bottleneck that we felt was actually going to be a blocker. And it is basically the problem of large inputs.

• 1:39Speaker 01

  And in the sort of like frame of language models, large inputs means long contexts. It's a question of how much content can be synthesized all at once by the model. And we know how to scale up the models to make them smarter by increasing the parameters. And we can do that very efficiently and at scale in large clusters. And also we know how to sort of scale up the total amount of knowledge ingested by the model, training them on larger and larger data sets.

• 2:04Speaker 01

  We do not have good, like, cost-efficient ways to scale up the context. And this is something that might be surprising, given how far everybody has pushed their context lengths in recent years. But actually, what a lot of people are doing is not solving the core, underlying problem behind the very expensive cost of long context. It's the fact that the KV cache is always growing. That's sort of the fundamental bottleneck, is that every single token that you see, you have to remember in the KV cache. And this makes every subsequent computation, like every additional token that you predict, get more and more expensive. And this growing cost just quickly becomes the bottleneck of the whole thing, and it winds up intractable.

• 2:44Speaker 01

  And rather than sort of like solve this at its core and rethink attention, people instead thought up various, quite useful, but ultimately workarounds, right? Band-aid solutions, things like windowed attention. So this is very common. Anybody who says they're using a transformer with a context length of, you know, 256,000 or more, they're not using a true transformer. What they're using is a windowed transformer that essentially throws out a huge amount of its information at various layers within the network. And this does kind of work, but it doesn't really work. We've tested these models extensively, and they're actually just much worse at processing information from further and further back in the past.

• 3:25Speaker 01

  And this is why you see the funny phenomena of long-context models sometimes being really good at using things at the very beginning of their context, and really good at using things at the very end of the context, but sort of messing up and getting fuzzy on everything in the middle. A lot of this is just purely architectural. These architectures just aren't designed to do uniform computation across the whole context length, and so they end up with these weird idiosyncrasies once you train them. And a lot of this, we had manifest only realized along the journey, you know? It's not like we had the full picture laid out at the beginning, but we did recognize the importance and sort of like the fundamentalness of scaling up the input as a problem. And so we founded the company to essentially tackle this at the deepest level. And what we ended up coming up with is a different family of architectures, replacing

• 4:10Speaker 01

  transformers entirely, ended up needing to be the way to go. We call it retention-based architectures. Power retention is the specific variant that we're about to release. And it basically works by instead of the memory constantly growing, this constantly growing KV cache, you have a memory that is a fixed size. And each new token simply gets compressed into this fixed-sized memory. So it doesn't grow. But crucially, the memory itself can be scaled up or down based on the difficulty of the problem and the compute budget. So in this regard, our architecture has a state, like a memory, that is very similar to how we oftentimes think about parameters in other models. Just like for a really big training run, you want to use a ton of parameters.

• 4:55Speaker 01

  Similarly, for a really big training run, you want to use a giant memory that's going to help you. But if you're very inference latency sensitive, for example, if you're going to be using a model to sample many, many tokens and you want to get the speed up and the price down, then you don't want to be using a large number of parameters. And similarly, you don't want to be using a large memory. And you certainly don't want to be using a constantly growing memory that makes each token more and more expensive. So this is sort of the space of ideas where power retention fits in. And yeah, I'm excited to see what people do with it.

• 5:26Alessio Fanelli?

  Yeah, that's awesome. Are there any fundamental differences? So you mentioned the KV cache, right? The KV cache does allow inference providers to offer cash prices on tokens. So if you kind of keep the prefix the same, then you only get incremental costs on the new tokens added. Are there big differences? So now you run inference with a power retention model versus a normal attention model? Or is it going to be for most, you know, for most models, like a pretty easy switch?

• 5:55Speaker 01

  It's actually going to be way cleaner on the engineering side to do inference with power retention and similar models. The KV cache, in addition to the fact that it's big, one of the real challenges, just infrastructurally, is that the size is dynamic, right? The more a user is querying the model or the more tokens they're querying it for, the larger the KV cache gets. And so allocating a GPU to multiple users becomes very tricky because each user has this constantly growing When you want to like remove a user from that GPU, there's a whole scheduling problem to decide which new users get their caches loaded in. You want to pick ones that fit, but not that are too close to the limit, because then you're going to have to rotate them out pretty soon, but not ones that are too far from the limit, because having a ton of extra memory is very wasteful.

• 6:40Speaker 01

  It's just a really challenging engineering problem to efficiently use a whole fleet of GPUs to serve inference for many users when you have these dynamically growing KV caches. But with power retention, this completely goes away. The retention architecture just comes with this fixed size memory. And so every GPU can just be partitioned into, say, eight pieces. This GPU has enough memory to serve eight users, and that's it. They never grow, they never shrink, you're never wasteful. When you have seven users, you have one extra slot. And when the eighth user joins, you put them there, no matter who they are, no matter what they're querying, no matter how long they've been using it before. It just doesn't matter.

• 7:21Speaker 01

  So it actually is going to meaningfully reduce inference-like complexity as well as costs. And just to give you some sense of how much, at training time, the sort of speedup that we get purely from the flop reductions and efficient hardware-level implementation of flash, of power retention over flash attention at a context like 64K tokens. Obviously, our speedups actually grow the longer the context is. And at 64K tokens, we get something like a 10x speedup at training. But at inference time, because you're not only saving flops at inference time, but also paging in and out of memory of the KV cache, you actually get 100x speedups from power retention over like flash infer at context lengths of around 64K, and only growing from there.

• 8:05Speaker 01

  So these are real material differences. It's going to be a massive, massive reduction in inference costs and complexity as a result.

• 8:14Alessio Fanelli?

  Yeah, and so you're also releasing, just to go from the bottom up, VDrill, which is a framework for CUDA kernels. So I'm assuming you're gonna release all the kernels for like power retention. And then I also know you rerouted the flash attention to kernels and actually faster than the ones that three DAW originally implemented. So talk about that level of the stack.

• 8:38Speaker 01

  Yeah, so the Power Retention Architecture is very flop efficient. But this is not actually useful if it's not hardware efficient too, right? If the GPU is getting wildly underutilized, then it doesn't matter that you're doing many fewer computations. It's going to take longer and therefore ultimately be more expensive, right? So one of the key goals we had at Manifest was to release an architecture that isn't just theoretically faster when you count the flops, but is actually practically faster in terms of wall clock time. And that required a huge investment into getting really, really good at writing ultra-efficient, hardware-friendly kernels. And Power Retention is a very tricky thing to get right. It's actually, in many ways, significantly more involved

• 9:22Speaker 01

  than Flash Attention or a lot of the other kernels that people like to write. And so we actually couldn't use something like Triton or Palace. That's one of these more higher-level Pythonic frameworks. Actually, our initial version of Power Retention was written in Triton, but we realized quickly that it just didn't offer the flexibility to really squeeze the performance that we wanted out of the GPU. So we took a step back and dove into CUDA. And before we knew it, in order to write the kernels that we wanted to write for Power Retention, we actually had to write an entire, fully-general framework for writing CUDA kernels. And that's what we call vitriol. And sort of the key intuition behind vitriol is the idea of just-in-time sweeping.

• 10:04Speaker 01

  So when you're designing a kernel to do a particular operation, mathematically, it's going to do one thing, right? You always want it to produce the same output. But computationally, there's a huge number of ways for most kernels to actually implement. The GPU has all sorts of different hardware features and the choice of basically like which cores to use to copy data, where to copy it to, when to send it to the tensor cores, all sorts of things like tile shapes and pipelining that really can completely change the performance of the actual kernel, even if the outputs are always the same. And for most kernels, most CUDA kernels, you basically, as a kernel designer, just figure it out. You just think really hard and try a lot of stuff. And you write a config that is, you write a kernel that is

• 10:49Speaker 01

  Configured to be, in general, making the right decisions. And then you release it and everyone uses it and gets pretty good performance and is happy. But we realized that this can be done much better. Because on any particular user's hardware, any particular problem shape, you know, like the shape of the input, the memory layout, et cetera, there is an optimal configuration. And it might not be one that the kernel designer thought of. But rather than having the current, putting it all on the kernel designer, we could just write the entire space of possible ways to execute any given mathematical operation on a GPU and let the user sweep over all the different ways to compile this kernel and empirically decide which is the best one. And this is a little expensive. It takes a few minutes.

• 11:31Speaker 01

  But ultimately, if it takes a few minutes to shave off 20% a sort of like call to this kernel on a kernel that's going to be called billions of times on a huge cluster over a three-week training run, then it is unbelievably worth it, right? And so Vidrial is all about writing the most generic possible kernel. Essentially, you're never just writing one kernel. You're writing every possible kernel that could implement this operation. And when you're actually executing it, you're sweeping over all the different ones, measuring the time for each one, and always choosing the best. And the result of this is that if you take something like flash attention and implement it in Vidrial, then on the problems where flash attention is really well optimized, we end up basically matching performance, 1% to 2% faster sometimes.

• 12:16Speaker 01

  But on the problems where flash attention is not well optimized, which is a huge number of problem shapes, just because there's so many possible problems, we can get 20 or 30% faster, literally just by making really intelligent decisions about how to use the GPU hardware by empirically testing different ways of setting things up.

• 12:34Alessio Fanelli?

  And the inputs to that are the model plus the GPU. What are like, like, you know, once you do this, does the same sweep kernel work for most people? Or is it very specific to like the number of GPUs, the size of the cluster, the type of run that you're doing, blah, blah, blah?

• 12:50Speaker 01

  Yeah, we find it's pretty robust to different setups. So yeah, we're also like releasing some pre-compiled, pre-optimized kernels. So if you're doing something very normal, right? Like if you're doing flash attention with a batch size of eight and a head dimension of 64 and eight heads on a sequence length of 1,024. Like this is a totally bog standard thing. If you just run nano GPT out of the box, you're going to be calling the kernel with this exact sort of input shape. And if you're doing that, you won't have to sweep yourself. We'll be releasing a pre-swept kernel that'll do it for you. But if you're straying from that and you want to use a sequence length of like 1,243 for some reason. Now, we're going to be way faster than flash attention

• 13:33Speaker 01

  because we're actually going to find the right way to configure the kernel to call it on that specific input shape.

• 13:39Alessio Fanelli?

  Great. And so people now can just do pip install retention, basically. And that's it. It's a pretty, pretty plug and play.

• 13:48Speaker 01

  Yeah, it's completely drop-in replacement for flash attention. So yeah, from the retention package, you can access our flash attention kernel written in VIGRIL. And you can also access power retention, which is the subquadratic architecture that I was talking about earlier, which not only is extremely hardware efficient because it's written in VIGRIL, but also has these huge flop savings because it's genuinely compressing the memory instead of memorizing every single thing that is seen. Import and a one-line change and it'll... You're off to the races. You'll see huge speedups if you're doing any sort of training or inference on long contexts.

• 14:27Alessio Fanelli?

  Yeah, and you know, the other thing is this is not a change that requires pre-training. So you can take a pre-trained model with some architecture and use this to kind of like, like you called the metamarcosis, turn it into like the new power retention architecture, basically.

• 14:44Speaker 01

  Yeah, so it's not something that requires...

• 14:48Speaker 01

  Pre-training at like a real pre-training scale. It does typically require a little bit of what people are now calling mid-training, basically fine-tuning, but on all the parameters, right? It doesn't require very much, though, which is why this process of metamorphosis, right? You start as a slow crawling transformer caterpillar, and you end up as a beautiful lightning-fast-on-long-context power-retention butterfly. And it actually, I can actually show you some training curves if you're interested. Takes surprisingly little compute there. So, yeah, maybe I can share my screen here. So, yeah, this.

• 15:25Speaker 01

  There we go. So here's an example. This is our in-house sort of run visualization platform. I call it Log Cabin. And this is basically showing, this blue line here is a model called StarCoder 3B. This was released by the Big Code Project, sort of hugging Face-affiliated, small-ish, but real meaningful-scale, 3 billion parameter coding model. And this is the baseline level of loss that it achieves on like a very standard, they call it the STACK dataset, and specialized to Python. And what happens when you switch it from flash attention to power retention? Well, you get from this blue loss right up to this orange loss. You take a big hit, like immediately the model is just not making the right predictions basically anywhere.

• 16:10Speaker 01

  But after just 10,000 steps of training, which this training run took about two hours, this orange curve, you see that it fully matches the original loss. And from there, you can start pushing the context length. So we did more training at 4K and then even more training at 6K context. And you can see that you can really keep pushing the loss down. And not only that, but the long-context training will actually be fast, where if you tried to train base StarCoder on 16, it would take way, way longer to do this. And here we have downstream evaluation. Maybe I can zoom in here a little bit?

• 16:44Alessio Fanelli?

  Yeah.

• 16:45Speaker 01

  And?

• 16:46Alessio Fanelli?

  And the base model is 8,000, right? 8,192 tokens of context window?

• 16:53Speaker 01

  I think it's just 4,096, actually.

• 16:56Alessio Fanelli?

  4,000, okay, yep.

• 16:57Speaker 01

  But they did a little bit of longer context post-training, but using windowed attention. So they didn't actually extend the attention to 6K. They kept it at 4K, but just used the sliding window approach to train on longer sequences. And yeah, so this is this is a downstream evaluation. This is the human evaluation, like human eval, negligence. And basically corresponds to the performance. So we're basically perfectly able to match after just two hours of training, the downstream performance on these tasks. And then continued training allows it to get even better. In the end, this converges to, I believe, about 35% accuracy on human eval, whereas the star coder baseline was about 30%.

• 17:40Speaker 01

  And yeah, so this is just like very manageable, very achievable for a lot of different groups. And when you get out at the end is this metamorphosized model that is just as good as star coder, but with power retention. So we call it power coder. And it has way cheaper inference costs as well at long context.

• 18:00Alessio Fanelli?

  What's the cap? So you have 16K context. Could it go to 32, 64K? Like, how much can you expand the context size?

• 18:11Speaker 01

  We actually have not found a limit. It just comes down to resources, like each subsequent training costs more than the previous. So this costs about two hours to train at 1K. And then this represents another about six hours of training. And then this represents another 16 hours of training. So it does get like, you know, you're training on more tokens, right? So it gets more expensive as you train along context, just because you're you're training on more tokens. So yeah, but we don't pay a quadratic cost. If you were looking at the star coder baseline, it would get even more, more expensive way more quickly. So it's actually the 16K runs become very tractable for a lot of groups. But yeah, we could push it to 32K. We could push it to 64K. We could probably push it beyond that, although that would then require doing some like tensor

• 18:56Speaker 01

  parallel or sequence parallel. So that just would be a little more implementation effort that we haven't done yet would be very doable.

• 19:03Alessio Fanelli?

  And you ran this on SF compute, I saw on your.

• 19:08Alessio Fanelli?

  On your announcement.

• 19:09Speaker 01

  They're they're one of the main compute providers we use for the research. And this actually one actually was not trained on that cluster, but yeah, they're they're great.

• 19:17Alessio Fanelli?

  Nice. Yeah, we had them on on the pod before, so I was trying to shout out previous guests. So you now have also built a power coder like, I mean, obviously you've shown the the loss, the loss curves, but you're like actually releasing the power code Python model, right?

• 19:35Speaker 01

  Yeah, I can show you a demo of that to have it over here. Actually, before I do that, I just wanted to show off real quick the difference in training speeds. So this is two runs. These are both at 32K context. And the only difference between these two is this is the star coder to baseline at 32K. And this is the power coder at 32K as well. And so I'm just going to kick these both off, really give this guy a little bit of a head start. And we can come back in a bit and see how much progress each of them has made. But yeah, so here is the power coder model in action. We decided to go for a slightly different interface. Everyone does a little like, you know, chatbot, whatever. So we want to do something a little bit different.

• 20:18Speaker 01

  And we made a tool called manifesto where basically you can point it at any repo, including a very large repo, and it will basically fix any mistake in that repo or do its best to, of course. It's only a 3 billion parameter model, so you know, it has its it has its limitations, but it can do some very cool stuff. So here I just set up that this is like the nano GPT repo. And we can just go look at, you know, like the model dot py file, for example. Right? This is like completely the standard model dot py. And you can see there's there's lots of other files in here, too, just like regular nano GPT setup.

• 20:58Alessio Fanelli?

  Can you make it slightly bigger, just so it's easier?

• 21:00Speaker 01

  For sure.

• 21:02Alessio Fanelli?

  Just zoom in a little bit. Yeah, that's good.

• 21:04Speaker 01

  There we go. Okay. So yeah, let me show that again. We just have like, I guess you can see it here, right? Now, GPT is just like with normal training files in it. And now if we go into model dot py, you can see it's just like a very ordinary

• 21:18Speaker 01

  carpathy style setup. And so let's say we just like deleted this class.

• 21:26Speaker 01

  And replace it with this comment and then maybe we can do one more here. So now the just not unimplemented functions. And now we can basically see what this does. And it will basically just take a look at the repo, read all the code in the repo and propose some repairs. So you see here model.py, the one we messed up and it starts repairing layer norm. And yep, it's basically just giving this seems like a valid layer norm class to me. Honestly, this probably like nano GPT probably appeared in its training corpus.

• 22:06Alessio Fanelli?

  Right.

• 22:08Speaker 01

  Yeah, but yeah, this is the idea and it's generally pretty good on Python repos. The distribution was weighted towards repos that have a majority of files ending in dot py. So it is mostly good at that. And again, that was just like a computational sort of efficiency thing. We just decided to target something smaller still because we're a pretty small lab and and a little resource constraint. But yeah, and look, it also made one more change. It added a logger too. So that and these will be released on hook is for people to play with along with the command line tool.

• 22:43Alessio Fanelli?

  Do you see more of these models being built for different languages? Do you think that's like a now that you have this new architectures, like is that a good direction for people to work on? Do power coder, TypeScript, power coder, Rust, blah, blah, blah.

• 22:57Speaker 01

  Yeah, I mean, ultimately I would just like a power coder, everything, right? Like you should just be able to train one sort of foundation scale model that does it all. And we are building up to that. That's something we plan on doing at the like 30 billion scale in the coming months. This is sort of just like a first demo, you know, just a taste of what's to come. And something that I'm really excited to see is since we're going to be completely open sourcing all of the pieces that you need to do this metamorphosis yourself. I think there's going to be a lot of groups who have sort of different needs and they want to metamorphose the different base models. Maybe someone wants to do Quen coder instead of star coder, or someone just wants to do a text model or a question answering model or an instruct tune model instead of a code model.

• 23:40Speaker 01

  And we'll release all the tools needed for them to do that. So they can just get a handful of GPUs.

• 23:48Speaker 01

  Basically, Pivensol retention to get our kernels. Modify the architecture. We'll be releasing guides and tutorials on exactly what the modifications need to look like, but they're honestly very, very straightforward. And they'll be off. They'll come out with a nice, shiny, new power retention architecture that has the same performance on whatever dataset they want as the original base model did. And my hope is that people will start to open source these. So just like right now, there's a large ecosystem of open source transformers on Hugging Face and other places that lots of people use and benefit from. I'd really like to see the same thing happen, like each of these models be transformed into a retention variant and exist alongside the original. And I think this would be really

• 24:30Speaker 01

  great, especially as people start thinking about things like a chain of thought where you're sampling tons and tons of tokens at inference time and inference costs really start to add up. I would really love to see people basically switch over to retention for those use cases because you're going to be able to save an enormous amount of both money and time on those long context inference use cases.

• 24:53Alessio Fanelli?

  So if you are Together or Fireworks or any of these inference providers, should you just be doing this for every single model? Like, what are like maybe the downsides? Otherwise, it just sounds like it's... Does it make sense to not do this?

• 25:09Speaker 01

  No. I think it only makes sense to do this. And yeah, I love the guys at Together. I think they're really great and very like open source friendly as well, which I really appreciate. So they're one of the people I think I would love to partner with. You know, I think they stand to benefit a lot basically from, you know, like, once a model is in wide distribution enough in the open source space that people like trust it and like it, then it becomes appealing for them to serve. And so I think they're a very natural partner here. We could work together to train some of these big power retention metamorphosis, metamorphoses, that they can then serve to their customers at

• 25:51Speaker 01

  much lower prices and much faster latencies than any attention model at long contexts.

• 25:57Alessio Fanelli?

  Cool. I know some of them are listening, so maybe they'll reach out.

• 26:03Alessio Fanelli?

  What else is on your roadmap, I guess? You know, so you started with this idea of like, you know, longer context should not come with a quadratic, kind of like a penalty, so to speak. Do you feel that from here, you know what's next to like start scaling this? You know, like what, how far do you think you are from being able to do GPT-5 on power attention, you know, kind of like when the frontier models do that? Is it a matter of having access? Is it a matter of, I know, obviously, some of them are like MOE models or not necessarily like dense transformers? Yeah, what, what's kind of coming up for you?

• 26:39Speaker 01

  Yeah, I think ultimately it's a matter of building community trust around this architecture. And this is something that is really important to us. And one of the main reasons for open sourcing as well is because we've run tons of experiments on this architecture. We've seen what it can do, but we're just a small lab. We need everyone to see what it can do. And that means just putting it in people's hands for them to play with it themselves. If you think about the history of the transformer itself, right? The transformer didn't drop and then immediately get scaled up to, you know, like the 700 billion parameter scale. Instead, it was something that the whole community used, played with. There was this gradual realization of how strong it was on lots of different tasks. There's things like flash attention from TreeDAO, which produced these massive, massive

• 27:23Speaker 01

  improvements in performance. And it was just basically like a big community effort for people to realize this is the architecture we want to be training at scale. And honestly, I wouldn't expect the big foundation model companies to want to overnight drop everything they're doing and pick up power or attention. That wouldn't, that wouldn't make sense. But I do think the momentum is, is, has already started to build. A lot of people are interested. And over the next couple of months, maybe year, I think we'll start to see some really big, successful models trained in this way. And, um, you know, there is admittedly a lot of, I guess I would call it institutional momentum at places that have invested massively in training a particular architecture, like the transformer at massive scale investment in terms of engineering resources, scientific

• 28:08Speaker 01

  effort, like plotting scaling laws, even just pre-trained weights that they already are confident in. And I do think the burden of proof is high in order to shift that equilibrium. But I think it's going to happen. And so, yeah, the first step is for everyone to just like do these metamorphoses, play with the resulting models and let everyone see how powerful it is. And after that, I think, you know, probably within six months to a year, we're going to start to see the really big foundation models being trained in this way.

• 28:37Alessio Fanelli?

  Awesome. Anything else we missed? Anything you want to shut out?

• 28:42Speaker 01

  So, yeah, the training speed difference, you can see, I don't know, maybe it's a little too small here, but it's about like 4.8 seconds per iteration for power retention at 32K. This is the 3 billion parameter star coder model, and it's about 23 seconds per iteration. So this is going to be about a X speed up at 32K. And you can see it's like, you know, way, way more advanced. This one's at step 20. This one's at step 90. And yeah, so there you go. That's like the, the training speed difference. And as I mentioned, the inference speed difference, I don't have a demo of it here, but it's even more massive. It's like another like X faster at inference relative speed.

• 29:23Alessio Fanelli?

  Nice. So if you're a inference provider that wants to metamorphosis their model, please reach out. Are you bringing on anybody else on the team? Or like any request for collaboration that you have?

• 29:35Speaker 01

  Yeah, so the team is right now small but mighty. We're definitely looking to add anyone who's interested in any of the stuff we've talked about today. Architecture research, pushing the limits of long context, making foundational architectural breakthroughs instead of just, you know, tweaking what already exists. If you are excited by those ideas, especially of a strong background in deep learning, CUDA programming, or anything mathematically adjacent to those ideas, please reach out. We'd love to talk to you. Also, another big thing that we're looking for is the right data sets. So this is actually maybe an interesting tangent, but the value of long context is deeply and intrinsically tied to the structure of the data.

• 30:19Speaker 01

  It's really easy to construct really long context data sets where the context is useless. And actually, in some ways, this is how most pre-training data sets are constructed today. If you take like any old, like the pile or one of the more modern ones, basically any common crawl derived scrape, open web text, any of these things. You're going to find that almost all documents are very short. There's some long context documents, but the vast majority of them, I think last time I measured it, it was something like 90% are of context length 2K or below. So if you're doing really long context training runs on that data, typically how people do it, they call it packing, right? They just sort of like stitch together different short documents until you get that long sequence.

• 31:03Speaker 01

  And sometimes you get a single document subsequence. So you do learn some long context stuff. But by and large, most of your long context compute is in some sense wasted because there is no long context structure in the data. And this just seems to be a property of internet text. In general, most internet text has mostly short-term structure. There's just not that much value in capturing long-term structure. And so compute-optimal models on internet text actually don't have that long contexts. And so of course, you're perfectly fine using transformers for that sort of situation. But internet text is not the only dataset that there is, although it is probably the most popular and most accessible. And in addition to internet text and code and all the other things that people like to use,

• 31:48Speaker 01

  we'd like to look for other datasets. So if you're somebody who has an interesting dataset and a unique long-context dataset, where you feel like the context actually matters, it's not just a matter of thinking more about a short-context window to make the prediction, but you actually need to incorporate information from very far in the past, I would love to talk to you as well, looking for the most exciting datasets out there to really showcase the power of this long-context unlock. In particular, anything involving human trajectories over really long periods of time. For example, a human completing some task, be it an administrative task or a coding task, a computer use task. Those sorts of trajectories we think are really, really exciting to work with.

• 32:31Speaker 01

  And if you are somebody who is in possession of a lot of them and wants to make the most of them, we'd love to collaborate.

• 32:36Alessio Fanelli?

  Awesome, man. Thanks so much for taking the time. I will put the links to the model, Pedro, Power Attention and all of that.

• 32:44Speaker 01

  Yeah.

• 32:45Alessio Fanelli?

  Thank you.