Transformers Need Glasses! - Federico Barbero

Federico Barbero (DeepMind/Oxford) is the lead author of "Transformers Need Glasses!". Have you ever wondered why LLMs struggle with seemingly simple tasks like counting or copying long strings of text? We break down the theoretical reasons behind these failures, revealing architectural bottlenecks and the challenges of maintaining information fidelity across extended contexts. Federico explains how these issues are rooted in the transformer's design, drawing parallels to over-squashing in graph neural networks and detailing how the softmax function limits sharp decision-making. But it's not all bad news! Discover practical "glasses" that can help transformers see more clearly, from simple input modifications to architectural tweaks. SPONSOR MESSAGES: *** CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting! https://centml.ai/pricing/ Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/ *** https://federicobarbero.com/ TRANSCRIPT + RESEARCH: https://www.dropbox.com/s/h7ys83ztwktqjje/Federico.pdf?dl=0 TOC: 1. Transformer Limitations: Token Detection & Representation [00:00:00] 1.1 Transformers fail at single token detection [00:02:45] 1.2 Representation collapse in transformers [00:03:21] 1.3 Experiment: LLMs fail at copying last tokens [00:18:00] 1.4 Attention sharpness limitations in transformers 2. Transformer Limitations: Information Flow & Quantization [00:18:50] 2.1 Unidirectional information mixing [00:18:50] 2.2 Unidirectional information flow towards sequence beginning in transformers [00:21:50] 2.3 Diagonal attention heads as expensive no-ops in LAMA/Gemma [00:27:14] 2.4 Sequence entropy affects transformer model distinguishability [00:30:36] 2.5 Quantization limitations lead to information loss & representational collapse [00:38:34] 2.6 LLMs use subitizing as opposed to counting algorithms 3. Transformers and the Nature of Reasoning [00:40:30] 3.1 Turing completeness conditions in transformers [00:43:23] 3.2 Transformers struggle with sequential tasks [00:45:50] 3.3 Windowed attention as solution to information compression [00:51:04] 3.4 Chess engines: mechanical computation vs creative reasoning [01:00:35] 3.5 Epistemic foraging introduced REFS: [00:01:05] Transformers Need Glasses!, Barbero et al. https://proceedings.neurips.cc/paper_files/paper/2024/file/b1d35561c4a4a0e0b6012b2af531e149-Paper-Conference.pdf [00:05:30] Softmax is Not Enough, Veličković et al. https://arxiv.org/abs/2410.01104 [00:11:30] Adv Alg Lecture 15, Chawla https://pages.cs.wisc.edu/~shuchi/courses/787-F09/scribe-notes/lec15.pdf [00:15:05] Graph Attention Networks, Veličković https://arxiv.org/abs/1710.10903 [00:19:15] Extract Training Data, Carlini et al. https://arxiv.org/pdf/2311.17035 [00:31:30] 1-bit LLMs, Ma et al. https://arxiv.org/abs/2402.17764 [00:38:35] LLMs Solve Math, Nikankin et al. https://arxiv.org/html/2410.21272v1 [00:38:45] Subitizing, Railo https://link.springer.com/10.1007/978-1-4419-1428-6_578 [00:43:25] NN & Chomsky Hierarchy, Delétang et al. https://arxiv.org/abs/2207.02098 [00:51:05] Measure of Intelligence, Chollet https://arxiv.org/abs/1911.01547 [00:52:10] AlphaZero, Silver et al. https://pubmed.ncbi.nlm.nih.gov/30523106/ [00:55:10] Golden Gate Claude, Anthropic https://www.anthropic.com/news/golden-gate-claude [00:56:40] Chess Positions, Chase & Simon https://www.sciencedirect.com/science/article/abs/pii/0010028573900042 [01:00:35] Epistemic Foraging, Friston https://www.frontiersin.org/journals/computational-neuroscience/articles/10.3389/fncom.2016.00056/full

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I think reasoning is just very ill-defined. A computer program will generalize arbitrarily well if you write it correctly, and we can prove that it's perfect and so on. Is it reasoning? I'm not sure. These models are probably not implementing algorithms, right? They're implementing heuristics that, you know, at least when they're training, are good enough to fit, right? But then, as soon as they go out of distribution, they generalize horribly. I like our results because we give a quantity that you can directly measure and like say, "This is happening in a transformer." Well, these other results are more like constructive kind of proof that you can represent this class of languages or so on. So they're kind of like very related, but very different approaches to similar issues in some way.

• 0:45Tim Scarfe?

  Federico, welcome to MLST. It's so nice to have you here.

• 0:48Speaker 00

  Thanks for having me.

• 0:50Tim Scarfe?

  So why do transformers need glasses?

• 0:51Speaker 00

  Probably should not be allowed to pick titles anymore because I think I come up with these wacky titles. But like the key idea, like why this phrase like kind of nice to me? It's like, they seem to be very bad. And this is kind of the point of the paper. It's like they seem to be very bad at detecting... Like if you care about a single token or whatever, at some point this breaks, right? And it's something about like, you know, they're not... They can't be, at least in the limit, as your context size grows. They can't be good at caring about a single token. This is kind of the... especially if it comes to the end.

• 1:32Speaker 02

  So Tofu Labs is a new AI research lab I'm starting in Zurich. It is funded from Path Ventures, involving AI as well. And so we are Swiss version of DeepSeek. So a small group of people, very motivated, very hardworking. And we try to do some AI research, starting with LLM and O1 style models. What we're looking for now is chief scientist and also research engineers. You can check out positions at tufalabs.ai.

• 1:59Tim Scarfe?

  So talk us through that. There's a figure, I think it's on page two, where you talk about the representations getting kind of squashed

• 2:07Speaker 00

  Right.

• 2:07Tim Scarfe?

  towards the end. Can you explain that figure?

• 2:09Speaker 00

  So imagine like a family of growing sequences. You repeat its last token. In the paper, for example, we have like, you know, can you count the number of ones in this sequence? And now you add an extra one, right? So, so now the two sequences have two different answers, right? Like essentially the one in which you repeat an extra one now has as an answer like n plus one, right? Or k plus one. There's actually an issue with this construction and it's like the fact that at some point, the influence of this final one gets lost, right? And this is, this is the fundamental idea, right? Like as, as these sequences grow, you have, you really care about that final one, that one that kind of gives you the right answer. But some point

• 2:51Speaker 00

  you, if you start measuring inside the model at the representations, you can, and you can also like, I guess we showed this mathematically as well as like, at some point, these representations get closer and closer, right? And, and then the issue is computers have precision, right? And, and if, if now this representation is too close, it's under the precision of the machine. Then you're forced, you're, you're mapping these two sequences to the same thing essentially. So one of them has to have a mistake in it.

• 3:21Tim Scarfe?

  So let's sketch this out. I'm not sure if I've remembered the experiments exactly, but it's something along the lines of you generate a sequence. So it could be 1, 0, 1, 0, 1, 0. Maybe you probabilistically generates the ones and zeros. So 70% of the time it's like a 1.

• 3:37Speaker 00

  Right, right.

• 3:37Tim Scarfe?

  30% of the time it's a 0. And then at the end, you ask a question, right? So you say, how many ones are there? Or what was the last thing? And, and you're kind of saying that as the sequence grows longer, the network becomes, it has a bit of a blind spot towards the most recent thing.

• 3:55Speaker 00

  To me, what was surprising is that like the way you construct this sequence, right? Of like, let's say, let's stick to copying, because maybe that's Kind of a more simple example, right? It's like, can you copy the last element of the sequence? This is kind of like a completely trivial operation, right? For a human, like a human will never make a mistake in this. Irrespective of, like, I think this is the nice thing, right? Like, if you're counting, right? Like, a massive sequence, a human will very likely make a mistake. But if you're just asking, what's the last element? You'll never make a mistake in this, essentially, right? Because it's like a completely trivially generalizable. You don't need to do any computation, essentially, right? It's just like, just look at the last thing.

• 4:37Speaker 00

  But then somehow the machine or the transformer, you know? We tried this with Gemini, you know, like, at the time of like the most powerful thing. At some point, it just, if you give it, for example, one, one, one, like a very long sequence of ones with a zero at the end, at some point, it starts outputting one. Like, I think this very nice explains, like, at some point, the fact that there's a zero in the sequence gets lost in the representation. So that, you know, and we measured this, right? And in a smaller model, of course, because like we need to have the weights and everything to do it. And, but, you know, this happens in, like, you can really view the representations kind of converge to each other. And I think this is a nice thing because it's kind of like saying, look, we can really find a quantity which explains why we're failing in this task, right?

• 5:21Tim Scarfe?

  Yes, and we'll get to your intuition and theory in a minute. Because I think it's really cool. It's like an intuition pump for people. But the reason why people at home will find it interesting is I think most of us feel that there's a kind of recency bias. And if anything, it's more likely to know about stuff that you've just said rather than stuff a long time ago. And certainly, when we look at some of these large models with large context, we see a kind of U-shaped curve, don't we?

• 5:47Speaker 00

  Right, yes, yes.

• 5:48Tim Scarfe?

  So how does that kind of fit in with your new model?

• 5:52Speaker 00

  Here, we have to talk a bit about one of the main intuitions, right? Of this paper, which is like, kind of related to this representational collapse idea. But it's like, I think in our paper, we're kind of developing this as its own idea, right? And it's the fact that, like, somehow, the way information flows in these transformers

• 6:12Speaker 00

  actually has an inherent mechanistic bias, right? And it's actually towards the start of the sequence. But when you train these models, right? Like, you You're training them to predict the next token. Very likely, this next token depends on the very nearest tokens, right? So in some sense, the training dynamics push it to care about the most recent things. But the mechanics of the information instead pushes it to care about, or at least retain information towards the start of the sequence. We believe this kind of nice, there was this kind of nice phenomenon which was observed.

• 6:54Speaker 00

  It's like if you ask or language models to do information retrieval, they tend to find it harder towards the middle, right? And to us, this kind of explains this phenomenon in the sense that mechanically, you transformers are good at the start, but they learn to care about the end and the middle is kind of lost somehow.

• 7:14Tim Scarfe?

  MLST is sponsored by Sentmel, which is the compute platform specifically optimized for AI workloads. They support all of the latest open source language models out of the box, like Llama, for example. You can pay on consumption essentially, or you can have a model which is always working, or it can be freeze drive when you're not using it. All of the models that they deploy support the OpenAI API specification out of the box, which means it's just a one-line change in your application to switch over to Sentmel and start saving money and make your application go faster. This is, we should linger on this for a bit. Maybe we should show a figure on the screen now, which is to do with this kind of topological information transfer through the network.

• 7:56Tim Scarfe?

  So the really interesting thing is that the earlier on you are, the more topological pathways that are to the prediction token. So if you think about it, you know, this self-attention transformer. Now, first of all, it has causal masking, which you should introduce, but that's just a thing so that when we're training it, it can't cheat by looking into the future. But subject to that causal masking, the further back you go, the more kind of pathways through this self-attention matrix it can take before it does the prediction. And what you're kind of claiming is that the more pathways there are, like the less likely the information is to become kind of like squashed.

• 8:38Speaker 00

  These kind of ideas, I think, are very popular in like graphical network literature, actually. And so it's not really out of the blue, this stuff. But like, as you mentioned correctly, it's like, it's really the fact that You have this causal mechanism in place, right? Which essentially means that attention now looks like a lower triangular matrix, right? Like, attention is only allowing you to look backwards. And this is really like a training trick, right? Like, of course, you can have transformers like BERT and so on that don't do this, right? But then, if we want to efficiently train autoregressively, you can't have a kind of your transformer look forward, right? Because then this makes it super inefficient to do things. And so what we point out is that actually,

• 9:22Speaker 00

  this choice of causal attention has an implication on how information propagates. And it's exactly the number of paths you care. Like intuitively, if you're the last token, the only way for your information to kind of survive through the attention mechanism is if your attention kind of on yourself, because this is the only kind of attention that preserves you, is quite high. But now this means that this forces, because there's this kind of sum to one constraint, that everything else is low. So there's this kind of trade off, right? Like, if you want yourself to survive and you're the last token, you can't essentially do attention, right?

• 10:04Speaker 00

  So I found this to be like, actually explaining a lot of interesting phenomenon. That this is why I quite like this kind of intuition.

• 10:13Tim Scarfe?

  And there's a really interesting literature here, because of course, your supervisor, by the way, was Michael Bronstein.

• 10:18Speaker 00

  Right, or still is, I guess.

• 10:20Tim Scarfe?

  I love Michael, he's amazing. But of course, Michael's done lots of work on this idea of squashing in GNNs. And even before that, people were talking about vanishing gradients and stuff in RNNs. And I think I read in your paper, you were saying that in the GNN literature and squashing, there's this notion of, I can't remember the word you used, but it's something to do with like the taxi distance or something. So like, the more steps you have to travel, like, the more squashed you get.

• 10:44Speaker 00

  This paper, right? It's like, taking ideas from this kind of work in graphical networks and building a bridge, right? And a very common way you think about, and actually, this is the work I used to do. I used to study information propagation graphical networks. So this work is not really a surprise, right? Like, that I kind of got into this. But it's like, essentially, the way people study graphical network propagation is by relating it a lot to spectral graph theory. And the...

• 11:18Speaker 00

  You relate this a lot to how kind of these, even like Markov chains, I guess. And now you get some natural quantities, like the taxi kind of distance you refer to is usually referred to as commute time. Yes. And it's, it's a quantity that measures, like imagine you're, you're on a graph and you, you define a random walk, which says, I'm a node with degree five, so I have five neighbors. I can jump to any of them with probability one over five, right, and so on.

• 11:57Speaker 00

  So this defines kind of this process, which, you know, you start at a point or the node and you sample and you can kind of create your little journey, right? And this commute time is essentially asking the question, let me pick two nodes on the graph and let me start at one of them. What's the expected number of steps I have to do to reach the other node and come back? So it's commuting because you want your random walk to hit that node and come back within, you know, and, and somehow this is super related to heat equations on graphs and so on. Maybe not surprisingly, because there's this kind of relationship between how heat spreads

• 12:41Speaker 00

  and how random particles move, right? So there's this kind of nice, very well-studied behavior. And, and it turns out that graphical networks, depending on how you choose them, like for example, graph convolutional networks are very similar to heat equations. They're actually, you can view them as a discretization of a heat equation over a graph. And this is actually very well-defined. Then it's not surprising that how sensitive they will be to two nodes talking to each other will be somehow related to this commute time, right? And, and this is how people have been studying graphical networks recently. And it's proven like to, we actually had a paper on this where like, you can kind of show that like

• 13:23Speaker 00

  on some tasks, just changing the way the graph is connected, heavily affects how easily the graphical network can kind of make two things. Communicate or how you can kind of transfer information between the two.

• 13:36Tim Scarfe?

  And just explain more about the the heat equations. Are you talking about things like second law of thermodynamics and Navier Stokes and stuff, but in the context of a graph where you have fewer degrees of freedom? Because you can you can speak to your neighbors, obviously, but it's a little bit different.

• 13:48Speaker 00

  Right, right. So you can literally define a heat equation over graph and you can define a Laplacian operator and evolve it, right? And this is like studying the spectrum of this operator, something that's there's just like super strong analogies between like continuous heat equations, you know, and like like you can really view graphs like discrete types of surfaces. And then like you can draw many, many parallels. And for some reason, magically, maybe not magically, it's super consistent. Like you can develop inequalities on graphs and heat equations on graphs that are consistent to continuous spaces, right?

• 14:29Speaker 00

  And then you can use these inequalities or whatever to tell bound to bound how a graphical network can kind of spread information or like how it's kind of information spreading operates.

• 14:46Tim Scarfe?

  You cited Muse, the lyrics of one of his songs, talking about how energy dissipates over time.

• 14:51Speaker 00

  Right, right.

• 14:52Tim Scarfe?

  Yeah, which was which, of course, was a reference to the second law of thermodynamics.

• 14:55Speaker 00

  I think this this citation is more due to Petar, so I can't take credit, but yes.

• 15:01Tim Scarfe?

  So Petar, of course, is famous because he invented graph attentional networks. And he's got loads and loads of citations on that. That's amazing. But Petar, for years, has been talking about the, I guess, the the potential limitations with transformers. And he's been looking into the domain of graph networks to improve that. Do you think that there's an opportunity here to build a better architecture?

• 15:25Speaker 00

  Yeah, I guess that that's the hope, right? Like somehow, I mean, maybe this is this is coming from the opinion of someone that started his PhD on graphical networks and then decided to kind of jump ship and go to like language models, whatever. And I think there's actually a lot of value in studying graphical networks because we can really have, I think we have a pretty deep understanding of how they work. At least some models are very kind of.

• 15:58Speaker 00

  You can really study them quite nicely because of these kind of relationships with physics and so on. And the hope is that you can take these approaches and then

• 16:12Speaker 00

  bridge, build a bridge to transformers. And transformers, if the, if language models were fully connected, this would not be as interesting. But the fact that they have this very specific topology kind of gives you opportunity to exploit these ideas. If you're fully connected, essentially you have arbitrary behavior. And so what can you say, right? But, but if you're not fully connected now, you can maybe say something more interesting. And, and this is, this is why I like causal attention mechanisms, because they're actually super interesting to, to study. And, and I think in our paper, we also have some efforts, maybe quite small.

• 16:53Speaker 00

  But I think they're an interesting direction in which, I mean, they're mostly in the appendix. But like, if you can develop a spectral theory for this kind of triangular attention matrices, you can come up with cool results or like kind of cute results. Whereas for example, we have this result, you know, in the limit, you only care about the initial token, right? And then so on. And these are kind of like really ideas from spectral graph theory that kind of naturally, and these are not maybe, I wouldn't say they're like super deep results, but like, I think they provide a lot of intuition for like what is happening, right?

• 17:26Tim Scarfe?

  There were so many things you said there. I don't know which ones to take first, but maybe let's just touch on the last thing you said, which is one of your, you know, kind of ending theorems in the paper is, is what you just said. I think it wasn't a strong theorem, actually. You said that you, you, you believe that in the limit, as the, the token length, you know, increases, increasingly it's only going to pay attention to the first token.

• 17:49Speaker 00

  Right, right. We had to make some assumptions there, right? And it's like, because everything is super nonlinear. But like, the main intuition is that like, what, what happens if you add more and more layers, right? Like, like this is kind of what you're trying to study. And if you assume that like, you can kind of ignore the contribution. So of course it's a pretty big limitation, but like, in general, I think it's pretty hard to study.

• 18:17Speaker 00

  Super nonlinear systems from this perspective, right, of linear algebra. But like, I guess the point is like, if you keep applying these attention matrices, and maybe this kind of leads nicely, but attention matrices cannot never, can never really be sharp, because for them to like, only like by sharp, I mean, for now, let's just say I wanted to like, you care essentially how much this, this process mixes things, right? And because you can only go backwards, this mixing goes only in one direction. And this direction is only towards the beginning of the sequence. And so the more layers you apply, the more you just care about the beginning of the sequence.

• 19:01Speaker 00

  And this is kind of what happened, right? And actually, there are very nice results here, and I was really happy to see this. There are some, there's a very nice paper by Carolini, which, these are very good folks that do like, security, right? And then, and there's this very nice paper where they actually found that as you repeat tokens and so on, it seems like your final output just becomes the beginning of sequence token. This was nice to see because these results, to me, point why, you know, like, why is this happening? Like, it seems kind of arbitrary, like, why do you, but actually mechanically, this makes a lot of sense from a spectral perspective. So this is why I'm kind of very excited about this direction, like, perhaps, you know,

• 19:45Speaker 00

  even if we make these kind of broad simplifications, there's something there that you can really, really helps your understanding. And so I'm hope that this kind of work could even kind of spill over to, you know, like, somehow better understanding leads to better security, or better attacks, or better defenses, right? And so I kind of view this as like, understanding is, comes first, and then application comes later.

• 20:11Tim Scarfe?

  I guess the obvious question is, we do have models now that have a 2 million token context, and, I mean, they don't work perfectly, but they seem to work quite well. So why don't we see like a massive mode collapse with those?

• 20:23Speaker 00

  I guess someone has to open up these models and figure this out, essentially, because I think at least spectrally, this should not work, right? And the issue is that then, like,

• 20:37Speaker 00

  You can view attention as a type of contraction. And but then you have all these kind of different components like MLPs coming later and so on, that then act potentially as expansions, right? So there's something there about like the balance that needs to be struck. Like, you want to counter the contractive effect of attention and you have residual connections, for example, that help with this, or you have MLPs that help with this, right? So is this kind of like some kind of self-stabilizing like process that you hope for? I imagine that, well, first of all, I think a lot of models which go much longer context

• 21:20Speaker 00

  alternate between windowed attention. So they kind of have a lot of times mechanisms, right? And maybe these are discovered by accident and this helps. You know, you do architecture search and you know, you discover that this helps, but whatever, like, why? I don't know. And then I also imagine that these models will learn a lot of things to kind of preserve the mixing. So for example, if you look into Llama or Gemma, a lot of the heads actually don't do anything. They, for example, they implement diagonal heads, right? These are quadratic no-ops essentially. These are expensive no-ops, right? And why is the model learning this? I think it's because it's trying to preserve itself.

• 22:03Speaker 00

  It's trying to not mix everything all at once. It's trying to still, I don't know, maybe once, like, it's trying to, yeah, just essentially not overmix in some way. And it can find ways, like, for example, a lot of heads would just attend to the beginning of sequence token. And this beginning of sequence will have norm very close to zero. Again, this is a no-op. So there's also, I think, a huge opportunity here to like figure out in some way, much more, like, this is a huge waste, right? And like model compression is a huge topic. And so maybe, you know, looking at these issues can kind of help with that, right? Like figuring out why is it learning no-ops, you know?

• 22:48Speaker 00

  Like, this is a huge waste.

• 22:49Tim Scarfe?

  Yeah, a few thoughts on that. I mean, first of all, This kind of theory can help us a lot, understanding training dynamics and coming up with principled ways to design architectures. But there's an element of sometime, I mean, even now, if you take Gemini and you trained it 100 times, presumably sometimes it would work better than other times because of like, you know, whatever the conditions are. And maybe sometimes you need to sort of give it a bit more capacity and it might not use that capacity. So when it trains, it will almost kind of like say, no, I'm not going to bother using the stuff over there. And even then, I don't think it makes sense to think about training dynamics in a global way because it's quite input sensitive. So there are certain inputs that might light up more of the network and perhaps in some cases and not in other cases. So it's, it's really complicated.

• 23:32Speaker 00

  Yes, yes, I guess that's a good way to summarize. I mean, I think if you ask Razvan, Razvan Pascano, who's also in this paper, he, he will tell you that he thinks you need a lot of heads because at the start, it's kind of a way to give it a lot of options. So it maybe then it learns that like some heads explore in this direction, some heads explore in this other one. And then at some point, heads that it doesn't need shut off and it just preserves the ones which for some reason found the best kind of setup. So I guess in some sense, you have this super over, you have maybe 500 heads or even many more than that.

• 24:14Speaker 00

  And yeah, as you're saying, like you, it's just like a kind of diversity strategy at the start to then try to just pick the ones you care about.

• 24:24Tim Scarfe?

  You were saying that, you know, we design these neural network architectures and it's kind of a little bit hacky. So there are multiple opportunities, multiple heads, multiple pathways, residual networks and so on. And I loved how you described it, that there's kind of like expansion and contraction in successive stages.

• 24:45Tim Scarfe?

  What's the intuition there?

• 24:47Speaker 00

  Yeah, I guess this is really like this spectral intuition that like, like if you look at the eigenvalues of these triangular matrices, right? Like, there will be only one, a single one, if you make some side assumptions, and everything else will be less than one, right? And, and kind of the, the, the effect of this, like, is if you take powers of this, then essentially you're just left with the one eigenvalue that's like the one, you know, that's a value one. And so this is kind of this eigen space that survives. And, and so this kind of acts like a contraction, right? So it's still like, not necess- like, I guess it depends how, I'm being kind of loose with the idea of a contraction.

• 25:31Speaker 00

  But it's like, to me, this is contracting in a sense that it's destroying information unless you're like in this kind of one specific starting token. And then instead, the MLPs can essentially have whatever lipshits constant they want. So MLPs, they can expand you as much as you want. So like, in this sense, like, imagine you, and I'm really viewing this as like, kind of read and write operations. And so, like, imagine your attention now cares about 100 different things. And so each coefficient is roughly one over 100. And now you're copying this information maybe to

• 26:14Speaker 00

  a different hidden dimension within the value vector. So you're just kind of like a copying operation. Now, everything in there will be scaled by one over 100, right? So, so then, this is kind of, you know, norm to me is very important into building very sharp operations. So if now everything is one over 100, this actually won't have that big of an effect. But now your MLP can take this and multiply everything by 100. And now everything is like, kind of strong again. And of course, now you're still fighting with like layer norms and so on. But like, in some way, this gives you a way to

• 26:48Speaker 00

  The attention can copy information, but the more it copies, the weaker the effect of this kind of copying operation is. So the MLP can kind of counteract this. And this is kind of the way I view these two operations as like, if you want to copy many things, you probably want some component that's able to like, kind of crank up the knob and like, try to make this copying more strong in some way.

• 27:14Tim Scarfe?

  When you cited Carlini and certainly in the experiments, most of the experiments you did, it was quite a low entropy sequence, which is to say, you know, it was ones or zeros or something like that. How much of an effect does the, you know, the amount of like different types of tokens or the amount of entropy in the sequence? How much does that matter?

• 27:33Speaker 00

  This matters. And I feel like it's also due to how tokenization works. Like, like if you, I think you have to be very careful when you're doing these experiments. Like, you know, for example, if you want to usually put a space between tokens, so that now the tokenizer, you know, most of them will kind of see a space and then give you a new token. Right? So you really have to be careful about this. And then also the kind of entropy of the sequence matters. And I think this just makes it very complicated, right? To, so I believe we do have kind of experiments that check that this still holds for like, you know,

• 28:17Speaker 00

  essentially random, like arbitrary sequences. But like, you just want to be kind of like, they definitely do have an effect. So for some reason, I found that digits such as nine or like larger digits will have usually larger magnitudes for some reason, or you know, it kind of makes sense. It's what it's trying to do. But like, you know, all of these different things can really have an effect on like how quickly you converge to, how quickly these two sequences converge to each other.

• 28:47Tim Scarfe?

  And can we explain what we mean by that? So if I understand correctly, you've got two sequences and then you do the L1 like difference of the softmax. And you're saying that that kind of converges towards like, you know, epsilon, some arbitrary small number.

• 29:04Speaker 00

  I guess what we really study is like, You care about the representation of the final token at the last layer. Because then this is what you slap on your linear projection and you extract your next token, right? So the key idea here is like, if you take two sequences and this kind of last, the representation of last token, last layer, doing these two sequences arbitrarily close, then you run into issues because now they get mapped to the same thing. And essentially, what you care about is the norm between these two things. You want to show that, you know, based on this family of sequence,

• 29:48Speaker 00

  like if you keep growing and growing, like at some point, this norm goes to arbitrary epsilon essentially.

• 29:56Tim Scarfe?

  That makes sense. So we want to have self-attention transformers that have the representational fidelity to distinguish, you know, differences in these situations.

• 30:06Speaker 00

  Right.

• 30:07Tim Scarfe?

  And here's another interesting thing. So you bring in this idea of numerical precision. So you say, we've got a problem, right? So we have these sequences and they get longer. And, you know, perhaps the sequences are quite low entropy to start with. But eventually, depending on the numerical precision of the neural network, so it might be FP16, it might be quantized. At some point, we kind of like dip below the threshold where the network can distinguish between the two things.

• 30:36Speaker 00

  Yeah, yeah, that's exactly the end. The fact that we have really heavy quantization in a lot of these models at the moment. I mean, they go to much less than 16 bits. This really, to me, points like towards, I mean, I think it's understandable that like, if you quantize models, you lose performance. I think anyone would probably expect this. But like, I think this points to very kind of mechanical issues of like, you know, if you really quant, like if you quantize a lot, at some point, you even have sequences that maybe were distinguishable at higher precision, but now they're indistinguishable and you make a mistake by force, right? So.

• 31:15Speaker 00

  There's this kind of very mechanical just complaint in some way about quantization in here.

• 31:23Tim Scarfe?

  Yeah, because doesn't this become catastrophic quite quickly because your research shows and by the way, Liron, he's the doom debate guy. He keeps saying, give me, give me examples of things that L L M's can't do. Copying and counting. They can't do more than a hundred elements.

• 31:40Speaker 00

  Right.

• 31:41Tim Scarfe?

  Frontier models can't do more than, and this is like before you quantize them. So now we quantize them down to four bit or whatever. And they, they just, they collapse straight away.

• 31:50Speaker 00

  Right.

• 31:51Tim Scarfe?

  This is a huge problem.

• 31:52Speaker 00

  Yeah.

• 31:52Tim Scarfe?

  And Liron, Liron would say, well, this isn't a problem. We just need to use tools, man.

• 31:57Speaker 00

  Yeah, I'm sure.

• 31:59Tim Scarfe?

  What, what's the problem with that?

• 32:00Speaker 00

  I guess this is a depends who you are and what you're kind of interested in. I mean, I'm interested in, um, well, first of all, understanding what's going on, but also like, I think a model should, you know, if we claim that, you know, we have A G R or whatever, you know, like I, I think a model should,

• 32:23Speaker 00

  should be able to do simple things that a human can do, right? And, and so there's this kind of argument that like, okay, it would be nice. And then of course you can equip models with tools and this is very successful. But like, I guess one of the main points we make in our paper to kind of argue against this directly without saying this, it would be nice is that,

• 32:43Speaker 00

  you often just want to copy things into tools, right? And so if you can't even do this, uh, then you probably have a problem, right? So, so somehow counting sure could be solved with tools, but copying is something that's super fundamental. And you want to be very robust at, at doing this.

• 33:07Tim Scarfe?

  So you also did an, well, you did two very interesting experiments. One was with chain of thought and one was where, and, and this actually has a bit of a history. Do you remember like G P T three had a problem with numbers and because of the, the byte pairing coding or whatever. And you, you put spaces between the numbers and it works better. So one of the experiments you did was you just kind of like interleaved the sequences.

• 33:28Speaker 00

  Yeah.

• 33:29Tim Scarfe?

  And that, that helped weirdly. Why did that help?

• 33:32Speaker 00

  Actually, it was not weird. Like actually this was us trying to. debug our theory in some sense. And it's like, the intuition there with oversquashing, for example, is that like, um, that if you're trying to copy the zero at the end, right? And you know now that the zero is being at the end means that it's kind of just very problematic now, because you lose this idea that there's a zero. Now, if you add more zeros in the sequence, this fact that there's a zero is not lost as quickly. In some sense, it's a way to counter the ones dominating the representation.

• 34:14Speaker 00

  So now adding these zeros, perhaps unsurprisingly, helps not lose the zero in the representation. What I think was very nice to see was that copying the first token seemed to be much easier than the last one, because then this is really kind of showing this idea of the paths, right? The representation of the zero, when you look at these partial derivatives, is much more influential when it's at the start rather than at the end. This to me is like, really like, okay, this is like, we have built some kind of understanding here, right? Of like, because I think prior to, without thinking of oversquashing these paths, like, this seems like absurd, right? Like, it's like, super random behavior. Like, why is this happening? Right? So, so I kind of like the intuition from these experiments.

• 34:59Tim Scarfe?

  And folks should definitely look at the paper. I mean, just kind of like verbalizing it, as I remember, there was kind of like a task where you told it to do something at the beginning and at the end. And you found that as you increased the sequence length, it was kind of like, still successfully attending to the beginning for quite a long time. But at the end, it dropped off very quickly. And one of the things as well you did was like, I think, zero shot and more than zero shot chain of thought prompting, because, you know, all of your experiments were basically showing that when the sequence length got to even a fairly small number, I think even past 10 or something, it would just, it would just kind of go crazy. And if you were counting, you would expect to see like a monotonic, like, increasing set of results. But it just collapsed very quickly.

• 35:43Tim Scarfe?

  How did chain of thought help?

• 35:45Speaker 00

  The experience we did were like, you, maybe the most simple one is like, just asking it to sum 1 plus 1 plus 1 k times, right? And, and I guess, okay, if you, I think it's kind of notorious that these models suck at this kind of task. Which to me is quite disturbing, but anyway. And, and with Chain of Thought, the idea was like, we were trying to ask it to break it down, right? So it's like, maybe do it in groups of five, right? And so on. But then you very quickly see that essentially starts diverging and like, just coming up again with like, nonsensical answers. And, and this was just to kind, this was not surprising to me.

• 36:30Speaker 00

  It was just like, it was quite expected to me, actually, because there's something about this repeated nature of the task, right? Where it's like, I think the model just gets very kind of quickly mixed up with like, what it's keeping track of. And it's just like, there's so many one plus ones, right? And it just becomes, just extremely hard for it.

• 36:51Tim Scarfe?

  So there was this amazing plot, and I think it was a counting task.

• 36:55Speaker 00

  Right, yeah.

• 36:56Tim Scarfe?

  And what it showed was this big mode smack in the middle. Could you, could you explain that?

• 37:01Speaker 00

  So, so what we did is just very simple. We, we essentially fed the model, can you sum one plus one, so on, five, 10, 15, up to 200 times? And what you see is that it's okay around 20-ish. And then it starts outputting numbers, not even multiples of five and so on. And then, uh, once you go past like a certain size, it just relax to output 100. And, and this to us was kind of pointing towards the fact that it's probably not mechanically counting.

• 37:41Speaker 00

  And I think there, there's been like, actually some very recent pre-prints out on like this kind of behavior. That these models are not like, probably not implementing algorithms, right? They're, they're implementing like, I think in this paper, they call it bags of heuristic, right? Like they, they, they're implementing like, kind of. Heuristics that, you know, at least when they're training are good enough to fit, right? But then as soon as they go out of distribution, they generalize horribly and they output always 100 because 100 seems to me like a pretty plausible, common answer for some, a sequence that's a big sum of ones, right? Like somehow 100 is like, makes sense as 100, right? At least from a training set point of view, right?

• 38:25Speaker 00

  So, so this, I think is a pretty surprising, but also not surprising result that, you know, this occurs.

• 38:34Tim Scarfe?

  Yes, and that paper you mentioned is arithmetic without algorithms, language models solve math with a bag of heuristics.

• 38:40Speaker 00

  Right, yeah.

• 38:40Tim Scarfe?

  And when I read your paper, there was this beautiful word that you used, I had to look it up because I've not seen it before. I think it was subitizing.

• 38:47Speaker 00

  Right, yeah.

• 38:48Tim Scarfe?

  And children, apparently, they, they learn to sort of like, you know, that's five apples. And they, they learn a pattern for counting, so they're not actually counting, we can just quickly kind of do that. And the suggestion is that language models are doing the same thing.

• 39:03Speaker 00

  Like, it makes sense that you care about rough estimates more than like precise numbers. I mean, I'm not sure I really understand the difference between 100 and 103,

• 39:14Speaker 00

  but I can understand the difference between 10 and 100, right? Like, at least, you know, so I think it makes, and I guess this kind of goes to a question like, I'm not even sure how I count, right? Like, I, I don't, I don't really know how robust my idea of counting is in my head, right? It's like, okay, I can mechanically do it, but, um, yeah. But it, it makes sense that raw estimates are probably quite important for children and probably for language models.

• 39:46Tim Scarfe?

  So in, in the introduction, you did a bit of a like literature review on some of the LLM skepticism papers, and you cited Vinyals talking about how under certain conditions, self-attention transformers can be Turing complete.

• 40:01Speaker 00

  Right.

• 40:02Tim Scarfe?

  Under which conditions can they be Turing complete?

• 40:03Speaker 00

  Right, so I think that paper in particular is not really formal about this, actually, but there are papers which are, and I think they come from folks who are more like in kind of...

• 40:18Speaker 00

  Kind of more computer science approach, right? Like of looking at them through automata and so on. And usually assumptions you make are about like hard attention. So this means that you assume that your your attention is one or zero, right? And then this is impossible, right? In practice, but it's it makes sense why this is a useful kind of paradigm. I think people relax this usually or sometimes to having. I'm not sure exactly was like softer hard attention where it's like, instead of having just a delta at some point, you have one over K, right? So you can attend perfectly uniformly, right?

• 40:58Speaker 00

  And so I think many of these results involve these constructions because they're simply easier to treat mathematically.

• 41:06Tim Scarfe?

  But shouldn't it be impossible in principle for a self attention transformer to be Turing complete? If it's finite precision and if it's doing a fixed

• 41:13Speaker 00

  Right, right. Yes. So they also have these assumptions on memory growing with sequence length usually and or precision growing. So by memory, I roughly mean hidden dimension times floating point precision. This is kind of the memory, how many bits you have, right? In a transformer and and and I guess you also utter this times the number of tokens you have, right? This is kind of like, but but usually I think a lot of people study this in the paradigm of like, you have your precision increase with with sequence length, like logarithmic or whatever. But obviously, precision is not increasing, right? Precision is what it is, right? The what you choose it to be, so.

• 41:53Tim Scarfe?

  Yeah, I spoke with Shmirtu about this. Of course, there was that Segalman paper in ninety five where she showed that if you have an infinite precision RNN, then of course it can mimic a Turing machine.

• 42:03Speaker 00

  Right. And if you have infinite precision, you don't have representational collapse. Like you can go as close as you want. You don't care, right? Because you can always distinguish the two, right? So. It makes sense that, you know, in some sense, these results are quite related, right? Like how this kind of, what I think the nice thing about our results, other than, you know, these results in automative formal languages and so on. I mean, there's this very nice line of work on this programming language called Rasp. It's a programming language that implements operations transformers can do. There's this kind of working conjecture on like how if you want to implement something, which is a short Rasp program, then you can, like your transformer can learn this relatively well, right?

• 42:48Speaker 00

  And like generalize better. So this is kind of nice relationship with like central programming languages, right? And transformers. But in some sense, I like our results because we give a quantity that you can directly measure and like say, this is happening in a transformer. Well, these other results are more like kind of constructive, which is obviously super interesting, but they're more like constructive kind of proof that you can represent this class of languages or so on. So they're kind of like very related, but very different approaches to similar issues in some way.

• 43:23Tim Scarfe?

  Yeah, there are so many ways to come at this. And by the way, I love that, is it Greg Delatang, the Chomsky hierarchy?

• 43:28Speaker 00

  Right, yes, yes, yes.

• 43:30Tim Scarfe?

  I think he mapped all of the various different models, like, you know, LSTMs and transformers to the automata hierarchy.

• 43:36Speaker 00

  Right.

• 43:36Tim Scarfe?

  I think he said the transformers were.

• 43:38Speaker 00

  I think quite at the bottom.

• 43:39Tim Scarfe?

  FSA's, yeah, I think RNNs could learn, like, you know, counting languages.

• 43:46Speaker 00

  Yeah, yeah, yeah.

• 43:47Tim Scarfe?

  But you also cited Peng talking about the limits of compositionality and transformers.

• 43:52Speaker 00

  Right.

• 43:52Tim Scarfe?

  Because that must, that must trivially be true, surely. Like, a language model couldn't generalize Mary loves John to Mary loves Jane because it doesn't have that invertibility.

• 44:04Speaker 00

  Right, yes.

• 44:05Tim Scarfe?

  Everything gets kind of spread out into all of these circuits, want to be better.

• 44:09Speaker 00

  Yes, yes. I think there's also been work, for example, that showed that even just switching the order of, of like, if you ask a language model to solve like first order logic, like just chains of implications, the order in which these are presented to the transformers. So if kind of this chain is very naturally present in the order you feed it in, greatly affects if you can like solve these kind of problems. So. Yeah, all these results point to like pretty bad generalization habits that our current models have.

• 44:44Tim Scarfe?

  So what kind of modifications could we make, you know, practical modifications to overcome this this collapse of representations?

• 44:51Speaker 00

  Vanished ingredients, since I think are quite related to this. And this is something that, you know, transformers are in some way a solution, you know, like some way transformers arise naturally after you discover that R N Ns have this kind of bias. But then we point out they have even causal mechanisms implement this bias in some way. And like,

• 45:11Speaker 00

  there's something I think just hard about

• 45:15Speaker 00

  having long sequences, like you have to essentially take a large amount of information and try to fit this in a finite number of bits, right? So I think this is just a hard problem. And probably the best solution would be to have ways to compress information and somehow just be left with less and and have your technique just care about less things, right? So in some way, windowed attention and these kind of processes, I think are quite well motivated in this aspect.

• 45:57Tim Scarfe?

  I mean, in your bones, deep in your bones, do you lean connectionist or do you lean towards we need to have a hybrid neuro-symbolic approach?

• 46:04Speaker 00

  I'm a big fan of having

• 46:08Speaker 00

  hybrid approaches. And

• 46:11Speaker 00

  actually, very recently, I think, maybe, I mean, now there's like the chess world championship going on and and actually some, like Peter and some other and these amazing researchers, they released like some kind of component in Gemini that's able to play chess.

• 46:28Tim Scarfe?

  Oh, no way.

• 46:29Speaker 00

  And it's actually like surprisingly strong and it's like a language model still. And it does have its own kind of specialized like chess component, right? So so this is a I think this is just a very principled approach. Like, imagine you could have your base language model, which is really like this orchestrator. And then you could have maybe a math, specialized math unit, or a specialized chess unit and so on. You could kind of let, ideally, you keep the base model fixed and you can swap out and improve your chess component at some point. I think this, to me, sounds super exciting and we know that it kind of works.

• 47:11Speaker 00

  I mean, in some sense, vision language models, you know, they have a specialized vision component, they have a specialized language component usually. And so I think this works. And of course, visual language models are trained in their own kind of way. So it's a bit tricky, but like, this, to me, sounds quite nice. Like, in some sense, you have this stream of tokens and these can be generated by very specialized things, but then you have this kind of base model that's processing the stream. But these tokens could come from whatever you want, essentially. It could be even not just, it could be like an RNN or something. If you find that RNNs are better accounting, maybe you could use those as like your maths component or whatever.

• 47:54Tim Scarfe?

  What are the trade-offs? I mean, I love this formal approach because it can give us solutions that are explainable, robust in some ways, verifiable and so on. But when you mix and match these things together, of course, it's very difficult to achieve learning.

• 48:09Speaker 00

  Yes, yes.

• 48:10Tim Scarfe?

  And it's kind of somehow bottlenecked by how we designed the, you know, the system two component, if you like. So there's no clean and easy way to build them together.

• 48:20Speaker 00

  I agree. I think this is why we don't do this at the moment. It's just pretty hard.

• 48:26Speaker 00

  I, I just think that transformers have pretty fundamental limitations that, like, at the moment. And so it makes sense to try to use other models as well. Like, but yeah, it's just super hard to do. And maybe once we find a better, better way to do it, then this will become much easier.

• 48:52Tim Scarfe?

  What is your definition of reasoning?

• 48:54Speaker 00

  I think reasoning is just very ill-defined to me. And like, I can talk to you about length generalization, which I view as like, some kind of... Subcomponent of reasoning. I mean, to me, reasoning is just a very kind of broad idea of like, we care about generalization and robust generalization. But then there's like very different, like, for example, a computer program will generalize arbitrarily well if you write it correctly, and we can prove that it's perfect and so on. Is it reasoning? I'm not sure. A human, I mean, there's this, I think this very nice example of like, I believe it's in Numberphile.

• 49:35Speaker 00

  They set off with a bunch of people and they're like, let's compute pi to, I don't know how many digits, right? And then, and they got like a ton of people, they spent a week doing this. And then I think the first time they pretty much, they had a mistake somewhere in the middle, right? And like, you know, and so this to me points to like, it's not surprising. Like, in some sense, even humans, as tasks grow, I mean, I'm sure you know how to sort an array or whatever, but like, good luck sorting an array that has a million digits, you know, you'll probably make a mistake, like somewhere, right? And then so are you then not reasoning? I'm not sure, right? So there's this kind of weird trade off, where it's like, yes, we care about langen generalization,

• 50:19Speaker 00

  but then we're kind of bad at langen generalization, right? Like, we're not good at processing large amounts of data. So maybe if we just care about langen generalization, we're not reasoning. So essentially, it's a very long way of saying, I think, maybe reasoning is not super well-defined, but we can do things that are well-defined. And I think that it's fine if language models have a different definition of reasoning to humans, because they are fundamentally computers, right? And they're built on a computer. So computers are good at things that humans are terrible at. This is why I think stuff like counting and whatever should be doable for, because they are fundamentally computers. And so I think reasoning for humans and machines, it's, we care about different things, right? So.

• 51:04Tim Scarfe?

  Shorley's coming here at the weekend. And, you know, he wrote this measure of intelligence paper. And he always, he never uses the word reasoning, which is interesting. And there was a tweet I was reading this morning. And, you know, so he thinks intelligence is knowledge acquisition efficiency.

• 51:20Speaker 00

  Right, yes.

• 51:20Tim Scarfe?

  And knowledge acquisition for him is all about this kind of creative, adaptive search for novelty.

• 51:27Speaker 00

  Right.

• 51:28Tim Scarfe?

  But he's always at pains to stress that. Just performing a skill isn't intelligent and you're not doing reasoning. So a chess computer isn't reasoning.

• 51:39Tim Scarfe?

  But AlphaGo is reasoning. And or AlphaZero, I should say. And there's something about the creativity. It's because it's actually kind of trying things. And it's acquiring new knowledge that perhaps the creators of the system didn't program into it.

• 51:54Speaker 00

  I'm really into chess, right? And like, I don't understand this comparison because

• 52:02Speaker 00

  what AlphaZero, for example, or you know, AlphaZero has produced very novel ideas that computers don't find. For example, right, like this kind of pushing the h pawns and so on. And that now are used in like all the time by grandmasters, right? And they are mechanically doing very similar things. Like both work on this kind of tree search idea. Both they just have different ways of a different way of computing heuristics, right? Like in Stockfish, for example, they will. I'm not sure about all the details, but like my understanding is that essentially a bunch of people just put in a ton of heuristics in there that like maybe are somehow extracted from statistics or so on.

• 52:45Speaker 00

  Right? Like you, you want your knights on average, not be on the side as well. But then these human heuristics will just bias the tree search, right? And then now if you don't have these human heuristics in like AlphaZero, it will find something different, but this is not surprising to me. And somehow I think that associating this finding novel ideas, not as being creative, but rather is just using a function that's not aligned with what humans necessarily think is.

• 53:19Speaker 00

  And even Stockfish will find completely crazy ideas, but somehow the fact that there's still like in some way, there's a human touch in Stockfish that's not there. And in AlphaZero, and I'm not sure, like, for example, if then you can say that a human playing chess is reasoning, right? Like, humans will be much less methodical when they think about a position. They will have a ton of kind of moves that appear in their head. And you can't even explain why, right? But like a computer and sometimes doing a tree search can roughly tell you why, like, okay, because deep down, you know.

• 53:54Speaker 00

  I'm not sure I agree necessarily with this chess, but maybe it's because I'm maybe too addicted to chess. And then like I have, but, but, but yeah, to me, yeah, it's just like you're, you're guiding through a heuristic that's not human. So then it feels creative.

• 54:13Tim Scarfe?

  I mean, if, if we were to design an optimally intelligent reasoning system, I mean, what would it look like to?

• 54:19Speaker 00

  Defining reasoning is, is important, but I, I think I would just like it to,

• 54:26Speaker 00

  in some way behave like a computer in the sense of like, it doesn't make mistakes which are like trivial in some sense. Like, like it, it doesn't make mystical trivial, but it also doesn't make like kind of conceptual mistakes, right? Like this, this to me is like really what, what these language models can do much better than humans, right? And like, if you can kind of leverage this fact that they are machines and also leverage this fact that we can make them behave like, or align them to think like humans, you know? I think this is essentially creating a super powerful combination, which a human is, would, it would just be completely out of reach for, for a human.

• 55:08Tim Scarfe?

  When I was speaking with Neil, I'm slightly skeptical about the Golden Gate Claude thing, right? So it, it said, okay, take an abstract feature. Here are all of the, you know, go, go back to the corpus. Here are all of the tokens that maximally activate that feature. And to me, it looked like a bit of a keyword matching scheme. And this is where the, the, you know, we have this kind of reasoning chauvinism. We think when, when we do stuff, it's reasoning. And when language models do it, it's not reasoning.

• 55:34Speaker 00

  Right.

• 55:34Tim Scarfe?

  And I think the, the meta component of reasoning is really important. So as you say, we could just do this tree search. And from a functional perspective, it's as if it's doing reasoning, because it's found all of these weird and wonderful trajectories and patterns. And it's doing reasoning, hey? But when we do reasoning, we're looking at abstract analogies in the real world. So we're saying the universe has these patterns, just like a kaleidoscope, or this composition of like knowledge being mixed together and so on. And when we look at a chessboard, we see this weird analogy with like bananas or something. And we're, we're Kind of doing that multi-domain analogical reasoning. And that feels like a very creative jump. So we call that reasoning. But when we see a computer do it in a simplistic way, we kind of say that's not reasoning.

• 56:15Speaker 00

  There's this illusion of reasoning that comes from like you. Humans are extremely good at like finding extremely complex patterns which are kind of unexplainable. And somehow these jumps are reasoning. But like, I think there's, actually chess is a great way to think about this. And it's like, what, what? For example, there's this kind of very notorious experiment where you ask grandmasters or whatever, like very strong chess players to look at a position and then try to recite it off memory, right? So you look at it for, I don't know, 20 seconds and then take fresh board, put all the pieces right. And then they asked also amateurs to do this.

• 56:54Speaker 00

  And the grandmasters were much, much better at recuperating this position. But then if you gave them our, these positions all came from games, right? These are positions that arise from playing chess. And instead, when they gave them random positions, they were equal, right? So it's not that grandmasters are extremely, they have crazy memory or whatever. Grandmasters have a much more compressed representation of the world. Because they've played and seen so many chess positions, they're able to compress structures, right? So if I have pawns on f7, g6, and h7, and a bishop on g7,

• 57:34Speaker 00

  this is a, and then I'm castled, you know, and it's like, this is like, I recognize it immediately as a fiancetta, right? And this is like one thing now, right? This is, but someone who's never played chess, they have no, no concept of this structure, right? But like, or I've seen this Sicilian pawn structure a gazillion times. I'm just remembering this is, or this is this position that arises from this special call or whatever, right? Like, this is one concept now. And it's just because I'm allocating so much more of my brain to compressing complicated, seemingly noisy information, but it's not actually noisy. It's something that I've just seen 10,000 times.

• 58:16Tim Scarfe?

  Yeah, but then there's the question of. It feels like, you know, we have all of this experience and then there's this diffused concept formation that you speak of. And then the intuition isn't reasoning, it's memorization, surely. But it's not a simple case, though, of one day, a chess player has a flash of inspiration and they just define this new piece of knowledge. And then from that point on, they can just intuitively grab that knowledge. It's a really diffused thing where you have all of these diverse experiences and like, you know, this concept emerges. It might happen emetically, it might happen spread out over lots and lots of different people.

• 58:51Speaker 00

  Yeah, it's somehow magical, right? There's the magical power of human brains, right? That you don't know why one day you look at this position and you come up with this brilliant move that you maybe would have not even considered. But I guess my point is like, this only happens once you have an extremely strong compressed representation. Then you can somehow ignore noise. You have, essentially, this gives you more capacity to compute things without thinking about like, oh, all these pieces are here, right? Like, you just look at it and you know, you can just compute with a much clearer kind of representation, essentially. And then this is when brilliancies occur and this is when these magical moves pop out.

• 59:35Tim Scarfe?

  So are you a fan of the flash of inspiration?

• 59:38Speaker 00

  Once you have these kind of moments where like, everything clicks, it's kind of like, you've climbed a little step now in like, your understanding. And I think they will only, at least from my experience, these kind of moments come once there are a lot of things, at least for humans, right? You have, you need a lot of things to be right for you to come up with these kind of next steps. So like, you need to be like, very much super focused on this task and then you need to have good sleep and then you need to kind of just be in good health. And then you can play maybe one day at just a much better level or whatever. You can do something at a much higher level. And then I think this kind of gradually accumulates, this kind of like

• 1:00:19Speaker 00

  sequence of inspiration moments or whatever that then like kind of build up on each other. But yeah, it's for sure not like something you can choose or like something that doesn't come unless you invest a ton of time into into something, I think.

• 1:00:33Tim Scarfe?

  Carl Friston's got a beautiful word for this. He calls it epistemic foraging. You know, which is it's a little bit like, you know, over time, we just discover this knowledge and the cloud of knowledge just increases. Federico, it's been an honour to have you on. Thank you so much.

• 1:00:46Speaker 00

  Yeah, thank you so much.