GSMSymbolic paper - Iman Mirzadeh (Apple)

Iman Mirzadeh from Apple, who recently published the GSM-Symbolic paper discusses the crucial distinction between intelligence and achievement in AI systems. He critiques current AI research methodologies, highlighting the limitations of Large Language Models (LLMs) in reasoning and knowledge representation. SPONSOR MESSAGES: *** Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/ *** TRANSCRIPT + RESEARCH: https://www.dropbox.com/scl/fi/mlcjl9cd5p1kem4l0vqd3/IMAN.pdf?rlkey=dqfqb74zr81a5gqr8r6c8isg3&dl=0 TOC: 1. Intelligence vs Achievement in AI Systems [00:00:00] 1.1 Intelligence vs Achievement Metrics in AI Systems [00:03:27] 1.2 AlphaZero and Abstract Understanding in Chess [00:10:10] 1.3 Language Models and Distribution Learning Limitations [00:14:47] 1.4 Research Methodology and Theoretical Frameworks 2. Intelligence Measurement and Learning [00:24:24] 2.1 LLM Capabilities: Interpolation vs True Reasoning [00:29:00] 2.2 Intelligence Definition and Measurement Approaches [00:34:35] 2.3 Learning Capabilities and Agency in AI Systems [00:39:26] 2.4 Abstract Reasoning and Symbol Understanding 3. LLM Performance and Evaluation [00:47:15] 3.1 Scaling Laws and Fundamental Limitations [00:54:33] 3.2 Connectionism vs Symbolism Debate in Neural Networks [00:58:09] 3.3 GSM-Symbolic: Testing Mathematical Reasoning in LLMs [01:08:38] 3.4 Benchmark Evaluation and Model Performance Assessment REFS: [00:01:00] AlphaZero chess AI system, Silver et al. https://arxiv.org/abs/1712.01815 [00:07:10] Game Changer: AlphaZero's Groundbreaking Chess Strategies, Sadler & Regan https://www.amazon.com/Game-Changer-AlphaZeros-Groundbreaking-Strategies/dp/9056918184 [00:11:35] Cross-entropy loss in language modeling, Voita http://lena-voita.github.io/nlp_course/language_modeling.html [00:17:20] GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in LLMs, Mirzadeh et al. https://arxiv.org/abs/2410.05229 [00:21:25] Connectionism and Cognitive Architecture: A Critical Analysis, Fodor & Pylyshyn https://www.sciencedirect.com/science/article/pii/001002779090014B [00:28:55] Brain-to-body mass ratio scaling laws, Sutskever https://www.theverge.com/2024/12/13/24320811/what-ilya-sutskever-sees-openai-model-data-training [00:29:40] On the Measure of Intelligence, Chollet https://arxiv.org/abs/1911.01547 [00:33:30] On definition of intelligence, Gignac et al. https://www.sciencedirect.com/science/article/pii/S0160289624000266 [00:35:30] Defining intelligence, Wang https://cis.temple.edu/~wangp/papers.html [00:37:40] How We Learn: Why Brains Learn Better Than Any Machine... for Now, Dehaene https://www.amazon.com/How-We-Learn-Brains-Machine/dp/0525559884 [00:39:35] Surfaces and Essences: Analogy as the Fuel and Fire of Thinking, Hofstadter and Sander https://www.amazon.com/Surfaces-Essences-Analogy-Fuel-Thinking/dp/0465018475 [00:43:15] Chain-of-thought prompting, Wei et al. https://arxiv.org/abs/2201.11903 [00:47:20] Test-time scaling laws in machine learning, Brown https://podcasts.apple.com/mv/podcast/openais-noam-brown-ilge-akkaya-and-hunter-lightman-on/id1750736528?i=1000671532058 [00:47:50] Scaling Laws for Neural Language Models, Kaplan et al. https://arxiv.org/abs/2001.08361 [00:55:15] Tensor product variable binding, Smolensky https://www.sciencedirect.com/science/article/abs/pii/000437029090007M [01:08:45] GSM-8K dataset, OpenAI https://huggingface.co/datasets/openai/gsm8k

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I think if someone wants to take one important message from this talk would be understanding the difference between intelligence and achievement. The field is currently focused very heavily on the achievement and the numbers and accuracies instead of trying to understand what does an intelligent system mean? What does it mean for a system to understand, to reason? Instead of tying it to a certain number for a certain benchmark.

• 0:28Tim Scarfe?

  Iman, I think you would agree that we need to have better abstract world models, right? We need to have better representations. How's that gonna work?

• 0:36Speaker 00

  To me, it looks nearly impossible to build an intelligent system that operates without an abstract model of the environment and the world and knowledge. But right now, there are many questions that need to be answered before that. One of the issues that I have with the literature right now, and also including me, is that we don't even have basic answers to these questions. So after AlphaZero came, what happened is that the chess event became more popular and the quality of chess improved. Not because people used chess engine and just memorized moves, it's because they tried to understand what AlphaZero is doing, other chess engines are doing.

• 1:18Speaker 00

  So grandmasters actually use these tools a lot, but they don't go and memorize the moves by chess engines. What they do is that they develop theory. So they work on novel openings, they work on novel moves at certain positions, novel strategies. They learn from these tools, not in a way that just memorizing. They understand, they create a knowledge, new knowledge, new theory. In image and computer vision, we had these benchmarks of ImageNet and all those benchmarks, and we saturated them and we thought, okay, the vision is solved. But now we see that self-driving car is not becoming a thing right now because it's very difficult. Like in real world, there isn't a specific frozen cut of reality

• 2:03Speaker 00

  that with fixed number of examples. You have to build an agent that understands and reasons. So that's why I think it's important to focus on, not the exact number.

• 2:18Tim Scarfe?

  MLST is sponsored by Two for AI Labs. Now, they are the DeepSeek, based in Switzerland. They have an amazing team. You've seen many of the folks on the team. They acquired Minds AI, of course. They did a lot of great work on ARC. They're now working on O1-style models and reasoning and thinking and test time computation. The reason you want to work for them is you get loads of autonomy, you get visibility, you can publish your research. And also, they are hiring, as well as ML engineers, they're hiring a chief scientist. They really, really want to find the best possible person for this role. And they're prepared to pay top dollar as a joining bonus. So if you're interested in working for them as an ML engineer or their chief scientist, get in touch with Benjamin Cruzea. Go to twoforlabs.ai and see what happens.

• 3:01Tim Scarfe?

  You know, we all accept there are limitations in reasoning in LLM systems. And some people say, oh, it's just because there's a missing feature, a missing tweak. We just need to tweak the transformer's architecture and then they can do copying and counting and reasoning, all those stuff. Other people say, all we need to do is let them access tools, right? So for example, like, I want to have a system that plays chess. It can just use a chess computer. What could possibly go wrong with that?

• 3:28Speaker 00

  So, yeah, that's one of the most common arguments I hear and debate with many friends and colleagues and people even at NURIPS. And I don't personally have any issues with using tools. Humans also use tools. Everyone uses tools. So, but going back to our previous part of discussion, it's not about solving a task. It's not about achieving certain accuracy on a task. Then it's about understanding. It's about creating novel knowledge, creating novel goals, achieving those goals.

• 4:07Speaker 00

  So there are two arguments I have against the tool use. So one is that...

• 4:18Speaker 00

  First of all, it's not just one tool, right? So there are, like, for chess, yeah, if your only goal in the world is to do well on chess and beat some humans, yes, you can look at, I don't know, a chess engine as a tool and just give moves to it and get moves and play chess and win. So it won't be just one tool for your whole life, it will be many tools, right? So if you look at some reasoning tasks, right, like solving a math, logical problem, planning, so it involves many steps, many states that you need to navigate and learn.

• 4:59Speaker 00

  And even with tools, you won't be able to use, you won't necessarily will use only one tool, you will need many tools. So it will be like, I don't know, at some point you will need to use five tools, ten tools, and then the tools may become complicated, they might not necessarily be like you give input and you get output, it will be complex. So essentially, it will reduce to the same problem, right? What if I need to plan for ten different actions? If you look, and so at that point, you still face the same problem, right? So that's one argument.

• 5:40Speaker 00

  And for the chess, also, there is this argument that if you use tool, it's fine to use tool, definitely, but you don't necessarily be able to know if the system that uses external tool understands that task. So looking at the chess engine, you could play, you could use a chess engine and play and win, but you won't necessarily understand why did you make those moves? Why there is this move in that position? What constitutes a good position, bad position, planning? So, uh, actually, one, because I'm a, like, a huge fan of chess, I, I watch chess competitions and,

• 6:25Speaker 00

  so I follow what's going on in the chess world. Uh, what happened is that after AlphaZero came, uh, so humans, instead of like, uh, everyone thought like, the chess, uh, as a sport and a competition will be, uh, kind of, uh, will lose its importance because there is the system that wins any grandmaster in the world. But what happened is that the chess event became, uh, uh, more popular, uh, and the, the quality of chess improved. Not because people used, uh, chess engine and just, uh, uh, memorized moves. It's because they tried to understand what, uh, Alpha, uh, AlphaZero is doing, other chess engines are doing.

• 7:08Speaker 00

  Uh, so, actually, they wrote a book on, uh, understanding the moves of AlphaZero, right? So the famously AlphaZero has this move that, uh, sometimes it tries to, uh, push from the, uh, left or right corner. And these moves have not been, uh, like, seen before, right? But you could say that, uh, I don't care, this is a move that goes, uh, like, gives me advantage and wins. Or you could understand why this move, um, this kind of moves, uh, improve chess, right? So grandmasters, uh, actually use, uh, these tools a lot, right? But they don't go and memorize the moves, uh, by chess engines.

• 7:53Speaker 00

  What they do is that they develop theory. So they work on novel openings, they work on, uh, novel, uh, moves at certain position, novel strategies. They learn from these tools, but they under, not in a way that just memorizing. They understand, they create a knowledge, new knowledge, new theory, and. I think this distinction of like how humans use chess engine versus how people think we should use chess engine is very important. So if someone claims that chess engine, like you can use chess engine for your L L M as an external tool, I'm fine with that if you want to develop that system.

• 8:34Speaker 00

  But my metric for measuring how good that system is, is that if you start, if you let that system or L L M or any system use that chess engine, after a while, it comes up with a theory. So one example would be that human, like in chess, there is this principle that in the opening phase, you should have a control on the center because you have control over more squares. It allows you to, like in the next moves, it allows you to have more squares and it prevents your opponent to develop its pieces, right? So development of pieces, controlling the center are

• 9:16Speaker 00

  the principle of the chess. So if you start with any system and a chess engine, and after a while, it comes up with why it is important to control the center. Because the openings, like you could train a model on all the games and it will learn as a distribution and statistics that the first moves are for white, e4, d4, or c4, right? Like the first, the center pawns. So the reason behind that is that they want to control center, right? If it comes with that new knowledge that understands why it matters using a tool, it's fine. But if it's just, you know, using a chess engine and gives

• 9:59Speaker 00

  moves and back and doesn't have any understanding on what's going on, then all you did is that you got like high rating on chess without really knowing what's going on.

• 10:10Tim Scarfe?

  How puritanical should we be here? So you know, L L M's, they act as if they are doing reasoning in many, many circumstances, but they're prompt dependent. You give them the wrong prompt, you put distractors in there, it brits it, it doesn't work. So how do we interpret that? Do we say it's not reasoning because of the brittleness?

• 10:29Speaker 00

  The short answer is yes. So the way I look at the prompting and the way and the reason I also don't necessarily think that like exploring different prompting helps in terms of really understanding like improving the system is that I look at the prompting is that I look at L L M's as learning as a system that knows a lot, lot, lot of distributions, right? And then the prompting to me looks like conditioning those distribution, nudging those distribution to like maybe specifically steer the model towards some direction.

• 11:10Speaker 00

  And so you could use prompt to do anything you want almost. But that doesn't mean that the system is understanding what's going on behind the knowledge. So my problem with the classical way of what the way we train the system is that we look at everything as a distribution. And when you learn a distribution by by construction, our last function is that you learn a distribution that minimal and minimize the distance between your distribution and the some distribution that you infer from data, right? And once you learn that this week, that's all the model cares.

• 11:55Speaker 00

  So once you learn that distribution, you just stay in that distribution. And that the problem is there are many problems with that. The first one is that by definition, by construction, these systems won't be able to understand what is beyond that distribution, right? You by construction, we train the models to minimize the loss and stay like within a boundary of a distribution. And that's what leads to all the problems, right? You change something, system breaks. You you you wanna measure like how the models perform on like different distributions, they can't do that. And like all these problems, because that's how we trained our model.

• 12:40Speaker 00

  How do you expect the model to stay in a box? And then like you ask it like what happens outside the box? And like you you constructed that system this way. So I think that's one of the issues.

• 12:55Speaker 00

  Another issue with the way we train, I think, our models. I don't have by the way answers to how to fix this right now. But another issue is that

• 13:07Speaker 00

  when the way we train with, for example, cross entropy loss, right? So let's say you are like, you teach, you are teaching models to do arithmetic, right? What we do is like two plus two equals, there are like, assuming there are like two is a token, plus is a token. So these are the tokens. And then in the text, there is four, right? So all the model has to do is learn after two plus two equal, it should output four. And some context maybe. So it doesn't matter for, for the system that this four comes from understanding the whole natural numbers,

• 13:49Speaker 00

  the, the, the line of numbers, addition, can we do this or not? Or it's just memorization or maybe some other thing, right? So there is nothing in our loss function saying like, it's important that you understand numbers and natural numbers as a concept, which later will be grow to, I don't know, rational numbers or like real numbers and all these concepts. All you have to do, all I care is that after two plus two, you should give me four, right? And because of that, we don't know if the model can come up with that understanding and building a world model or kind of concepts of what's going on.

• 14:34Speaker 00

  But yeah, I think these are the main issues with the, like, all those, these are the consequences of, I would see, all the, all the things you are doing to train the systems.

• 14:47Tim Scarfe?

  So you said to me that you're overwhelmed by the lack of progress in, you know, the sphere of intelligence and AI research at the moment. Tell me more.

• 14:58Speaker 00

  Oh, okay. Yeah, I mean, there's like, there are two sides to this story. One is that, obviously, I think we are living in a very important time and very exciting moment. And there are lots of progress being made. Lots of people are investing in AI right now. There's lots of research being done. So from that side, it's very exciting. On the other hand, I think like, In my opinion, uh, the way we are doing research in the field is not necessarily optimal. And the way we designed our uh metrics and uh the way the system works in general is not necessarily optimal.

• 15:42Speaker 00

  So, for example, uh, right now, the the current way of doing uh research, most of the uh research is that uh people believe that uh research is an uh always an incremental pro- process. And I don't disagree with that. Uh, you you will always build on top of uh a work of other people. Uh, but my my issue with how the research is being conducted right now is that we don't uh by reading newspapers and uh learning all the newspapers, we still compared to, I don't know, two years ago, three years ago, four years ago, we still don't have a better understanding

• 16:26Speaker 00

  of uh how these systems work, right? We do uh not know that like whether we are moving towards the right direction or not. Uh, so uh, for example, like, most of the papers these days uh have a hypothesis is that like, if you do this, uh, then this will happen. And then the wrong experiments show that like, yeah, if you do this, then this happens. Then there will be another paper saying that, if you do this, then this happens. And then there will be other papers saying that all these papers had this assumption. And then if you have a different assumption, uh, this will happen. And then still, after like reading, I don't know, ten, twenty papers, you still don't have a coherent picture of like how these wo- uh, systems work, like,

• 17:10Speaker 00

  what is the uh general hypothesis? So I think what uh, what ha- what is, I think, good about many other fields like physics is that you have a hypothesis saying like, this is my model of, I don't know, a transformer, right? This is how I think a transformer works. And then after that, you say, okay, if my hypothesis is true, then there will be these consequences, right? And then I'm gonna do a uh, design an experiment and measure that. But this is not how not how things work. Uh, the the way people approach this problem is that They they look for solutions before understanding the problem.

• 17:51Speaker 00

  So I think that's one of the issues that I hope like at least people spend more time on like understanding things rather than focusing on the solution and what we can do. Uh I I I am always like uh uh. I think it's more important to understand what is what what is happening right now. What are the problems? What are the uh uh pros and cons of these systems? How do the how do these systems work, right? And then once you have a like at least hypothesis of mental model or like how these systems work, then you can uh build on top of that. But otherwise, you will maybe run into like a papers or like research that think like

• 18:35Speaker 00

  instead of trying to understand how like prompting, like you could say my model of prompting is that it will nudge the distributions, it preconditions the model. And then if there are a few distribution and a model is interpolating on that, then this will happen, right? And then you could run a few like uh test this hypothesis. But like what happens is that someone explores that what happens if you keep a language model? And even if it like increase your accuracy by five percent, you still don't know what's going on, right? Like you after reading that work that increased the performance of the model like a few percentage, like all those work, you you still don't have a better understanding of the uh models.

• 19:20Speaker 00

  And then that will be makes it very difficult to uh improve those systems, right? And then there are other, I think, uh aspects like uh uh. I think like from the theory perspective, I'm not an expert in theory at all, but I think maybe our theory is also too rigorous, I think, for the moment because the problem is too hard. And the the current theory lacks far behind the state of uh the the practitioners. And then they wanna stay rigorous, very rigorous and like develop it, but they it's very hard if the theory uh doesn't like become more relaxed.

• 20:00Speaker 00

  So it can at least be uh be like mo- prog- have a faster progress. So I think those are the uh those are the main issues. And then from the other uh aspects, I think like it's given the amount of interest and investment and uh in the A I literature. Then there are many people working on the field, which is really, really great. But uh there will be other challenges, like the reviewing process, the peer reviewing process is really difficult and noisy. Uh I don't have a unfortunate solution for that to propose here, but I think we can at least uh think about these problems. How can we improve those pros- processes?

• 20:44Speaker 00

  And uh yeah, I think these these are the main uh I think challenges of the A I literature and the A I research community right now in terms of research uh and how the research is being conducted.

• 20:58Tim Scarfe?

  How can we bridge symbolic A I and connectionism?

• 21:03Speaker 00

  Hmm, yeah, that's a uh.

• 21:08Speaker 00

  I think uh.

• 21:11Speaker 00

  I mean, uh we there are a couple of ways to think about it, like uh I think you could have a symbolic model. I'm not, right now, I'm not uh saying that like this model, like symbolic model versus uh non symbolic models or deep learning is good, bad, or like think work. Uh I I'm thinking about more like uh fundamental ways of like, given any system, is there a way to understand if this system is uh understanding a concept is able to reason? But in general, I think it is fine uh to start and like integrate these two. As like maybe we start with a separate module and then.

• 21:54Speaker 00

  Then maybe eventually, like by marrying these two fields, like it will be become more integrated and improve over time. But it's very important to like not look at these things as two separate uh system. Otherwise, it will lead to problems of, you know, using an external tool and all those things. So there should be some notions of understanding uh and like the model itself should be able to have a word model should be able to understand if I need to update this or not. So one uh one other thought experiments that I have is that like

• 22:34Speaker 00

  imagine right now that we are talking, I tell you the area of a square is P R

• 22:42Speaker 00

  to the half, like the power of half, I don't know, something like that. P time, pi times two R, right? Uh so you wouldn't accept my argument, right? Like you could prove mathematically that this is not correct. But if I tell this uh the area of uh the area of squares uh pi R to the power of three uh to an L L M, because there is no module in the model system. That's saying like, is this making sense? What is, what do I think about the area of this uh circle? It should ideally won't be, it shouldn't accept it. But we don't have that system. So I think that's one of the things that I think uh one of the examples of why we need another component,

• 23:26Speaker 00

  why we are missing at least one component that after a while learns uh uh comes up with its own belief system and like knowledge representation. And if when something comes, it may update its belief system or may disagree. I wouldn't uh agree that, I don't know, the area of circle is something else because I can't prove it to you right now. That that's it. If you have another proof, I will read it, but I will doubt that that proof is correct, right? So. That's very important, but there are other belief systems that I'm open to change. Like the best, the best place to eat in Vancouver, I'm open to change and try your, your suggestion.

• 24:12Speaker 00

  Uh, so I think it's important that the, the systems should to not be separated and like be looked at as two separate components. It should be a like integrated loop.

• 24:24Tim Scarfe?

  Yeah, and I think it is, it is interesting that you say that humans don't always reason, but we're capable of doing some privileged form of cognition that we might call reasoning. And also you could argue that the L L M's have a spectrum of modes. So sometimes they're doing surface statistics and maybe in certain circumstances, they're doing at least some approximate version of reasoning. So there seems to be a spectrum. Tell me about that.

• 24:49Speaker 00

  Oh, it's, yeah, uh, I think, uh,

• 24:55Speaker 00

  if you look back, uh, obviously we can admit that, uh, kind of the, what these systems today are capable of, uh, kind of surprised the field and everyone, I think. Uh, but I think, uh, we kind of sometimes, uh, get, uh,

• 25:16Speaker 00

  read too much into it and what these systems are capable of doing. And, um, the fact that like there, there, some claims say that the models can reason and like there's a spectrum. Uh, I'm not sure about that actually, that, uh, these models can system. What they are good at, uh, I think is, uh, doing some sort of interpolation. So it's not that they're just memorizing things. They can't do anything beyond training data. They learn many things from different places and they learn different distributions. And they do interpolate, uh, between these, uh, things that they've learned.

• 25:56Speaker 00

  So I think that's what, uh, gives, uh, I think, uh, the, uh, illusion of, uh, what the, these systems are capable of reasoning. And, uh, if the domain is very limited, uh, and closed. For example, a specific domain where the overall space is closed. Uh, the interpolation uh of the model seems enough uh to uh assume that these models are capable of performing in that domain. But I think um in general, uh, there is a very huge distinction between the achievement and capability of a system and the intelligence and other like

• 26:40Speaker 00

  interning the uh capabilities of the system. So these two are different. But currently, what we do is that we mix those two together. So for us, uh, uh, right now, it's like if a model performs good at a coding benchmark, it means that it has reasoning. If it does good uh on math benchmark, it means it's doing like some sort of reasoning. And when we mix this together, it's uh, it makes uh arguments and like uh, understanding these systems very difficult. So I think it's important to discuss uh the distinction between intelligence and achievement.

• 27:21Speaker 00

  So uh, intelligence uh by uh, by default, it means like by definition, it means uh about the capability of system, not and how it can uh grow and at some point eventually becomes capable. Not necessarily what uh it is uh, how capable it is right now, right? So achievement is about measuring what the uh system is doing uh on a specific domain or task or uh benchmark. So these two are different. So the, if a, if a system is intelligent, eventually this uh, it will be able to do well on some sort of benchmark.

• 28:06Speaker 00

  And uh but necessarily the other way around is not uh correct. So if the model is doing well on a benchmark, it doesn't mean necessarily it is intelligent. But because it is hard to measure, right, to the intelligence in general, this is an open question. We we set a benchmark and say an intelligent system should do good on this benchmark. And then when a system gets good, we say this is doing reasoning. Uh so these are different. Uh I think intelligence uh is about uh

• 28:43Speaker 00

  is about how a system in in the long run is able to uh uh perform well. So um there was a good uh very interesting uh picture uh in Ilya's talk at NURIPS this year. And and that figure showed like uh the there was a uh regression line between the body mass and brain mass. And that uh figure showed like uh how the uh different species scale in that sense. So my take from that picture was that intelligence is not about where you are in that scale right now. It's about how like the slope of uh that figure.

• 29:26Speaker 00

  So if you are thinking about the scaling laws, intelligence is the slope of uh scaling, not what point you are at.

• 29:35Tim Scarfe?

  Yes, I I I love all of this. I mean, um you probably know I'm a huge fan of Francois Schrulli. And he's always at pains to say that intelligence is not skill. It's adaptation to novelty, it's skill acquisition efficiency. Um so that there's something about the the macro adaptability which is important. And actually, um when we were discussing this before, you came up with your own test for intelligence, which I'm gonna call the the the Iman Moon test. Which is, you know, basically, um imagine we start from cavemen and we land on the moon and and and sort of how quickly can we do that?

• 30:11Speaker 00

  Yes, so yeah, that that's and that came from a thought hypothesis, right? So uh just to again reiterate the distinction between intelligence and achievement, right? So imagine uh a caveman. Uh. So if you go back in time and give MMLU or G S M eight K to a caveman, I would argue that the performance would be near zero. And current models get near like eighty percent, ninety percent accuracy on these benchmarks, right? And do we really believe the current systems that we have are more intelligent than a caveman? So intelligence is about the system that a uh a human uh possess,

• 30:55Speaker 00

  not how well it's doing on a benchmark, right? Even if you go even further, like uh some great figures in history and science, like Aristotle, right? If you give M L U to Aristotle, I don't think it will perform nearly as well as uh current L L M. But do we really believe uh those L L M's uh that we have are more intelligent? Because intelligence is not about how much you achieve on a certain task. If it's about how well that system, that human can, if it spends time, learn something and grow.

• 31:33Tim Scarfe?

  So how could we measure this? Because you know, like, Sholay had this formalism for for measuring intelligence, but it it wasn't computable. And benchmarks, they just get saturated, you know, that they they get good hearted. How can we build a new suite of benchmarks?

• 31:48Speaker 00

  Yeah, that that's uh, I think, I don't have a uh concrete answer to that. I'm thinking about it a lot. Uh, but uh, I don't think uh right now I do have an answer to that. But we can maybe not, we can at first uh start with not something that is not necessarily objective. But maybe we can start by asking, what kind of system do we want? What does intelligence mean to us? What does uh achievement mean to us? And then uh if we maybe start with a kind of axiomatic definition and characteristics,

• 32:28Speaker 00

  then we can, at first it won't be objective, and I know it will be not concrete and open ended. But if you start thinking about these important problems, we may come up with some uh uh some way of measuring it. Uh. So it is uh. In general, measuring intelligence is very difficult. It's even for humans, it's not trivial to measure intelligence of humans. But I think we can have some, define some desired properties that we want from a system. So those desired properties might mean something like the system being able to perform well on novel tasks.

• 33:12Speaker 00

  And then definition of novel itself is not easy, but I think we can start from something here and then build on top of that and think more on that. So because there was a nice paper by Gilles Gignac et al. And it's called on definition of intelligence.

• 33:38Speaker 00

  I think that was a really good read. It's not in like circulated very well in the machine learning community because they have background in psychology and cognitive science. But it's really, really nice paper. It defines very formally what is the definition of intelligence? What is the definition of artificial intelligence? What are kind of forms of intelligence we are looking for? And we can start from there and think more on those problems. And maybe we come up with something better.

• 34:15Speaker 00

  So yeah, that's what I'm thinking on these ways. But in general, I think the direction should be towards measuring novel tasks. So for example, if you're training a model to do coding, at least what you could do is that for me, a better measure would be how fast that system can learn a new programming language from scratch, right? So a system like NLLM could do very well on Python or all the languages it's been trained on. If I create a new programming language and ask they to write a new language, a new program in that language,

• 34:56Speaker 00

  uh how fast it can learn that language and how good it can uh uh write programs in that language, compared to a human that never seen a uh language before. Because I think one other things that we can do is that compare the uh how he how humans learn and reason and compare it and contrast it with how these machines uh are learning and reasoning. Uh so, yeah, I think we can think towards that direction, but I don't have a concrete.

• 35:26Tim Scarfe?

  Yeah, it's very slippery. Uh uh there's a great paper by Pei Wang um on defining intelligence and and he said that it's very anthropocentric. And, you know, in different fields, people use different techniques. So, you know, we we could have this basically a a copy of a human brain. And that's the most anthropocentric. That's not particularly useful, is it? Or we could have notions of behavior or more abstract capabilities, so being able to write Python programs um or functions, so having things that do planning and reasoning and all these abstract cognitive functions, or maybe even having things that have certain principles like, you know, um um emergence or certain characteristics or something like that. Very, very slippery. But just before we move off that, you you cited this paper from psychology. I'm I'm very interested to know what

• 36:11Tim Scarfe?

  what their definition was. For Pei, his definition is very much about adaptivity. That that seems to be one of the core things for him.

• 36:19Speaker 00

  So, yeah, uh actually, uh that paper discusses lots of other uh works on uh studying and understanding and defining intelligence. I highly recommend that paper. Uh so, yeah, uh the the focus of that paper, the main uh uh emphasis is on novelty. So.

• 36:41Speaker 00

  The the definition of intelligence in that uh paper is that the maximum capability and capacity of a system uh to achieve a novel goal uh given some time, right? Uh obviously, you can define many uh different uh programs and like uh systems that given infinite time, eventually they could uh reach to a goal. But uh the the time is also important. That's why I also mentioned the slope of scaling, why and why it matters.

• 37:15Tim Scarfe?

  Do you think agency and autonomy comes into intelligence? So you know, basically the ability to set your own goals? Or do you think we should think of intelligence surely, you know, purely in the frame of this is a thing, it has a goal and can it achieve it?

• 37:26Speaker 00

  No, I think uh actually uh agency and the system are having uh some concept of what uh it wants is very important. So I was reading a book uh recently, it's called uh How We Learn, uh Why Humans Uh Learn Better Than Machines For Now by Stanley Duan. And in that book, uh he discusses lots of great uh uh topics and how humans learn, how machines learn, and it comes from neuroscience background. So lots of new insights, at least for me. And one of the things that uh seemed really interesting to me was that um that one of

• 38:11Speaker 00

  the pillars of learning for humans is active engagement. And essentially, you, some humans like uh have very uh it's very hard for humans that to just observe and don't get involved and just watch something to learn compared to having them uh set their own goals, being able to actively engage in the environment, explore, maybe later exploit and all these things. So uh without that, that's actually, according to that book, is one of the like necessary conditions. Yeah, it's not something that is good to have, it's something that you must have. So I think that's uh one of uh the important topics and I think.

• 38:55Speaker 00

  For me, the the the implication was that uh if you want to build a really intelligent system, uh supervised learning won't be enough. So uh you need an agent to ask for what should I do then next and be able to ask questions. It's not like you just observe and see, okay, I learned this, I learned that. It's sort of, it should be something like, now I learned this, what do I want to learn? I maybe I haven't understood this well enough, I I need to explore more and like.

• 39:27Tim Scarfe?

  Let let let's explore this, this is fascinating. So um controlling the center is is an abstract category, right? It's it's an analogy and Douglas Hofstadter said that, you know, um a concept is a bag of analogies, essentially. So there's, moving to the center is something that we all understand at some abstract level, but what does that actually mean? I think many concepts in chess are quite fuzzy, right? So what that actually means in many, many different situations is is slightly different. So the question is, what's the difference between um AlphaZero, it discovered this move thirty seven, and if you play it millions of times, this move thirty seven in its behavior space emerges as a mode?

• 40:09Tim Scarfe?

  So you could argue that it is doing reasoning, but it's doing an emergentist form of reasoning. So rather than having some notion of in the center, its behavioral profile acts as if it has a notion of in the center. And how is that so different from the way our brains work?

• 40:26Speaker 00

  Hmm. Yeah, I think um so again, like uh that if so there there is this thing, like AlphaZero, AlphaGo and all these systems are able to understand the environment they're put in, right? Uh so they they like they explore the environment, they play uh uh a lot, and they are able to eventually uh explore all the, like, most of the kind of uh positions in the Go or chess or many of them. So uh eventually they will be able to take rational decisions because they understand the value of each uh state and do that.

• 41:10Speaker 00

  But uh that doesn't mean they come up with this concept. So for for.

• 41:16Speaker 00

  For a chess engine, it is that this move improves the value of this position, right? But it doesn't necessarily transfers to other aspects of life, right? So in chess, for example, you could argue that controlling center means controlling important or strategic positions, right? You could, you could, once you learn this, you could also apply it to other aspects, right? This is not just about four or five squares in the center of a chessboard. It's about importance of specific positions in different parts.

• 41:55Speaker 00

  So it could be about controlling an important road in a country, controlling an important passage. So that, that, that form of abstraction is, I think, and the way we store knowledge and learn is what allows us to scale very fast. Once we learn this in chess, we are able to scale, like use it in across other domains. But systems like AlphaGo or AlphaZero are not able to do that because the world to them is just that and they don't have this kind of abstract representations and knowledge. So I think, I'm not saying that learning abstractions is necessarily the only way

• 42:38Speaker 00

  that you could build an intelligent system, but certainly it's helping humans to scale way faster and improve faster than machines.

• 42:50Tim Scarfe?

  On, on the, the matter of generalizing out of distribution, so what we're starting to see is, I mean, obviously chain of thought is, is used a lot, which means that you can take something which is trained on a distribution and you can kind of like manipulate, I suppose you're, you're doing some kind of directed retrieval to create, you know, compositions in, in some fixed way. So there's, there's chain of thought prompting. Some people are doing program induction. And it certainly seems to be the case that with a few examples, maybe with chain of thought, you can get a language model to induce a very rich and diverse set of programs. And many of those programs seemed to have some kind of abstraction.

• 43:32Tim Scarfe?

  Another thing we're seeing is transduction, which is where you do some kind of active fine tuning. So you take the the test instances and then you you kind of like modify the existing model a little bit so that you get generalization in this domain. So there's loads and loads of approaches that seem to be working quite well for making these things do better. O O D. What do you think about that?

• 43:55Speaker 00

  Yeah, OK, uh, so, yes, uh, so you could do, like, there are lots of ways to explore this, right? Like chain of thought, program synthesis, all these topics. So my uh, my question, which I'm trying to think about like these days a lot about, uh, is that these, all these systems, all these methods, all these directions have are built on some assumptions. And the assumption is that there is something going on in the model, some kind of uh, understanding that we need to improve that, right?

• 44:35Speaker 00

  Uh, I'm not quite sure if uh, right now there is that uh, if that assumption is correct, right? So I'm not saying that it's correct or incorrect, I'm not sure about it. So if the that assumption is incorrect, then all the things you are doing, uh, chain of thought, uh, all the other methods for improving this kind of reasoning uh, uh, may not be helpful, right? So I'm like, what I'm thinking about like is that like, what, like taking a step back and really asking this question, like, is there anything there, like beyond?

• 45:14Speaker 00

  Some interpolation between distributions, right? If there is, then sure, we can continue and build up on top of that. If there's not, then what are we doing? Like, uh, why are we trying to, uh, do all these cons- uh, like, methods that? It's very difficult, by the way, like, like, uh, because we are also doing it in a kind of ad hoc way. There is no coherent model of what the transformer model is doing at scale. Uh, so it's very, very tricky, I think. Uh, so right now, I'm leaning towards that the, the models don't have a correct representation.

• 45:56Speaker 00

  And the way we build these models by construction is, uh, limiting. So I don't really see how, uh, other, like, extending this, uh, would be helpful. Uh, like, there was an old example of, like, uh, if you wanna land on moon, you can do it with an airplane, right? So, uh, there, there are two questions. If you have an airplane and you wanna, uh, land on moon, then, uh, you could improve, like, the speed of plane. You could make it, uh, lighter, you could, like, make it go faster or all these things. Uh, but it doesn't necessarily help you land on the moon.

• 46:37Speaker 00

  So you could also work on, like, how would, uh, like, wings should look like? How would, like, uh, the, the, what kind of, uh, uh, runway should we use to have a faster takeoff? All those things, uh, but that doesn't necessarily help you, uh, with your goal, right? So I'm saying, like, I'm not saying that, uh, it is impossible, like, right now. I'm saying that, like, we should ask a question, is this the right, uh, vessel? Is it like, uh, does it make sense? So, yeah, in general, it's, I think, uh, we should take a step back and think harder on that problem.

• 47:15Tim Scarfe?

  Onto scaling laws. So, uh, you know, Noam Brown is talking about this, this new, um, test time scaling law. Um, you've said that it's very, very difficult to convince people that, that scaling laws, you know, don't work. Because you'll always have a group of people that say, oh, you know, just over the next hill, just, you know, if we, if we ten X the parameters, ten X the compute, then we'll finally get there. How do you convince people that they're wrong?

• 47:40Speaker 00

  Oh, yeah, I mean, I can't really convince, because the the the issue with the scaling law is that if I spend millions or billions of dollar and train a model, like, I don't know, ten trillion parameters on one hundred trillion tokens and come and show you, hey, this is the model, and it can't do even simple mathematics beyond what it's been trained on, right? It can create new knowledge. And then someone can claim that, hey, eh, this is not the right scale. Like, if you scale it to fifty trillion parameters, there will something emerge. And then I, because my my argument is not theoretical, I can't say, no, you're wrong.

• 48:23Speaker 00

  And because I can't be, I won't be able to train that model at fifty trillion and spend a few billion dollars, I can't say you're wrong. So there will be always this debate that something will emerge. And by the way, I don't have issue with scaling, right? Like, scaling as in a sense that, like, I'm not saying that it will hurt or it won't be helpful at all. I'm saying, like, if you are thinking about the scaling, that that you should focus on the slope of your scaling. You shouldn't care about where you are right now, right? So if you wanna compare two different systems, or like, one architecture with another, you shouldn't care about, like, uh, one, what happens at the one hundred trillion, or will something emerge?

• 49:08Speaker 00

  You should say, look how fast these models will be able to learn and do novel thing. So right now, I think, in that sense, the slope is near zero for all the architectures and methods we have. But we can focus on the slope rather than the points and endpoints.

• 49:24Tim Scarfe?

  On the slope, then, how how does the slope of human scaling compare to L L M scaling?

• 49:29Speaker 00

  Uh, yeah, that's one of the interesting things, like, uh, we won't, uh, we weren't able to do that. But it would be interesting to, there, I think there are many studies, I'm not an expert in the field, but there are many studies of, like, uh, showing that how humans versus, uh, models, uh, react to novel environments, right? Uh, so if you compare those, I think, I don't have any number, but I think there are many studies showing that humans can adapt to new, uh, environment. They can learn new environment very easily. Much easier than uh at least uh L L M. Uh so I think the slope of human is faster uh like uh better than the slope of L L M.

• 50:12Speaker 00

  Uh but it could be that there would be a better uh species with better scaling, better slope. And I I look forward to it.

• 50:23Tim Scarfe?

  Very cool. So um Iman, I think you would agree that we need to have better abstract world models, right? We need to have better representations. How's that gonna work?

• 50:31Speaker 00

  So I think uh yeah, I mean it to me it looks nearly uh

• 50:39Speaker 00

  impossible to build an intelligent system uh that operates without uh without an abstract uh model of the environment and the world and knowledge, right? Uh but

• 50:54Speaker 00

  right now, there are many questions that need to be answered before that. And we don't even have like that one of the issues that I have with the uh literature right now. And also including me is that we don't even have basic uh basic uh answers to these questions. So imagine you wanna learn a function, right? So one way to represent that function is representing with table, right? Like input X and output is Y, right? And then imagine there is another way of representing uh the function is representing it in a way like this is a polynomial, right?

• 51:35Speaker 00

  Like uh Y equals X squared plus some number, something like that, right? And we we don't know like right now, there is no objective measure. And we don't even have an answer to that, which one is better, right? You might come say like uh the the second one where it has a form, it's like it's abstract or like something like that, uh this one is better. But we don't know, like uh why would someone claim this one is better than the other, right? I also believe the second one is better, but uh we don't have a like, we couldn't quantify this, right?

• 52:14Speaker 00

  The second one is certainly not like someone can can claim that the second one is compressed in a compressed form. You don't need a table, you just need a polynomial. But uh I don't think that's a right uh.

• 52:29Speaker 00

  That's necessarily correct, because in order to represent something with polynomial, it's not about the number of characters in a string. It's about concepts, right? So in order to define a polynomial, you need to understand what does a function mean? What does function x squared mean, x to the power of n? And then that requires understanding what does continuity mean? What does a real number mean? Like, all these concepts. So if you wanna encode everything that it takes to reach to y equals x squared plus some number, then that maybe even is being being larger than the table, right?

• 53:09Speaker 00

  And then if your only goal, again, is achieving something, referring back to our previous discussion, then these two are equal, right? You give input, and both of them give you output, and that one is even faster, you just have a lookup. So we don't have a way of comparing two representations. So if I wanna sum up, is that in order to answer all the questions you ask, we need to start with understanding what, like, how to compare two representations. Once we do have, then, if you have a good measure of comparing two representations, then we can build on top of that. But we don't currently have that.

• 53:50Speaker 00

  Actually, I was talking to a friend, like, how do we compare these two representations? And he said, like, I don't have a measure. But similar to the idea of, like, at some point in time, someone someone came up with the idea of, like, formalizing the concept of generalization. Before, like, 200 years ago, like, generalization wasn't a concept in a thing. It was a word, but, so it says, like, one representation could be more beautiful than the other, right? Like, A polynomial is more beautiful than a table. But we don't have any formal way of defining what constitutes something being more beautiful. But we are thinking about these problems all the time, so.

• 54:34Tim Scarfe?

  I guess the core here is whether deep in your bones you are a connectionist and you believe that it's possible in principle for this kind of thing to work. Because obviously, when we'll talk about your paper in a minute, you've designed a symbolic test to prove that certain types of reasoning can't happen in, or don't happen in an L L M. In nineteen eighty eight, there was this famous connectionism critique by Fodor and Polish. And they argued that these systems don't have systemat化ity, they don't have compositional generalization, they don't have invertabilities, they can't, you know, explain why they did things, they can't canalize their knowledge in the way that you're talking about. But other folks like Smelensky, get his name right, in nineteen ninety, and certainly people like Benjio, they argue that, yeah, it's not a problem.

• 55:17Tim Scarfe?

  You know, all of this symbol use can just emerge, you know, at some level of complexity or whatever. So do you think that in the future we can make L L M's do this kind of symbolic reasoning?

• 55:29Speaker 00

  Well, I think when we are talking about symbols, like the notion of symbol is not, like, we have to define the notion of symbol. Like, is symbol something concrete and predefined? Or something like, do you agree that, maybe do you accept the, like, medial activation of a model? A deep learning model can represent a symbol or not? So if you agree on that, that could be looked at as a formal symbol, then I think, yes, the model will be able to do that. Otherwise, if you think that symbolic is like the form of, like.

• 56:11Speaker 00

  Separate external system, then no. I think I I'm more leaning towards like uh symbols being emerged as a part of uh uh computation inside and.

• 56:25Tim Scarfe?

  Oh, that's very interesting. Yeah, because I suppose that there's a spectrum of views. I mean, that there are some people who think that um just pure connectionism, sim- symbol use can emerge. Um. There are people who are quite sort of like in the middle. I think I'm in that camp, which is that we can build L L M systems. So maybe um uh sort of agent L L M's that talk with each other, or maybe neuro-symbolic architectures where the L L M's can can use tools or some combination of of the above. But but you're you're in the camp that we could in principle just have a pure play neural network and it could do, you know, symbol use.

• 57:00Speaker 00

  Yeah, I mean, ru- uh yeah, right now I don't have like any re- I I don't see like in theory at least, right? Like, I don't see in theory why such a system won't be able to uh create that kind of symbols theoretically. But like, it could at some point in like create symbols and do computations on those symbols. I'm not saying like this is the only way. We shouldn't have anything like not very experienced in that sense. Uh but I don't have uh this kind of thing. The same goes with like other like like when we when I discuss like architectures, right?

• 57:39Speaker 00

  So I don't, I don't know if the like maybe transformer is the right architecture or not. But right now, in theory, at least given the context and assuming the model is generating tokens because it becomes too incomplete. I don't have any reason in theory to believe that's a limitation right now. It could be that other uh architectures are better, like we can develop better architectures. It's just that right now I don't see that as a limitation.

• 58:09Tim Scarfe?

  Okay, but you do think we need to have Turing completeness for simple use? So we could somehow come up with a neural network which is Turing complete?

• 58:17Speaker 00

  Yeah.

• 58:17Tim Scarfe?

  Okay. We're not there yet, but maybe.

• 58:20Speaker 00

  Yeah, I think like, yeah, when we're defining intelligence, like at some point, like Turing complete, like being able to do Turing complete is like a operation is like a necessary condition.

• 58:32Tim Scarfe?

  Yes, I completely agree with that. We should talk about your GSM symbolic paper. So this is a landmark paper. It did the rounds on socials. Millions of people in my community said that I had to interview you because it was amazing. Sketch it out for me.

• 58:47Speaker 00

  Oh, okay. So a little bit of background, like we were working on understanding reasoning and we were exploring a few ideas about improving the reasoning of the models in terms of like improving the, like increasing the amount of computation a model does per output token. And for that, we needed to evaluate the model and also be able to have a robust evaluation and then eventually have maybe better training data. So we started with the evaluation, like creating a small sample from GSM symbolic just template.

• 59:30Speaker 00

  And then we're just to do a static checks, we ran the experiments and see just like how close these numbers are to GSM 8K. And we observed that there is a huge, for some models, there is a very huge gap. For example, 52 had like 20% gap, 14% gap, something like that. And then there's also huge variation, right? So, yeah, we like kind of got sidetracked and like explored this immediately to understand like what's going on, right? And one of, one of the second, like after we had the initial GSM symbolic, the second

• 1:00:14Speaker 00

  thing I was working on was the GSM NOOP version, where you try to trick the model essentially and that you add one clause.

• 1:00:28Speaker 00

  To the question where that clause carries no logical or like arithmetic operation. It's like if you completely ignore that clause, you will be fine, right? So that's why it's called GSM no op. And then that was the second one. In the paper, we have a different presentation with like uh uh we explored the second. And then we observed a very huge uh uh performance draw. So then we came back and tried to understand like something in between. Like we try to uh uh be more uh like try to create like easier benchmarks than GSM no op. By just adding one clause that carries operation and then adding two clauses that carries operation.

• 1:01:13Speaker 00

  So you can't ignore that and see how the models are performing that. And to like among all those uh benchmarks and experiments that we designed. The the the surprising thing to me at least is that the the the variance of the performance. We observe across the model. So I wish we could at least do some sort of uh cross evaluation across students. Of how students would perform if the uh if you change the me like the words in the problem. Without changing the numbers, right? Like if you instead of saying Amy has three apples, you can say John has three bananas, right?

• 1:01:58Speaker 00

  They're the same like in terms of logical uh reasoning. And we I wish we could do that. Uh and I think I hope someone else explore this thing like how would human student perform? If you just change orange to banana.

• 1:02:14Tim Scarfe?

  Yeah.

• 1:02:14Speaker 00

  Uh compared to an L L M. And so to me, the the the most fascinating part is the the variation in in the sense that. Uh why does this like gap exist even though you only change the numbers? So we have a benchmark called GSM names where we only change the proper names of the question. We don't uh change the numbers or add anything to the question. So, yeah.

• 1:02:43Tim Scarfe?

  The front end model, I mean, to be honest, they still did. Quite well. I mean, I would expect them to just like drop off. Did you know Sabaro Kamahati has done some experiments with planning? And he did this mystery blocks world where he just changed the name of the symbols to like random things and it dropped off a cliff. And the O one model kind of like, you know, made it up to non trivial performance again, but there was a huge decline. And I guess, reading your paper, I was still somewhat intrigued that the frontier models, they did obviously like much worse, but it was still pretty like non trivial performance.

• 1:03:20Speaker 00

  There are two things, like personally, I don't know, like, I think overall the GSM eight K should be very simple benchmark. Given, we also, there is also this factor that we don't exactly know on what kind of data these models have been trained. It could be, I'm not saying it is, but it could be the case that these models are also being trained on some sort of synthetic data generated from

• 1:03:49Speaker 00

  questions similar to math four problems like GSM eight K. And we also know that like all the companies that are trying to build L L M, they are also getting lots of human created question answers in math and other domains, right? So we don't exactly know, like, that's one of the issues with not being like transparent about how what kind of data, at least the model's been trained on. But it could be the case that the frontier models have like access to better data quality and data format similar to GSM eight K. And it could be that they have unlocked some kind of emergent ability.

• 1:04:32Speaker 00

  But if that were true, then at least, I'm not a believer in benchmarks, but there are again different benchmarks that are similar to GSM eight K, slightly more difficult. And in that, those benchmarks, I think someone tried a similar idea that we did on GSM symbolic on math dataset Hendrix math. And then they observed a larger performance drop for frontier models as well. But overall, it's. Again, for me, it's not about performance. It's not about, like, I prefer a model that if you make the question difficult, the performance drops ten percent.

• 1:05:13Speaker 00

  But there is no variation, because if it understands the question, it understands the question. To a model that drops like one percent, but there is a like large variation. So it's, to me, it's not about the number of, like, the accuracy number. It's about why would a system that understands, is able to do a set of logical steps, tries to do differences of logical steps and gets it wrong if you change orange to banana in a question, right? And that probably is coming from because we trained the system to learn a distribution. There is no concept of, in this question, it doesn't matter what kind of object that is.

• 1:05:57Speaker 00

  It's, it's about the number of objects, right? So these two for us seems trivial, but for a system that is being trained to predict the exact an object in the question during the training, uh, it's non-trivial. So, yeah, I think overall, uh, in all these systems, uh, I don't think they dare truly understand, uh, this concept. Like, you could change any, like, I wasn't like doing a search on those kind of things. But you could, like, if you spend enough time, any question that they get them wrong.

• 1:06:38Speaker 00

  I truly believe if you spend enough time, you could change those, that question to a form that the models, including frontier models, get them wrong. But unfortunately, the field looks at these things as an accuracy number, not necessarily what does it mean for a number to drop, or what does it mean for a variation to increase? And they kind of, now, I think the field is trying to move from GSM HK because the performance is reaching like ninety-five percent, near hundred percent. And now I think like yesterday in the workshop, there was like, oh, we have a new difficult benchmark called frontier math, and this has been designed so that's like now frontier models get ten percent. Now let's increase that ten to ninety.

• 1:07:23Speaker 00

  I feel, I, like, my problem with benchmarks is that they always, they're like a cut of reality. And once you freeze that, you could change the system in a way that indirectly you changed the system, but it has like impact on the performance. So eventually that benchmark will be always saturated, but it doesn't, at least to me, it doesn't matter. Like that system, unless we change something fundamental, that system may get ninety nine percent on that benchmark, won't be able to create new knowledge, won't be able to understand what's going on. And to me, that's what matters, right? So in image and computer vision, we

• 1:08:06Speaker 00

  had these benchmarks like ImageNet and all those benchmarks, and we saturated them. And we thought, okay, the vision is solved. But now we see that self-driving car is not becoming a thing right now, because it's very difficult. Like in real world, there isn't a specific frozen cut of reality that with fixed number of example, reality will change. You have to build an agent that understands and reasons. So that's why I think is important to focus on, not the exact number.

• 1:08:38Tim Scarfe?

  Yes, and benchmarking is a big problem. So as you say, I'm sure many of the frontier models have basically memorized GSM AK. And certainly, we should be moving towards more of a generative benchmark type thing where, you know, of course, it's not deterministic, but we have some generative system and we sample from it enough times and we report some kind of average or something like that. But you did show some very interesting stuff. So first of all, there was, I think, a change on the no-op dataset where you actually H-shot it, which means the model should be able to kind of filter away the distractors, but it didn't, which was very, very curious. But more broadly, though, isn't it interesting that you sample from these models a whole bunch of times and you see this huge variation?

• 1:09:22Tim Scarfe?

  You know, and what's the implication? Is the implication that when we use language models for doing reasoning, we should actually be kind of like sampling a hundred times and taking the average result or something? Because many of us use language models and we just sample once. We just assume that, oh, it's doing reasoning, that's the right answer. And we don't really think about, well, if I asked it another hundred times, it would give me a bunch of different answers.

• 1:09:43Speaker 00

  Yeah, I mean, uh, so, yeah, I mean, about sampling, uh, there are a couple of things. Like, sometimes sampling in general doesn't make sense. So again, back to our example, two plus two equals, there's no sampling, it should be four, right? Like, if you increase the temperature, it could become five, but why, why do you wanna do that? Like randomly picking a number doesn't make sense at all in that sense. That's why, yeah, for our study, we always stick, uh, to, uh, greedy decoding, no sampling. Because like, if you are doing arithmetic or you're doing reasoning, it has to be, uh, greedy. But in general, the other argument against, uh, sampling and, uh, majority booting and all those, uh, kind of thing is,

• 1:10:27Speaker 00

  uh, the example, I remember you had, uh, uh, an episode, uh, where you, there was an example like, if you, uh, let ten thousand drunken people go to, uh, uh, after the bar go to home, eventually some of them may reach the home, but that doesn't mean they are able to like, understand what's going on. Like, given enough samples, you will eventually, uh, reach to destination probably. But, uh, that doesn't mean that's, uh, that's the issue. So actually, there was, uh, a work that argued exactly the same thing. Like they said, like, if you measure the GSM eight K and math accuracy and you, uh, sample the model hundred times, it will be like twenty percent better.

• 1:11:12Speaker 00

  But for the same argument, I don't think that's what we should study.

• 1:11:17Tim Scarfe?

  Cool. Well, um, Iman, it's been an absolute honor to have you on MLSD. Thank you so much for joining us.

• 1:11:22Speaker 00

  Thank you. It was great talking to you.