Test-Time Adaptation: the key to reasoning with DL (Mohamed Osman)

Mohamed Osman joins to discuss MindsAI's highest scoring entry to the ARC challenge 2024 and the paradigm of test-time fine-tuning. They explore how the team, now part of Tufa Labs in Zurich, achieved state-of-the-art results using a combination of pre-training techniques, a unique meta-learning strategy, and an ensemble voting mechanism. Mohamed emphasizes the importance of raw data input and flexibility of the network. SPONSOR MESSAGES: *** Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/ *** TRANSCRIPT + REFS: https://www.dropbox.com/scl/fi/jeavyqidsjzjgjgd7ns7h/MoFInal.pdf?rlkey=cjjmo7rgtenxrr3b46nk6yq2e&dl=0 Mohamed Osman (Tufa Labs) https://x.com/MohamedOsmanML Jack Cole (Tufa Labs) https://x.com/MindsAI_Jack How and why deep learning for ARC paper: https://github.com/MohamedOsman1998/deep-learning-for-arc/blob/main/deep_learning_for_arc.pdf TOC: 1. Abstract Reasoning Foundations [00:00:00] 1.1 Test-Time Fine-Tuning and ARC Challenge Overview [00:10:20] 1.2 Neural Networks vs Programmatic Approaches to Reasoning [00:13:23] 1.3 Code-Based Learning and Meta-Model Architecture [00:20:26] 1.4 Technical Implementation with Long T5 Model 2. ARC Solution Architectures [00:24:10] 2.1 Test-Time Tuning and Voting Methods for ARC Solutions [00:27:54] 2.2 Model Generalization and Function Generation Challenges [00:32:53] 2.3 Input Representation and VLM Limitations [00:36:21] 2.4 Architecture Innovation and Cross-Modal Integration [00:40:05] 2.5 Future of ARC Challenge and Program Synthesis Approaches 3. Advanced Systems Integration [00:43:00] 3.1 DreamCoder Evolution and LLM Integration [00:50:07] 3.2 MindsAI Team Progress and Acquisition by Tufa Labs [00:54:15] 3.3 ARC v2 Development and Performance Scaling [00:58:22] 3.4 Intelligence Benchmarks and Transformer Limitations [01:01:50] 3.5 Neural Architecture Optimization and Processing Distribution REFS: [00:01:32] Original ARC challenge paper, François Chollet https://arxiv.org/abs/1911.01547 [00:06:55] DreamCoder, Kevin Ellis et al. https://arxiv.org/abs/2006.08381 [00:12:50] Deep Learning with Python, François Chollet https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438 [00:13:35] Deep Learning with Python, François Chollet https://www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438 [00:13:35] Influence of pretraining data for reasoning, Laura Ruis https://arxiv.org/abs/2411.12580 [00:17:50] Latent Program Networks, Clement Bonnet https://arxiv.org/html/2411.08706v1 [00:20:50] T5, Colin Raffel et al. https://arxiv.org/abs/1910.10683 [00:30:30] Combining Induction and Transduction for Abstract Reasoning, Wen-Ding Li, Kevin Ellis et al. https://arxiv.org/abs/2411.02272 [00:34:15] Six finger problem, Chen et al. https://openaccess.thecvf.com/content/CVPR2024/papers/Chen_SpatialVLM_Endowing_Vision-Language_Models_with_Spatial_Reasoning_Capabilities_CVPR_2024_paper.pdf [00:38:15] DeepSeek-R1-Distill-Llama, DeepSeek AI https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B [00:40:10] ARC Prize 2024 Technical Report, François Chollet et al. https://arxiv.org/html/2412.04604v2 [00:45:20] LLM-Guided Compositional Program Synthesis, Wen-Ding Li and Kevin Ellis https://arxiv.org/html/2503.15540 [00:54:25] Abstraction and Reasoning Corpus, François Chollet https://github.com/fchollet/ARC-AGI [00:57:10] O3 breakthrough on ARC-AGI, OpenAI https://arcprize.org/ [00:59:35] ConceptARC Benchmark, Arseny Moskvichev, Melanie Mitchell https://arxiv.org/abs/2305.07141 [01:02:05] Mixtape: Breaking the Softmax Bottleneck Efficiently, Yang, Zhilin and Dai, Zihang and Salakhutdinov, Ruslan and Cohen, William W. http://papers.neurips.cc/paper/9723-mixtape-breaking-the-softmax-bottleneck-efficiently.pdf

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Test time fine tuning is a new paradigm to deep learning, right? It's something completely outside of the deep learning paradigm. What's the most efficient way to learn at test time? That's a very interesting question.

• 0:11Tim Scarfe?

  How did you encode the problems?

• 0:13Speaker 00

  The whole point of ARC is they're gonna trick you. Whatever specialization you put in the input, you can create a problem that's adversarial to that tokenization scheme or special representation scheme for our problems, because it's so arbitrary. Problem is so different, right? It's a very new problem. But the really cool thing that you get out of this is you train this model to be a very weak dumb contextualizer, maybe, right? But that's what you're tuning for.

• 0:39Tim Scarfe?

  Transformers, even in a trivial sense, cannot do counting or copying. They just can't do it. You count up to 100 and you just say, can you count these numbers up? And it just fails abysmally.

• 0:48Speaker 00

  What we do, which is really interesting, we prompt everything into the forward pass, all at once.

• 0:59Speaker 00

  We're looking for people that are interested in changing the paradigm, going into test time compute, that like working in small and nimble teams tackling really big problems.

• 1:10Tim Scarfe?

  Two for AI Labs is a very, very exciting new research lab that's just started in Zurich. They are looking for amazing MO engineers to join their team. It's a very small team. If that sounds like you, go to twoforlabs.ai. You might remember when we interviewed Mohammed last year. He's part of Minds AI, along with Jack Cole and Michael Hoddle, a couple of legends. Of course, they were acquired by Two for Labs. They got the highest score on the ARC challenge, about 58%. And they've just released their paper where they spilled the beans on how they did it. It's called "Don't Throw the Baby Out With the Bathwater: How and Why Deep Learning For ARC." Now, the prevailing view has been that neural networks lack the necessary capabilities for abstract reasoning tasks. And at least to some extent, they proved

• 1:55Tim Scarfe?

  that wrong. Consider this particular ARC puzzle. The task requires inferring complex transformational rules from minimal examples, a challenge where vanilla LLMs like GPT-4 get no better than around 10%. Now, Mohammed thinks That ARC puzzles are fundamentally perceptual reasoning problems. They incorporated the optimizer directly into the evaluation process, which allowed the overall approach to develop new abstractions during test time. And their methodology introduces two principal techniques. Number one, we've spoken about this a lot on the show, test time active fine tuning, or what we've been referring to as test time active or transductive fine tuning, where you generate synthetic training data derived from each puzzle's examples,

• 2:40Tim Scarfe?

  and you fine tune the model as you go. The second approach is what they call augment inference reverse votes, where you apply transformations to input puzzles, generate predictions, reverse the transformation, and implement a voting mechanism to identify a consistent solution. They found that the latter improved performance by 260%, with test time active fine tuning providing an additional 300% improvement, which is how they yielded the highest score on ARC, about 58%. Now, another thing they found is that the model architecture scale has a greater impact than pre-training for building new abstractions. Larger models are simply more expressive, enabling better abstractions and reasoning during inference.

• 3:24Tim Scarfe?

  Now, we are counting down the days until the version two of the ARC prize. We are launching it on MLST on Monday, and I hope you're excited because we've got Francois coming over and he's going to tell us all about it. Suffice to say that all of the frontier models are going to be going down to negligible performance on it, and I'm excited to show you. See you on Monday and enjoy the show. Mo, you've been on the show before, but basically you won the ARC challenge with Jack and Michael. Technically, you didn't win because you didn't choose to share your solution, but certainly in terms of the leaderboard, you guys did the best solution. Welcome back to MLST.

• 4:02Speaker 00

  Thank you so much for having me. It's always a pleasure to, it's really nice seeing you here in beautiful Vancouver. I feel like you're kind of a guest to me in Canada here. It's really nice seeing you.

• 4:15Tim Scarfe?

  Amazing, amazing. So congratulations on the incredible result. Tell me more.

• 4:20Speaker 00

  Thank you so much. We've been obsessed with the ARC challenge for a very long time. We, yeah, me and Jack have been working on it for two years as part of the same team. Michael has been working on it and we're now also for two years, by the way. So like, we've, we've always thought that this benchmark was going to get more important, more important. And, you know, this is the case now, you know, at Europe, lots of people know about, lots more people know about the ARC challenge. And yeah, there's been lots of great, you know, popularization of it happening.

• 4:58Speaker 00

  Really, really happy to see that. Yeah, we continued developing our methods based on kind of similar philosophies that we had, and I'm really excited to kind of dive deeper into those.

• 5:14Speaker 00

  And also, yeah, give you a little bit of some, some of our results that we're, we're planning to share very soon in a paper.

• 5:23Tim Scarfe?

  Very cool. So by the time this goes out, you probably would have released that paper. So you can, you can probably tell us some of the headline.

• 5:30Speaker 00

  The paper is not going to be super interesting technically because we, you know, we've shared our ideas before. We've shared them on MLST and, and we're always, we always like sharing. And, and, you know, if you look at like the, out of the top 10 current leaderboard on ARC, I don't know how many, but like maybe close to 80% were using similar, similar ideas, very similar ideas in certain cases. So we were always like happy to share. And, yeah, we've been pretty open about that. And we're happy to see everyone kind of converging on test time, fine tuning, and, and the voting mechanism that we found to be super useful.

• 6:14Tim Scarfe?

  Very cool. Well, why don't we break down, like, what, what are the key innovations? And maybe you can't talk about all of them, but what are some of the key innovations that led to your, to your win?

• 6:27Speaker 00

  Yeah, I, I'll, I'll preface this by saying. Like lots of uh the current methods are using very similar things uh right so it's

• 6:36Speaker 00

  like and I'm again I'm very happy to see that everyone is kind of leveled up and you know uh I I'd be remiss to say it's it's not uh because of the the amount of sharing we we were doing so I'm very happy to see that and I'm happy to see what new innovations are gonna come up now that we've all leveled up here let's break it down so the innovation so I think we should start with uh test time fine tuning because it's really the big one and um I'm uh I I uh I like framing it in this way okay so just to get the general uh idea so there is a sense that you could say that uh test time

• 7:17Speaker 00

  fine tuning is a new paradigm to deep learning right it's something that's completely outside of the deep learning paradigm right that you know you're changing parameters at test time that's not really uh that's a fair thing to say right but there is a way to look at it in which it exactly fits the deep learning paradigm and that's how that was uh how we we uh we arrived at it so the idea is we really see arc as this uh perceptual problem right you have an interpretation problem it's it's it's kind of subjective in a way because you have these biases right you're looking at the problem and you have an inf like an almost infinite set of possible transformations from inputs to outputs

• 7:59Speaker 00

  right and um uh it's really hard to narrow down what you should look at right it's really hard like imagine uh a riddle that there is an a box and inside it there is another object and you know there are different types of objects and it's really hard to find the right level of representation uh to start to even search around the solutions right so you kind of have to take it in all at once and uh make like hope that you know something uh pops up it's very similar to uh looking at an image there is an almost infinite set of different colors and different lightings and they're all the same thing but you have to contend with this infinity uh before you Can kind of proceed to abstract over the image itself to say like, okay, now there are four apples.

• 8:44Speaker 00

  You start counting or first you have to identify the apple in infinite lighting, infinite coloring, right? And that perception part is really important in images, but we think it's really important in ARC also. So what like you could easily imagine, once you have the right level of abstraction, search becomes much easier, right? Uh, searching for the right function is very, very easy. If I know this is the relevant object, I need to count or I need to do something around this specific object or this object to object mapping, right? So very, very important to get the right level of perception. So, okay, so what's the best way that we know of to tackle a novel perceptual problem? It's the deep learning paradigm, right?

• 9:25Speaker 00

  If you have, um, if, if you want to gain it, like learn a new skill, right? Uh, a perceptual skill, you want to classify mugs, start with an untrained network and you train it on a bunch of images of these mugs, right? Uh, and so we take that idea, but apply it uh on ARC. So the, the claim with ARC is completely novel uh at test time examples. So uh, the logical thing to do and, and you know, you have to really uh take seriously this idea that ARC is uh, the, the most important or, or one of the very difficult uh parts of it. Not the most difficult is this perception problem.

• 10:06Speaker 00

  So then you just apply, uh, apply that paradigm, but in uh, in test time, right? And uh, you know, it's, it's known to acquire skills really well. So you apply the whole paradigm to acquire skills at test time.

• 10:20Tim Scarfe?

  Right. So that's, that's certainly true. So there are, there are things that neural networks can do that we cannot write programs to do. Right? Just, just because they are, um, Sholay kind of distinguishes between perceptual problems and like type two problems or, or whatever. So many, many of those problems are, are perceptual problems. One, one really interesting thing though, is that you guys counter to people like Kevin Ellis. You are fans of solution space prediction. So you don't create intermediate Python functions. Is, is that, is that the case?

• 10:53Speaker 00

  Yeah, that's true. Yeah.

• 10:54Tim Scarfe?

  So that, that's a pretty huge kind of like strategy right, right there. So, um.

• 11:01Tim Scarfe?

  I have this intuition, and Sholay does, that there's something special about Python programs, right? And the special thing is mostly that they have this kind of compositionality, which means they can be composed together, they can be decomposed into small parts, you can construct a library and you can take bits together and so on. And we intuitively feel that neural networks, for whatever reason, they don't have this compositionality. Maybe we can make neural networks that do in the future, but right now, they don't. So people who are really bullish about neural networks, and certainly I think you guys are in Tufts as well, think that, no, actually, they can do this kind of compositionality, they just need to be coaxed in the right way.

• 11:41Speaker 00

  Yeah, that's a very poignant point, that it's absolutely true that neural networks by default are not compositional, and you need to do a lot of work to get them to be compositional. And that's, you know, not easy to do, and we did do that work, and it took us a lot of time to really understand that point, right? That, you know, by default, they're gonna just learn statistics. And it's not elegant, because they're not composable, but I think, and I'll speak to this later, that, you know, for a certain domain, you can get the biases deep enough,

• 12:24Speaker 00

  and the biases are really important, such that you're able to tune the reasoning part really easy. And in that sense, you do get compositionality, but it's not an elegant solution that, you know, without the test time aspect, without putting the biases deep enough, you don't get that. So now, what do you do and how do you do that more efficiently? These are all things that we want to explore in the future, and we're gonna do that at Tufts Labs. With the Python programs, there's something, again, that's really important. You can perceive the problem, if you perceive it right, you can take action in Python or in the neural space. I think that's fine, as long as you see the right level of abstraction. You have that dynamism inside the model to be able to...

• 13:10Speaker 00

  Do some correlations from input to output and then match those correlations and force them in the other input to output and then, you know, make sure it generalizes and then you can, after that, you can either output in Python or in a direct output manner, right?

• 13:23Tim Scarfe?

  Well, let me, let me press on this a little bit because I think this, this is potentially, you know, the, the most exciting part. So, um, we have an intuition that Python programs are compositional. And I interviewed Laura Ruess the other day. She, um, she's working at, at Coher and she, she did this paper showing that basically you, you train influence functions on neural networks and then you get them to do fact retrieval or you get them to do reasoning tasks and you can see how much of, of the, you know, like how many of the source documents light up essentially, um, based on the task. So when doing reasoning tasks, it had a very diffused activation in, in terms of the, the source documents.

• 14:03Tim Scarfe?

  And what she noticed is that it was looking at code. So when calculating the slopes of lines and so on, it was looking at things on Stack Overflow and it was looking at procedures of like how to perform some reasoning and it, and it was applying that in a very diffused way to a specific problem. So, so the, the fascinating thing is that even though it's not explicitly generating code, you can fine tune it on code and code has some form of compositionality. And you can get, you can coax the neural network to do an approximate form of compositionality, even though you're doing solution space prediction. That is fascinating.

• 14:41Speaker 00

  Yeah. One of the things that we do for, uh, for pre-training. So code, uh, I think there's something, there's another thing that's really, uh, another way of saying what you're saying about code. And by the way, I know about that work. It's really amazing work.

• 14:54Tim Scarfe?

  Yeah.

• 14:55Speaker 00

  So with code, it's really hard to get, to predict the next token. You have to contextualize a lot more. Like you have to know, okay, what are we doing? What is the exact name of the variable? What is the exact relevant variable here? With language, that's not the case. With language, you can shortcut, you can cheat easily, right? You can, you can use like an inaccurate word and it'll probably be fine. You know, with code, you have to be very precise. And so you have to be very contextual, right? And I think that's, that's like, uh, but, uh, another way of, of kind of looking at the influence of code. And, and code has been, code pre-training has been shown to improve reasoning across many domains and, and a few new papers now. So this thing has been reinforced. But yeah, it's.

• 15:39Speaker 00

  Yeah, training on code is really interesting. Um, I also wanna kind of go to this idea of attuning the reasoning. So, uh, you know, you, you, you mentioned that the neural networks, while they're solving this reasoning task, uh, they're, while they're solving this reasoning task, they're looking at code, right? Um, I think framing the problem also matters in a very important way here. So, um, so yeah, so first of all, uh, we established that test time training gives you the generalization ability.

• 16:19Speaker 00

  Okay, so, um, so what do you have to do to, uh, what's the best way to, what's the most efficient way to learn at test time? That's a very interesting question, right? Um, and so what we do is we prompt everything in context, right? We're just speaking about contextualization ability. And that's a really important thing for us to maximize, um, for ARC, right? Because if you're prompting things into ARC, like in the ARC format, you need, you need the model to be as dynamic as possible, uh, right, Tim? So you're, you have your inputs and outputs, uh, and inputs and outputs. And now I have this test input and the problem is so different, right? It's a very new problem.

• 17:01Speaker 00

  So contextualization and the ability of the model to kind of be steered is very, very important. And what we do, which is really interesting, we prompt everything into the forward pass all at once. So inputs and outputs, input and outputs, input and, and then test input, uh, like new input, uh, is all in the forward pass all at once. So it's, one, one way to look at it is a measure of the contextual, the novel contextualization ability of the transformer, right? But the really cool thing that you get out of this is you train this model to be a very weak, you know, dumb contextualizer maybe, right? It doesn't, the, the modeling ability in the forward pass is not that good. But that's what you're tuning for. It's kind of like a meta model in a sense, right? That's what you're pre-training for, okay?

• 17:43Speaker 00

  And so the model is gonna be kind of, it's gonna learn reasoning patterns. It's not gonna learn exact transformer. Like Clem's work, right? So Clem's work is the function learns just the transformation, right? But our model, you know, by by putting everything in the forward pass in pre-training and and and you're training it over many different arc riddles, right? So you're kind of saying, okay, the thing to learn here is this meta meta task of looking at the context and then generalizing from it or doing your best at modeling from it. And and and so you have this weak meta model.

• 18:25Speaker 00

  And now that's much easier to tune than a Clem type model where it's you just have. So for Clem's model, the encoder only receives one pair of input to output. You get stronger induction or regularization. I don't know what what you want to call that. But with with that model, but you do, but but it's like, you know, there are many possible, like you don't get that that meta ability, right? There are many possible sets of transformations given one only one instance, right? You get you need two or three instances to kind of narrow that infinity down. So you're training this meta model and now tuning that meta model is a much easier task, right? Like, that's really interesting.

• 19:09Speaker 00

  You kind of like, again, the thing is you're you have a predictive model that's gonna be kind of wrong. You can just tune tiny pieces of it to get the reasoning to click, right? So that's really, really key. And it's it's work that is not super present in the literature. I didn't survey it too super well, but there was one interesting paper, the MLC paper by Brandon Lake in Nature, which was looking at reasoning. They don't do test time training or anything like that, but implicitly there's a way to look at it where it's exactly this, actually. It's exactly doing what we're doing. So for that paper, what they do is they have this input to output kind of task that they want to learn.

• 19:51Speaker 00

  What they do is they retrieve similar things, they put them in context, and then they put the new thing and then they they do. That's their forward pass, okay, and then they train over that and that helps them do much better. So they're kind of training over that. So that's kind of test time tuning. If you take that whole paradigm and put it at test time. So we train over that, but at test time. And it's the framing of the model as a meta model and now the tunings that you have to do are are much smaller. So I think that's a maybe one of the biggest differences between our work and Clem's.

• 20:26Tim Scarfe?

  Can you just summarize that again, just so the audience can understand it? So you said that you're doing more kind of training of the model before you do anything at at test time. Can can you explain that?

• 20:37Speaker 00

  Yeah, so we we do pre-train the models on Arc.

• 20:40Tim Scarfe?

  So you you start with a language model, you know, like, I don't know what you use, maybe Lama or something like that.

• 20:46Speaker 00

  T five variant, long T five.

• 20:48Tim Scarfe?

  So you start with a T five language model?

• 20:50Speaker 00

  Yeah.

• 20:50Tim Scarfe?

  And what kind of a language model? It's not a normal like auto-regressive language model.

• 20:55Speaker 00

  It's an encoder decoder. So that's, yeah, so it's an encoder decoder. But it's and it's pre-trained. So we don't train it from scratch. Pre-trained on language, which you think has that contextualization aspect a little more, right? And we also train it on code further. So so then it also gets stronger emphasis on this contextualization ability, right? You want to train the best kind of forward pass steerable model, basically dynamic model that that can change a lot based on its inputs. Not and and that's that is possible with transformers, but only to a certain extent. So so that's what we we start with.

• 21:34Tim Scarfe?

  Even that is very interesting though, because no one else has done that. So you're you're you're starting with an antique model from about 2020, which is an encoder decoder, which is something that doesn't really exist anymore. I mean, those were originally used for like machine translation tasks about a million years ago.

• 21:51Speaker 00

  Yeah.

• 21:51Tim Scarfe?

  So and and it's presumably trained on a tiny, tiny corpus compared to modern thing. I mean, it might be trained on like, I don't know, like less than a billion tokens, probably.

• 22:02Speaker 00

  Yeah, yeah.

• 22:02Tim Scarfe?

  So so you so you start with that. But the benefit of it is, I'm guessing it's a tiny model, which is like, what? In the millions of parameters, like 340 million parameters or something.

• 22:10Speaker 00

  Exactly, yeah.

• 22:11Tim Scarfe?

  So it's a tiny model.

• 22:12Speaker 00

  Yeah.

• 22:13Tim Scarfe?

  Start with that, then what?

• 22:14Speaker 00

  So then we have this pre-training recipe that involves code. It involves auto-generated ARC tasks in various ways. And Jack did lots of really good work there, Jack Cole. So there's this magic recipe, but the

• 22:35Speaker 00

  the bottom line is it has lots of code. It has lots of kind of synthetic tasks. By the way, the amount, like the total number of new tasks is not big, right? It's not like we are able to, you know, sit down and generate so many synthetic Riddle generators, right? Like the total number of new concepts in pre-training is low, but there's there's an important thing that we think is happening during pre-training, even with the few number of like number of concepts trained. But we have, okay, so we have a recipe that's that has some code and it has some synthetic tasks. And then we get our like pre-trained ARC model, right?

• 23:20Tim Scarfe?

  Okay.

• 23:21Speaker 00

  Now we go into the test time where we feed everything into the, in the forward pass, like we were doing in pre-training. And then, so all of the instances are in the forward pass. And then we get the test input, the new input that we need to predict the output for, that's also in the forward pass. And we need to predict the output. And, and what I was saying about tuning the reasoning is the framing of the problem is really important, right? Everything in the input, you need to kind of learn, okay, how do I compare the input? And then kind of get at a very almost okay function, but probably not okay in very important ways.

• 24:07Speaker 00

  That's already very, very close to your solution. So now you can test time tune and kind of search over the reasoning, tune the reasoning, rather than

• 24:18Speaker 00

  a different way of looking at it, right? And then, okay, so first of all, that compares to like the MLC work. It compares to lots of, lots of ideas where you're, you're kind of, when you frame the problem in a meta way, you give the model more to learn, it kind of is always the best thing to do in the forward pass. And then you have to kind of scaffold and tune and, and do everything else, but you're kind of. You know, halfway there, you know, with that, with that step. And so the tuning is much less work at that point.

• 24:49Tim Scarfe?

  Okay, at, at test time, so you call it the forward pass. So you, you, you put the test instances in there. You represent it in a really clever way that kind of helps the, the language model. And, you know, the, the seek to seek model do its thing. And then you do some green blating, for want of a better term. You, you do lots of sampling until you get ones that fit all of the, the specifications. Do you do any augmentation in the forward pass?

• 25:13Speaker 00

  Yeah, so the forward pass is just like prompting the model with the, with the representation of the board. And the representation isn't super special. It's just a plain representation. So just a plain representation into the model. We don't do any filtering, if that's what you mean by green blating. It's, it's a very transductive, like you just produce the output directly. So we can't do any filtering, actually.

• 25:36Tim Scarfe?

  Right, but, but do you do loads and loads of sampling until you get ones that you think are good?

• 25:43Speaker 00

  Yeah, so we have a method of voting that we also introduce in the paper. And voting, I think, is especially suited to ARC. So, uh, and it follows this idea of there are many ways that you can be wrong about an ARC riddle, but there's only one correct solution. So, you know, just try all of these ways. And then hope that, you know, the, the, the majority vote is gonna go to the only right, one right way to do it right. So, yeah, we do that.

• 26:15Tim Scarfe?

  And tell me more about the voting.

• 26:16Speaker 00

  I think that summarizes it pretty well. The, uh, we do augmentation. So, so there are many ways to sample things out of a model. One, one way to sample something out of a model is beam search. Right? Uh, or you could do temperature based sampling, which, um, I don't think is a good idea for ARC. Because again, like if you just get bad RNG, uh, you know, you're kind of done for. But beam search is really interesting because ARC is, is very special because it's only one right way to do it right. So you can imagine all of these beams and the model. It's kind of uncertain about a certain next token, right? And that's that's totally fair. So we're gonna take a beam, we're gonna try all of them.

• 27:01Speaker 00

  And then iteratively, if you've made, if you made the wrong decision, you're gonna be kind of lost, right? Like in the pixel space, if you're outputting, if you've made the wrong decision, you're gonna be very lost because you don't know, um, like, a mistake is already done, obviously these models can't backtrack. So, okay, how do you really continue after a mistake? It's unclear, things are ambiguous, so that the the probabilities are gonna disperse. And then these are gonna uh fall, uh, these beams are gonna fall. But if you've done the right thing, you're gonna be more and more certain of the the next token. So that's one aspect that uh, one way that we do uh, we get the samples. But we have like a kind of a variety of ways to that we get samples for voting.

• 27:45Speaker 00

  We go into it uh, a little bit more in the in the paper that will hopefully be out. Um, and yeah, so that's the voting.

• 27:54Tim Scarfe?

  That was the sampling. How do you do the voting?

• 27:56Speaker 00

  So it's just majority vote.

• 27:58Tim Scarfe?

  Oh, okay. And you said that there's only one way of getting getting it right. But aren't there uncountably many ways of getting it right?

• 28:06Speaker 00

  Yeah, well, there's only one way with the human priors and with the R core knowledge of getting it right, hopefully. Otherwise, it's an ill-designed riddle, um, right? Like for an arc riddle, there's only one correct answer officially. But of course, you can have different uh, systems that have different biases that they think this is the right way and so on. But hopefully, it's very clear for the human biases and the, you know, core knowledge priors that there's only one way of of getting it right.

• 28:39Tim Scarfe?

  We should explore that. So the the ridiculous example is I could.

• 28:45Tim Scarfe?

  I could have a program which says, if you see this specification, then give this answer. If you see this specification, then you give this answer. And if you see the third one, then give this answer. So it's just like explicitly encoding the answers without any like generalization or whatever. And you're making the argument that because we have a certain structure of priors that can be composed in a certain way. You think that it's very unlikely that it will find a solution other than the ones which humans would agree on in that compositional space.

• 29:18Speaker 00

  Well, that's what we hope to do with the training, right? So with the pre-training, that's what we hope to encode, right? We hope to encode two things, the priors really, really well. And by the way, I really want to get into prior encoding also like and what that means. Because there's lots of theorizing about that that we get. And there's lots of people like saying, okay, you're memorizing certain reasoning patterns and so on. So let's pin that for later. But that's what we're hoping to do with the pre-training. We're hoping to get the right priors and the right kind of, yeah, core knowledge into the model in such a way where, yeah, it only outputs the right solution.

• 29:59Speaker 00

  But the second thing we want to do is want to make the model steerable, right? For a new riddle, to not rely on the priors too much and to be able to actually produce the right one that generalized. So it's kind of this very difficult balance. You want the model to be contextual, but also have the priors very deeply in there to be able to use them to search.

• 30:22Tim Scarfe?

  Yeah, so there's a trade off between flexibility and correctness. The reason I ask this is on Wending and Kevin's paper, they said that on the, because they did an ensemble with induction and transduction, but they favored the induction. And I say transduction, they're both transduction. But they said they had a nine percent false positive rate on the, you know, the explicit function generation, which meant that it was creating functions which gave the correct answer, but was actually wrong.

• 30:46Speaker 00

  Yeah.

• 30:47Tim Scarfe?

  So it worked on the test specification, but it was actually wrong.

• 30:50Speaker 00

  Yeah, yeah.

• 30:51Tim Scarfe?

  But have you seen that?

• 30:52Speaker 00

  So we don't output programs, so we haven't seen that. But you can have transductive.

• 30:59Speaker 00

  In my definition of like you can have intrariddle transductive Python programs, meaning just like you said earlier, like if this is the question, give me this answer. If this is the input board, give me this answer. You can have that with Python programs. You can have overfit Python programs, right? Which which just means that they didn't get the right prior from this one and this one and this one to be able to generalize to new ones.

• 31:23Tim Scarfe?

  Exactly.

• 31:24Speaker 00

  Yeah.

• 31:24Tim Scarfe?

  Exactly.

• 31:25Speaker 00

  Yeah.

• 31:25Tim Scarfe?

  Yeah. So is how much of of an issue was that for you guys?

• 31:30Speaker 00

  Well, we kind of just like sidestep this whole thing and we just kind of find different ways of giving feedback to the model, i.e. test time fine tuning. So to kind of hope that the model, the the built in model kind of has the best chance to to generalize to the new test riddle. And and so basically, the answer to that is everything we do with test time, right? We we test time tune, which means, you know, you start with the initial kind of guess from the model based on the I C L, the in context learn or whatever you want to call the the contextualization ability of the model. And then you tune that.

• 32:13Speaker 00

  And what what's happening is there is an implicit model inside like there is an implicit kind of transformation function, you could say, inside the model that is getting tuned, right? It's give it that implicit model is giving a guess and then the guess is incorrect in maybe major, maybe minor ways. And it's getting your feedback through gradient descent and it's updating that implicit model through the weight and then it's giving another guess, right? Another guess. So that's test time tuning. And then the hope is, you know, that ARC has a perceptual problem and that neural networks can learn these generalized perceptual problems if you put them in domain and if you tune the reasoning that way. So that's, I guess that's how we deal with it.

• 32:53Tim Scarfe?

  How did you encode the problems?

• 32:55Speaker 00

  It's a very plain like numbers as text. So it's it's there's nothing special there.

• 33:01Tim Scarfe?

  Okay.

• 33:01Speaker 00

  Absolutely, nothing special. This is a very interesting question. You shouldn't, like, in my opinion, you shouldn't generate any special representation for ARC. The whole point of ARC is they're gonna trick you. Whatever specialization you put in the input, you can create a problem that's adversarial to that tokenization scheme or, or, you know, like, special representation scheme for our problems. Because it's so arbitrary, right? The main point of ARC is raw, like, you get, you, you, you get the, the problem raw and then be flexible about combining it because it's gonna be novel and it's gonna be raw.

• 33:42Speaker 00

  So don't, like, this is another thing, like, VLMs are really bad for ARC, right? Maybe this is like giving a lot of Alfred people. But, because I think like, for some people, for a big majority of people, the first thing they try is, is, oh yeah, we need, this is a visual problem, let's use a VLM, right? Well, that's a really bad idea. And it gets, and that gets to some of the stuff I was seeing at NERIPs, which is, people are coming up with lots of data sets where VLMs do really bad. Like the six finger problem, right? If you have a hand with six fingers, you give it to a VLM, ask it how many fingers? It'll tell you five, right? Why? Because the VLM is a fixed representation machine.

• 34:26Speaker 00

  This, this really gets to the core of ARC. Like, if you wanna understand ARC, you need to understand that you need raw representations in, in, into the transformer rather than, you know, pre-encoding some stuff. So the fixed, the frozen encoder in most VLMs, it will take some kind of perspective, pre-existing perspective on the ARC problem. But you haven't seen the rest of the, the, the input-output pairs yet. You don't know what the right framing is. You're doing that, you're guessing, and there are many ways that you could be wrong. It's the same thing with the six finger, right? With, it, it just, the VLM only knows a certain thing, uh, certain things and it's gonna give you in its, in the output tokens, a representation of only that framing on certain things, like whether it's a hand or like...

• 35:10Speaker 00

  Doesn't like it's not flexible enough to be able to recombine. It doesn't know what the question is also like with the VLM.

• 35:20Speaker 00

  Like that the you already encoded the image before asking the question, oh, I need to count the fingers in the image. So that the transformer model and the VLM encoder can't condition on that.

• 35:30Tim Scarfe?

  Couple of things on that. I mean, there's the notion of, I mean, you know, Greenblatt did GPT four O, the vision model, but. I I'm not entirely sure how that works, but you know, the the best way to do a visual model is to have it multimodal. So it's a language model and it's a vision model all in one, and it can do some kind of crossover between them. So there's there's the notion of whether having a vision model actually helps with ARC or whether it's better to think of it as a reasoning problem and whether to kind of like skip the the interpretation part and give it a structured representation, that maybe we should we should start with that. I mean, do you do you think in principle whether we had a strong multimodal model which could transfer between the reasoning domain and the visual perception domain, it will work?

• 36:11Tim Scarfe?

  Or the other point of your of your statement was that that it's not flexible enough. But could we not do test time training and all of this transduction stuff with with a VLM as well?

• 36:21Speaker 00

  Yeah, yeah, you could. I just don't think it's elegant. I think it it it could get you to like sixty percent or whatever, but it's not going to get you to hundred percent of the way there unless you change the architect. So like I'm I'm making a statement about a specific architecture, just to be clear, like multimodality might help. The thing that I'm advocating for is you need a flat representation and an intermixing. So you could have vision patches intermixed with language tokens and they all condition on each other. That's that seems reasonable and but if you kind of have a pre-encoding that is frozen and you don't test and tune, that's just that's just a bad idea. If you do test and tune, maybe that works.

• 37:01Speaker 00

  But again, it's it's this crutch. It's not reliable. It's better to just really focus on contextualization and on being flexible, because that's the point of ARC, right? And it's a very, very good benchmark because of that, because it it found a way to test that. And I think one one thing that's really good about. Our approach is there's a very different way of seeing it like, okay, if you want to measure the the contextualization ability, the true contextualization ability of these models, of these architectures, then one thing you could do is just prompt it in the forward pass and see if it's able to solve it. So that's a really cool way of attempting solve ARC.

• 37:41Speaker 00

  Obviously, that's what we're doing. So I I think that, but uh, but uh, but I, and but that allows you to ask certain research questions that I'm really excited to answer at Tufa Labs. So yeah, like, so it's not just about solving ARC. You could start framing things about measuring what transformers are doing or the the contextualization ability. I hope more people start doing that as well.

• 38:04Tim Scarfe?

  So one of the really cool things about Test Time Compute, I don't know if you saw Hugging Face have just released this kind of like O one type thing. And they're showing that they can make a one billion parameter llama solve the same types of math problems as, you know, like a vanilla eight billion llama model. So what it seems to do is it gives you this ability to make a small model behave as if it's a big model. And that seems to be a superpower.

• 38:31Speaker 00

  I am not familiar with with that release that Hugging Face did, so I I can't comment on it. I can give you some

• 38:43Speaker 00

  some more details about how our models scale on the hidden test set. So I think that's really interesting, something people really want to know. So

• 38:54Speaker 00

  yeah, so what do the scaling laws look like for the hidden test set that is just completely uncontaminated, right? By the way, the public

• 39:03Tim Scarfe?

  Not completely.

• 39:05Speaker 00

  The the hidden test set?

• 39:08Tim Scarfe?

  Well, yeah, I think I think there there is a potential objection that because we have now hit the hidden test set a lot of times, that there's some unintentional information leakage. Could you comment on that?

• 39:21Speaker 00

  I think the information leakage, like from an information theory perspective, is very, very low. But I'll I'll tell you what we are trying to To do with the information leakage, we're trying to uh make it algorithmic, right? Like, were this this information leakage about the type of things that work on ARC? And that was the kind of the hope, but I I don't see it as significant at all, honestly. Uh, the, so the ARC uh dataset was actually available for a hundred submissions a day for like a couple of years, uh, and people didn't seem to kind of like gain too much from that. Uh, we didn't gain too much from that. Uh, we weren't honestly using it that much. I think it's, I I see it very minimal.

• 40:05Tim Scarfe?

  Yeah, because um Sholay thinks that the reason he needs to make the next version of the ARC challenge is that if it's not overfit now, it it kind of is, but just in a latent way. So he says that when you look at the ensemble of all of the various different approaches to ARC, even back in twenty twenty, you know, the ensemble was getting about forty nine percent. So he said that, you know, if you did a targeted, you know, imagine we're like security researchers, that that's what they do. They they like, they they look for sources of entropy and they mix things together in a targeted way. And and so he said, even though it hadn't really been attacked in this way yet, it's only a matter of time until it was.

• 40:43Speaker 00

  Yeah, yeah, I mean, uh, I think that's fine. I think it's it's good to refresh the dataset and also maybe to remove the the brute forcible uh ones. So, uh, yeah, I mean, when I uh, I spoke to Francois a little bit and he mentioned that even in in his talk, uh, he also mentioned this, the aspect of, hey, I'm gonna find out the brute forcible ones, right? Because that was the main approach in twenty twenty and and I'm gonna remove those. I think that could be a good thing that you do. I do think uh the the domain is so huge and the the possible variation in the ARC that is so big that uh it's totally fine to to kind of use it, but maybe, yeah, like, uh,

• 41:27Speaker 00

  again, from an information theory perspective, the bits are not that much, but, I mean, yeah, who knows? I'm totally fine with a new ARC dataset that is harder. Uh, I think uh we're up for the challenge. The refreshment is good, the the the guys, I spoke to them a little bit, they're doing really good calibration with human testing to make sure that uh you know, the data sets are well calibrated. So yeah, it's um it's gonna be fun.

• 41:56Tim Scarfe?

  Sholay is quite bearish about some of the the sort of the test time compute strategies that that are being used. And maybe could you just reflect on on why that is and what you think he would rather people did?

• 42:11Speaker 00

  Yeah, he he spoke about that a little in his talk. So he's bearish on test time compute? Or the specific strategies that people are using for test time compute?

• 42:20Tim Scarfe?

  Uh I think both. And and I asked him like, what would you what would you do? Because he's he's doing his own startup now.

• 42:26Speaker 00

  Hmm.

• 42:26Tim Scarfe?

  And he has advocated almost for something that resembles the original Dream Coder. So he thinks that we should have, uh so first of all, he he's a big program space guy. So he doesn't, he he thinks that there are inherent limits in in compositionality and whatnot using neural networks. So we should do the program um space thing. Whether it's a DSL or or an actual programming language, I can't remember exactly what I said, let's say programming language. And and he says that we should have a neurally guided search, which is what the original Dream Coder thing did.

• 42:57Speaker 00

  The reason I don't like uh Dream Coder type approaches is not because of Dream Coder specifically. And I spoke to like even Kevin Ellis about this and they uh like I it's it's interesting their current approach, right? When when you consider that that uh he was the first author there, but so I think there are two kind of things that you have to focus on, especially when talking about Dream Coder. The the output space, that's what Kevin Ellis mentioned, it's it's too restrictive. Right? Uh this lambda calculus, uh it's not, it's it's very uh inflexible, right?

• 43:34Speaker 00

  And and that's something also that I think uh uh like in my opinion, generating Python programs is hard. If if I ask you to uh write a Python program for certain, our creators could take you up to like thirty minutes, even if you are a good. Maybe ten minutes to thirty minutes, but if I actually color them in, you can do that instantly. And that gets to like Piaget's uh theory of like incremental kind of development. So you can be in a game without being able to perfectly describe the rules of the game, but you can act in the game. So you can color the things, you can represent the thing and act in it. But the level of being able to fully specify the rules of the game and write them down,

• 44:18Speaker 00

  you don't have to have that. That's a higher level according to Piaget. So uh so that that exactly like uh hits it home uh to me. So that's the first problem with Dream Code. The second problem is it just doesn't focus on the perception, right? It could, and if it did, maybe it would do well, but it doesn't. And I think the main thing should be, okay, how are you tackling being flexible in the perception space? And then uh you can go on to do whatever you like to learn uh programs in the sleep phase and and compose them and all of that stuff is fine. But first the first question is that, so.

• 44:54Tim Scarfe?

  So that that's exactly right. And what's also interesting is that Kevin has abandoned using DSLs. So he has with open arms embraced language models. And and why is that? Well, it's because, as you were just saying, Python code is too incomplete. You can represent any concept. It's incredibly flexible. The reason why he didn't generate Python code in the first place was because there was this intractable search problem. You know, it wouldn't have been possible. Now with language models, we can. And it's because language models encode our knowledge, our notions of of what's interesting, even just basic priors that we couldn't even put into words, like the the complexity of a program and and the the intuition and creativity and all of this kind of stuff.

• 45:34Tim Scarfe?

  So now we can generate these programs and we can mix them and remix them and do all of this kind of stuff. Everyone at Europe is is doing the same thing. I guess the question to you is, this was a restricted compute benchmark. So of course you made certain trade offs that you wouldn't make otherwise. Like now at two for labs, you've got all of the compute in the world, all the time in the world. What would you do differently?

• 45:56Speaker 00

  That's a great question. Okay, I think the most interesting thing to me right now is the thing I mentioned about prompting everything in the forward pass at once and measuring the transformer ability, innate ability, and then tuning the reasoning, these types of lines. So there are so many angles to tackle ARC. So it's not about, it's the compute to do many experiments and to tackle the different angles and to get people on that will help and that are really interested in tackling these things and interested in thinking along these same lines, right? Like transformers are really bad at ARC. That's the statement that Shulay or like language models really bad at ARC and they are.

• 46:39Speaker 00

  But let's measure it and let's try crazy ideas to get them to improve. That's what I'm after.

• 46:46Tim Scarfe?

  Tell me about this paper that you're writing with Jack and Michael.

• 46:49Speaker 00

  Me and Jack are working on a paper to outline kind of the test time tuning. There's been papers now that outline the technical content of what we have, but we wanted to put the paper out there for people to cite the original kind of source. And people right now, by the way, are citing our MLST podcast.

• 47:11Tim Scarfe?

  No way.

• 47:11Speaker 00

  Yeah, like lots of papers.

• 47:13Tim Scarfe?

  Really?

• 47:14Speaker 00

  Yeah, the test time paper by MIT by Ekin cites the podcast. Like lots of people cite that MLST video podcast, which is not great, right?

• 47:23Tim Scarfe?

  Good for me. It's probably because you didn't have it in writing anywhere.

• 47:28Speaker 00

  Yeah.

• 47:29Tim Scarfe?

  Folks, you can continue to cite the MLST podcast as much as you want.

• 47:33Speaker 00

  I mean, that's actually amazing. You should have like a kind of a counter. That'd be, I mean, I hope people do that more, but I also want to put a paper out there so that people can cite the written work and we can track it better. But yeah, I mean, I think, I mean, this is a great format. We shared a lot about our method and people got inspired by it and implemented. And and put a lot of papers out, so we want to put our paper there.

• 47:57Tim Scarfe?

  Yeah.

• 47:57Speaker 00

  But like technically, it's very similar, like with the stuff we're doing now. It's it's not far away from the stuff everyone else is doing. And you know, some people are complaining about us not open sourcing. But again, like again, the stuff is already out there. And it's, you know, I'd be more worried about someone, some prof told me this. I'm more worried about OpenAI not open sourcing rather than Mind AI. So uh, so I think uh, that's the the key thing. And the the key, the another key thing is we're working on more papers, not just that one. Uh, that explore different angles and we're going to be putting lots of stuff out there in the very near future. And yeah, I'm super excited about that.

• 48:39Tim Scarfe?

  Quick point on that. Why did you not open source the paper, you know, the approach and already put more out there?

• 48:46Speaker 00

  We were in a very tough position from many different um angles. So the guys, there was the the requirement for open sourcing was was a little too extensive. So they required weights, but also training a code or training examples. And the entire test time code, so there was a lot of things. We felt like, uh, you know, with Michael's DSL that so many people are now using and have cited for everything arc related. For test and tuning and voting that everyone is using to get the top scores. We've contributed a lot to the community. Um, and um, and yeah, the the there was kind of a little too much, especially for us.

• 49:31Speaker 00

  Targeting that eighty five percent, right? So it's just kind of the incentive structure really didn't make sense. If you look at the prize money and the potential gain. So it was kind of, um, I think lots of people were making the comment, the incentive structure. If you don't get eighty five percent from the first competition, just doesn't uh, wasn't wasn't great. So uh, but uh, you know, uh, we're super excited to uh, do that moving forward. And also the competition team have told us that they're working on that absolutely. And they're uh, they've taken that feedback.

• 50:07Tim Scarfe?

  Just to be clear to the audience, what was the incentive? Given that you got, was it around fifty six, fifty seven percent?

• 50:13Speaker 00

  Yeah.

• 50:14Tim Scarfe?

  What was the incentive? I mean, if if you did open it up, what would you have got?

• 50:18Speaker 00

  Yeah, that's a very good question. So, um, so by the way, just for reference, we started out at thirty three percent at the beginning of the competition. So we did all of the work to get it to fifty five point five. We even got fifty eight uh score on the hint set, but it wasn't on the leaderboard because the time had had passed. But but so we we moved up a lot and what we would have gotten was twenty five thousand uh US dollars. Right? Like after taxes, you know what I mean? Like not not a not a huge incentive. But again, like uh I'm just wanna say it was a great competition and the the guys did a lot to

• 50:58Speaker 00

  put the word of ARC out there and into the world and they've they've taken that feedback and we have also uh up leveled the community and shared a lot. So I feel like we're both happy and uh we don't need to like kind of dwell on it too much and we can um just look forward for next year.

• 51:18Tim Scarfe?

  You guys are now working with Tufa AI Labs in in Zurich. As I understand, Tufa have acquired Minds AI.

• 51:25Speaker 00

  Uh yeah.

• 51:26Tim Scarfe?

  Is that? Yeah, tell me the story.

• 51:27Speaker 00

  Yeah, we are now. So the whole team is now working uh out of Tufa Labs. We have a lot of funding. We have a lot of compute coming in. Uh so uh so, okay, so we're gonna be purely focused on ARC for the first around year. And different angles to ARC, again, we're working on not just this paper, but another one and a couple of other angles also that we hope turn into paper. So we are gonna be putting stuff out there and we're hiring and we have compute coming. So uh do apply if uh if all of the if any of that sounds interesting to you. We are going to uh we have also plans for uh things to do after ARC. Uh like, again, we we spoke a little bit about this with the the

• 52:11Speaker 00

  the compositionality uh right of these large language models. It's not there. How can we get them there in a more general format? Uh, you know, lots of uh system to uh goals uh that and test time obviously is a very ripe area and more generally uh for uh research. So we're gonna be exploring all of that uh beyond ARC.

• 52:35Tim Scarfe?

  Yeah, all of that is very, very exciting. So yeah, the compositionality problem, Clem was saying the same thing. I mean, that is the golden ticket. If we can solve the compositionality problem, I think reasoning opens up. But presumably, though, the approaches that you're going to be working on now are going to be slightly broader than what was appropriate for this particular benchmark. So you're going to be doing a few things differently. What particular kind of strategies and approaches are you going to be looking at now?

• 53:03Speaker 00

  Different creative ways of doing test time is really interesting that I'm also interested in. Okay, first I'll say you always have to target the hundred percent with ARC. You can't, it's very hard to optimize, you know, for a certain competition for something like ARC, or it's a really bad idea. It's not gonna give you lots of score because it's because of the like how private the dataset is and you can't make guesses about it. It's just formulated in a really nice way where you have to kind of go for broke. Whether you are working in a small team with very little funding or with a lot more funding. I think it kind of is the same, but what we get now is different angles, right?

• 53:45Speaker 00

  And research angle and the competition angle and these are like super well aligned, by the way, like they mesh together really well. So yeah, so just more angles of that. I'm also like, yeah, just thinking about different benchmarks around ARC. I have, I think, some really good ideas there. The ARC is a special format, I think, that allows you to benchmark certain things. Maybe we can go into that at another time.

• 54:15Tim Scarfe?

  What do you think will happen when Sholay releases the new version of ARC? And by the way, one thing that Kevin and Zenner said, which I didn't predict, was that has ARC become a lame duck benchmark now that Sholay has said he's going to invalidate it next year?

• 54:30Speaker 00

  Invalidated meaning?

• 54:31Tim Scarfe?

  He's making a new version of it. So it's almost like, are you wasting your time working on ARC one when everything changes with ARC two?

• 54:38Speaker 00

  So I think ARC two is gonna be the same format as ARC one, right? So I think what What's gonna happen is it's just gonna get harder.

• 54:47Tim Scarfe?

  So what you said to me was that he's employing loads and loads of humans to design and select tasks that are at the appropriate level of difficulty.

• 54:58Speaker 00

  Yeah.

• 54:59Tim Scarfe?

  So some tasks are insanely difficult and most humans don't get them right. And some are like too easy. And even then, there's, there's an interesting overlap between what's easy for a human and what's easy for a computer.

• 55:10Speaker 00

  Yeah.

• 55:10Tim Scarfe?

  Shalay, of course, is a big believer that they are a proxy for intelligence, that there should be some kind of G factor or generalization

• 55:17Speaker 00

  Yeah.

• 55:17Tim Scarfe?

  between them. The other thing I wasn't entirely clear on is, is he, is he protecting ARC two, you know, the second version of ARC through diversity? So just reducing information leakage through sheer diversity? Or is he talking about creating a dynamic benchmark? Which is almost entirely impervious to overfitting?

• 55:37Speaker 00

  I will say, uh, this, so I was talking to Shalay and I asked him, are you going to have a dynamic benchmark where you iteratively? Or, or you have some kind of iterative framework where you can ask for more? And, and he did say, no, it's going to be the same format, just different, uh, data and possibly like, uh, very likely more difficult in, in, in the right, in, in whatever way they, they are calibrating for. So I think one thing that Shalay speaks about related to your first question is the, getting the right level and so on, that this, this idea of idiosyncratic riddles, right? Meaning like one-off riddles, right?

• 56:17Speaker 00

  Those riddles that are, uh, so creative. I think it requires real creativity to come up with those riddles, right? Like they're so creative, they're just kind of very novel, right? Like you can't, you can't label them. If you were working on a labeling approach for ARC and you're trying to find the right label, it, it would be hard to categorize, right? They're one-offs. And, and I think what he, what he's gonna do is have a lot more of these idiosyncratic riddles. And I think that's interesting. I think that's a good thing.

• 56:44Speaker 00

  But I, I also think that the original ARC formulation was already good enough and it's already enough for us to come up with new methods. And even if Remain the same. I think it's still, there's still lots of angles where you can measure, you know, generalization transformers, improve on it, even with this fixed thing. And I, I do think it gets solved with scale. Like, I think ARC V one, you know, give us four thirty nineties or two thirty nineties and more time on, on the problem uh next year and absolutely uh we get to eighty five percent. Um so

• 57:25Speaker 00

  for ARC V one. So I, like, I do think it scales and, and I do have some data on that with the hidden test set.

• 57:33Speaker 00

  That it does scale, actually, uh in a very interesting way that, that you wouldn't expect. But yeah, no, I think it does scale. And I, I, I, and I'm, I'm okay with more reduce Socratic riddles. I think uh maybe that would tune our signal for what is generalization a little bit better. So they're putting a lot of hard work into that and I'm excited for it. I just, maybe if, if they do listen to this, just one request or hope that they also open up the V one again. So that we can have a kind of, just a benchmark for progress on the methods, right? Like, we, we know the scores really well on V one. If we can still submit to V one, even though now there's a V two and it's very different, we can still submit to V one. That'd be really good to,

• 58:17Speaker 00

  to have maybe as like a validation set or a kind of another benchmark, you know what I mean?

• 58:22Tim Scarfe?

  Yeah, yeah. I mean, Sholay's measure of intelligence was, was so fascinating because he was talking about this um uh adaptation, you know, this like, and being able to create a, a new skill program in the response of, of novelty. And if you think about it, it's, it's not a foregone conclusion. This is actually a very difficult thing to do from a psychological point of view. Cause you're trying to create a, a general benchmark that, given the kind of like the base knowledge of, of the average person, that they would be able to like, you know, do this generalization. You gave the example of the one offs. That would be why our man, manhole covers round. You know, like these ridiculous riddles that they used to use at Microsoft to hire people and only one in a thousand people. So

• 59:03Tim Scarfe?

  the information gain on that is basically zero. So you need to create a set of programs that like the average people with their base, basis priors would be able to do. And ARC solved that really well. And I think like, maybe Sholay doesn't get enough credit that he did so cleverly select challenges that, that, that would work well for. And of course, now he needs to diversify across that. One other quick question is, in your experimentation, did you notice patterns? Like which, which types of tasks are you failing on and which ones are you doing well on?

• 59:34Speaker 00

  That's a good question. So Prof Melanie Mitchell has a great benchmark.

• 59:39Tim Scarfe?

  Concept arc, yeah.

• 59:40Speaker 00

  Yeah, yeah. Uh, yeah, she's, she's uh been thinking about this for a long time. She has lots of great ideas there. And uh, one thing I'm noting is, well, I've spoken to lots of people that are in the benchmark and are using neural methods. And they all say, yeah, concept arc, counting is the lowest, right?

• 1:00:01Tim Scarfe?

  Yeah.

• 1:00:02Speaker 00

  What, like, it's just, it's just a thing. And you try to, if you try to add more counting based thriddles or uh, I don't think you should add priors. We haven't tried that, like in, in priors in terms of like feature engineering and that kind of a thing is, is not good. But yeah, just counting for some reason um is really abysmal for neural networks.

• 1:00:24Tim Scarfe?

  Yeah. And I can tell you why. So I interviewed two guys at DeepMind on Monday. And they've, they've studied this in self-attention transformers. They actually say it's because of representational squashing.

• 1:00:39Tim Scarfe?

  I, I won't spoil the surprise. You, you can watch the interview, but um, there's, there's a couple of problems with transformers that because of the way they're set up, they, um, almost all of the attention gets kind of like focused on, um, in the limit, actually, if you, if you scale transformers all the way up, all of the attention gets focused on the first token. And there's also another problem with the softmax function that like creates this kind of like directedness.

• 1:01:03Speaker 00

  Yeah.

• 1:01:03Tim Scarfe?

  And for certain types of, of reasoning tasks, you want it to be directed.

• 1:01:07Speaker 00

  Yeah.

• 1:01:07Tim Scarfe?

  And for certain types of creative reasoning, you want it to be diffuse. But, but they were basically saying that transformers, like even in a trivial sense, cannot do counting or copying. They just can't do it. Like you, you can give it a trivial example where you say, you know, you count up to a hundred and you just say, um, like, can you count these numbers up? And it, and it just fails abysmally straight away. Copying is an interesting example because even if you use tools, you know, like people say, well, it's okay. You can just use tools. You can just like, you know, stick it into a Python tool. If you can't even copy the tokens into the tool, then you can't do that. But the, but the upside of this is all we need to do is fix this problem. Right, all we need to do is whatever the problem is in these architectures, if we can make them copy and count, then maybe all of these problems will just disappear.

• 1:01:50Speaker 00

  Yeah, it's exactly that. You need to kind of dive really deep into the architecture and see where the problem is. And in this case, yeah, it can be very clear what the problem is, right? The softmax is a max, kind of a cheat code way to kind of achieve something similar to a max. And you can see how that would bottleneck things. You can also see how more layers.

• 1:02:13Speaker 00

  You know, there is someone, some really brilliant postdoc told me this. He told me, you want to do the processing in, in a single layer. You don't want to do the processing of, of these things like in one layer. You want to do them as you go up the layers. And, and what he means is if you add, or if you're counting all in one layer, that means you're overfitting, right? Why? Because you're not really running an algorithm, right? You're just kind of taking this feature, this feature, and you're doing it in this one layer. But if you're, if you just have one layer, right, that goes and attends to this thing and updates itself and then goes and updates this thing. It's, that, that's the general algorithm, right, that you want.

• 1:02:58Speaker 00

  It's not this like heuristic kind of MLP in one single layer that will just kind of, I get this feature and this feature and this feature and do everything in one layer. That's not what you want. And so lots of ways of doing that, that I, again, super excited to explore. And I think, yeah, you hit the nail on the head there with, with what you just described.

• 1:03:22Tim Scarfe?

  Most. It's been amazing to have you here. Thanks so much for coming on.

• 1:03:25Speaker 00

  Thank you so much.