Daniel Franzen & Jan Disselhoff - ARC Prize 2024 winners

Daniel Franzen and Jan Disselhoff, the "ARChitects" are the official winners of the ARC Prize 2024. Filmed at Tufa Labs in Zurich - they revealed how they achieved a remarkable 53.5% accuracy by creatively utilising large language models (LLMs) in new ways. Discover their innovative techniques, including depth-first search for token selection, test-time training, and a novel augmentation-based validation system. Their results were extremely surprising. SPONSOR MESSAGES: *** CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Check out their super fast DeepSeek R1 hosting! https://centml.ai/pricing/ Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/ *** Jan Disselhoff https://www.linkedin.com/in/jan-disselhoff-1423a2240/ Daniel Franzen https://github.com/da-fr ARC Prize: http://arcprize.org/ TRANSCRIPT AND BACKGROUND READING: https://www.dropbox.com/scl/fi/utkn2i1ma79fn6an4yvjw/ARCHitects.pdf?rlkey=67pe38mtss7oyhjk2ad0d2aza&dl=0 TOC 1. Solution Architecture and Strategy Overview [00:00:00] 1.1 Initial Solution Overview and Model Architecture [00:04:25] 1.2 LLM Capabilities and Dataset Approach [00:10:51] 1.3 Test-Time Training and Data Augmentation Strategies [00:14:08] 1.4 Sampling Methods and Search Implementation [00:17:52] 1.5 ARC vs Language Model Context Comparison 2. LLM Search and Model Implementation [00:21:53] 2.1 LLM-Guided Search Approaches and Solution Validation [00:27:04] 2.2 Symmetry Augmentation and Model Architecture [00:30:11] 2.3 Model Intelligence Characteristics and Performance [00:37:23] 2.4 Tokenization and Numerical Processing Challenges 3. Advanced Training and Optimization [00:45:15] 3.1 DFS Token Selection and Probability Thresholds [00:49:41] 3.2 Model Size and Fine-tuning Performance Trade-offs [00:53:07] 3.3 LoRA Implementation and Catastrophic Forgetting Prevention [00:56:10] 3.4 Training Infrastructure and Optimization Experiments [01:02:34] 3.5 Search Tree Analysis and Entropy Distribution Patterns REFS [00:01:05] Winning ARC 2024 solution using 12B param model, Franzen, Disselhoff, Hartmann https://github.com/da-fr/arc-prize-2024/blob/main/the_architects.pdf [00:03:40] Robustness of analogical reasoning in LLMs, Melanie Mitchell https://arxiv.org/html/2411.14215 [00:07:50] Re-ARC dataset generator for ARC task variations, Michael Hodel https://github.com/michaelhodel/re-arc [00:15:00] Analysis of search methods in LLMs (greedy, beam, DFS), Chen et al. https://arxiv.org/html/2408.00724v2 [00:16:55] Language model reachability space exploration, University of Toronto https://www.youtube.com/watch?v=Bpgloy1dDn0 [00:22:30] GPT-4 guided code solutions for ARC tasks, Ryan Greenblatt https://redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt [00:41:20] GPT tokenization approach for numbers, OpenAI https://platform.openai.com/docs/guides/text-generation/tokenizer-examples [00:46:25] DFS in AI search strategies, Russell & Norvig https://www.amazon.com/Artificial-Intelligence-Modern-Approach-4th/dp/0134610997 [00:53:10] Paper on catastrophic forgetting in neural networks, Kirkpatrick et al. https://www.pnas.org/doi/10.1073/pnas.1611835114 [00:54:00] LoRA for efficient fine-tuning of LLMs, Hu et al. https://arxiv.org/abs/2106.09685 [00:57:20] NVIDIA H100 Tensor Core GPU specs, NVIDIA https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/ [01:04:55] Original MCTS in computer Go, Yifan Jin https://stanford.edu/~rezab/classes/cme323/S15/projects/montecarlo_search_tree_report.pdf

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Tim Scarfe?

  Jan and Daniel, it's amazing to have you on MLST. Welcome, both of you.

• 0:02Speaker 01

  Thank you for having us here.

• 0:05Tim Scarfe?

  So you guys are the official winners of the ARC challenge. The results were finalized, I guess, about a month ago or so?

• 0:12Speaker 01

  Yeah.

• 0:13Tim Scarfe?

  And why don't we just start by, you know, tell us a little bit about your solution?

• 0:19Speaker 02

  It was a process with lots of ups and downs. Like, we started with a simple LLM fine-tuning and had pretty good results.

• 0:30Speaker 02

  But during the competition, we tried to add

• 0:35Speaker 02

  additional computation steps outside of the LLM, which helped us to increase our scores substantially. Our LLM alone reached a score of 41 points, I think. But with the additional external help, we could get the LLM to improve substantially to like 56 points. But they didn't count, so 53.5 points at the end.

• 0:57Tim Scarfe?

  So can you talk us through the overall architecture?

• 1:00Speaker 01

  Yeah, so we actually started with a basic LLM that was trained on language. We started out with a rather big model. It was a 12 billion parameters model. And what we essentially do is we just take the ARC tasks, which are essentially located on a grid with pixels of different colors. And we just have one token for each color. And we tokenize them line by line and we then put them directly into a LLM. We do no program search or other preprocessing. We just convert them to text and put them into the LLM. And that worked amazingly well, better than I did expect. And of course, there are quite a few tricks we need to do to make it work better on these tasks,

• 1:39Speaker 01

  because the reasoning capabilities of LLMs without essentially fine-tuning on the specific task are not so good. So we essentially did some test time fine-tuning. So we had essentially two training processes. We first did a long pre-training on the official ARC training set.

• 1:58Speaker 01

  We later replaced it by Rearch, which can generate more tasks, more examples. And this model did not perform so good on the validation tasks. So what we do is test and training. We do another training process on the examples of the validation set, which we get during inference, without the final challenge, which we then try to predict with the model. And that gives a big improvement to the score. There were quite a lot of other improvements that we tried during the challenge. Some did work very well, but there's also a lot that didn't really turn out well.

• 2:34Speaker 03

  So Tufa Labs is a new AI research lab I'm starting in Zurich. It is funded from Past Ventures, involving AI as well. And so we are Swiss version of DeepSec. So a small group of people, very, very motivated, very hardworking. And we try to do some AI research, starting with LLM and O1 style models. What we're looking for now is chief scientist and also research engineers. You can check out positions at tufalabs.ai.

• 3:01Tim Scarfe?

  MLST is sponsored by Centimel, which is the compute platform specifically optimized for AI workloads. They support all of the latest open source language models out of the box, like Llama, for example. You can pay on consumption essentially, or you can have a model which is always working, or it can be freeze drive when you're not using it. All of the models that they deploy support the OpenAI API specification out of the box, which means it's just a one line change in your application to switch over to Centimel and start saving money and make your application go faster. Yeah, it must have been a fascinating process, kind of trying all of these different ideas and seeing what stuck and what didn't. I've updated so much. So, I mean, Melanie Mitchell had quite an interesting paper out talking about her concept arc and, you know, what it, what it means for LLMs to understand and all of this.

• 3:46Tim Scarfe?

  And the, the common sense rationale was very much that language models will never be able to do this. And the only way it could possibly work is if we generate an intermediate program, because symbolic things are better, they're more compositional, they generalize better. And certainly, I think if you took GPT-4 and just did zero shot, you know, solution space prediction, you were looking at around 10%, which is non-zero. It's certainly much better than something like GPT-3.5. But many of us, myself included, just ruled out the possibility that these things could be reasoning. What did you discover on that journey?

• 4:25Speaker 02

  We started, I at least started with the same assumption. But we quickly found that the L L M's have far more computational capability than we thought. For example, all tasks seem like 2D vision tasks, right? They are on a 2D grid. If you see them, you use your perceptual knowledge to move blocks or move pixels around.

• 4:50Speaker 02

  And my heuristic was an L L M is on 1D, like a text. Text, we have slash N for go to the next line. And it seemed crazy to me that the L L M just functions in that space. Like it has to learn to infer the 2D structure of the problem without ever working in 2D. And the more we trained the networks, the more it seemed like that just wasn't a problem. The L L M's were strong enough, had enough capability to infer this structure implicitly. And we did some experiments where we tried to help the L L M's, giving them more information about the structure, like pending dimensions or coordinates.

• 5:36Speaker 02

  But it wasn't needed. It rarely improved the score. And if it improved the score, it did so by a very small amount. That didn't, wasn't worth the additional compute time, basically.

• 5:47Tim Scarfe?

  So I think one of the intuitions there is it's a 2D grid. I mean, certainly in transformers models, you know, we put in positional tokens and stuff like that. So the remarkable thing is that it generalized to different grid sizes. And you're saying without you needing to explicitly give it any like compasses or markers or anything like that?

• 6:08Speaker 02

  We actually tried to do a 2D positional encoding on the data, but it didn't really improve the results. So it just learns to detect the end of line markings and to output lines with the correct length. It just worked fine. And we then stopped with the precision encoding approach because it was too complicated and didn't give us any advantage.

• 6:31Tim Scarfe?

  You did fine tune the model. I think you ended up using a Lama eight billion. Is that, is that what you're using?

• 6:39Speaker 01

  That's what we actually started with. Then we went to a 12 billion model. And then because this model was too big to do much things on Kaggle. We actually went back to a smaller model. But it was the Lama 3.23B model. Which we found to be essentially as strong as the 8B model. But being much faster in execution. So we could try a lot of more things.

• 7:04Tim Scarfe?

  Yeah, I'm very excited about that new Lama model. It's amazing how much they've gotten out of it. So just with that model, out of the box, zero shot, you know, in solution space. What kind of accuracy are we talking about?

• 7:17Speaker 02

  Without training?

• 7:17Tim Scarfe?

  Without any pre-training?

• 7:19Speaker 01

  Any training, I think close to zero or zero.

• 7:22Tim Scarfe?

  Interesting.

• 7:22Speaker 01

  The model can't even predict the structure really. Sometimes it's just output its end of sentence tokens indefinitely. Sometimes it predicts some numbers like it should with end lines. But then some lines are too long, some lines are too short. Hardly produces any solution that is even possible without any training process.

• 7:45Tim Scarfe?

  Very interesting. So you, you trained it on Rearch, which is Michael Hodel's dataset. Maybe talk a little bit about that.

• 7:52Speaker 02

  It was very useful to have a dataset that is basically unlimited. Like we have the four hundred tasks on the easy dataset. And Rearch helps to generate more examples for each challenge. It is in a way, at some point you saturate because you don't get novel ideas in these tasks. But it was very useful to just have virtually infinite generatable training data.

• 8:19Tim Scarfe?

  Very cool. So Michael Hodel's thing is a dataset generator. So given a task, you can create a whole bunch of additional, you know, augmentations of that task. So you had about 10,000 of those across all of the, and the ARC tasks in the, in the public set. And you fine tuned, let's say Lama 3.2 billion on that. And, and then zero shot, you know, solution space out of the gate. What are we looking at?

• 8:47Speaker 01

  Oh, let me think for a moment. With the LAMA model, I think we are maybe about ten to twenty percent performance, I think.

• 8:57Tim Scarfe?

  Interesting.

• 8:58Speaker 01

  Uh, I'm not sure, because we did some uh multiple inference steps and something uh. So you could get the zero-shot model with some tricks to maybe twenty or twenty-five percent. That was the max we could get out, I think.

• 9:11Speaker 02

  On the, on the evil data, that's right.

• 9:12Speaker 01

  Yeah, on the evil data.

• 9:14Tim Scarfe?

  Well, that brings us to the magical test time, test time inference, or you know, there are lots of different names for it.

• 9:20Speaker 01

  Yeah.

• 9:21Tim Scarfe?

  People have been using different names for it, but tell us about that.

• 9:24Speaker 01

  Yeah, what we actually did uh was after the model was fine tuned the first time, we did an additional training step on the evaluation data, so not on the full evaluation data, because we only had the examples and not the challenge itself. But the examples essentially look the same than the challenge, so you could just pick one output from the example as the final example, and that's just train on these inputs. On the KAN contest, we essentially did a retraining on uh some of these examples. And uh after that, we did the inference step, and this uh improved the results quite a lot.

• 10:03Tim Scarfe?

  I suppose you could argue that's, I don't know whether you'd call that cheating or not, but in, in the real world, I suppose you would be doing an online prediction. And what you're doing is you're, you're making your predictive model as good as it would have been at the end of a line of doing online prediction.

• 10:19Speaker 01

  You're correct with on that, yeah, but uh we tested different versions and it's also possible to essentially retrain on every single task. So that would uh essentially be a fairer version, so the model doesn't see the other evaluation tasks. That works just as good, but in our experiments, it required more time to retrain than just putting in all the tasks at the same time. And because time was so limited in this challenge, we choose the option to train on multiple tasks at once.

• 10:51Tim Scarfe?

  Yeah, so there's there's this thing that is transductive active fine tuning, which is when at test time, you you have the specification and you generate augmentations from the specifications. And then you fine tune on those augmentations and then you do the prediction.

• 11:10Speaker 02

  Do you mean from test, do you mean from the from the from the new challenges?

• 11:13Tim Scarfe?

  Yeah, so you you get, let's say you get three new specifications for a task. And then do you augment that and fine tune on it?

• 11:21Speaker 02

  Yeah, we use, we heavily use augmentation in basically all our inference steps, also in training. We also use it for for the pre-training in VR actually.

• 11:33Speaker 02

  But it helps a lot to get enough data. During the retraining or test time training, we are heavily data limited, right? Because each challenge only has three examples. But we can use the symmetry of the problem because all the problems work on a two D grid. And there are some implicit biases in it. So we can rotate it, we can mirror it, we can even shift the colors in specific ways without changing the challenge actually. And since the model is not rotation invariant, right? It's a one D L L M. We can create the rotated augmentations and use them to train. Yeah, the challenge too.

• 12:18Speaker 02

  And it's it's looks like a novel task for L L M because it's rotated differently.

• 12:24Speaker 02

  And which helps a lot with the amount of data we have.

• 12:26Tim Scarfe?

  So from the test specifications, you do symmetry augmentations and fine tuning. Do you do any any additional augmentations?

• 12:34Speaker 02

  We were thinking about a lot of augmentation stuff.

• 12:39Speaker 02

  Most of our tests didn't pan out or were too time intensive. Like, for example, we have a selection step where we

• 12:51Speaker 02

  take the generated example, like our solution, and then we do an augmentation again, like we asked the LM basically, does this solution look correct from a different perspective too? And theoretically, you can do more augmentations there. Like you can shift into another space where the problem looks simpler, as long as there's some consistency in the solution. Like, for example, you could many, many problems have the same solution if they were in black and white instead of in color, for example. And in theory, you could use that to filter wrong solutions very easily. But we didn't have the time to get it to work, basically.

• 13:32Tim Scarfe?

  So, other than the symmetry augmentations, there were no additional augmentations?

• 13:37Speaker 02

  Symmetry, color shift, and shifting the examples, because the transformer is not invariant to the order of the examples. So we also shuffled the examples a little bit around.

• 13:49Tim Scarfe?

  Very interesting. So then we get to the, we're getting close to putting the, you know, the pin in the middle of the dartboard here, which was like the main thing that you guys did. So you do some kind of a search and evaluation process as part of your test time inference. Can you, can you tell me about that?

• 14:08Speaker 02

  There are actually different methods of sampling. For LM, you could just do greedy sampling. This means also you always take the token with the highest probability when sampling. Or you could do sampling in a stochastical way, by just taking it according to its probability that's predicted by the network. And all these sampling methods didn't work really well if we want to generate lots of different candidates. So what we actually did...

• 14:39Speaker 01

  We implemented a selection process later on, which we can maybe talk about later. So the focus here in this generation step was essentially to generate lots of candidates to increase the chance to find the correct solution. And yeah, we first started out with sampling and we also tested beam search because often there were different paths through the network which give different predictions and one of them is correct. But in the end, we settled for a depth first search, which we implemented ourselves because we haven't found it anywhere.

• 15:15Speaker 01

  And that worked quite well and it had quite some advantages over beam search. So what we do is we just treat the tokens that are predicted from the network as a search tree. And we then search through this tree to find all solutions which has a sampling probability above a certain cutoff value. So for example, we set this probability to 10% and then we get a variable amount of candidates. This could be up to 10 for 10%, of course. This search has quite some advantages. The first one is that it's very memory efficient.

• 15:59Speaker 01

  You only need to store exactly one path. So it's memory efficient as a single inference step and much more efficient as beam search which needs one essentially the same memory for each beam.

• 16:16Speaker 01

  And the other advantage is that it gives us multiple solutions at once for the problem. And the third one is that we have a cutoff. So if there are paths that are not promising, so in these cases, the search stops early and doesn't even follow them. And this gave us a big efficiency boost in generating solutions, or solution candidates, I would say.

• 16:44Tim Scarfe?

  This is so fascinating. I interviewed these guys at the University of Toronto and they were talking about the reachability space of language models and they had this famous Roger Federer game, which was like, you know, you try and you make the language models say great, Roger Federer is the greatest and you have to kind of find what prompt will reach that word. And their theorem essentially is that the capacity or the reachability of transformers is incredibly flexible. There's a remarkable divergence between the way that we sample tokens from a language model and what they're capable of. And you know, like a clearer way of saying that is that there is a misalignment between

• 17:26Tim Scarfe?

  how we sample language models and the epistemic truth or the correctness of the program that we are sampling. And I find that fascinating. It's also related to this creativity problem, which is that when we do this greedy sampling, which means we just take the best token and the best token and the best token, we're kind of missing out on all of these creative possibilities. And this is something that a lot of people aren't really thinking about yet.

• 17:52Speaker 02

  I think that's a, it's a very interesting problem. And we have to distinguish between like a language context and the ARK context. There's a substantial difference, namely that we only have 10 possible tokens that might be the next correct answer. And also we know there's only one correct answer. So in language, you can have millions of ways of rephrasing the same idea and all of these are valid or good continuations. In our case, which is one of the reasons why the DFS works so well, we only have one correct continuation and we have to find it.

• 18:34Speaker 02

  It's very easily evaluatable and it's very easily searchable, also because the set of possible continuations is much, much smaller. We can even calculate how probable each solution is by simply taking the solution, doing a forward pass, calculating the logits, summing them, and then we know the cutoff needs to be this low for our DFS to find the correct solution guaranteed. And that makes the problem very tractable in this sense. And all the sampling algorithms are built for language, because the LLMs work on language, obviously. Which is also why we didn't find the DFS, because we had re-implemented it ourselves, because it's

• 19:17Speaker 02

  it's not a valid way of sampling a language, it just can't work. It's the language has too much possible paths, and each path, as long as it's valid, should roughly have the same probability. So if you have, if you have one million possible continuations, each probability is one by one billion, million. So it's very hard to use it in this context. But because our problem is discrete, small, we can sample very, very efficiently.

• 19:45Tim Scarfe?

  This raises so many interesting questions, because you could argue then that the reason why these models are misaligned between program or knowledge correctness and how we sample them is by your argument, because there are too many degrees of freedom in, in natural language. So, you know, it's almost like they've, we've, that they're dealing with this ambiguity in their training, therefore, when we sample from them, we're, we're not very likely to get the correct answer.

• 20:12Speaker 02

  You have to be careful about that, because we have to differentiate between the probability of a path and the probability of the answer.

• 20:22Speaker 02

  You can have a very improbable path that leads to a specific answer, and then you can have multiple variations of it, you know. But the, the probability of the answer is not dependent on any path, but on the sum of all paths that lead to this answer. So in, in language, you have, have this, the, the, this broad set of paths that all lead to the same solution. And when you use standard sampling techniques, you sample the final answer correctly, like that's the, that's the use of sampling.

• 21:02Speaker 02

  In the ARC contest, there are no alternative intermediate steps, right? So. You don't need to marginalize all possible paths, basically. You can just, you need to get the correct path, basically. So in normal L L M's, this is completely fine. But I think in the age of reasoning models we get into, like O one or O three, you also need the correct reasoning steps, the correct intermediate steps to get to the right solution, especially if you use it for code generation or answering complex mathematical questions. So this will be, I think in the future, this will be a much larger problem.

• 21:48Speaker 02

  But up till now, sampling was simply the correct approach.

• 21:52Tim Scarfe?

  A couple of things on that. So certainly, perhaps in your solution, the way that you traversed that sparse space, and the reason I say sparse is it's a bit like in reinforcement learning, that you need to take several steps without knowing what the value is going to be. And this is really, really difficult when we have some kind of traversal optimization algorithm where we rely on a monotonic signal. So we're getting better, better, better, better. Like in this case, the space is sparse and we have to take several steps into the unknown. But the thing that really interests me, though, is even if we don't know what the reasoning steps are. Ryan Greenblatt's solution really impressed me and surprised me

• 22:33Tim Scarfe?

  because I assumed that this is just an exponentially large problem and that using a language model to guide the search would not be significantly better than just exhaustively finding programs. And what he demonstrated quite succinctly is that it's tractable. I mean, yeah, we still need to generate loads of completions and do some search, but a reasonable amount of search gets you to the solution. And what one's saying there is that even though there is an apparent orthogonality between correctness and searching this space, the correct solution isn't that far away.

• 23:10Tim Scarfe?

  That's fascinating.

• 23:11Speaker 02

  Yeah, I think the interesting thing about Ryan Greenblatt's solutions from my perspective is that he showed that code generation can work, like that the models can see the visual problems, like the one D representation of the grids, and then produce code that works to move these two D objects even though they were never seen, even without fine tuning. On the other hand, code generation has a massive advantage compared to our approach because you can just test your code. Like, you can check, does the code that's generated perform in the way I want it to perform? And if it does, then it's probably a good solution.

• 23:53Speaker 02

  Whereas our approach doesn't have that. Like, we had to find a way to select which candidate we want to submit without the possibility of having any guarantee that it's right.

• 24:07Tim Scarfe?

  Well, yeah, this gets us on to the most delicious part of all. So as you just articulated, Ryan Greenblatt's solution was technically a neuro-symbolic architecture. So, I mean, Sabaro Kambohati would have been delighted because, you know, he loves having formal guarantees. He loves actually knowing for sure. I mean, actually, in this case, we don't know for sure because many correct programs are not correct, of course. But but at least, you know, we can run a program on a Python interpreter and we can get a yes or no answer. It might be a false positive. Now, you guys have done something very interesting. You're you're using the language model to verify its own value as you traverse through this tree structure. Can you can you tell me about that?

• 24:50Speaker 01

  Yeah, so our generation process essentially generates multiple solution candidates. Could be, usually it's up to 10 or 20 in practice. And now we need to find a way to select the correct one. And what we actually did is we used our model also for judging how good these solutions are. And the interesting part is it's difficult if you do it without augmentations. Because a model of course favors its own predictions. Because what we use as score is the sampling probability. Probably we forget it in the first place. So what we essentially do at this point, we also use a lot of augmentation for the judging.

• 25:34Speaker 01

  So we essentially use sixteen different augmentations of the problem and put them through the model to calculate scores for each augmentation. And then we average them. And this is an interesting part. We tried different algorithms. One was just for example to sum up the actual probabilities. And the other one was to sum up the logarithmic probabilities, which would essentially be equal to multiplying the probabilities. And the second one worked much better than the first one in selecting. And yeah, we really asked this, what, why this is.

• 26:13Speaker 01

  And so our hypothesis is essentially that the good, the correct solution looks mostly correct from every angle. No matter how you rotate it, it gets maybe not a really high score, but it doesn't get an extremely low score. And the incorrect solutions, they might also look good in some perspectives. But usually there's one or two or even more perspectives where they get an extremely low score, like zero point zero one percent or something in probability. And so essentially these low scores are important when doing the calculation. And you can't get these scores during the generation process because these things are never sampled.

• 26:57Speaker 01

  So we actually need the extra scoring step to calculate these low scores to sort out the false solutions.

• 27:04Tim Scarfe?

  This is absolutely fascinating. So you you generate sixteen augmentations for, let's say, you know, a test specification. And as you said before, you had already fine tuned the model on symmetry and, you know, permutations, which means it's it has awareness of different symmetry transformations. And you're saying that using it to do self reflection on the value of those symmetry augmentations gives it different perspectives on what good looks like. And then you can kind of look at those, you know, look at the coherence of the different symmetry perspectives to give you an idea of which one is correct.

• 27:40Speaker 02

  Yeah, exactly. Ironically, it's very useful to us that L L M does not have symmetry in its predictions. And it's the interesting part is that it's useful to us that the language model does not have the two D symmetries. Like if it had two D symmetries, we couldn't use our augmentation for scoring because the score would be the same every time. So in a way, we abused the fact that L L M is not perfect. It's two D tasks. Also because it's it's always generating from left to right, from up to down, right?

• 28:19Speaker 02

  So some parts of the puzzle get generated later with more information and some parts get generated earlier with less information. And if we rotate it, we generate other parts of the solution first. So a lot of the score differences between the different augmentations come from the fact that it can generate maybe complex parts or parts of the solution first or later. Like it helps that the helps that sometimes a solution depends on previous input. And

• 29:04Speaker 02

  If we generate the wrong solution and then rotate it, it sees it much earlier, like it can see, ah, it makes no sense that like the line that goes from left to right here ends at an empty pixel, it should never happen. And because we rotated the problem, it can see it from a new perspective and then decide that it's an invalid solution or very improbable solution.

• 29:25Tim Scarfe?

  How much does this depend on the symmetry augmentation before test time? So do you guys say that before we even get to any test time computation, we take all of the evaluation tasks? And then we do some symmetry transformations and we fine tune. If you don't do that, how does that affect this evaluation?

• 29:47Speaker 02

  We have never tested that.

• 29:50Speaker 01

  So you mean not doing the augmentations during training?

• 29:53Tim Scarfe?

  Yes, yeah.

• 29:55Speaker 01

  Yeah, we have never tried that out. We always use augmentations from the beginning.

• 29:59Tim Scarfe?

  Because I'm a bit confused, because you seem to be saying it's actually a good thing that it doesn't know about the augmentations. So doing the augmentations, it seems to me that that would be a good thing, but maybe it isn't a good thing.

• 30:11Speaker 02

  It's a trade off, like we would lose a lot of training data. And it's also, it is good if it can just generate the correct solution from every perspective, right? But some problems are simply easier from some perspective than others. And in theory, the network should converge to a correct solution from every perspective. If it's deep enough, if it's strong enough, but sometimes it just can't. And then we can use that fact by generating different augmentations and scoring them correctly.

• 30:46Tim Scarfe?

  I'm trying to understand as well, so that there are different perspectives on language models. So Sabaro Kambahati, he called it jagged intelligence, I think, which is what Andre Karpathy called it. You know, and that's this idea that pointillistically, they're actually very good, which means they have these islands of knowledge. So you can say, I'm going to do a symmetry transformation and ah, it knows. So it, given an example of something, it knows, knows if it's good or bad, but it doesn't know how to get there. So there seems to be like these concentric circles where they are better at discriminating than generating. And they also are capable of knowing if they don't know how to get there.

• 31:28Speaker 02

  I think in some ways, yes, it depends, it depends so strongly on the task like always, right? There are problems where it's true that they have a much easier time of discriminating the solutions than generating them, but at the same time,

• 31:46Speaker 02

  it depends how you measure it, basically. Like, we measured the score of a task or of a solution by calculating basically the probability that it would be generated from normal sampling. So in our case, generation probability or generation and evaluation is exactly the same. Um, the only way we can then use the discriminatory abilities of the L L M's is by using these transformations. Because like Daniel said, L L M's prefer their own outputs. So we have to, we have to trick the L L M to think that it's a output it would have never done.

• 32:26Tim Scarfe?

  There's also this thing that Thomas Dietrich was telling me that um, it is very interesting looking at the divergence between epistemic risk and aleuteric risk and that that's simply that there's a divergence between things that are true and things that the language model will kind of like, you know, will give you if you stochastically sample them. And, and he said that before R L H F, there is much more of a correspondence between the likelihood of a trajectory and epistemic factfulness. So whether the thing is actually true or not, and R L H F deranges that. But I'm thinking that doesn't really affect your situation because you're doing fine tuning on the top.

• 33:08Tim Scarfe?

  So you're kind of, I mean, what you're doing is, is either overriding or orthogonal to any R L H F training. So you're, you're actually deliberately saying, you know, here is what good looks like. And if you can generalize from that example of goodness, you can use the likelihood of a trajectory as a proxy for, you know, correctness.

• 33:30Speaker 01

  We actually used a Lama unsensored model in our experiments. And we also compared it to just to the normal model and the unsensored model was a little bit better. So maybe that makes a little difference, yeah.

• 33:47Tim Scarfe?

  Was it retrospectively uncensored or was it just never RLHF trained?

• 33:53Speaker 01

  Retrospectively, it was from which one?

• 33:57Tim Scarfe?

  Yeah. Yeah, because I've I've played with uncensored models, you know, like the the Gwen, the Gwen models. And I found that they are lobotomized. And for whatever reason, like the however, they are uncensored, I think there are various different approaches to uncensoring them. But they just seem to degrade to the performance of a model, you know, half the size. So like a seventy two billion model will now have the reasoning abilities of a thirty two billion model. But that's just my experience.

• 34:26Speaker 02

  We are actually not sure if it's at all important that we use a pre-trained model. Like one of our experiments in the pipeline at the moment is just resetting all the weights and then training to check if we even use the pre-trained language capabilities or if it's just the architecture, actually.

• 34:46Tim Scarfe?

  That's very interesting. Yeah, I I interviewed Randall Bellistriero at NERIPs and he made the same discovery. So he's got this really interesting paper out. And, you know, we are taught that we need to have these large pre-trained models and we're leveraging the the knowledge inside the model and so on. And he apparently no one has really done this experiment that you can just kind of and this is for discrimination, not generation, right? So if you're just doing some kind of classification problem, you can take a very large model. You can train it from scratch for doing some like discriminative task. And you can actually get better performance than like frontier models pretty pretty much with hardly any training and hardly any data. Like, isn't that and and and it almost goes back to those papers from years ago, you know, that

• 35:30Tim Scarfe?

  there are so many inductive biases just in the architecture of the model itself that, you know, you don't actually need that much training.

• 35:37Speaker 02

  I think one of the interesting parts is that we we lobotomize our L L M tool like at the end, we have an L L M that has basically only one hundred forty tokens or something. Like we remove all language capabilities from the model, just delete it.

• 35:56Speaker 02

  And really make sure that it can only argue in the space of Arc. Like it can only output numbers zero to nine, end line tokens and some letters. And, um, we did so mostly because of computation requirements. Like it saves surprisingly a lot of RAM. But yeah, we, we remove all language capabilities. So we are often asked, do we do any pre prompting? Is it like, can I thought thinking? It can't, it's all gone.

• 36:27Tim Scarfe?

  Well, that's really interesting because certainly the Minds AI team, I was talking with Jack. And, you know, they're, they're very bullish on this idea of like multimodal or, you know, cross domain transfer. So they, you know, I think Jack was saying, for example, they might even give English language descriptions of the Arc challenges. And they're, they're looking for as many ways as possible to leverage the base model in, in the language model. Uh, you know, and it, there are two views on this. Like one view is that these things are a general intelligence. And the whole point of O three, for example, is that it's kind of like, um, it's creating a novel skill program by doing some kind of compositional generalization. And it's like, you know, working in this novel situation.

• 37:08Tim Scarfe?

  Whereas this, this alternative view is very much that they are a jagged form of intelligence and very specialized form of intelligence. And they're mostly just working on specific instances of very similar things that they've seen before.

• 37:22Speaker 02

  Indirectly, I think we could say that we found

• 37:28Speaker 02

  that the model performed much better at tasks where it had seen similar tasks. Like, um, in our paper, we have, uh, we have claimed the performance of seventy six percent.

• 37:42Speaker 02

  Seventy two percent on, uh, left out part of the evaluation data set.

• 37:48Speaker 02

  But the performance drops or dropped at the time noticeably if we never trained on any evaluation data set tasks. And we believe that to be because there is some conceptual leakage because it's from some tasks of the relation data set to other tasks. Like they, they're not the same, but they have similar ideas. Which is probably also a reason why the additional data sets like concept dark or bar helped. Because they maybe introduce some novel ideas in some way. And if the L L M has never seen an idea.

• 38:25Speaker 02

  Conceptually in training, then it's very unlikely that it will perform well on it. So for example, if we train the LLM on arc challenges that are, I don't know, fail flat algorithms or something like that, then it will not transfer its skills into a counting task or something like that. But as long as it has seen some counting tasks, it might be able to combine the ideas to form like novel challenges.

• 38:54Tim Scarfe?

  It is very interesting that, I mean, just take multiplication, for example. Frontier models, they don't generalize very well. You know, I mean, they do a little bit, you know, three digits, maybe four digits or something like that. But you can take GPT two and you can fine tune it up to 20 digit multiplication and it works better than any frontier model. So there's always this notion that doing fine tuning for a specific task is better. Architecturally, that doesn't work very well, right? Because, you know, we're in this world and we're subject to novelty all the time. And we can't possibly be fine tuning in every possible situation. So maybe that's when this active fine tuning, like this test time computation comes into the play. Because increasingly we'll build architectures where we are learning online.

• 39:40Tim Scarfe?

  And maybe sharing that knowledge in a clever way to other models. But it kind of feels that with this new online version of prediction, we can kind of have our cake and eat it.

• 39:51Speaker 01

  So the interesting thing is for the multiplication, tokenization plays a big role because the models tokenize numbers differently. And the model we used was using for some digits, just one token. But sometimes it combined digits to one token. So essentially there were digits zero to nine, it could tokenize. But then there were some two digit numbers and some three digit numbers. And yeah, I think newer models do this differently or probably have.

• 40:26Speaker 01

  All three-digit numbers integrated. But for us, this was a problem in Arc, so we also removed all this from the tokenizer. So we just have single digits. Because this would have been a big problem, I guess, if the length changes of the outputs.

• 40:43Tim Scarfe?

  Oh, interesting. So you modified the tokenizer to remove all of the compounded.

• 40:48Speaker 01

  We essentially removed anything except numbers, end of line, and end of problem. And input and output prefix.

• 40:59Tim Scarfe?

  Very interesting. Yeah, and of course, you weren't deranging it because you were doing the training on the top. So you're kind of like doing a high-resolution token version of training.

• 41:08Speaker 02

  Exactly, yeah. And I think for the multiplication task too. It's a common problem I see on Twitter is that people misunderstand how complicated a problem is for LLMs. Like, I think OpenAI has almost every three-digit number as a token. So if you tell ChatGPT to multiply two three-digit numbers. It has to basically remember one million multiplication pairs. Like, it's a very, very complicated task. And then people are like, yeah, it can multiply three-digit numbers, but it's a hard task for ChatGPT. The same with the typical strawberry task.

• 41:50Speaker 02

  Like, in, like, my way of explaining it is imagine, like, every syllable is a token in a, well, a letter in a language you've never seen. And you have to remember how many Rs are in each of these tokens. And you have to remember that for one hundred and thirty thousand different things. Like, it's a complex task. And then you have to learn it without ever seeing the letter R. It's hard. LLMs are very, very strong problem solvers if you fine-tune them. But they are not the intended solution of Arc in the way that we use them. So in a way, it's a hacky solution. But because of the challenge, you have to use the best-performing model.

• 42:33Speaker 02

  Like, you could do a very creative solution that goes further into the direction of AGI, right? But you wouldn't win. You win with LLMs and abusing, like, fine-tuning speed and stuff like that. So LLMs are. Too strong for this contest to enable any other solution, basically.

• 42:54Tim Scarfe?

  Interesting. What about the thing that right now, um, this is quite an interesting coincidence that ARC was designed to be solvable by humans. And one of those restrictions is that they shouldn't be more than about thirty times thirty cells on a grid, because otherwise humans would start making loads of mistakes. But that also just almost coincidentally happens to be the upper bound of what we can, you know, correctly do with an L L M. If it was fifty by fifty or even sixty by sixty, no one would be using L L M's to solve ARC.

• 43:26Speaker 02

  We would have only solved the small problems, yeah.

• 43:29Tim Scarfe?

  Yeah, because even now, like when we look at the failure modes of O three, there's definitely like you can see it tailing off with solution size. So it starts to make more and more mistakes, you know, when it gets to thirty by thirty. Um, but it does raise the question, though, of the complexity of the problem. So again, my intuition is that this problem has a kind of exponential complexity to it. And I would have said the same thing about language. It's interesting that language. If you think about all of the ways that a reference can hierarchically, you know, um, connect to each other. It it seems like as the length of even a book, for example, the

• 44:07Tim Scarfe?

  the information complexity of a book should be exponentially large, because there are all of these like connecting things between all the paragraphs and the sentences and the words and so on. And language models don't have any problems understanding and explaining books. And I want to understand what what what that is. So there's there's the potential complexity of all of the possible connections between the terms. And then there's the actual embedded complexity of of language. And I I think you you're making this argument earlier that that you feel that actually the the the sort of the embedded complexity doesn't scale exponentially with the problem size.

• 44:47Speaker 02

  Yeah, it's um, in many challenges at least, like you can you can see that, like, many problems are.

• 44:55Speaker 02

  Mostly trivial, there are just a few pixels where you have to do the right decisions. We even measured that and we found that in a intermediate state, stage of our solutions, where we still did normal sampling, where we didn't use DFS. We found that we had to get a model to choose the second highest probability token three times and then we would have solved eighty percent of the challenges. But we had to get it to choose these and we don't know which of the nine hundred tokens, the second best is the correct one, you know. The space of solutions the LLM generates is much, much smaller than the theoretical

• 45:41Speaker 02

  large space because a lot of the pixels are trivial. Like the background is often trivial, like many problems are move this object here. And if you start object at the position, then the LLM generates completely the rest of the object correctly, but has to decide where to put this object and that's the hard part. But the number of decisions we have to do to generate the correct solutions is surprisingly low.

• 46:09Tim Scarfe?

  So does that imply that there's no reason in principle why an LLM wouldn't be able to scale to a hundred by a hundred or even a thousand by a thousand?

• 46:18Speaker 02

  I think it's one of the reasons why Daniel's DFS solution worked so well for us because when you do normal sampling, each token has a probability of being selected wrongly. Like there are many ways of fixing stuff like that, like min p sampling, top k sampling or something like that. But we have for every token we generate, we have a small chance of doing it wrong. And our DFS solution just generates all solutions that are.

• 46:51Speaker 02

  Above a certain probability. So we are basically guaranteed to get the most likely solutions and some below that, which reduces a lot of the problems with the exponential problems of sampling, as long as our probability bound is low enough.

• 47:08Tim Scarfe?

  How did you decide on the depth of the search and the threshold?

• 47:13Speaker 01

  We decided about the depth by just essentially just testing it out. So how deep can we go to find in most cases the correct solution? And it's of course a trade-off because if you search down to 1%, you can get 100 solutions in theory. So in practice you get much less. And if you search to 10%, you only get 10 solutions and you get much less solutions. So in some problems where the solution is clear, it doesn't make a difference. You just find the solution. But when the solution is unclear, you get a lot of samples for going down to 1%.

• 47:55Speaker 01

  And that would have been computationally infeasible on Kaggle. So in the end, it was a trade-off that we just tested out. And we ended up with different values between 10% and 17% in the end that we tried in practice. And the solution was mostly in there. Going down from 10% to 1% gave an improved, I think, from 70% to 71% on the score. So not much.

• 48:24Tim Scarfe?

  And still technically a variable computational budget. Because if you think about it, at 1%, when you do it 100 times, you could get a huge variance in the amount of searching that you're doing. But it's kind of vaguely budgeted. Is that correct?

• 48:41Speaker 01

  So it's not clear before and how many solutions you will get from the DFS. Yeah, so this can be more or less depending on the problems. And something that was very interesting is, in the end of the contest, we switched back to a larger model because we fine-tuned a lot.

• 49:00Speaker 01

  And when you do normal sampling, switching to the larger model, there's a certain factor by how much more compute you need for the larger model, because the inference is essentially just in time something like a compute. But for the DFS, because the larger model gets better at deciding what the correct solution is, it could actually prune earlier in the DFS. So it still took a little longer, but not as much longer as we expected it to take from our previous tests. And that was quite surprising and that also enabled us to do more in the short time, short compute time and cackle.

• 49:41Tim Scarfe?

  Did you guys just, just for fun at home, try a 72 billion model or something bigger?

• 49:47Speaker 01

  I tried it on the evaluation set after the contest, but I did just one pass. I don't think it was a 72 billion model, but a 32 billion model or something in that range. But I didn't have time to fine tune it really. So I ran it with the same parameters and it didn't make much of a difference in the setting. But of course, that's not final because I hadn't really time to tune it.

• 50:15Tim Scarfe?

  Yeah, there seems to be an interesting relationship between the strength of the base model and how much test time computation you do, which is to say, there's an argument that it's almost better to do, if, if, if, if the budget allows, it's better to do more test time computation on a smaller model.

• 50:33Speaker 02

  Yeah, I think so, yes. I think the going up in size is not necessarily always good in this case. The more parameters you have, the more parameters you have to fine tune. And for fixed compute budget, I think the smaller models are much better. You can generate more, you can evaluate more, you can filter more.

• 50:57Speaker 02

  But also, I think in some experiments, larger models just didn't perform better. Like, they converged roughly to the same score.

• 51:09Tim Scarfe?

  So then there's this notion of adaptability. So let's imagine that you were deploying this thing. Let's say you wanted to make some money and you deployed a web service called Arc Solver. And I'm guessing what you would notice over time is that if you persisted the L L M and you were always continuously fine tuning it and adapting it and so on, you would probably notice its improved knowledge by virtue of the fact that it was doing less tree searching because it found the solution quicker. Do you think that that would just be the case, that it would just get generally better at Arc? And it would just know the solution more quickly? Or do you think you would get weird things happen where it might learn distractors on some tasks and get better and get worse at other ones?

• 51:51Speaker 02

  It's a good question.

• 51:53Tim Scarfe?

  I mean, part of the reason I'm saying that is I think that that's what's happening with the O one models. So I think the reason why we're seeing a dramatic improvement, even in three months, is because they are learning from the users. So they're continuously adapting and augmenting their knowledge. And the models are getting better quicker, needing to do less thinking over time.

• 52:15Speaker 01

  Yeah, maybe I have an interesting fact to share. So for the fine tuning, we tried different number of tasks we trained on at the same time for the second fine tuning. So we did something with one task, just a single task with fifty tasks on Kaggle. And we also tried the full four hundred tasks of the evaluation set. And that led to very much degraded performance. So up to fifty was fine, but maybe for more tasks, it just didn't work. Maybe it was too much for the model to.

• 52:52Speaker 02

  Store at the same time. I'm not sure, we also use LoRa, which also has limited number of parameters. So if we would do some online training and start with new examples, maybe it would have to forget the old ones to solve the new ones, that is quite possible.

• 53:07Tim Scarfe?

  Yeah, you know, we are always told in the the theory of neural networks that there's a problem with catastrophic forgetting and continual learning and all the rest of it. And, you know, even Shalay wrote in his book that when you do fine tuning, you have to be really careful. You have to like turn the learning rate right down because you'll just destroy anything it already knows. And it will only learn the thing that you're teaching it and so on. And we're just becoming quite red pilled on OpenAI now because that, you know, we we believe that these things are like magical memory machines and they're just sucking in all of this knowledge. And the more knowledge you give them, the better they get. And that seems to fly in the face of practical experience.

• 53:44Speaker 02

  And transformers are just very, very good at learning facts, I think. It's frankly incredible what they can store. Um, but there's also a slightly different thing that I find interesting, like in our fine tuning, we use LoRa, obviously, because fine tuning on the whole network is very expensive. And um, there's a small subtle thing with LoRa. If you use weight decay on a LoRa on a network, then it's the greatest weights of the LoRa. But you always have the base network below it. So you can't, you can't fall below specific performance like you, you're always around the correct

• 54:28Speaker 02

  full network and just um, move a little bit in another direction. So if you um, do test and training or fine tuning with a LoRa, you can in a way have the safety blanket of your base model below it, which can be very useful, I think, in our challenge. Like we merged, I think, the first LoRa of the pre-training into the weights. And then did a test and training on another LoRa. So at the worst case, um, it would degrade to the base model again.

• 55:04Speaker 02

  Like the base model from pre-training.

• 55:06Tim Scarfe?

  And just for the audience, can you quickly, because Laura is a low water rank approximation. Can you explain how it works? It's approximation or adaptation?

• 55:15Speaker 02

  Adaptation.

• 55:15Tim Scarfe?

  Adaptation, yeah.

• 55:16Speaker 02

  It's a very simple idea, very elegant idea. You have all these quadratic weight matrices in the attention layers, right? And instead of fine-tuning all the weights on them, you fine-tune to smaller matrices that you can multiply to get the full rank matrix. And that's mostly enough, like if you have a 1000 by 1000 matrix, instead of fine-tuning 1 million parameters, you say 1000 by 1 and 1 by 1000 matrix, that would be rank 1, and fine-tune those two small vectors. And because you

• 55:58Speaker 02

  go down to a smaller problem, because you have a simpler problem you want to learn, it's enough to get the network to go in the right direction, basically.

• 56:07Speaker 02

  I think we had a surprisingly high rank, right?

• 56:10Speaker 01

  Tried out different ranks from, I think, 64 to 256. And the original model, I think, has a rank of 4,096. And anything above 128 didn't actually make a difference for file performance. We also tried full fine-tuning once. It was very memory-consuming, and that also didn't make a difference for us. Yeah, so this low rank is probably enough for the problem.

• 56:42Tim Scarfe?

  And just to give folks an idea of how long it took, I mean,

• 56:47Tim Scarfe?

  you know, are we talking like half an hour, two hours a day?

• 56:51Speaker 01

  So competing in the challenge?

• 56:52Tim Scarfe?

  And as in to do, you know, like, let's say at the beginning, when you're fine-tuning the model, how long did it take?

• 56:59Speaker 01

  Oh, at the beginning, we had smaller models and we had also shorter training times, especially when we're still working on the original training set, which we couldn't put in too often without overfitting. So then it was a few hours, maybe two to four hours. And it got longer and longer during the challenge. So at some point, we were at two days on an Nvidia H100. And the final model was essentially eight days on an Nvidia H100. So that was quite long. And the interesting thing is when the heavy arc dataset appeared, when we found it, there were only, I think, four days left till the end of the contest.

• 57:44Speaker 01

  So what we did there, we actually started with multi GPU training to do the work of eight days in four days, in two days. So luckily, we got a lot of compute. Lamta offered us to use a machine with eight times H100 GPUs, so Lamta Labs. And yeah, so we could still train the model in time and submit it on the last day. That was the model with the fifty six point five score, which sadly didn't finish in time then.

• 58:19Tim Scarfe?

  Yeah, they say that necessity is the mother of all invention.

• 58:24Tim Scarfe?

  I'm one of these guys that, you know, like what you're supposed to do when you're coding is you add one feature at a time. You check it and then you like commit it and so on. And it's, I have this kind of, not ADHD, not quite, but you know, like I'm just, I've got so many cool things I want to do and I want to just like stick them all in there. Where do you fall on that spectrum?

• 58:42Speaker 02

  Normally, I would fall similar to you, but in the setting of the contest, we just didn't have time. Like it didn't matter. We had to push like new solutions. We had to experiment quickly. And trying to do clean code or trying to refactor in a way that makes it more flexible would have taken, I don't know, one or two days. But that was too long. Like we had, we had so many ideas we wanted to experiment on and try that we had to use the code base that we had.

• 59:17Speaker 01

  So when we would have put in new features again, we would have to certainly refactor it again because, yeah, we didn't know exactly what things we would try out during the time. So that was probably a little bit pointless to refactor it all the time during the contest.

• 59:32Speaker 02

  It felt a little bit like a mad dash.

• 59:35Speaker 01

  So there were some reflectors. So when it got too complicated, it's like two or three steps where I completely refactored it in between during the challenge. But in between these reflectors, we just used it over and over and just added stuff essentially.

• 59:51Tim Scarfe?

  And would you say your intuitions were fairly correct throughout? Or because you know, like, we get a bit superstitious about things. So we think, oh, that thing's really important. We need to keep that in there. Were there any situations where you realized to your horror later on that something you thought was good was actually not good?

• 1:00:06Speaker 02

  Yeah, but sometimes you were you were constantly like testing different things, like leaving some stuff out and putting some stuff in. And we also used a lot of our submissions for testing, like what happens if we remove this feature, what happens if we remove that feature. Often, our our favorite ideas that we tried didn't pan out. And then we had to decide do we invest more time in making them work? Or do we just say, screw it, we don't have the time to investigate it deeper. We have to push the things that work. And often, it was the right decision to ignore our crazy fun ideas and just do the thing that works better.

• 1:00:51Tim Scarfe?

  And what were the top two things out of all of this stuff that you did that you think moved the needle the most?

• 1:00:56Speaker 01

  So the things that were most important, I guess, in our implementation were first of all, the scoring process. So scoring with the augmentations, which allowed us to select the social solutions reliably. So this was one thing that gave a big jump. I think it was from thirty to thirty seven at that point when we implemented it. And the other thing, I think what is the depth first search algorithm, because it gave us such a performance gain that we could change a lot of things after, like using a bigger model, doing more augmentations, and things like that, as we would couldn't have done without it.

• 1:01:37Tim Scarfe?

  Well, that's interesting, because part of experimentation is actually having a useful signal. And you were saying that certain features actually unlocked a signal on other things you could try.

• 1:01:45Speaker 02

  Yes, the DFS was a very natural extension of the scoring, I think. After we had a good working scouring and selection algorithm, the challenge became to generate good candidates. And we tried a lot of stuff to do that. Some some LLM stuff and some external stuff like combining snippets of different solutions to generate new solutions.

• 1:02:13Speaker 02

  But what really worked well was Daniel's DFS algorithm. It was very quick and gave us so many different candidates we could score and very few bad candidates. So, yeah, the two approaches worked very well together. It was a very synergistic approach.

• 1:02:34Tim Scarfe?

  Did you ever see any catastrophic failure modes with the DFS where it would just search for an unreasonable amount of time before hitting the threshold?

• 1:02:42Speaker 01

  No, not really, because we limited the search depth to 10%.

• 1:02:48Tim Scarfe?

  Oh, I see.

• 1:02:48Speaker 01

  So it was bad. It essentially happened that it searched quite a lot of solutions there sometimes, but we set the limit in such a way that it didn't lead to a big problem.

• 1:02:59Speaker 02

  Because the probability mass has to distribute over all paths, right? So if the the lower limit is 10%, then it can't be more than 10 paths that are valid at the same time.

• 1:03:10Tim Scarfe?

  Do you think it would be possible, before you do the depth first search, to predict what the computational budget is or to predict what what the search tree might look like?

• 1:03:21Speaker 01

  That's an interesting question. I think it might be quite difficult, because you don't know until you have actually done some searching. So could probably predict it in an estimation after you've searched a part of it.

• 1:03:39Speaker 01

  Or you could go for iterative deepening, like you first go to 10%, then to 5%, something if you want to search deeper. It would also be a possibility if you want.

• 1:03:50Speaker 02

  To them as a compute, but.

• 1:03:52Tim Scarfe?

  I'm just thinking you could take the, you prompt the model with that solution. And then you could just use a classifier or something and train the classifier on different topologies of trees that you found previously. And you might be able to sort of predict a priori, what, how, how much is this going to take me? You know, how mu- how much computation should I invest in this problem?

• 1:04:12Speaker 02

  If we have the solution, then that works. Like you can, can just check the, the logic for each pixel and then measure basically the entropy, like how distributed is probability. And if the probability is distributed enough, then that is a branch in the tree. But then you only have like the one path through the tree and then branches from there. Predicting the whole tree is probably basically equivalent to predicting the correct solution in some ways. Like you have to know the solution to know what the tree looks like.

• 1:04:45Tim Scarfe?

  We didn't touch on this before, so you were doing a minimum entropy search. But what would happen if you tried?

• 1:04:51Speaker 02

  I'm not sure if we do a minimum entropy search.

• 1:04:54Tim Scarfe?

  It's a little bit like, you know, Monte Carlo tree search. So it's not doing a minimum entropy, but like if you look at the, the entropy distribution, it's like it's, it's searching the space of low entropy.

• 1:05:06Speaker 02

  The, the problem is, the problem here is, I think that

• 1:05:11Speaker 02

  we have, we have like, you can imagine we have a rank order for each pixel for the next prediction, like pixel color one is most probable, then pixel color zero. And for basically every problem, there's never more than three possible continuations. So you have to be very careful if you like to tweak it or try different things that you don't get into the seven things that are basically impossible. Like for many problems, there are, for example, only three colors. And if you do a maximum entropy or very broad search, you would get colors that are just impossible. And you have to safeguard against it, but at the end, every smart algorithmic improvement we tried.

• 1:05:57Speaker 02

  Perform worse, like the LLMs are just smarter than us.

• 1:06:01Tim Scarfe?

  There must have been some relationship between you you stick a candidate's solution in there and you've got this entropy distribution and you're saying in in some circumstances it's quite it's quite pointed. So it's like the the model will say I want to go in this direction and no other direction and sometimes it's spread out. What can you infer from that?

• 1:06:20Speaker 02

  In general, I think the pointed ones, like if we have very few candidates, they are often just correct. Like the model has a very good idea of how the problem is structured. So if there are several solutions, then they are mostly because there's some some link missing in the understanding of the problem or or if the model just doesn't understand the problem at all. Like we we did a lot of testing and the problems that are hardest for models were often counting or I think fluff, no fluff will work very well, but mostly counting or size estimation problems.

• 1:07:06Speaker 02

  And in that case, because the model was very unsure, it predicted often a large amount of possible candidates. In the other cases, when when when there are few candidates, then they're almost always correct, I think.

• 1:07:23Tim Scarfe?

  And how did you deal with false positives? So I guess there are some situations where you found solutions which, you know, now now when you look at the the the sum of the log probabilities, you know, like it's telling you this is the solution and it might just not be the solution because it's not the solution or it might be a false positive because it's, you know, like the right solution for the wrong reason or whatever. What what do those failure cases look like?

• 1:07:47Speaker 02

  It depends strongly on the problem, I think. The one of the most interesting things I've seen in analysis was a task where you have color shifting, like you have a you have four colors in the upper left corner and the goal is to shift the colors in an object clockwise direction. And the model solves that problem.

• 1:08:10Speaker 02

  The second best solution is also a color shift just in the wrong direction, which I found very fascinating because it means that the model conceptually understands the problem and even the second best solution is a conceptually right solution just goes in the wrong direction. In the cases where the model has a false positive or doesn't solve the problem, it's almost always still conceptually in the right direction. Like it understands, I should fill the space with something or I have to generate an object here.

• 1:08:47Speaker 02

  But sometimes it just doesn't have said which object. Sometimes it fills the wrong space, stuff like that.