Francois Chollet - ARC reflections - NeurIPS 2024

François Chollet discusses the outcomes of the ARC-AGI (Abstraction and Reasoning Corpus) Prize competition in 2024, where accuracy rose from 33% to 55.5% on a private evaluation set. SPONSOR MESSAGES: *** CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. https://centml.ai/pricing/ Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. Are you interested in working on reasoning, or getting involved in their events? They are hosting an event in Zurich on January 9th with the ARChitects, join if you can. Goto https://tufalabs.ai/ *** Read about the recent result on o3 with ARC here (Chollet knew about it at the time of the interview but wasn't allowed to say): https://arcprize.org/blog/oai-o3-pub-breakthrough TOC: 1. Introduction and Opening [00:00:00] 1.1 Deep Learning vs. Symbolic Reasoning: François’s Long-Standing Hybrid View [00:00:48] 1.2 “Why Do They Call You a Symbolist?” – Addressing Misconceptions [00:01:31] 1.3 Defining Reasoning 3. ARC Competition 2024 Results and Evolution [00:07:26] 3.1 ARC Prize 2024: Reflecting on the Narrative Shift Toward System 2 [00:10:29] 3.2 Comparing Private Leaderboard vs. Public Leaderboard Solutions [00:13:17] 3.3 Two Winning Approaches: Deep Learning–Guided Program Synthesis and Test-Time Training 4. Transduction vs. Induction in ARC [00:16:04] 4.1 Test-Time Training, Overfitting Concerns, and Developer-Aware Generalization [00:19:35] 4.2 Gradient Descent Adaptation vs. Discrete Program Search 5. ARC-2 Development and Future Directions [00:23:51] 5.1 Ensemble Methods, Benchmark Flaws, and the Need for ARC-2 [00:25:35] 5.2 Human-Level Performance Metrics and Private Test Sets [00:29:44] 5.3 Task Diversity, Redundancy Issues, and Expanded Evaluation Methodology 6. Program Synthesis Approaches [00:30:18] 6.1 Induction vs. Transduction [00:32:11] 6.2 Challenges of Writing Algorithms for Perceptual vs. Algorithmic Tasks [00:34:23] 6.3 Combining Induction and Transduction [00:37:05] 6.4 Multi-View Insight and Overfitting Regulation 7. Latent Space and Graph-Based Synthesis [00:38:17] 7.1 Clément Bonnet’s Latent Program Search Approach [00:40:10] 7.2 Decoding to Symbolic Form and Local Discrete Search [00:41:15] 7.3 Graph of Operators vs. Token-by-Token Code Generation [00:45:50] 7.4 Iterative Program Graph Modifications and Reusable Functions 8. Compute Efficiency and Lifelong Learning [00:48:05] 8.1 Symbolic Process for Architecture Generation [00:50:33] 8.2 Logarithmic Relationship of Compute and Accuracy [00:52:20] 8.3 Learning New Building Blocks for Future Tasks 9. AI Reasoning and Future Development [00:53:15] 9.1 Consciousness as a Self-Consistency Mechanism in Iterative Reasoning [00:56:30] 9.2 Reconciling Symbolic and Connectionist Views [01:00:13] 9.3 System 2 Reasoning - Awareness and Consistency [01:03:05] 9.4 Novel Problem Solving, Abstraction, and Reusability 10. Program Synthesis and Research Lab [01:05:53] 10.1 François Leaving Google to Focus on Program Synthesis [01:09:55] 10.2 Democratizing Programming and Natural Language Instruction 11. Frontier Models and O1 Architecture [01:14:38] 11.1 Search-Based Chain of Thought vs. Standard Forward Pass [01:16:55] 11.2 o1’s Natural Language Program Generation and Test-Time Compute Scaling [01:19:35] 11.3 Logarithmic Gains with Deeper Search 12. ARC Evaluation and Human Intelligence [01:22:55] 12.1 LLMs as Guessing Machines and Agent Reliability Issues [01:25:02] 12.2 ARC-2 Human Testing and Correlation with g-Factor [01:26:16] 12.3 Closing Remarks and Future Directions SHOWNOTES PDF: https://www.dropbox.com/scl/fi/ujaai0ewpdnsosc5mc30k/CholletNeurips.pdf?rlkey=s68dp432vefpj2z0dp5wmzqz6&st=hazphyx5&dl=0

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I've never been in the purely symbolic camp. Like, if you go back to my earliest writing about, yes, we need program synthesis. I was saying, we need deep learning guided program synthesis. We need a merger of intuition and pattern recognition together with discrete step-by-step reasoning and search into one single data structure. And I've said very, very repeatedly, you know, for the past eight years or so, that human cognition really is a mixture of intuition and reasoning, and that you're not going to get very far with only one of them. You need the continuous kind of abstraction that's provided by, you know, vector spaces, deep learning models in general,

• 0:41Speaker 00

  and the more discrete symbolic kind of abstraction provided by graphs and discrete search.

• 0:48Tim Scarfe?

  So why do your detractors see you as a symbolist when you're clearly not?

• 0:53Speaker 00

  Well, I'm not sure. I've been into deep learning, you know, for a very long time, since basically 2013. I started evangelizing deep learning very heavily around 2014. And back then, the field was pretty small. You know, especially with chaos, I think I've done quite a bit to popularize deep learning, make it accessible to as many people as possible. I've always been a deep learning guy, right? And when I started thinking about the limitations of deep learning, I was not thinking in terms of replacing deep learning with something completely different. I was thinking of augmenting deep learning with symbolic elements.

• 1:31Tim Scarfe?

  What is your definition of reasoning?

• 1:34Speaker 00

  I don't really have a single definition of reasoning. I think it's a pretty loaded term, and you can mean many different things by that. But there are at least two ways in which I see the term being used, and they're actually pretty different. So for instance,

• 1:52Speaker 00

  If you're just, let's say, memorizing a program and then applying that program, you could say that's a form of reasoning. Like, let's say in school you're learning the algorithm for multiplying numbers, for instance. While you're learning that algorithm, then when you're tasked, you're actually applying the algorithm. Is that reasoning? I think yes, that's one form of reasoning. And it's the kind of reasoning that LLMs and deep learning models in particular are very good at. You're memorizing a pattern, and that test time you're fetching the pattern and reapplying it. But another form of reasoning is when you're faced with something you've never seen before, and you have to recompose, recombine the content building blocks you have access to,

• 2:35Speaker 00

  so your knowledge and so on, into a brand-new model and do so on the fly. That is also reasoning. But it's a very, very different kind of reasoning, and it turns out it has very different kinds of capabilities. So I think, you know, the important question about deep learning models and LLMs in particular is not, can they reason? There's always some sense in which they are doing reasoning. The more important question is, can they adapt to novelty? Because there are many different systems that could just, you know, memorize programs provided by humans and then reapply them. What's more interesting is, can they come up with their own programs, their own abstractions on the fly? Broadly speaking, I think programming from input-to-output pairs

• 3:17Speaker 00

  will be a widespread programming paradigm in the future, and that will be accessible to anyone, because you don't need to write any code, right? You're just specifying what you want the program to do, and then the computer just programs itself. And if there's any ambiguity, by the way, in what you meant, and there will always be ambiguity, right? Especially if the instructions are provided by a non-technical user. Well, you don't have to worry about it, because the computer will ask you to clarify. If you tell it, "Okay, so I created basically the most plausible program, given what you told me, but there's some ambiguity here and there. So what about this input? Currently I have this output, does that look right? Do you want to change it?"

• 4:00Speaker 00

  And as you change it iteratively, you are creating this correct program in collaboration with the computer.

• 4:08Tim Scarfe?

  Is there just a massive new type of architecture we need to build for this?

• 4:13Speaker 00

  Yeah, I think we're gonna need a completely new type of architecture to implement lifelong distributed learning, where you have basically many instances of the same AI solving many different problems for different people in parallel, and looking for commonalities between the problems and commonalities between the solutions. And anytime they find sufficient commonality, they just abstract this commonality into a new building block, which goes back into the system and makes the system more capable, more intelligent.

• 4:42Tim Scarfe?

  I think I've got it now. So what you're building is a globally distributed ARC on the basis that we find a good solution to ARC.

• 4:49Speaker 00

  Well, I can't really tell you exactly what we're building, but it's gonna be cool.

• 4:55Tim Scarfe?

  Yeah, it sounds pretty cool. What's your theory on how O1 works?

• 5:00Speaker 00

  Well, you know, we can only speculate. I'm not sure how it really works. But what seems to be happening is that it is running a search process in the space of possible, the space of possible chain of doubt,

• 5:19Speaker 00

  trying to evaluate which branches in the tree work better, potentially in backtracking and editing if the current branch is not working out. It ends up with this very long, unsophisticated and plausibly near-optimal chain of doubt, which represents basically a natural language program describing what the model should be doing. And in the process of creating this program, the model is adapting to novelty. And so I think something like O1 is a genuine breakthrough in terms of the generalization power that you can achieve with these systems. Like, we are far beyond the classical deep learning paradigm.

• 6:29Tim Scarfe?

  MLST is sponsored by Centimel, which is the compute platform specifically optimized for AI workloads. You might have seen the interview we did last month with Gennadi, their CEO and co-founder. He spoke about some of the optimizations on Centimel, which only they have done, which makes it dramatically faster than the competition. Anyway, if you're interested in hacking around with Centimel, I'm going to be doing a live with one of their engineers in the coming weeks, so feel free to jump on that live and you can ask their team any questions. Two for AI Labs is a very, very exciting new research lab that's just started in Zurich. They are looking for amazing MO engineers to join their team. It's a very small team, it's very exciting.

• 7:11Tim Scarfe?

  They are doing things like test time compute, inference time scaling. They want to get models to reason, to do this system two amazing thing that we've been talking about for years on MLST. If that sounds like you, go to twoforlabs.ai.

• 7:26Speaker 00

  I love the show in general. You know, I think of it as like the Netflix of machine learning. Well, you're going to learn about Arc two, you're going to learn about what we've learned from running the Arc prize competition in 2024. And maybe you're, you know, you're going to pick up on some of my grand ads on abstraction generation and how to implement a GI.

• 7:48Tim Scarfe?

  And you're going to find out what Francois's doing next?

• 7:52Speaker 00

  Well, a little bit about it.

• 7:53Tim Scarfe?

  A little bit, a little bit. Wonderful. Francois Chollet, it's an honour to have you back on the show. Thank you so much.

• 7:59Speaker 00

  Thanks so much for having me.

• 8:00Tim Scarfe?

  So we're now at the end of the Arc AGI prize and you just released a technical report about it, but can you reflect on the competition?

• 8:10Speaker 00

  Right, I think we learned a lot and overall it's been a big success. I think, you know, in 2024, we've seen a huge shift in the narrative around AI, where previously, you know, the mainstream narrative was that we could just train larger models on more data, you know, 100x larger models, 100x more data, and get something that's, you know, basically a GI. And while more recently, you know, in the past year or so, there's been this realization that actually we were going to need something akin to system to reasoning, that it is not something that will simply just emerge

• 8:53Speaker 00

  from pre-training larger models on the larger data sets, that you needed to add it to the system somehow. And of course, you can use test-time search, you can use program synthesis, you can use this sort of techniques to do it. And I think Arkrise was really part of that narrative shift. And it's part of the reason also why Arkrise has been, you know, very popular, with lots of teams entering, lots of people talking about it and using it as a kind of reference for whether, you know, we might have achieved the GI or not, which by the way, is not what Ark is intended to be. It's not intended to be an indicator of whether we have the GI. It's just intended to be a tool, a research tool that gets you to think about the right problems,

• 9:38Speaker 00

  to focus on the right directions. And really, the reason why it's been successful is because there was a latent demand for something like this. Many people had this intuition that, yes, like plain LLMs weren't going to get to it, that we needed something more, either as a replacement to LLMs or some kind of superstructure around them that will implement system too. And there was this intuition floating around and I think lots of people just latched onto Arkrise as a concrete sign that their intuition was right.

• 10:18Tim Scarfe?

  So there were two flavors of the ARC prize. There was a compute restricted one and there was the main one. Can you reflect on the difference between the two in terms of the entries?

• 10:28Speaker 00

  Sure, so we had the main track for the competition on Kaggle. And this was only for submissions that were self-contained. And the reason why is primarily because we need to keep the private test set fully private. Right, so we cannot send it to a server via an API. And so on this track, people are submitting notebooks effectively that are going to have to run on a VM in less than 12 hours and the VM has just a P100 GPU. So this is equivalent to a total of roughly 10 bucks worth of compute per submission. Right, and then we had the public leaderboard, which was targeted at frontier models.

• 11:12Speaker 00

  And it was evaluated on a different set of tasks because of course, you know, we cannot evaluate on the private test set that we just leaked the private test set to a server. So instead, it's evaluated on an entirely new set of tasks, which is called a semi-private test set. So it's my private because it's not published anywhere. So it's not, it's not public, but it's also not entirely private since we are actually sending the task data via the API to our company, I am tropic and so on. And each submission on that leaderboard can use up to $10,000 in API credits. So that's 1,000 times more in terms of compute worth than the private leaderboard.

• 11:57Speaker 00

  And on a per task basis, it's actually a little bit less because we're evaluating on more tasks. We are evaluating on the 400 tasks from the public eval and then the 100 from the semi-private eval, which is really what we're looking at. And so on a per task basis, that's about 200 times more compute. And what was really remarkable and frankly quite shocking is that the scores you end up seeing on the on the public leaderboard track what you're seeing on the private leaderboard. Like in both cases, we had about 55%.

• 12:35Speaker 00

  And this tells you that really it's not just about throwing more compute at the benchmark. Compute is really a multiplier for ideas. And of course, if you have, you know, infinite compute, you can solve the benchmark in a very stupid way, like yeah, brute force search, for instance. But having better ideas really gives you dramatically more leverage for your compute, right? And so, which is why we end up with with solutions consuming 10 bucks of compute that are doing like 55%. And meanwhile, solutions are consuming 10K words of compute, doing the exact same. They're just, they're, they're just not nearly as computational.

• 13:17Tim Scarfe?

  So which was the most successful method which worked across the board?

• 13:20Speaker 00

  There were really two categories of approach, two categories of approaches that worked well. So one is deep learning guided program synthesis, which is really my, my favorite approach personally and what I've been advocating for for many years. And so most people nowadays are doing deep learning guided program synthesis using L L M's. They're using L L M's to generate code, L L M's to iteratively debug codes. Some people are trying to,

• 13:50Speaker 00

  you know, do deep learning guided processes using building blocks from a D S L. I think this is a very underexplored approach, but I think it should be an effective one. And the other category of approaches has been test time training, where you're going to be using an L L M to directly try to predict the solution given a task description. So you're looking at a set of demonstration pairs. And then you're looking at an input grid and you're directly trying to generate the output grid, which is, you know, a process as we call transduction, to opposite to induction, like program synthesis, program induction. You're trying to come up to write down the program that will map the input grades to the output grades.

• 14:34Speaker 00

  And in transduction, you're just trying to directly predict the output grid. And of course, if you try to do this with L L M's and you stay within the classical deep learning paradigm, where you have this big model that's pre-trained on tons of data. And then at inference time, it is static and you're, you're, you're really just running for parts of the model. If you're staying within that paradigm, you really cannot adapt to any meaningful amount of novelty, right? You are stuck to memorizing patterns in a test time, fetching and reapplying the patterns that you've memorized. And to go beyond that, people have started using test time training, which is the idea that you start by pre-training this base model that

• 15:17Speaker 00

  knows about ARC, knows about ARC tasks. And at inference time, on each new task that you see, you're going to try to fine-tune the model, the base model, on the demonstration pairs, to basically try to recombine the knowledge contained within latent space into a new model that is adapted to the task at hand. And if you don't do this, if you're going to do this test time adaptation, then LLM-based models, transaction models, they are stuck below 10% accuracy roughly. But if you start doing test time training, then you're unlocking a dramatically higher level of generalization, and you can go well into the 50%, 55% probably and 60% soon.

• 16:04Tim Scarfe?

  So the big question is whether that's in the spirit of the challenge or not. So I've done some interviews on this transduction with active fine-tuning, where you take the test instances, you do some dataset generation, some augmentation, you fine-tune the model, you do really well. And certainly, as a methodology to broadly generalize to lots of tasks, it's good. But the problem is, it still has human supervision. So you've stressed from the very beginning in your measure of intelligence that we need developer-aware generalization, which is simply that we can't have a human supervisor specializing the thing for every downstream task. We need to make a system that can itself generalize to tasks that the developer of the system wasn't aware of. So by that metric, do you feel that it's not in the spirit of ARC?

• 16:47Speaker 00

  No, I think it's a completely legitimate way to approach a challenge. And I also think it represents a very significant breakthrough in generalization power and in the ability of these models to adapt to something they've not seen before. And I don't think the supervision that we are talking about is really done by humans. You're using the demonstration pairs to fine-tune the model. This is actually fairly autonomous. Of course, this needs to have been programmed by a human aware of the task format. But the same would be true for a program induction type approach. So I think it's very much in the spirit of the challenge.

• 17:29Speaker 00

  And further, I think it does demonstrate a legitimate breakthrough in generalization.

• 17:36Tim Scarfe?

  So pressing on the legitimacy thing, I mean, it stands to reason that we do some kind of active inference.

• 17:42Speaker 00

  Of course.

• 17:43Tim Scarfe?

  You know, so we're always adapting to novelty, building new models and so on and so forth. So, so what is the difference between transductive active fine tuning and what we do?

• 17:54Speaker 00

  Well, I'm not sure what we do exactly. Of course, we are doing active inference. What does that mean exactly? What algorithms, what data structure are we leveraging? We don't know. So I can't, I can't really tell you, you know, what's the, what's the difference? It does, you know, it does seem to me, I will say one thing. So when you're doing a test time training within LLM, you are letting a gradient descent process do the knowledge recombinations. So to adapt to novelty, it is necessary to take the knowledge that you have and recombine it in some way. And there are multiple different things you could be, you could be doing to achieve that. You could be doing a program search, right?

• 18:38Speaker 00

  Where the thing that is adapted to the new task is a program and you are building this program via a search process. You could also, you know, and something like what the O1 model from OpenAI is doing is similar to that, where you are doing search effectively in the space of chain of thoughts. And you are writing down this chain of thought, which is basically a natural language program for the model to execute. And, and you're doing this search via and a sort of like, you know, I find zero star search process, like tree search process. So that's one, that's one take. You can also just use, use this good program search to write down the program.

• 19:22Speaker 00

  And the program is the artifact that models the task at hand.

• 19:26Speaker 00

  Or you can try to modify the weights of the model, modify the representations of the model to create a new model that's adapted to the task. In this case, the, the artifact. Adapted to the task is the model itself. And that's what the stamp training does, and it does it via gradient descent. And well, my take is I don't think humans adapt to novelty by recombining what they what they know via gradient descent specifically. I think the way, I think the the the level at which we represent knowledge, especially in the context of solving arc puzzles, is much more abstract and symbolic in nature. And the way we combine it is much closer to function composition than it is to what you what you can achieve with gradient descent.

• 20:10Speaker 00

  Right? In general, I don't think it's a good idea to try to use gradient descent as a replacement for a programming process. So I'm actually more of a fan of what a system like O1 is doing than than trying to to to to the test on training.

• 20:26Tim Scarfe?

  We'll get to O1 in a second, but there are folks who are really bullish on this transduction thing. And I think you and I agree that the reason it doesn't work in principle is because language models are just finite state automata. They don't have this compositional generalization, even in even in principle, right? But we know there's evidence that transformers on their own, they can't do basic things like copying and counting and all of these kind of things. But some people are bullish because they think we could improve the architecture so that it could do those kinds of things. And then a transductive approach might work in the future. I mean, would you rule that out?

• 21:02Speaker 00

  No, I think I think it's entirely plausible. You know, as you point out, even given lots of data, there are many algorithmic tasks that you cannot train a transformer to do. Right? Or even even if you can, it will learn a solution that does not generalize very well. It will work on inputs that are pretty close to what it's been trained on. If you try and input as very far away from train distribution, which will just fail. And while people who think we can move past these limitations, they're saying that we can make architecture tweaks. And well, you know what? They are right. Like, it is always possible to take a deep learning model, modify the architecture to bake into it some strong structural prior about the.

• 21:46Speaker 00

  Algorithmic problem that you're trying to solve. And now you can actually use gradient descent to find a solution that will generalize, right? But the the way this works is by asking a human engineer effectively to first understand the the task at hand and convert that understanding, that symbolic understanding, into a better architecture, an architecture that is in some important ways isomorphic to the causal structure of the problem. And so the, of course, you know, if you want to autonomously adapt to novelty, you cannot just require a human engineer to intervene and write in your architecture, right? The process has to be has to be fully autonomous. And so the question is, can you create

• 22:31Speaker 00

  an architecture search or architecture generation machine that will take a problem, identify the the key elements of the problem structure that you need to bake into in your architecture and then generate the architecture? If you can do that, then sure, maybe maybe you can leverage that to to to achieve strong generalization. But I think that's a problem is at least as difficult as

• 23:02Speaker 00

  problem system, problem session in general case. Yeah, it's also pretty overfit, as it turns out. So you know, on the on the public leaderboard, we are evaluating on the public eval set, but we're also evaluating on this semi-private eval set. And the reason we're doing that is to test for overfitting. Some solutions might be overfit to the to the public eval. And it was actually the case for their solution. They scored something like, you know, ten, ten percentage points lower on the semi-private set. And other solutions, solutions based on on program synthesis in particular, are not featuring that drop at all. They're actually scoring the exact same on both sets. So it kind of tells you that whatever they did was in some important way

• 23:48Speaker 00

  overfit to the data that they had.

• 23:51Tim Scarfe?

  Can we touch on that as well? So you said when you take an ensemble of all of the, you know, the original Kaggle 2020 competition results, it got to about 49%.

• 24:01Speaker 00

  That's right.

• 24:02Tim Scarfe?

  Tell me more.

• 24:03Speaker 00

  Right, so in the very first Arcadia competition on Kaggle back in 2020, the highest score by a single submission was only 20%. So that was the winner, IceCube, and he was doing just basic brute force program iteration. But if you looked at all of the submissions in the competition and you assemble them together, you would see a high score of 49%, which would have been, you know, until very recently, state of the art. And that was, again, like four years ago, right? And what that tells you is that there's about half of the private test set that's easily brute forcible.

• 24:48Speaker 00

  Because every single entry in the competition back then, in 2020, was doing some kind of brute force program iteration. That is not the solution that we are looking for. And so the fact that doing this kind of stuff, doing this kind of stuff at scale, could get you 49% is a very strong sign that the benchmark is flawed. And well, today, you know, if you look at the 2024 competition, the state of the art for any single submission is about 55%, right? So you could say, okay, so we are very far from solving the benchmark, since the bar is 85%. And we also know that humans can solve very close to 100%.

• 25:31Speaker 00

  Like if I showed you the private test set, you'd probably do like, you know, 97, 98, 99% type of thing. And if you take an ensemble of everything that was submitted in the competition in 2024, you would get to a high score of 81%, right? Which is pretty close to what we are looking for. And I don't think, you know...

• 25:56Speaker 00

  Anything that was that was tried in the competition this year is really close to AGR in a unique full sense. And the ensemble of everything is still is still is still not close to AGR. It just shows that scale, just brute force compute scale, will eventually crack the challenge. And that what that says is really that the benchmark is flawed and now it is close to saturation. And that we need something else. And that's why we're we're working on Arc 2. So Arc 2 is not exactly a novel idea. It's not a reaction to the results that we got in in 2024. I first publicly announced Arc 2 in early 2022. Right, so a while back.

• 26:42Speaker 00

  And it was back then kind of a reaction to the to the 2020 competition results, where I was aware that the benchmark had flaws, that it was not quite diverse enough in terms of task diversity. Not every task was quite unique. There was some amount of redundancy. It might not have been quite challenging enough. And so I wanted I wanted to do V2. And so back in 2022, you know, in partnership with Lab 42 in Davos, we started crowdsourcing a bunch of new tasks. And since then, you know, we've kept making a lot more new tasks. And we've started, you know, filtering them, try to analyze, you know, which ones were difficult for humans, which ones were difficult for AI and so on.

• 27:27Speaker 00

  We've collected a lot of human testing data as well. Like, we hired people to actually try to solve them. And that gives you a lot of information about, you know, how many attempts they use for different tasks. How many times one task was solved, you know, by by by how many people? And you can turn this information into a kind of human facing difficulty rating. Then you can try to cross correlate it with what AI can do. So we are going to be releasing Arc 2 early next year. It's going to be addressing all of the flaws of Arc 1. So it's going to be slightly more data. It's going to be leveraging three sets.

• 28:10Speaker 00

  So that's going to be the public eval, of course. That's going to be the semi-private eval and that's going to be the private eval. And one problem with the first few competitions on Kaggle is that we've been reusing the same private test set across every competition. And anytime anyone made a submission, they could immediately see their score on the private eval. And so this can actually lead to some amount of information leakage about the private test set over time. And in fact, there are very well documented techniques about how you can actually start to reconstruct, given enough submissions, how you can start to reconstruct the contents of the private eval.

• 28:52Speaker 00

  So we want to avoid this in the competition next year. And the way we're doing this is, you know, obviously that we're going to be evaluating on the semi-private eval during the competition. So when you submit something, you get your semi-private eval score. And only at the very end of the competition to create the final leaderboard, are we actually going to run the submissions on the fully private test set. And another nice thing, if you do that, is that this enables you to make a direct apples-to-apples comparisons between the Kaggle leaderboard and the public leaderboard with all the frontier models. And of course, you know, it's not, it's not going to be the same amount of compute, but the score is going to be apples-to-apples.

• 29:33Speaker 00

  You're going to be able to say like, okay, so mind the eye on this test set. Their scoring, you know, as well as, I don't know, like, or one pro or whatever, you know.

• 29:44Tim Scarfe?

  So you know, the Kaggle 2021's 49% when you take the aggregate. First of all, which ones were more brute forceable? Because I suppose one way of looking at it is certainly with the mosaic patterns that in the solution space, there's an exponential number of combinations. But of course, in the compositional space, it might be brute forceable. But we're also starting to see with the induction transduction methods that they can solve different types of problems. And maybe when you do human evals as well, that humans can solve different types of problems. And there are all these overlapping Venn diagrams. How do you think about that?

• 30:18Speaker 00

  Yes, it's actually one of the most interesting findings of the 2024 competition is that program induction and doing transduction with deep learning models, typically in LLM, that's going to lead to solving substantially different sets of tasks. And so this was surprising to me, but you know, in retrospect, it makes sense, especially if you start looking at the task and analyzing what makes them different. There are some tasks that are very perceptual in nature, right? There are effectively a pattern recognition problems. And this is the kind of task that you solve well with transduction methods. And there are other tasks that are much more algorithmic in nature, much, much more discreet in nature.

• 31:01Speaker 00

  And you cannot provide an easy solution based on pattern recognition, but it is very easy to just write down an algorithm to produce a solution. And in reverse, if you look at puzzles that are very perceptual, it is very, very challenging to write solution programs for them, right? Because the program would have to formalize a lot of conceptual concepts that make intuitive sense for us, but that are actually very difficult to express in program form, right? Imagine trying to recognize, let's, let's take something outside ARC. Let's say trying to recognize the letter A, for instance.

• 31:44Speaker 00

  If you're just looking at letters, you can immediately see it. If you have a sort of like pre-training knowledge, pre-training visual knowledge of the, of the shape of letters, this is true to your problem. You just recognize letters, just where you use pattern recognition. But if you try to write down an algorithm that would take a handwritten A and we, and we tell you, you know, an handwritten letter and tell you if it's an A, for instance, it's actually a tremendously difficult problem. Right.

• 32:11Tim Scarfe?

  Couple of thoughts on that. I mean, first of all, on that Kevin Ellis paper, the induction transduction paper, maybe for political reasons, it feels like they wanted to find an inductive explicit function. So in their ensemble method, they first searched the inductive functions by Greenblatt-ing it, and then they failed over to the to the transduction. And the thought occurs that just because we can't write a Python program to recognize a digit, surely such a Python program could exist. And shouldn't we be thinking about making models generate such a program? Maybe the reason they can't is because nothing like that is in distribution.

• 32:51Speaker 00

  I think the reason why it's difficult is that we are talking about an input space that is structurally continuous and where decision boundaries are fuzzy, basically. So you're talking about a problem that is fundamentally a pattern recognition problem. And your networks are just intrinsically a good data structure to approach this type of problem. And discrete symbolic programs are not. So I really think, you know, it depends on what problem you're looking at. There are problems where vector spaces are the right data structure and problems where symbolic discrete programs are the right data structure.

• 33:29Tim Scarfe?

  I guess an interesting thought experiment is there must exist a Python program which does what an MNIST model does.

• 33:36Speaker 00

  Sure.

• 33:36Tim Scarfe?

  And what is the simplest possible representation of that program? Would it still be ridiculously complicated?

• 33:45Speaker 00

  I think the simplest representation of the program will look a lot like what the Comnet is doing, to be honest.

• 33:53Tim Scarfe?

  Yeah, I think that that's the clincher, isn't it? So there's no way of kind of decomposing it into a much simpler version.

• 34:02Speaker 00

  Yeah, I think it really depends on the nature of the problem.

• 34:06Speaker 00

  And for some problems, problem synthesis is just a bad idea, right? And perceptual problems are certainly in this category. And the other way around, for some problems, trying to use a pattern cognition machine is just a bad idea. Problems that are algorithmic in nature.

• 34:23Tim Scarfe?

  How do you think we could effectively combine induction and transduction methods?

• 34:27Speaker 00

  Well, the way, you know, Kevin Ellison's team are doing it in their paper is basically they're starting with induction and then they're falling back on transduction when it doesn't work. And I think that's a very smart strategy because induction is actually formally verifiable. You know, you can try to run your candidate program on the demonstration pairs that you have access to and see, well, first of all, does it run? And second, does it get you the right result? And if it does, you can have a fairly high degree of confidence that it is going to generalize, right? Whereas, you know, when you're doing transduction, it's more like you're guessing where the answer might be.

• 35:08Speaker 00

  And you don't really have any way to verify that this is the right guess. So one thing you can do is just, you know, increase the sample size, like can make many independent guesses. And then you can look at which answers come up the most, the most often. But, you know, you're basically making the assumption that wrong guesses are all going to be wrong for different reasons. So you're going to end up with different wrong answers. But correct guesses are all going to be correct for the right reasons. So the correct answer will show up more often. But really, you have no way to make sure 100% that your guesses are correct.

• 35:48Speaker 00

  So it's much smarter to start with induction because then you can have a high degree of confidence that the solution you have is the right solution. And then fall back when it just doesn't work. So basically, induction is just the method you should prefer and you should only use transduction if it's not working.

• 36:05Tim Scarfe?

  Should we think of them as being completely different? So hypothetically, if you used a shared model for doing induction and transduction, do you think there could be some crossover between it?

• 36:14Speaker 00

  Absolutely. In fact, this is something that some people in the competition have tried. They're using, like this is the so-called OmniARC approach, where the team is using the same model to solve a range of different ARC-related tasks, so including writing down the program, but also interpreting programs, doing transduction, generating more inputs and so on. So all these different tasks with one single model. And that does lead to learning better representations for the concept of the OmniARC.

• 36:51Tim Scarfe?

  Can you give me some more intuition on that? So in a sense, you can get the network to think about the symbolic version at the same time as the solution space?

• 37:00Speaker 00

  The basic intuition is that if you look at the same problem from different angles, you are more likely to come up with the true shape of the problem.

• 37:11Speaker 00

  And if, this is especially true if your data structure of choice is a neural network, because neural networks have this tendency to latch on noisy statistical regularities. And if you're only using, you know, one, if you're only targeting one problem and only using one input modality, you're much more likely to overfit to elements of noise within that problem. But if you're forcing the same representations to work across many different views of the problem, it acts, it acts as, well, first of all, you get better information about the problem, right? Because there is some knowledge transfer, some information exchange

• 37:56Speaker 00

  between things like, you know, trying to predict the output grid and trying to generate more input grids, for instance, right? But also, it acts as a regularization mechanism, where the noise that you might be learning with one of the modalities is going to be countered, but which you're learning with another modality.

• 38:16Tim Scarfe?

  Another approach from Clement Bonnet, I think is his name is, is searching the latent space better. So not necessarily greenblating, but actually searching the latent space for quite a while before you present a prediction. What do you think about that?

• 38:31Speaker 00

  Yeah, this is actually, personally, one of my favorite papers that was submitted in competition. It is this very, very original idea that you're going to be learning a latent space of programs. And then at test time, so of course at test time, you need some adaptation to the problem at hand. You need some recombination of your knowledge. And while some people use test time training, some people use search. And with Clement Bonnet and his colleague Macfarlane are doing, it's very original. So they are learning this latent space of programs. And then at test time, they are doing gradient descent in latent program space to basically move around latent program space and find the point of the space that

• 39:16Speaker 00

  actually matches best the task. I think this is a good idea. There are many ways, I think, to improve the idea, but it's a very, very original take on test time adaptation. That is not search, that is not fine tuning. So I like it a lot.

• 39:35Tim Scarfe?

  Would you still call that process thinking? Is thinking a system to process? Or would you call that thinking as well?

• 39:42Speaker 00

  It is a form of test time search, except it's not discrete search. It's based on gradient descent. So sure, I don't see why you couldn't implement some form of system to processing with that.

• 39:56Tim Scarfe?

  Yeah, I wonder whether that breaks the analogy with human thinking. That's it's doing perceptual deliberation. It's quite an interesting category.

• 40:06Speaker 00

  It is doing deliberation latent space, yes. So I think one way to improve the process is that you could also try to decode your latent programs back into a symbolic discrete form. And then you can start doing a local discrete search around the decoded programs. And the benefit of that approach Is that you would have the ability to actually run the programs and verify whether they work. And as long as you stay in latent space, even if you're doing this gradient descent guided, you know, search within latent space to find the best possible points. That represents a target program, you are very much limited to guessing. You have no way to assert.

• 40:51Speaker 00

  That's what the latent space is telling you matches the reality on the ground. So the ability to decode back into real program space, right? And run these programs would be a very good addition to the system.

• 41:05Tim Scarfe?

  Are there any potential issues with that approach? So I guess it helps if the latent structure is quite homogeneous and the modes are easy to find. Could it be improved?

• 41:16Speaker 00

  So what do you mean by that exactly?

• 41:17Tim Scarfe?

  So he does gradient search over the latent space and find some, you know, some optimal position and then does the inference from there. But wouldn't that work very well if it was quite a convex space, but not so well if it was a very heterogeneous space?

• 41:31Speaker 00

  Of course, in order to be able to do gradient descent, you need a relatively smooth surface. But I think, you know, that's why they're using a VAE. And I'm not sure if they've tried just directly learning a program embedding space where one point is one program.

• 41:52Speaker 00

  As it turns out, you know, this is not what they are doing. They are using a VAE. And the reason why is because when you're using a VAE, you are learning much more structured, much smoother latent spaces. And I think this is key to making a test time gradient descent work.

• 42:07Tim Scarfe?

  So other unexplored avenues of ARC, I mean, maybe a better way to ask this is, you know, if you yourself spent a year working on ARC, what would you do?

• 42:16Speaker 00

  I would be doing a deep learning guided program synthesis. And, you know, I think the way people are doing deep learning guided program synthesis today is wrong. So everyone is leveraging LLMs, which of course makes sense because there are these very powerful tools. They contain a lot of useful knowledge and it can be replied on any problem. We've invested billions and billions of dollars into creating this tool. So not using these tools would feel like you're missing out on a lot of power. But I think it is not the right approach to treat program synthesis as just token by token code generation. I think the right way to think about a program is as a graph of operators.

• 43:01Speaker 00

  And program search, program synthesis is basically a tree search process. I think you're better off trying to use deep learning models and in particular L L M's to guide that tree search process. This is not something that many people are trying today, but I think this would be closer to the right approach. And another thing that people are not doing today, but should be, is that if you look at the way humans solve arc puzzles, they are not trying many different solution programs in their mind. They are only trying a few. I think humans have the capability to first describe a model with their thing

• 43:43Speaker 00

  and use this model, basically describing like a grid, for instance, in terms of the objects it contains and their properties and their relationships with other objects, with a focus on causal relationships in particular. So you can use these descriptive models to constrain the search space when you're finally looking at input to output programs. And that's the reason why we only need to consider a handful of programs before finding the correct one. So in that sense, it might be possible to do enough modeling

• 44:28Speaker 00

  of the task to almost make search irrelevant, to almost entirely remove the need for search.

• 44:37Tim Scarfe?

  It was so interesting what you just said. We should we should meditate on that just for a second. So first of all, loads of people I've interviewed this week, especially in the neural evolution kind of space, you know, under Jeff Clune, for example, many of them are latching on to L L M's because they say we need to have a measure of interestingness or novelty. And L L M's, because they're trained on all of the data in the world, they they capture our instincts really, really well, our intuition and so on. So they're a great way of generating programs. But but you said, though, that it's not such a good idea just to generate the program. The next step of evolution is guiding the search. And I think we're starting to see this enlightenment in the use of L L M's in many commercial bits of software.

• 45:19Tim Scarfe?

  So, for example, the original L L M in an app was you just have a chatbot and you just stick it in there. Now, things like Cursor, for example, they're exposing a low level A P I and they're using, you know, tool use and so on. And the L L M is actually guiding the low level, you know, A P I interactions in the app. And so you're advocating for a similar evolution here, where the L L M actually guides the discrete search process rather than just generating code.

• 45:45Speaker 00

  That's right. And the idea being that by creating your program via this iterative discrete search process, you actually have the ability to make targeted modifications to your program graph. And that will be significantly harder to make, I think, if you just treat the program as just a sequence of tokens. And also, you can, you're you're changing the nature of the space in which you are making addition decisions, right?

• 46:16Speaker 00

  And graphs are just the natural data structure to represent programs. Programs are not sequences of tokens.

• 46:23Tim Scarfe?

  Another thing we're seeing is an intermediate solution. So Kevin Ellis, he did this thing called remixing, where you have a hundred handcrafted solutions and for every single ask, arc task, you have a generator which can generate new instances. So, you know, they can generate sprites and layouts and so on. And then they did this kind of like expansion where they used retrieval augmented generation. And it was doing an implicit form of library learning and it was like mixing all the solutions together. Is that an intermediate to what you're talking about?

• 46:54Speaker 00

  Not quite. I think that's a separate avenue. The idea being that, yes, so in order to get LLMs to perform well on data set, you need to expose them to as much, as dense a sampling as possible of Arc space. And of course, there's not a lot of Arc tasks available, so you have to make new ones. And very easy to make new ones is to leverage an LLM to sort of like extract the programmatic concepts found in, for instance, the training set and then remix them into new tasks. But you know, at the end of the day, this leads you to probably see the other feeling. And this is exactly what we are seeing with their solution.

• 47:35Tim Scarfe?

  Because there's been a bit of an interesting evolution where even Kevin, in DreamCoder, there was an explicit concept of awake and, you know, asleep, you know, dreaming. And in his newer work on this, you know, learning by examples, by example pairs, he's kind of made that dream-sleep process implicit. And it just feels that maybe we could achieve some of the same stuff by kind of coming up with a proxy or implicit version of it.

• 48:08Speaker 00

  Yeah, yeah.

• 48:09Tim Scarfe?

  Yeah. Any other avenues for Arc that you're interested in? By the way, I think you spoke with George from Symbolica, and he had this kind of program verification approach that he said he discussed with you.

• 48:21Speaker 00

  Yes. So what we discussed is, what you described to me is basically this idea of using a symbolic process to turn a problem definition, a task, into a deep learning architecture and then training the deep learning architecture. I think that's a very, very original approach. I don't think that's anyone else, to the best of my knowledge, working on something similar. So I'm very curious about what he's going to be doing with it. It sounds fascinating.

• 48:46Tim Scarfe?

  Yeah, absolutely. Do you think, you know, just talking about benchmarks in general? Do we need to incorporate the compute budget in the benchmark?

• 48:55Speaker 00

  Yes, absolutely. And I think this is going to be a very pressing need in the in the future and in particular next year for the Public Leaderboard of ARC, because it's always possible with, you know, test and test and compute, like test and search, test and training and so on, to buy higher performance at the cost of more compute. And typically, you're going to see a logarithmic relationship at test time between compute and performance. And it kind of means that if you want an apples-to-apples comparison between two systems, you have to look at the compute budget. You can only compare systems that are using the same amount of compute, right?

• 49:38Speaker 00

  And

• 49:41Speaker 00

  for instance, if you look at the O1 model from OpenAI, you cannot really attribute to it a fixed score on ArcGIA unless you're also kind of limiting yourself to a certain amount of compute. It's always possible to logarithmically improve your performance by just throwing more compute to the problem. And of course, this is true for O1, but even before that, it was also true for brute force program search systems. Assuming that you have the right D S L, then extremely crude, basic brute force program enumeration can solve ARC at human level. It would just take, you know, hundreds of millions of dollars of compute

• 50:24Speaker 00

  to crack the entire dataset that way. So it's an extremely inefficient, extremely stupid idea, to be honest. But in theory, it's possible, right?

• 50:33Tim Scarfe?

  Philosophically speaking, do you think there's always a commensurate relationship between the amount of compute that is taken to do a task? What I mean by that is when we use language and when we use cognitive tools, we might not think we're using compute, but the universe presumably expended a lot of compute in order for those things to emerge. So in some sense, is it really possible to compress the amount of compute that we use?

• 50:57Speaker 00

  I think so. I think humans are tremendously compute efficient. And you see this in the fact that, let's say you're solving ARC tasks, for instance, you can solve the entire prior test set.

• 51:10Speaker 00

  And and only expend a few calories, basically.

• 51:14Speaker 00

  And you you could say, okay, but it's because, you know,

• 51:18Speaker 00

  we we are just using extremely little energy per operation that our brain does. It's actually not true at all if you're comparing transistors and neurons, for instance. You you find that neurons are tremendously more energy angry than transistors. It just so happens that we are managing to solve extraordinarily hard problems using a comparatively small amount of neural depersonations. And we're just tremendously energy efficient compared to current AI. And we are going to have a GI when we get to to the same level of energy efficiency.

• 51:54Tim Scarfe?

  What's your opinion on, you know, like, using a programming language like Python, a Turing complete language versus using a DSL in these approaches?

• 52:03Speaker 00

  Well, I think using a DSL like for R, for instance, is fundamentally limiting. No matter what you do, no matter what base language you're using, you should be able to learn the functions that you're applying from the data that you have. In fact, you should be able to do this as a lifelong process. So every time you find a new task and you're solving it, in the process, you're going to be coming up with useful abstractions, maybe abstractions that relate to to problems you've seen in the past. And so you're going to want to turn that into reusable functions, reusable building blocks and store them so that the next time

• 52:45Speaker 00

  you come across a similar problem, you can reapply the same building blocks and save compute, like solve an equally difficult problem in fewer steps. So no matter what you do, you want to learn the language that you're going to be using. And of course, that that could mean learning the DSL. That could also mean using something like Python, but within it increasingly writing higher order functions and classes and other reusable building blocks to enrich your language.

• 53:15Tim Scarfe?

  I wondered if you are softening your position at all. So you you tweeted, it's highly plausible that fuzzy pattern matching, when iteratively, when when sufficiently iterated many times, can asymptotically turn into reasoning. And it's even possible that humans basically do it in that way, but it doesn't mean it's the optimal way to do reasoning. Is is that a shift in your position?

• 53:34Speaker 00

  Compared to what position? I don't think it's a shift.

• 53:38Tim Scarfe?

  I suppose it's saying that, first of all, it's possible that we think in this way.

• 53:45Speaker 00

  I think we do, yes.

• 53:46Tim Scarfe?

  Oh, interesting, because I would have thought that because of the spell key view on things, that you would have thought that we do this high level reasoning and we don't do the fuzzy matching.

• 53:59Speaker 00

  I think the fundamental cognitive units in the human brain is actually fuzzy pattern cognition. That's the core thing that you do. And then when you're doing something that's more akin to reasoning or planning, when you're doing basically system to processing, this sort of like slow, logical, step by step processing, what you're really doing is iteratively applying your intuition, but in a structured form. And by the way, this is exactly what deep learning guided program synthesis is about, which is the approach I've been advocating for since 2017. What are you doing when you're doing deep learning guided program synthesis? You are building a program, so basically a graph of operators,

• 54:44Speaker 00

  but you're building it step by step. And at each step, when you're choosing what to edit in your graph, what to add, what branching decision to make, you're applying your intuition, you're applying a guess, provided by a deep learning model, right? So you're iteratively guessing to create this highly structured, symbolic, discrete artifact. So this program, and when you're running this program, that system too. I think this is basically the way we do system to as humans, is that we are iteratively guessing, iteratively applying fuzzy pattern cognition to construct an artifact that is in fact symbolic in nature. Like, let's say you're playing chess, for instance.

• 55:30Speaker 00

  When you're calculating in your mind, you're unfolding some moves step by step, but you're only going to be doing it for a few of the moves that are on the board. And how do you do this selection of which moves to look at? You're applying pattern cognition. And then when you're sort of like... Simulating one move into the future and not going to be simulating the entire board, you are going to be focusing on some areas. So again, that's that's pattern recognition. And sometimes, by the way, any sort of pattern recognition is basically a guess at heart. So it might be wrong, right, in some way. And while sometimes in chess you're calculating and some of your intuition about future states

• 56:11Speaker 00

  of the board is wrong, and then you play the move and then you realize, oh, oops. Right. And so I think this is basically how humans implement system. So this is not a shift in my position. I've been thinking about these ideas, you know, for quite a while. In fact, this is the basis of my current favorite theory for how to interpret consciousness. It's this idea that, well, in order to, for something like system two to arise from iterated fuzzy pattern recognition, that iteration sequence needs to be highly self-consistent. So everything you add, you need to double check that it matches what came before it,

• 56:56Speaker 00

  you know. If you're just iteratively pattern matching with no guardrails whatsoever, you're basically hallucinating, you're dreaming. This is exactly what happens when you're in dream state, by the way. You're just repeatedly intuiting what comes next, with no regard whatsoever for consistency, for self-consistency with the past. And I think this is the reason why any sort of deliberate logical processing in the brain needs to involve awareness, needs to involve consciousness. Consciousness is this sort of like self-consistency check.

• 57:38Speaker 00

  It's the process that forces the next iteration of your intuition of this pattern recognition process to be consistent with everything that came before it. Right. And the only way to achieve this consistency is via this sort of like back and forth loops that are sort of like brain Bringing the past into the present, bringing your prediction of the future into the present. So you have this sort of like nexus point in the present, this thing you're focusing on. And that that nexus is basically your consciousness. So consciousness is the process that forces iterated pattern recognition to turn into something

• 58:19Speaker 00

  that's actually like reasoning, that's actually self-consistent.

• 58:22Tim Scarfe?

  Yeah, so I think the issue was you haven't changed your position. It's just a case of people understanding what your position is. So I think people dichotomize system, you know, like symbolic people as thinking it's all like this discreet world. And then the the alternative, you know, the the connectionist approach is that it's all fuzzy matching.

• 58:41Speaker 00

  I've I've never been in the purely symbolic camp. Like if you go back to my earliest earliest writing about, yes, we need program synthesis. I was saying we need deep learning guided program synthesis. We need a merger of intuition and pattern recognition together with discrete step by step reasoning and search into one single data structure. And I've said very, very repeatedly, you know, for the past eight years or so, that human cognition really is a mixture of intuition and and reasoning. And that you're not going to get very far with only one of them. You need the continuous kind of abstraction that's provided by, you know, vector spaces, deep learning models in general,

• 59:22Speaker 00

  and the more discrete symbolic kind of abstraction provided by graphs and and and discrete search.

• 59:29Tim Scarfe?

  So why why do your detractors see you as a symbolist when you're clearly not?

• 59:34Speaker 00

  Well, I'm not sure. I've I've been into deep learning, you know, for a very long time, since basically twenty twenty thirteen. I started evangelizing deep learning very heavily around twenty fourteen. And back then, the field, the field was pretty small. You know, especially with chaos, I think I've done quite a bit to popularize deep learning, make it make it accessible to as many people as possible. I've always been a deep learning guy, right? And when I started thinking about the limitations of deep learning, I was not thinking in terms of replacing deep learning with something completely different. I was thinking of augmenting deep learning with with symbolic elements.

• 1:00:13Tim Scarfe?

  So you commented as well in this tweet that you were just talking to about about consciousness. You suggested that all system to processing involves consciousness.

• 1:00:23Speaker 00

  Yes.

• 1:00:24Tim Scarfe?

  Explain more what you mean by that.

• 1:00:26Speaker 00

  So any sort of explicit step by step reasoning needs to involve awareness. And in reverse, if there's any cognitive process that you're running unconsciously, it will not have this strong self consistency guarantee. It will be more like a dream, an hallucination.

• 1:00:47Speaker 00

  It's basically the idea that if you're just iteratively guessing, unless you have this this strong self consistency guarantee, you will you will end up drifting, where you end up, you know, diverging. And consciousness is is the self consistency guardrail, basically. And this is why you cannot have a system to without consciousness.

• 1:01:14Tim Scarfe?

  What is your definition of reasoning?

• 1:01:17Speaker 00

  I don't really have a single definition of reasoning. I think it's a pretty loaded term and you can mean many different things by that. But there are at least two ways in which I see the the term being used and they're actually pretty different. So for instance, if you're just, let's say, memorizing a program and then applying that program, you could say that's a form of reasoning. Like, let's say, in school, you're learning the algorithm for multiplying numbers, for instance. Well, you're learning that algorithm, then when you have tests, you're actually applying the algorithm. Is that reasoning? I think, yes, that's that's one form of reasoning. And it's it's the kind of reasoning that L L M's and deep learning models in particular are very good at.

• 1:02:01Speaker 00

  You're memorizing a pattern and at this time you're fetching a pattern and reapplying it. But another form of reasoning is when you're faced with something you've never seen before and you have to recompose, recombine the cognitive building blocks you have access to, so your knowledge and so on, into a brand new model and do so on the fly. That is also a reasoning. But it's a very, very different kind of reasoning and it generalizes very different kinds of capabilities. So I think, you know, the important question about deep learning models and LLMs in particular is not can they reason? There's always some sense in which they are doing reasoning. The more important question is can they adapt to novelty? Because there are many different systems that could just, you know, memorize programs

• 1:02:46Speaker 00

  provided by humans and then reapply them. What's more interesting is can they come up with their own programs, their own abstractions on the fly?

• 1:02:57Tim Scarfe?

  And what would it mean for a system to come up with its own abstraction?

• 1:03:02Speaker 00

  To start coming up with abstractions, first of all, you start, you need to be solving novel problems. So solving a novel problem means that you're starting from some base of knowledge, some building blocks, then you're faced with a new task, you are recombining them into a model of the task and you're applying this model and it works. And in the process, as you solve many problems, you will start noticing that some patterns of recombination of the building blocks that you had happened often, right? And when you start noticing this, well, it means you can take them and abstract them, refactor them into a more reusable form.

• 1:03:47Speaker 00

  And then this reusable form, you can just add it back to the set of building blocks that you have access to, right? And so next time you encounter a similar problem, you're going to be able to solve it in fewer steps, expending less energy, because you have access to this higher level abstraction that's fit to the problem.

• 1:04:07Tim Scarfe?

  Is there a way of measuring the strength of reasoning?

• 1:04:12Speaker 00

  So again, you would need to start by defining precisely what you mean by reasoning. I think you can define, for instance, you can define precisely generalization power, which is basically the amount of novelty that you can adapt to.

• 1:04:29Tim Scarfe?

  Yeah, it's quite interesting because I suppose you define it in terms of performance as in how good is my model. Another way of describing it is let's imagine we think of reasoning purely as traversing the deductive closure, the closure, so just composing together knowledge we already have in new configurations and then we make that that leap in solution space because we've found a new model that works really, really well. Is there an intrinsic way of measuring the type of model rather than its generalization power?

• 1:05:00Speaker 00

  No, I think you you really have to observe what the model does. You cannot just inspect the model and tell you, you know, how strong, how strong it is at reasoning.

• 1:05:10Tim Scarfe?

  So there's no, there's no intrinsic form of this is good reasoning?

• 1:05:14Speaker 00

  Given two, two models of a problem, for instance, which, which model is better? Can you just look at them and tell which one is better? I think it's very much goal dependent, right? You cannot really evaluate a model, like a simulation of a thing, for instance, if you don't have something that you want to do with it. But if you do have a goal, then you just look at what are the causal factors required to achieve that goal? And then the best model is going to be probably the simplest model that retains these causal factors.

• 1:05:50Tim Scarfe?

  Francois, what are you doing next?

• 1:05:53Speaker 00

  Well, so I just left Google a few weeks ago and so I'm starting a new company, a new research lab with a friend. And yeah, so I can't really share much for the time being, but we're going to be tackling program synthesis and in particular, the planning guided program synthesis. And we're currently building the team.

• 1:06:13Tim Scarfe?

  Amazing. Are you looking for people to join?

• 1:06:15Speaker 00

  Yes.

• 1:06:17Tim Scarfe?

  Tell me more. What are you allowed to tell us?

• 1:06:19Speaker 00

  So, yeah, I mean, what I can tell you is, you know, I've been talking about some of these questions, like the best way to get to get to a GI about the planning guided program synthesis, about

• 1:06:33Speaker 00

  ArcGIS, methods surrounding ArcGIS, but what the next benchmark after ArcGIS might be. So then I think about this question sort of like on the side while at Google, where my full-time job was developing cars, right? And now I feel like the time has come to just focus full-time on the research question. So make the research not a side project, but the main thing.

• 1:06:59Tim Scarfe?

  And is the focus creating the new benchmarks or beating the benchmark?

• 1:07:03Speaker 00

  I think both. I'm a pretty strong believer in the idea that you need to co-evolve the solution together with the problem. And that was actually the motivation for creating ArcGIS in the first place. It's to have the right challenge that forces you to focus on the right questions, on the main bottlenecks to achieving strong generalization with AI. And so I don't think, you know, ArcGIS is the last benchmark. And of course, there's going to be a V2 of ArcGIS. But that also is not going to be the last benchmark, I think. We are always going to be needing new benchmarks, exploring new things that are hard for AI and easy for humans.

• 1:07:42Tim Scarfe?

  Is it cheating in any way if you, you know, do work on your own benchmark?

• 1:07:48Speaker 00

  I don't think so. The benchmark is meant as a tool for research. It's again, like something like ArcGIS, for instance, is not really meant as a binary indicator telling you, oh, do we have HGI or not? It's really meant as a research tool. It's a challenge that forces you to work on the right questions that kind of directs your attention to the right problems and help you make progress. So you could say that ArcGIS is basically a compass towards HGI. It's not like a test for HGI.

• 1:08:19Tim Scarfe?

  And, you know, there's a spectrum of possible solutions to work. So for example, because you know what's in the private test set, you could just put the answers in directly.

• 1:08:27Speaker 00

  Of course, but I'm not going to be like entering Arc prize in any way. I mean, the obviously not.

• 1:08:33Tim Scarfe?

  Right.

• 1:08:33Speaker 00

  I'm running out prize anyway, so why would I enter it, you know?

• 1:08:37Tim Scarfe?

  But on that spectrum of generalization, there's the moonshot, which is going for extreme generalization.

• 1:08:43Speaker 00

  Yes.

• 1:08:44Tim Scarfe?

  Or there's one notch below that. Where are you going for?

• 1:08:48Speaker 00

  I would like to build HGI. I would like to build something with human-level capability.

• 1:08:53Tim Scarfe?

  And what would that mean? What kind of things do you think you could achieve with that kind of AGI?

• 1:08:58Speaker 00

  Well, the most obvious thing is solving programming, right? If you solve AGI, then you can just describe what you want to a computer and the computer will build it for you. And if it's really AGI, then it will scale to the same level of complexity, the same level of code-based complexity that you can do with a human software engineer. And you know, probably it's not going to stop there either.

• 1:09:21Tim Scarfe?

  How long do you think, you know, what's the role of humans in software engineering when we start to get there?

• 1:09:27Speaker 00

  Well, we'll see. I think we're gonna start creating entirely new tools, entirely new interfaces to work with this technology when it's ready. We are, we are still pretty far from it. Like we are talking about something that doesn't quite exist yet. I don't think frontier models, not even O1, are quite at that level.

• 1:09:48Tim Scarfe?

  Right now programmers are very technical. Do you think that programming might be democratized in the future?

• 1:09:54Speaker 00

  I think so, yes. I think in the future, anyone should be able to basically develop their own automation processes based on their own, their own, you know, domain expert knowledge of the problems they're facing. Everybody should be able to program, not, not, not really in the sense of writing down code, but describing to the computer what they want to automate and how they want to automate it and the computer will just do it.

• 1:10:19Tim Scarfe?

  A big thing in software is tackling complexity at different scales. It's certainly what, what you've been doing for your, for your entire career. Do you think we'll always have this problem that we'll always be on the boundary of this incredible complexity?

• 1:10:34Speaker 00

  What do you mean by that?

• 1:10:35Tim Scarfe?

  Well, as in, even if we democratize one or two steps up the hierarchy, wouldn't we still always just build software which is just really complicated?

• 1:10:44Speaker 00

  Yeah, quite possibly, but the idea that we're gonna be able to offload that complexity to an external, you know, complexity processing AI.

• 1:10:55Tim Scarfe?

  So we will transgress into a future where we no longer understand what the code that's being run in any way.

• 1:11:01Speaker 00

  Absolutely. You know, I think to a large extent, this is already true. If you look at any sizable code base, there is no single software engineer that actually understands it all. We always, we're always limited due to a fragmented understanding of what we are doing, which is fine as long as we have a good grasp on the high level goals and constraints of the system.

• 1:11:23Tim Scarfe?

  So where should the source of agency be there? Because you know, you're describing a blind man and the elephant type challenge, where loads and loads of developers have their own perspective on a very small part of the system. But when we have the AGI version, how could that change?

• 1:11:37Speaker 00

  Broadly speaking, I think programming from input output pairs will be a widespread programming paradigm in the future. And that will be accessible to anyone because you don't need to write any code, right? You're just specifying what you want the program to do. And then the computer just programs itself. And if there's any ambiguity, by the way, in what you meant, and there will always be ambiguity, right? Especially if the instructions are provided by a non-technical user. Well, you don't have to worry about it because the computer will ask you to clarify. If you tell you, okay, so I created basically the most plausible program, given what you told me. But there's some ambiguity here and there. So what about this input?

• 1:12:21Speaker 00

  Currently I have this output. Does that look right? Do you want to change it? And as you change it, you know, iteratively, you are creating this correct program in collaboration with the computer.

• 1:12:32Tim Scarfe?

  So these future systems will have program synthesis as a core component, an explicit component. But how will the humans interface with it? Are we still going to describe with natural language? And you know, gestures and images and things like that.

• 1:12:44Speaker 00

  It could be natural language. It could also just be drawing interface elements on your screen. It could be.

• 1:12:54Speaker 00

  You know, you could always try to generate a high-level representation of the program that's being generated that is at a level where it can be visualized and understood by a non-technical user. I could show, for instance, a kind of data flow graph and as the user for input about it.

• 1:13:15Tim Scarfe?

  Very cool. So one of the characteristics of LLMs at, you know, the hyperscale is, in a sense, it's not that difficult because they just have to, it's like a CDN, they just copy all the weights and they move it all over the place. You're describing something which is very sophisticated. It might be a little bit akin to a globally distributed database where the skill programs move around all the different nodes and so on. Is there just a massive new type of architecture we need to build for this?

• 1:13:42Speaker 00

  Yeah, I think we're going to need a completely new type of architecture to implement a lifelong distributed learning where you have basically many instances of the same AI solving many different problems for different people in parallel and looking for commonalities between the problems and commonalities between the solutions. And anytime they find sufficient commonality, they just abstract these commonalities into a new building block, which goes back into the system and makes the system more capable, more intelligent.

• 1:14:12Tim Scarfe?

  I think I've got it now. So what you're building is a globally distributed ARC on the basis that we find a good solution to ARC.

• 1:14:19Speaker 00

  Well, I can't really tell you exactly what we're building, but it's going to be cool.

• 1:14:24Tim Scarfe?

  Yeah, it sounds pretty cool.

• 1:14:28Tim Scarfe?

  How do you think that folks like OpenAI are going to start incorporating not only like test time inference, but you know, some of your ideas realistically into their system?

• 1:14:37Speaker 00

  How might frontier models incorporate program synthesis, for instance? Well, you know, I think something like O1 is already doing precisely that. When you look at what O1 is probably doing, that it's writing its own natural language program describing what it's supposed to be doing and it is executing itself this program. And the way it's writing this program is via a very sophisticated search process. So this is already a program synthesis in natural language space. There are other ways you could leverage your prompts. And this is you could do prompts in latent space potentially, you know, kind of like what Clément Bonnet and friends are doing.

• 1:15:22Speaker 00

  You could also just be generating actual programs. Like, why do we use natural language? Sometimes, you know, maybe you might want to use an actual programming language. So I think we are definitely seeing a shift towards leveraging, you know, more and more test time compute, and that's going to accelerate. It's a phenomenal trend, it is not.

• 1:15:48Tim Scarfe?

  On the transductive active fine tuning, though, that's a little bit more architecturally difficult, isn't it? Because my model is always being fine tuned. So I can imagine they might build something a bit like Docker, where there's the base layer and then there's, you know, my fine tuning layer and another fine tuning layer. And it's very kind of fragmented.

• 1:16:07Speaker 00

  So the difficulty in applying test time training in actual like frontier models is not so much the infrastructure. So it's definitely true that current serving infrastructure is absolutely not set up for doing like per task fine tuning. But you could re-engineer it for it. The main bottleneck is actually the task format. You can only do test time training if you have pretty clear inputs and targets. You basically need input output pairs, right? And for most prompts, you don't have that. You have that for ARC, obviously, but for most problems, you don't have that.

• 1:16:42Tim Scarfe?

  Very cool. What's your theory on how O1 works?

• 1:16:46Speaker 00

  Well, you know, we can only speculate. I'm not sure how it really works. But what seems to be happening is that it is running a search process in the space of possible, the space of possible chain of thought.

• 1:17:05Speaker 00

  Trying to evaluate which branches in the tree work better, potentially backtracking and editing if the current branch is not working out. It ends up with this very long, unsophisticated and plausibly near-optimal chain of thought, which represents basically a natural language program describing what the model should be doing. And in the process of creating this program, the model is adapting to novelty. And so I think something like O1 is a genuine breakthrough in terms of the generalization power that you can achieve with these systems. Like, we are far beyond the classical deep learning paradigm.

• 1:17:49Tim Scarfe?

  So one school of thought, which I think you agree with, is that there's some kind of active controller at inference time. So it's actually doing multiple trajectories in an isolated way.

• 1:17:59Speaker 00

  Yes, it's doing search.

• 1:18:01Tim Scarfe?

  Okay, because some people think that it's, you know, there's process supervision and whatnot at training time, but at inference time, it's all just one, you know, one forward pass.

• 1:18:12Speaker 00

  No, that is certainly not plausible because of the amount of compute that's being expended at test time. Like, it is very clearly doing search at test time.

• 1:18:22Tim Scarfe?

  Very interesting.

• 1:18:22Speaker 00

  I think it is trained at training time

• 1:18:27Speaker 00

  to reproduce the best available chain of thought for the current problem, right? Kind of like, you know, AlphaZero style training, basically. But it's also doing search at test time in chain of thought space. And this is kind of obvious in the, the telling sign is just the compute it's expanding, like the amount of tokens and the latency.

• 1:18:51Tim Scarfe?

  Are there any other telltale signs that that kind of thing is happening? Like, for example, it might have explored a particular area, and then after the consolidation, that is now gone. So when you talk to the model, it's almost like it's forgotten part of its thinking.

• 1:19:07Speaker 00

  Yeah, honestly, this is a little bit too specific. You know, I don't have any insider info about what O1 is doing, so I can only speculate.

• 1:19:15Tim Scarfe?

  Okay. So people like Noam Brown, they're really bullish on this new scaling law for test time. And certainly, I love O1 Pro. I think it's really, really good. It's qualitatively a big improvement.

• 1:19:30Tim Scarfe?

  What do you think there?

• 1:19:32Speaker 00

  Sure, so the test time scaling law is basically this observation that if you expand more compute, if you search further, you see a corresponding improvement in accuracy. And that relationship is logarithmic, by the way, right? So accuracy improves logarithmically with compute. And while this is not really new, like anytime you do test time search, you will see this relationship. Like if you're doing brute force program search, for instance, you will find that your ability to solve a problem improves logarithmically with the amount of compute. If you have more compute, you can just search further into the space of possible programs and logarithmically you find more solutions.

• 1:20:17Speaker 00

  So anytime you do any kind of test time search, you will see this relationship.

• 1:20:22Tim Scarfe?

  What is your current go to model and what do you use it for?

• 1:20:26Speaker 00

  For the most part, I'm using Gemini Advanced. And I've actually just started using the new Gemini Flash, like the latest one.

• 1:20:35Tim Scarfe?

  Me too.

• 1:20:35Speaker 00

  So I'm paying for Gemini Advanced. I'm also using Cloud 3.5, I think it's very good for programming. So these are the two I'm using.

• 1:20:45Tim Scarfe?

  What's your programming workflow with L L M's?

• 1:20:48Speaker 00

  I don't use L L M's all that much when I'm programming. But typically, you know, if I'm currently facing a problem that I feel might be a good fit for L L M's, I will just, you know, open my brother and prompt the L L M and ask it for like a function that does X, Y, Z. And while it usually doesn't really work on the first try, but you know, after a little bit of debugging and the change, I think it's a big time saver.

• 1:21:16Tim Scarfe?

  What kind of failure mode do you see when you're programming?

• 1:21:21Speaker 00

  With L L M's, you mean, well, so the failure modes are kind of different based on the model that you're using, right? I think in general, SNES 3.5, Cloud 3.5 SNES is the best one.

• 1:21:33Tim Scarfe?

  Yeah.

• 1:21:34Speaker 00

  But sometimes, you know, you might have code that's there for absolutely no reason, like variables that are not used or assumptions that are being made by the code that are not very far by the data that comes in, you know? So it's pretty clear that there is no.

• 1:21:54Speaker 00

  The code is generated in terms of statistical likelihood. There's no effort to actually make it self-consistent, make it correct, try to execute it beforehand and so on. So I think there's actually a lot of room for improvement there. And you could imagine LLM-based software developer assistants in the future that actually do all these things as you're prompting them, that actually write you the code, but then try to debug it themselves before actually showing it to you.

• 1:22:28Tim Scarfe?

  What's your opinion on LLM agent systems, which are all the rage at the moment?

• 1:22:33Speaker 00

  Right, so agents have been all the rage for quite a while now. People started talking about agents being the future, something like, you know, almost two years ago, like a year and a half ago-ish. And so far, agents have not really taken off. So agents have some... So the fundamental problem here is that LLMs are not quite reliable. You know, if you look at one forward pass of an LLM, that's basically... You can think of the LLM as a guessing machine, right? And the guesses it makes are much better than random, obviously. They're very useful guesses, they're directionally accurate, but they have some probability of being wrong.

• 1:23:17Speaker 00

  And when you look at an agentic workflow, you are chaining many of these guesses. And so the likelihood that you will end up not where you would like to be gets dramatically higher as you chain more guesses like this. And so this is the big bottleneck. Agents are just not reliable. They just don't have a sufficient level of autonomy. And so people say that with better models, this will get fixed. I think it's an impeccable question. You know, I'm waiting to see when agentic workflows actually start working. I don't think we're there today.

• 1:23:52Tim Scarfe?

  I've softened my position a tiny bit on this. I agree because of the ambiguity problem, they are misdirected. And when you chain this, they're very misdirected. But there is something to be said for just having more computation at hand. So I interviewed the guys that did the AI scientist paper. And certainly, if you take Claude Sonnet 3.5 and you say, generate me an entire paper, it will be banality beyond belief. It would just be a sketch. And what they did was they decomposed it into lots of agent workers that just, you know, on the Google Maps analogies, just kind of zooming in and zooming in and zooming in many, many, many times over. And it produced dramatically better results.

• 1:24:28Speaker 00

  Yeah, yeah, this makes sense. I think this basically tracks the idea we were talking about earlier of system two being something like iterated system one with strong guardrails. And the guardrails are very important. So in this case, the superstructure is provided by the human programmer. The human programmer is kind of breaking down the problem into the right sub-problems and sort of like orchestrating the whole thing in the right way. And then each sub-problem can actually be solved by guessing and producing good enough guess.

• 1:25:02Tim Scarfe?

  Amazing. When's Arc V2 coming up?

• 1:25:05Speaker 00

  Early next year, probably. So we are currently finalizing human testing. So as I mentioned, you know, every puzzle is going to be solved by a bunch of humans that we know it's solvable, that we have some data to be able to detect hard difficulties for the average human. And the goal is going to be to have three sets that are difficulty calibrated. So if you get a score on the public eval, and you're not overfit to that data set, you can be very confident that you're going to get a very similar kind of score on the other two sets.

• 1:25:37Tim Scarfe?

  When you tested with humans, you know, you wrote about this in your measure of intelligence, that, you know, one school of thought in intelligence is that there's this G factor. And another school of thought is that it's very specialized. Did your experiments reveal that, you know, a group of humans did generally perform quite well between all of the tasks? Or did you see huge specializations?

• 1:25:55Speaker 00

  No, it's absolutely the case that people were, there are people who are just more intelligent and they are just better at solving hard tasks. And you do see that. Yeah. In the human testing data.

• 1:26:05Tim Scarfe?

  What about on the long tail? Do you see specialization in the types of tasks? Or is it fairly, fairly flat?

• 1:26:10Speaker 00

  Yeah, I think it's pretty flat. I mean, either you're good at it or not.

• 1:26:14Tim Scarfe?

  Interesting. François Chollet, it's been an honour to have you on the show. Thank you so much.

• 1:26:18Speaker 00

  Thanks so much for having me. It's been great.