New top score on ARC-AGI-2-pub (29.4%) - Jeremy Berman

We need AI systems to synthesise new knowledge, not just compress the data they see. Jeremy Berman, is a research scientist at Reflection AI and recent winner of the ARC-AGI v2 public leaderboard.**SPONSOR MESSAGES**—Take the Prolific human data survey - https://www.prolific.com/humandatasurvey?utm_source=mlst and be the first to see the results and benchmark their practices against the wider community!—cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economyOct SF conference - https://dagihouse.com/?utm_source=mlst - Joscha Bach keynoting(!) + OAI, Anthropic, NVDA,++Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlstSubmit investment deck: https://cyber.fund/contact?utm_source=mlst— Imagine trying to teach an AI to think like a human i.e. solving puzzles that are easy for us but stump even the smartest models. Jeremy's evolutionary approach—evolving natural language descriptions instead of python code like his last version—landed him at the top with about 30% accuracy on the ARCv2. We discuss why current AIs are like "stochastic parrots" that memorize but struggle to truly reason or innovate as well as big ideas like building "knowledge trees" for real understanding, the limits of neural networks versus symbolic systems, and whether we can train models to synthesize new ideas without forgetting everything else. Jeremy Berman:https://x.com/jerber888TRANSCRIPT:https://app.rescript.info/public/share/qvCioZeZJ4Q_NlR66m-hNUZnh-qWlUJcS15Wc2OGwD0TOC:Introduction and Overview [00:00:00]ARC v1 Solution [00:07:20]Evolutionary Python Approach [00:08:00]Trade-offs in Depth vs. Breadth [00:10:33]ARC v2 Improvements [00:11:45]Natural Language Shift [00:12:35]Model Thinking Enhancements [00:13:05]Neural Networks vs. Symbolism Debate [00:14:24]Turing Completeness Discussion [00:15:24]Continual Learning Challenges [00:19:12]Reasoning and Intelligence [00:29:33]Knowledge Trees and Synthesis [00:50:15]Creativity and Invention [00:56:41]Future Directions and Closing [01:02:30]REFS:Jeremy’s 2024 article on winning ARCAGI1-pubhttps://jeremyberman.substack.com/p/how-i-got-a-record-536-on-arc-agiGetting 50% (SoTA) on ARC-AGI with GPT-4o [Greenblatt]https://blog.redwoodresearch.org/p/getting-50-sota-on-arc-agi-with-gpt https://www.youtube.com/watch?v=z9j3wB1RRGA [his MLST interview]A Thousand Brains: A New Theory of Intelligence [Hawkins]https://www.amazon.com/Thousand-Brains-New-Theory-Intelligence/dp/1541675819https://www.youtube.com/watch?v=6VQILbDqaI4 [MLST interview]Francois Chollet + Mike Knoop’s labhttps://ndea.com/On the Measure of Intelligence [Chollet]https://arxiv.org/abs/1911.01547On the Biology of a Large Language Model [Anthropic]https://transformer-circuits.pub/2025/attribution-graphs/biology.html The ARChitects [won 2024 ARC-AGI-1-private]https://www.youtube.com/watch?v=mTX_sAq--zY Connectionism critique 1998 [Fodor/Pylshyn]https://uh.edu/~garson/F&P1. PDF Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis [Kumar/Stanley]https://arxiv.org/pdf/2505.11581 AlphaEvolve interview (also program synthesis)https://www.youtube.com/watch?v=vC9nAosXrJw ShinkaEvolve: Evolving New Algorithms with LLMs, Orders of Magnitude More Efficiently [Lange et al]https://sakana.ai/shinka-evolve/ Deep learning with Python Rev 3 [Chollet] - READ CHAPTER 19 NOW!https://deeplearningwithpython.io/

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Hey Spotify listeners. When you travel with American Airlines, you'll enjoy comfort and convenience at every altitude. Before takeoff, unwind in the Admirals Club Lounge with curated self-serve offerings of hearty bites and refreshments. In flight, stay connected with high-speed Wi-Fi. Sponsored by AT&T. Your journey is designed to feel easy from the moment you check in to the moment you arrive. No matter where you're going, American gets you there. Book now at aa.com.

• 0:27Speaker 01

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required. Compatibility and availability varies. 18+

• 0:52Speaker 02

  You can describe every single ArcV2 task in 10 bullet points of plain English, most of them in five bullet points. And I think this actually gets to the heart of Arc, right? Everything is quite simple. It's not very hard. And I think this is also how we do it too, right? Like, when we look at these Arc graphs, we're coming up with these bullet points in our head and we're checking them, okay, this was right, this was right. And Python doesn't have these features. It's just not as expressive as natural language. MLST is sponsored by Cyberfund. Link in the description. Actually, even more fundamentally, like, the ideal system would be we have a set of data. Our language model is bad at a certain thing. We can just give it this data and then all of a sudden it keeps all of its knowledge and then also gets really good at this new thing. We are not there yet.

• 1:35Speaker 02

  And that, to me, is a fundamental missing part. Really, what you want is a more expressive program. And so that's why I switched from Python to English, which is a much more expressive program. You can always teach a language model a skill, right? But it's the meta-skill. It's the skill to create the skills that is AGI. And to me, that's reasoning. Like, reasoning is that meta-skill. And so, to put it another way, I think if you fundamentally learn the skill of reasoning, you should be able to then apply that skill to learn all the other skills. That is the meta-skill. Kick whatever weights out you need to align the model to reason. And then, from there, you have a foundation from which you can actually build general intelligence.

• 2:17Tim Scarfe?

  Okay folks, hot off the press. Many of you would have seen last week that Jeremy Berman, who is a research scientist at Reflection AI, is now the winner of the Arc AGI V2 leaderboard, the public version of the leaderboard. He's using an evolutionary approach. Now, remember last year in December, he published a similar approach, generating Python functions and then refining those functions in a kind of iterative loop. His new architecture is generating descriptions of algorithms rather than code and iteratively, in an evolutionary sense, refining those ones and discarding the ones that don't work. He's now at the top of the leaderboard. It's a really, really cool and elegant algorithm.

• 3:02Tim Scarfe?

  And by the way, he works for Reflection AI, so he's doing reinforcement learning with verifiable feedback. And he's trying to address the biggest gap in AI at the moment, which is that we want systems that can synthesize new knowledge and new understanding. Current systems just get trained with a whole bunch of data and they only know what they've been trained on. They can't kind of think outside the box by creatively synthesizing new knowledge. Prolific are really focused on the contributions of human data in AI. And the reason this is important is actually the dirty secret of Silicon Valley, the extent to which human data is used to evaluate and fine tune AI models. The reason for that, as we discuss in today's show, is that current AI

• 3:47Tim Scarfe?

  does not understand the world in a grounded way. It doesn't have a deep, abstract understanding of the world, which is why the only way that we can make AI work effectively is by grounding the generation and supervising the training of AI models with human data. Prolific are putting together a report on how human data is being used in AI systems, and they need volunteers. You can just go and fill out this form to help them produce this report and you will get privileged access to see the report before anyone else. The link is in the description. Oh, there was an amazing part in, I think it was in your first paper, where you said, "A parrot that lives in a courthouse will regurgitate more correct statements than a parrot that lives in a madhouse."

• 4:33Speaker 02

  Thank you, thank you. My sister, who doesn't know anything about language models or AI, she pointed that out and said that was a great line. So at least I have that.

• 4:43Tim Scarfe?

  I've already used it. I credited you. I'll be using that quite a lot. Well, Jeremy, it's amazing to have you on MLST. I've wanted to have you on ever since you released your first blog post. You know, it was December last year. I was at NURIPS at the time. And at the time, you actually got the highest score on the the public ARC V1 leaderboard, just before the famous O3 launch. Do you remember when when they did this ridiculous $200 per task thing and they knocked you off the board? But Jeremy, can you just tell the audience a little bit about yourself? And and and maybe we should start with your with your first ARC solution?

• 5:22Speaker 02

  Yeah, sure. So I actually have only been working in research for about eight months. Before that, I had a company right out of college. I was got into Y Combinator. And so I've been running a company for the last four and a half years as CTO. And I've always been very interested in reasoning, in the brain. And I actually picked up Jeff Hawkins' book, A Thousand Brains. And I read that. And at the same time, I was kind of coming into language models. And something just clicked inside of me. And I just knew I had to be working on this. And, you know, I believe that general intelligence, artificial general intelligence, will be the most important invention of hopefully my lifetime. And so I decided to drop everything. I stepped down as CTO. Company's still going well. So it was

• 6:07Speaker 02

  a difficult decision. And I actually had gotten in touch with Mike and Francois, because I thought ARC AGI was such an elegant way of describing the problems with current language models, and their difference between the human brain. And so I just kind of dug in. That was my first research project independently. And yeah, I ended up getting the top score on that. And that was really great. And after that, I got recruited into Francois and Mike's AGI Lab, India, where I was working on program synthesis. And as you described earlier,

• 6:42Speaker 02

  over time, I've become more convinced that language modeling with reinforcement learning will yield

• 6:50Speaker 02

  Uh, generalization, um, far beyond what we see today. And so, um, I decided to, uh, move to a company that was focused purely on language models. And that's where I am now. So I'm, uh, currently working on reasoning and post-training at Reflection, which is, uh, we're building frontier, uh, foundation models.

• 7:08Tim Scarfe?

  Very cool. Maybe we should save that bit for a, for a tiny bit later, but, you know, one of the take-home messages in your, well, no, I mean, it's, it's super interesting. And one of the take-home messages on your new approach is that rather than producing, um, explicit programs, you are evolving descriptions of programs. And Francois is a neuro-symbolic guy. He thinks that we need to have a symbolic substrate where we, you know, represent the, the kinds of problems that, that we can do. And we need to do this kind of compositional form of, of, of intelligence. So we need to kind of be working in the symbolic layer, but perhaps guided by, um, you know, deep learning models. But maybe we, we should get to that, um, in, in a minute.

• 7:51Tim Scarfe?

  So in your, in your first solution, it was an evolutionary approach. It was using Sonnet three point five. And you had, um, about four iterations, I think. And, and essentially you, you know, you were working on the ARC challenge and you were producing these programs through evolution. Maybe just for folks that don't know about the ARC challenge as well. Could you introduce that and, and get into your solution?

• 8:12Speaker 02

  Sure. Yeah. So the ARC challenge is kind of like an IQ test for machines. It's a set of input-output grids and the whole point is to be able to figure out how to transform input grids into output grids, given a common transformation rule. And so, uh, what's interesting is these are really easy for humans, right? The average human gets around seventy-five percent accuracy on ARC V one. And at the time, the best language model is GPT four, Sonnet three five, uh, was getting maybe five percent. And so, yeah, basically the idea is you have a few training examples and then you're, you're, uh, you're trying to extrapolate the transformation rule, uh, on the final test example. And, uh, so I approached this, I was actually inspired by Ryan Greenblatt,

• 8:53Speaker 02

  who had a solution earlier, which was to generate a ton of Python programs that would encapsulate the transformation. And Python programs are great because they're deterministic and you can pretty quickly check whether or not the Python program works or not, which is really cheap. So it's cheap to verify and you can be relatively sure if the Python program works on all the training examples that it'll work on the test example. So I started with his approach, but then I noticed that, you know, the language models actually struggled on first attempts, even if you ask the language model a thousand times to generate Python programs. They were always off by small amounts on easy tasks, which I thought, you know, presumably it's in their distribution. They should be able to solve this.

• 9:34Speaker 02

  So what I found is that actually by taking the top programs, the top performing programs and then running that in a revision loop. So asking Sonar three five, hey, here's what you got wrong. Here are the cells you got wrong. Here's your original Python program and prove it. That started to really work well. And then I thought, well, why not just increase the depth, right? Why not ask it 10 times, you know, to revise until I'm happy with the solution, passes some sort of accuracy threshold. So that's kind of how I was inspired by it. And, you know, I didn't think of it as evolutionary at first. I was just thinking about, you know, broadly what would work. And over time I kind of understood, you know, there was something a bit deeper going on here,

• 10:16Speaker 02

  which is that, you know, evolving solutions is a powerful technique generally. And I think it's actually going to play a role, you know, in future technologies. But yeah, that's generally guided by just intuition.

• 10:30Tim Scarfe?

  Yeah, I had Ryan Greenblatt on the show. I'm a huge fan of his. He's a very, very smart guy. And I asked him a similar question because he did this iteration, right, where you have a certain depth of iterations. And I guess one approach is that you have a like a shallow method, right? So you just try 200 different variations. And in your, in your blog post, you kind of said, there's a, there's a Goldilocks zone where you want to have a certain number of tries of, you know, different, you know, different variations of things. But you also want to be able to refine your solution because that allows you to do this kind of composition. And composition is very, very important. For things for problems that require iteration.

• 11:13Tim Scarfe?

  And indeed, the second version of the ARC challenge. And I think the tasks were selected so that they had at least a couple of iterations in them, which meant that they needed to have this depth. Can you talk about that trade off?

• 11:25Speaker 02

  First of all, the second, the ARC V two is in a sense fundamentally different than ARC V one. Because of what you're describing, they're compositional. There are many rules that you have to go through. And this is partially why I found that my solution on ARC V one did not perform well. Yeah, so there's a constant trade off between how deep you go, how many revisions you take, and then how how broad you start out. The problem with, you know, going deep and not going so broad is there are some edge solutions where you'll never you'll never get to, right? But then, of course, most solutions end up being somewhere within the bounds of your first broad attempt. So that's generally the trade off. And the trade off is different for ARC V one or ARC V two.

• 12:06Speaker 02

  Interestingly, I found that for ARC V two, it was more important to be broad. And I think this is surprising to a lot of people. And partially, this is because the models now think, and that's great. So the models actually do a lot of the deep revision for you in their thinking block. And this is a fundamental change from when ARC V one existed. And when I, you know, just started out in the field, I think I'm a bit embarrassed by a lot of the things that I wrote in my first post, because it was two weeks before O one was released. And, you know, everything about O one changed how I think about these things. Which is before you could kind of simulate emulate thinking to the quote you you described with the parrot, the stochastic parrot.

• 12:49Speaker 02

  I think before you had reinforcement learning over. Before you actually taught the machines, the language models to think with reinforcement learning, you were almost doing this stochastic guessing. That was not a very efficient revision loop, basically internal revision loop. And so you needed to artificially create that revision loop with code. But in V two, I was able to use a very powerful thinking model, which has a lot of the the deep revisions for it. So I found it was best to kind of increase entropy, let it explore the space itself. And then I'll add a revision loop on top of that. But the revision loop is less important in V two.

• 13:27Tim Scarfe?

  Yeah, so the first one was Sonnet 3.5, so that didn't have this thinking thing built into it. So in your prompt, you told it to think step by step. And I think you were inspired by Ryan Greenblatt's prompt, right? So you had a whole bunch of ways in there for representing the board state. And you said, I want you to think now and I want you to, you know, that this is an abstract reasoning challenge and I want you to think from first principles. And it would kind of go through that and then it would give you the answer. But you're saying that on the RL-trained models, it was significantly better at doing that.

• 13:57Speaker 02

  Right, exactly. You could think of RL-trained models as having inbuilt revision loops. They are trained to explore the space in a deep way, thinking generally for themselves in a way that, so you really don't, you know, thinking, you don't need to prompt thinking models to think step by step, they already do it.

• 14:19Tim Scarfe?

  Yeah, I mean, I wanted to challenge you on this a tiny bit, right? So you kind of said in your, I think it was in the second version of the blog post that you just released last week, that at the moment, the models can do domain-specific thinking, so they can do math thinking and they can do code thinking. And what we want to do is imbue like the core machinations of thinking into these models. And I'm a little bit skeptical. I feel that these models, because they're not Turing complete, because they're not symbolic, you know, similar to what Francois believes, that, I'm sure you read that LLM Biology paper as well, they were talking about these circuits that we can find in papers that do things like multiplication and addition.

• 14:59Tim Scarfe?

  And what we saw was that they are quite patterned, they're quite templated, they're not thinking in a very general sense. And my suspicion is it will always be that way because the models don't have semantics, they're non-symbolic and so on. Do you think we could ever make them truly think in a general way?

• 15:16Speaker 02

  Yeah, I think fundamentally, taking a step back, the fact that our brains can do it and our brains are generally running similar algorithms.

• 15:27Speaker 02

  To me, this means that we will eventually be able to inject general reasoning into the language models. I don't think there's a fundamental reason why neural networks can't behave like biological neural networks. So that's, I guess, the higher level point. And then, zooming in, right now, the models are as bad as they're ever going to be. There's generally more compute going into pre-training than there is reinforcement learning. And of the compute going to reinforcement learning, a subset is going into specific general reasoning. And so I think that over time, you're going to see the models get better and better at general reasoning. But I guess a question I would have for you is, do you think there's a fundamental difference between the way the brain works, where there's some sort

• 16:09Speaker 02

  of symbolic nature to the brain, and it's not possible to inject that type of nature into an artificial network?

• 16:16Tim Scarfe?

  Yes. Yeah, I mean, you mentioned Geoff Hawkins. I interviewed Geoff. He's absolutely amazing. And of course, his HTM algorithm is computationally stronger than a neural network. It's Turing complete. And our brains, even though they are finite, they run a Turing complete algorithm, which means our brains know how to expand their memory, right, our memory. We can go and write things on a whiteboard, and we can go and get another notebook. And that is a special type of algorithm, which is not reversible with stochastic gradient descent. So the rough argument is, yes, there is a difference there. And I also wanted to touch on this RL with verifiable rewards thing, which is that we do that at training

• 16:58Tim Scarfe?

  time. I'm very excited in the future about an active inference version of that, like an agentic version, where we're actually doing this kind of transductive active fine tuning in an agential way, right? So I take an action, I get some new information from the environment, and I update my weights. And that would be truly adaptive. That would be intelligent. But what we do now is, we do all of this stuff at training time, and the resulting frozen artifact is still an LLM. It still has just a bunch of patterns in there. And I think that while that can uplift reasoning in many ways, I don't think it has the intelligence. And according to Sholay, intelligence is simply the ability

• 17:38Tim Scarfe?

  to search through the space of Turing programs, right? And I don't think that's what's happening with these LLMs at the moment.

• 17:46Speaker 02

  I I think you're generally correct that it's not happening at the moment. But I still think like fundamentally, I don't think there is a like a fundamental blocker physically for why they won't be able to do it in the future. And it's possible that S G D, right? Like stochastic gradient descent is is an issue fundamentally. And I think we're going to overcome that. I guess what I would say is artificial neural networks, I think, have the structure capable of, yeah, basically being as smart in every way as a human brain. And I subscribe to Francois's definition of of general intelligence as well.

• 18:23Tim Scarfe?

  Yeah, I mean, I I think we mostly agree. I I think, I mean, you know, let's look at Alpha Zero or Mu Zero or something like that. They did this training loop where they were actually updating the, you know, like the the value network and the policy network. And then it was frozen and they did some kind of, you know, Monte Carlo tree search. So they were achieving adaptivity through exhaustive search during the actual games. And in an ideal world, we would have this adaptivity that's actually updating the weights. Now, I believe the only reason we can't do that at the moment is just computational tractability, right? We have these huge models. We couldn't possibly have a dynamically updating model for every single person that's using Chat G P T.

• 19:07Tim Scarfe?

  It it would it would just be ridiculously slow. But I think you and I agree that if that were possible, that would be an entirely different kind of form of intelligence.

• 19:16Speaker 02

  I don't think that's so intractable, actually. I think my guess is, and, you know, Nvidia just put in a hundred billion dollars into OpenAI, OpenAI, Sam Altman's plan is to produce a gigawatt of what of of compute a week, something like that. I actually don't think with, you know, ever efficient algorithms that that is like crazy far off. I mean, right now, you could buy GPU, you could have it running in your house and it could be running OSS one twenty B, right? And fine tuning is relatively trivial compared to, you know, the entire process for pre-training. I actually think that is a that is totally within the realm of possibilities in the next ten years.

• 20:00Speaker 02

  And I think that actually is is potentially where this goes.

• 20:04Tim Scarfe?

  Yeah, I mean, you know far more about this than I do, but I think the reason why fine tuning is so expensive is, you know, we have this continual learning problem. And when you fine tune a model on OpenAI, they're not just fine tuning it on the data you give them, they, you know, just to stop this catastrophic forgetting problem, they presumably have to sample in a bunch of the original training data and maintain the distribution and so on. And if they did this for everyone, it would be insane. But I am excited about it, just like you are, because I interviewed the architects and I think they got first place on the on the private version last year. And they were doing this transductive active fine tuning, right? So they and they actually said, by the way, that this is a curious oddity with transformers, that if you start with a, you know, almost

• 20:47Tim Scarfe?

  like a virgin eight billion transformer, it almost doesn't matter what it knew about before, you could just pretty much start training it from scratch on the arc challenges. So they did a whole bunch of augmentation and active fine tuning. And they built an intelligent artifact. I mean, intelligence is domain specific, as per Shulay, and they actually built the system, which was per task adapting and solving the tasks. And they were updating the weights and it was beautiful. So that was an existence proof of nothing else that this thing could work.

• 21:13Speaker 00

  Hey Spotify listeners, when you travel with American Airlines, you'll enjoy comfort and convenience at every altitude. Before takeoff, unwind in the Admiral's Club Lounge with curated self-serve offerings of hearty bites and refreshments. In flight, stay connected with high-speed Wi-Fi, sponsored by AT&T. Your journey is designed to feel easy from the moment you check in to the moment you arrive. No matter where you're going, American gets you there. Book now at aa.com.

• 21:40Speaker 01

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required compatibility and availability varies 18 plus.

• 22:06Speaker 02

  And that was on a Kaggle notebook.

• 22:08Tim Scarfe?

  Yes.

• 22:09Speaker 02

  You know, in 10 years, this is going to be what like the, you know, Apollo mission computer. I think that, I think what you're describing, I'm actually not even totally convinced that continual learning is fundamentally the blocker. But I think if it is the fundamental blocker, that's actually incredible because we will solve continual learning. Like, that's something that's physically possible. And I actually think like, it's not so far off. Now, the forgetful issue, that is a much more fundamental issue in my mind. And not just the fact that you need to, every time you fine tune, you have to have some sort of very elegant mixture of data that, you know, goes into this fine tuning process so that there's, there's no catastrophic forgetting. This is, I think, actually a fundamental problem.

• 22:52Speaker 02

  So the, and it's a fundamental problem that, that, you know, even OpenAI has not solved. Right? And I think Francois is a great example, and I think this is an important example. You know, if you have the perfect weights for a certain problem, and then you fine tune that model on more examples of that problem, the weights will start to drift, and you will actually drift away from the, from the correct solution. His answer to that is, well, we could make these systems composable, right? Like, we can freeze the correct solution, and then we can add on top of that. I think there's something to that. I think actually, it's possible that there's a research direction that where, you know, maybe we freeze experts, or maybe we freeze layers for a bunch of reasons that isn't possible right now, or, but people are trying to do that.

• 23:36Speaker 02

  But I, yeah, I think fundamentally, compute is not the issue. I think it's this catastrophic forgetfulness.

• 23:41Tim Scarfe?

  Yeah, so I'm inclined to agree. I've, I've long dreamed about there being a Docker for language models. Right? You know, in Docker, you can kind of freeze drive a state of, you know, like, let's say, Linux operating system with an application with the security updates. You have these kind of immutable layers. And the composability that we often talk about could actually happen at the architectural level, and we could do dynamic model merging between different layers and whatnot. That would be very, very exciting. So, you know, but also, just to come back to what you said before, I've never really heard this before, you're distinguishing like forgetting and learning. Right? When, when we were talking about, you know, catastrophic forgetting and continual learning. Can you just sketch out that distinction a bit more?

• 24:24Speaker 02

  So the way I think about it, you have a neural network and it has all these weights inside of it, right? Anytime you update those weights, you are pushing some weights out. And presumably, you are pushing some correct answers that you've previously trained and they are getting pushed out. And the benefit, I think fundamentally, of symbolic systems is that doesn't happen, right? Symbolic systems are deterministic. When you get the right answer, you can be sure you have the right answer, you stash it away into your library of correct solutions. This is the problem with continuous structures. But this is also actually now why I think it's important to draw from the brain, which is this similar thing happens actually with the brain. I believe the brain is much more composable than neural networks biologically, but I

• 25:07Speaker 02

  think there's no reason why we can't, we won't be able to figure this out, right? Like, again, it could be it's as easy as we end up freezing experts. Again, like the freezing of the layers. I think this is an underexplored area and I think it's actually, I think we're going to go through basically this RL S-curve. And then I think this is the next S-curve, is figuring out how to make language models composable, like figuring out how to, actually, even more fundamentally, like the ideal system would be, we have a set of data, our language model is bad at a certain thing, we can just give it this data and then all of a sudden, it keeps all of its knowledge and then also gets really good at this new thing.

• 25:48Speaker 02

  We are not there yet and that to me is like a fundamental missing part of general intelligence.

• 25:53Tim Scarfe?

  Yeah, completely agree. So it sounds like we have very, very similar intuitions and Sholay talks about this as well. Interestingly, in his Measure of Intelligence paper, it was actually about the measure of intelligence. He's never really spoken about the machinations of intelligence. He talks about it just casually. He says, you know, we need the spell key prize and we need to, so those are like the basis functions and we need to do library learning and library transfer and we do some kind of like, you know, symbolic compositional process, you know, to adapt to novelty. And he's kind of sketched out the mechanics of it, but he's never actually formally spoken about it. I assume that's what he's building at his company. But there was a famous guy called Jerry Foden. In nineteen eighty eight, he had this connectionism critique, he had this beautiful

• 26:37Tim Scarfe?

  paper, and he was basically saying that symbolic systems have systematicity and productivity. And systematicity is this compositional thing, it's that, you know, it's the ability to generalize between Mary loves John and Mary loves Jane, right? So you have semantics, you have these kind of like, you know, these symbolic relations, and they have certain computational properties, like you can do a variable binding and quantification over potentially infinite domains. Like, we intuitively understand that symbolic things have very interesting properties. And then what we're trying to do is, like, we know neural networks are really good, and we want to somehow graft this capability onto neural networks.

• 27:16Speaker 02

  Yes. Yeah, and I actually think neural networks are in some ways a superset of symbolic systems. I think you can generally, you should be able to encapsulate a symbolic system with a neural network. In the same way, I think that you can do the, I think you can do the same thing with the brain as well. I think there's, there's nothing fundamentally blocking, but of course, once you have, you know, you have this symbolic system in the neural network, it might catastrophically forget when you fine tune it. Like, I guess that's, that may be where we disagree, but I think everything you're describing is totally possible, but then when you're, when you're coming to train it again, there's no guarantee that it keeps the same structure.

• 28:00Tim Scarfe?

  I think it's possible, because a neural network is not too incomplete, so I, I think in principle it can't do many of these things, but you can build a controller, right? So you, you could just build a very simple kind of envelope, just as you did with your solution. So you, you had a bunch of code, and it was doing this, you know, this basically compositionality in code on top of the neural network substrate, and that gives you many of those things. You know, for example, we often talk about library learning and library transfer, and I'm not sure if you've seen Eric Pang's solution. I'm speaking to him in Hong Kong, actually, in a couple of weeks, but, but he, rather than the Dream Coder approach, where they do this explicit library learning, he was

• 28:42Tim Scarfe?

  doing it in a kind of implicit way using the L L Ms. And I think there's a whole spectrum between, you know, you don't have to do it explicitly. I think you can kind of use neural networks, and you can do some kind of implicit composition, and you can get many of these features.

• 28:58Speaker 02

  Also, I want to say, generally, when I speak about language models, I assume that they have basically a Python terminal that they can run.

• 29:06Tim Scarfe?

  Oh, okay.

• 29:07Speaker 02

  So, not, so I guess, two things. The first is, I think, if you have a large enough neural network, I think, generally, almost everything is, you could, you can represent a symbolic system, but of course, it's not Turing complete. But given a neural network, plus the ability to write programs, then, I think we're basically at the, we're at the human brain equivalent.

• 29:35Tim Scarfe?

  So, yes, so that is a hybrid system, and that certainly is significantly more powerful. I'm just regurgitating my co-host, Dr. Dargar, because this is his favorite point he always likes to make. But, but he says that, that's true, but stochastic gradient descent does not find the algorithms that allow the systems to behave as if they are Turing machines. God knows how it happened in our brains. There was some dint of evolution or something where, you know, we suddenly got the merge operator, or god knows what happened, and we've got this incredible Turing complete algorithm in our finite brain. And so we're getting into that trainability thing. So, yes, maybe there is an out, you know, there is a set of weights that we might find one day, and it can access like a Python tool,

• 30:19Tim Scarfe?

  and it can do all of those things. And, you know, its capability now, is it now effectively searching the space of Turing machine programs? I think it's not. Like, there's lots of problems there. Like, how would it know which ones halt and which ones don't? And how would it be able to efficiently search that space? It feels like there's a gap now, but I agree with you that it's significantly stronger than not being able to use the tools.

• 30:40Speaker 02

  Yeah, I think, I, but, but you think that the human brain is running a Turing system?

• 30:45Tim Scarfe?

  Yes, I think the algorithm that runs in our brain is a Turing machine algorithm. So, you know, like, a Turing machine has a codebook, which is a finite state automata, and then it has this, like, you know, read-write access to these two potentially infinite tapes. And, you know, the algorithm that you put in that Turing machine, that is very difficult to find.

• 31:08Speaker 02

  I don't disagree with that, but I don't, why, why wouldn't we be able to find that algorithm for neural networks? Right? Like, why wouldn't, you know, we, we train neural networks much bigger than the brain, we put a lot of compute towards them. You just don't think that finding the same algorithm is possible with, with SGD?

• 31:26Tim Scarfe?

  I think with SGD, because the fascinating thing is that, you know, if you look at all of the FSA algorithms, a tiny sliver of those algorithms are capable of controlling, you know, a Turing machine and expanding their memory and so on. So it's in the space. And I don't know if you saw that amazing paper by Kenneth Stanley, the Fractured Entangled Representations paper, and he had this beautiful diagram and he said that, you know, SGD finds the algorithms over here and neuroevolution algorithms find the ones over here. And it just so happens that the neuroevolution algorithms find ones that have these factored, you know, representations, which means they find representations that are about the world, that are grounded in the world, that carve the world up by the joints. And if only we could find those things. You know, when I spoke to Schmid Huber, he said the

• 32:10Tim Scarfe?

  same thing. He said, like, you know, it is actually possible to find the right weights in a neural network to make it, you know, effectively Turing complete with some caveats and so on. But when we do SGD, because there are all of these shortcuts, right, it's a bit like Goodhearting. It will always just find the wrong thing.

• 32:27Speaker 02

  I need to think about that a bit more.

• 32:31Tim Scarfe?

  Okay, so on the first one as well, you were generating Python programs explicitly. And because of all the things that we're just talking about, I'm a big fan of that, because I feel intuitively, and I think you did, that there's something special about Python programs. And then you did this iterative updating of those programs and you converged on the right one. You also had this amazing diagram in your first blog post where you kind of visualized the space of all the possible programs and you kind of showed what was happening in every iteration.

• 33:01Speaker 02

  In the first one, I used Python programs because Python programs are deterministic and it's really easy to verify whether or not, you know, it's correct. Did it run? And then did it run on the training examples and produce the correct outputs? So it's like a perfect program, right? Like it is a program. The problem is, it, you know... Python programs are brittle in that, you know, there are many things that are very difficult to describe with Python. Arc grids in V2 being one of them, right? So you have some grids that are very easily described by Python, but then almost the majority, overwhelming majority in Arc V2 are very hard to describe in Python. The correct Python formulation is, you know, lines and lines and lines.

• 33:45Speaker 02

  And really what you want is more expressive program. And so that's why I switched from Python to English, which is a much more expressive program. You can, you can describe every single Arc V2 task in 10 bullet points of plain English, most of them in five bullet points. And I think this actually gets to the heart of Arc, right? Everything is quite simple. It's not very hard. And I think this is also how we do it too, right? Like we, when we look at these Arc graphs, Arc grids, we're coming up with these bullet points in our head and we're, you know, checking them. Okay, this was right, this was right. And Python doesn't have these features. It's just not as expressive as natural language. And I think another way to put it would be, you know, you have this inductive, transductive trade-off, right?

• 34:27Speaker 02

  You could think of language models as, you know, being trained inductively and then they have an inductive bias and you almost want to let that inductive bias express itself fully in a way. And the way you do that is to give it the full power of how it was trained. And I think this is the same thing with humans too, right? If I told you to solve Arc with Python programs, you'd do a way worse job, even if you were an expert at Python. And so I think fundamentally it's more general and it leads to general and better solutions. I mean, the accuracy is much higher when you use natural language. Now, the problem is you actually have to then verify whether the instructions are correct. You can't run natural language on Arc grids. This was the fundamental problem with the solution.

• 35:12Speaker 02

  This is what made iteration challenging, especially because, you know, for each grid, for each training example, you have to run the natural language instructions and it takes a really long time, especially with this thinking model. So I originally started with a weak model. You know, it's the checker model, it's the checker agent. Let's just use GPT-5 mini, whatever. And it did terribly. So I ended up, it was actually more important that the checker was stronger than the actual instruction creator, which I think is interesting. But yeah, that just highlights, you know, the trade offs with using natural language. It's, you can express, you know, much more concisely programs that you want to run,

• 35:54Speaker 02

  but then they're not runable programs. You actually have to check them inductively. So that was the trade off, but it was worth it for Arc V two.

• 36:02Tim Scarfe?

  Yes, so fascinating. And just for the audience, we've been using transduction and induction to distinguish predicting the solution space versus predicting a program. I had this discussion with Clem and Bonnet, need not detain us now, but I think in traditional machine learning, transduction means that the test example is a function of your prediction. I had this discussion with the architects as well, that when you have this natural language description, natural language is more expressive, which simply means that there are more degrees of freedom. And this is the beauty of L L M's, that there's this huge kind of space that you're traversing around. And when you use natural language, you can just traverse to more places in that space more easily. So it seems like it would be a win.

• 36:44Tim Scarfe?

  And I'm really fascinated to find out whether that is just like a huge component of your solution. Because on Eric's solution, he's still predicting programs and still doing very well. So I'm not sure about that. And the other thing is, I wasn't entirely sure whether you are actually using a transductive method. So in your solution checker agent, is it directly going to the solution space? Or is it generating a program and testing it?

• 37:08Speaker 02

  In the checker, in the checker, it takes in the natural language and then it outputs a grid. That's all it does. It just outputs a grid.

• 37:16Tim Scarfe?

  Okay, cool. So you've moved to a transductive modality. Did you see any errors in that? So did it sometimes produce the wrong grid?

• 37:26Speaker 02

  All the time. Yes, all the time. And it's worth noting, actually, you know, the Python solution was, I obviously tried my V one solution on V two, right? And

• 37:42Speaker 02

  It wasn't so much worse, but part of what I wanted to do with V2 is show that as language models get more powerful and we get to use thinking models, we can start using more general solutions. And I just thought there was something elegant about using natural language. And then it also happened to be that there were problems that I could tell the Python functions were never going to get. And there are basically no programs that ArcV2, my solution for ArcV2, won't get. So when Grok 6 comes out, or GPT-7 comes out, you can use my V2 solution and it will win, it will beat Arc. That is not the case for my V1 solution.

• 38:17Tim Scarfe?

  The other important thing is you are now using Grok 4, which is very, very powerful. I assume you chose Grok 4 because it just happened to be the winner on the leaderboard for the base ArcV2. And how much uplift is coming from that? So for example, if you used Grok 4 on your solution last year, how much better would it be?

• 38:38Speaker 02

  So I actually don't think it would be so much better. For some reason, and this is what I talked about in my blog post, these language models are very spiky in certain things where they were trained heavily on, right? So I think what happened with Grok is there was a distribution of similar shaped tasks, right? Grid tasks, just reasoning in the type of general direction that allowed Grok to have a special capability in this area. And I actually like tested each model. So I tested Grok versus GPT, you know, I didn't just go by the leaderboard. And Grok definitely outperformed. The problem is, you know, for my V1 solution, you also have to generate code. And Sonnet 3.5 is really good at thinking about code and generating code. And I prefer Sonnet to Grok for code

• 39:21Speaker 02

  generation. So my guess actually would be that if you use my V1 solution, it's highly possible, you know, OPUS 4.1 would be the best. I haven't tested that. That would be very expensive on OPUS 4.1, but maybe it's worth testing. But I think that the general idea is that these networks are very spiky. When you get into specific domains, it actually very much matters which model you use. And the Arc is a great example of this, right? Like the leaderboard is super spiky in ways that other benchmarks are not.

• 39:50Tim Scarfe?

  I did an interesting interview at Europe's last year with the Google guys. And and they were talking about this adaptive temperature and language models for reasoning. Because you know, there's this constant trade off between reasoning, we want to be quite constrained. Right. So so we actually want to kind of like go go a particular pathway. We want to be constrained by our knowledge. And when we're being quite creative and flexible, we want to, we want to be able to go in in different places. And I'm really interested in creativity, for example, and and I think creativity is like you. It's very similar to reasoning, as Sholay talks about, you know, you're composing together these constraints. There's this phylogeny of of knowledge and you need to respect it as much as possible. Because if you don't respect it, you're not grounded anymore.

• 40:33Tim Scarfe?

  So it kind of feels to me that intuitively code is great. Because it means that I'm actually respecting the constraints and and the semantics are correct. And it's grounded in in the real world. Do you feel in any way that by using these natural language descriptions? That you're kind of creating something which might by dint of chance or search? Find the right solution, but isn't correct and verifiable?

• 40:56Speaker 02

  Yes.

• 40:57Tim Scarfe?

  Okay, tell me more.

• 41:00Speaker 02

  Yes, I for sure. I think generally when models think in natural language and they output in natural language, they they are higher entropy, right? I think the when you, the second you start prompting with code, they go into code mode. And this is, you know, there are a lot of papers that show right? Just by prompting it in a certain direction, it activates certain weights that are, you know, just naturally lower entropy. But that was part of the thing that I wanted. I actually wanted to introduce entropy because, you know, still, most are tasks for V two. The models don't get close, right? You know, my solution was the top and it's at thirty percent. So I actually wanted to inject as much entropy as possible, which is partially why my prompts are so broad.

• 41:45Speaker 02

  You know, I could definitely improve my accuracy on a few tasks by making the prompts more specific. But I wanted to just constantly berate it, more entropy, more entropy. So I actually found that to be a positive, not a negative.

• 41:58Tim Scarfe?

  Interesting, interesting. On the efficiency of the solution, so the O three model from OpenAI, that was about $200 per task. And that was, I think it was, did we ever find out? I think it was sampling, right? So they just sampled it a bunch of times, they had a basic verifier, is that correct?

• 42:15Speaker 02

  I don't think we ever figured that out.

• 42:18Tim Scarfe?

  So you think it could be so because when I interviewed Cholay, he was he was being quite kind of not cagey, but it it seemed like he was suggesting they were actually performing a search algorithm. And and I think the OpenAI guys said on Twitter, no, they were just doing very basic sampling. And I think they even published the code that they used. Yeah, I'm not quite sure.

• 42:41Speaker 02

  Yes, I also spoke to the OpenAI guys about this, and I I I'm not sure after all of this what they were doing. I think it's probable that they were doing sampling. I mean, it took a very, you know, hard to imagine they weren't doing. They were certainly doing sampling. I'm not sure what else they were doing, whether the model is fine tuned. My guess is my best guess is that they were sampling and it actually was not a fine tuned model.

• 43:03Tim Scarfe?

  Oh, very interesting. Yeah, I remember there was that big hoo-ha at the time that they even though it was scandalous that they were training on the training set. But anyway, that's one side. There's also like the thought occurs that if they did, if they did do something like what you are doing. So, you know, like this approach of iterative refinement with verification at every single step, would they have done even better?

• 43:26Speaker 02

  For sure. Yeah, for sure. I think OpenAI generally, they want to do the right thing, and they want their solutions to be very general and broad. And this is the sense I get. I think it's to their culture. And I spoke so I spoke to the OpenAI guys, they did include the training data in that O three model. But I think that's fair game, right? They also I don't think they fine tuned on it, right? So it's just part of the corpus that when it's pre-training, which to me is fair game, right? This is this is totally fine.

• 43:54Tim Scarfe?

  Okay, very cool. So so on ArcV one, their efficiency was $200 per task. What was your efficiency?

• 44:01Speaker 02

  Oh, on ArcV one, I maybe $10, something like that. I need to something like that.

• 44:07Tim Scarfe?

  Okay.

• 44:08Speaker 02

  In order of maybe, I need I need to check.

• 44:11Tim Scarfe?

  Yeah. So talk to me more about this. So and Eric Pang's solution, he he was slightly came in slightly worse than you, but I think he was a fair bit more efficient. I think his one, I'm going off my memory now, was it about $8 per task? Was yours about $30 per task on ArcV two?

• 44:27Speaker 02

  Oh, this is the V two. So my, yeah, my latest solution. Right. My latest solution was around $30 on V two and $8 on V one.

• 44:34Tim Scarfe?

  Oh, okay. And I think Eric's one was slightly more efficient. And and I and he was indicating that it was because he was doing the library learning and transfer. And even that, I was left kind of thinking, first of all, it's interesting that you got better results. And is that because there isn't much transfer? You know, where does the library transfer come into this? Because, you know, like, maybe like the broader question is, if you were to make your solution significantly more efficient, what would you do?

• 45:02Speaker 02

  I had a version that does do library transfer. Basically, I would save the traces from training and try and basically pull those in during test time. And I actually, just out of simplicity's sake, because I was getting such high scores with the simple solution, I wanted to just push the simple solution. And I might actually, it's, we'll see if someone's going to beat my score, I might bring that back in. It's for sure that will improve the score, and it's useful. There is a lot of transfer efficiency. I just found what I was doing very elegant. And so I actually liked like to keep it. But, you know, no third-party dependencies or anything like that. But that for sure helps

• 45:44Speaker 02

  accuracy. I think the fundamental reason why I got higher, and also, I could match his efficiency and I would still get higher, because I was using natural language. Natural language is a much more efficient area to play in.

• 45:58Tim Scarfe?

  Yeah.

• 45:59Speaker 02

  That's at least what I found.

• 46:00Tim Scarfe?

  Yeah. I just wonder, how close do you think are we getting to the kind of Pareto optimal of this approach? I mean, just to give you a few examples, we interviewed the Alpha Evolve team. That was fascinating. And maybe you can contrast with those guys. Sakana AI yesterday, Robert Lange was the first author. They've released this, I think it's called Shrinker. And that was a kind of similar kind of evolutionary, you know, program thing. And they had some cool features in there, like, you know, using Bandits and using UCB. And I guess, like, are we getting to the point where we're going to really figure out what is the most optimal way to do this? By the way, they were also switching between different foundation models.

• 46:40Speaker 02

  I think improvements will be log, right? Will be will be logarithmic. So I wouldn't expect using these, basically using the language models we have today, I would not expect anyone to break, let's say, 40%. But you could probably make my solution twice as efficient, I would say. You wouldn't get such so much, yeah, you wouldn't get, you wouldn't get more than a few percentage points more accurate is my guess, but you could make it a lot more efficient. There's a ton of efficiency gains to be made.

• 47:09Tim Scarfe?

  We've been dancing around this a little bit, that, you know, Chollet's measure of intelligence was all about resisting memorization. And there is this question now, you know, which is, to what extent are we actually building systems that we might call intelligent? And he says that intelligence is simply like the efficiency of knowledge acquisition. And I'm really on board with that. And I think it's fair to say at the moment that, let's say, like, you know, your solution and Greenback's solution, it's quite ephemeral and stateless, which is to say that when you have a new task come along, you kind of start again from scratch, which means it's not really like adapting and acquiring

• 47:50Tim Scarfe?

  new knowledge and transferring that knowledge. So maybe you would agree that in the spirit of Chollet's measure of intelligence, at the moment, it's more of a kind of searching approach. But what do you think we would need to do to kind of, you know, make it more adaptable?

• 48:06Speaker 02

  Right. So I think test time fine tuning would be the way to like fundamentally make it adaptable. But I also think, you know, Chollet hits it a core problem with language models, which is they, their reasoning is domain specific, right? I kind of, in my blog post, I described that when you train a language model to reason about math, for some reason, most of the reasoning circuits it just gained live in the math weights. And then you try and train it on science, and it had some generalization, but not as much as you would want. And I think not nearly as much as what humans get. Humans have like this, this, you know, generalization engine that is our reasoning capability.

• 48:47Speaker 02

  And this is, this is the fundamental hole in language models today. And I think

• 48:54Speaker 02

  In fact, actually, I would say, I actually, I generally agree with Francois, and he says, you know, language, you can always teach a language model a skill, right? But it's the meta-skill. It's the skill to create the skills that is AGI. And to me, that's reasoning. Like, reasoning is that meta-skill. And so, to put it another way, I think if you fundamentally learn the skill of reasoning, you should be able to then apply that skill to learn all the other skills. That is the meta-skill. And we need to figure out, and so that is the fundamental problem, and you need to do whatever you can, you know, kick whatever weights out you need to align the model to reason. And then, from there, you have a foundation from which you can actually build general intelligence.

• 49:38Speaker 02

  So I guess, I don't know if that was a, that was maybe a higher-level answer to your question, but I think, you know, how we, and what I'm focused on is really just fitting all of reasoning into these models, and I don't really care what else is left, I just want all of reasoning in.

• 49:51Tim Scarfe?

  Yes, I pretty much agree. And, I mean, you probably know that I'm Chollet's biggest fan, so I've obviously, you know, been a huge fan of his for years. But, by the way, he's just released Revision 3 of his Deep Learning with Python book, and I recommend folks to read Chapter 19. You can actually read it online for free. And he sketches out this entire vision, you know, it's so exciting, and I think that just to see it so beautifully articulated. Because there is a bit of an elephant in the room, in the scene at the moment. I think so many people just don't have such a crisp understanding. But the only departure that I make with Chollet and yourself, Jeremy, is that I think it's, you know, Chollet really focuses on behavioral tests of intelligence, you know, like, so

• 50:33Tim Scarfe?

  it's reasoning, if it can pass the test and it can actually, like, get the right answer. And I think we need to go, this is where I was kind of talking about the systemat化 and the symbolic AI. I think how you got there is important, right? So I think it's possible to get the right answer for the wrong reasons. And I think that if we have a system which has semantics, so we actually know what these symbols mean, and we've composed them together in a principled way, not only to get the right answer for the right reasons, but to be evolvable, so to have, like, an efficient epistemic base that allows us to go on to acquire new knowledge in the future. And that, to me, points to this need to have a mechanistic, like, you know, like, how are we acquiring this knowledge view? Would you agree with that?

• 51:18Speaker 02

  Yes, and I think about it a bit differently, so let me know if what I say is aligned with what you think.

• 51:24Tim Scarfe?

  Okay.

• 51:24Speaker 02

  So, to me, pre-training is kind of the opposite of what you described. I view, there's two types of knowledge. There's knowledge that is memorized, like the capital of New York, or the Spanish language. And then there's knowledge that is deduced. So that's physics, special relativity, general relativity, right? From axioms, you can deduce these things, and it's a causal tree. And then there's another type of knowledge, what is the capital of North Dakota? That is a knowledge network. It's not deductive. It's not a tree. And I think pre-training treats all knowledge as a knowledge web. It's embeddings that are connected, but there's no guarantee that you have the correct causal relationship between things. And this is where the memorization comes in.

• 52:07Speaker 02

  And I think this is actually where compression fits into intelligence. So my view is actually, intelligence is compression in that you should be able to deduce, you should be able to build a knowledge tree based on almost, you can build a knowledge tree based on almost nothing, right? You can deduce so much of math, you can deduce special relativity from the very, you know, the very roots of physics. And Einstein was extremely intelligent because the hints that he needed to come up with special relativity are zero, right? He could start from almost nothing and build up this deductive tree. And I think it's almost like reinforcement learning and reasoning is the process of pruning our knowledge network and replacing it with this tree.

• 52:48Speaker 02

  And until we have weights that represent the actual deductive nature of knowledge, we won't actually get generalization. I don't know if this fits in, but this is kind of how I think about reinforcement learning, which is replacing a knowledge web with a knowledge tree.

• 53:03Tim Scarfe?

  Yes, yes, this is brilliant. We're getting to the center of the bullseye here, absolutely. And I remember I read in the first version of your blog post that you were talking about, we need to do this kind of deduction where we synthesize hypotheses and then we test them and we do this kind of generate test loop. And that is what creativity is, it's what reasoning is. Because, you know, when I first read Chollet's paper, you know, years ago, I was kind of like, I didn't understand whether he was talking about acquisition. Position or synthesis. And I now understand he's talking about synthesis. So reasoning is like you, just like Lego. You you build you build this kind of tree, this epistemic tree. And actually, this is what we do. So there's a difference between

• 53:42Tim Scarfe?

  knowing and understanding. So knowing is is kind of like at the high level and understanding is is actually like there's this whole, you know, just imagine this big block of this big Lego structure and you're tracing down the structure with all the building blocks of how you got there. And I actually think even when you teach kids at university, what you're really doing, obviously, like you teach you teach them facts, but then they kind of synthesize their understanding over time. So they're doing this composition and they're kind of getting there the way that they get there. And and we need to build systems to do this. And there's the perennial problem that you were saying that in deep learning, what we do is we kind of start with this big pattern network and we kind of sparsify it. And I think reasoning should be more about synthesizing from building blocks. And

• 54:25Tim Scarfe?

  I I think that when you synthesize, you can actually do types of reasoning which are not in the training data. Right. You can build things that simply are not there. You can just think about things and and and figure things out. So do you think of that as a gap?

• 54:41Speaker 02

  Yes, I think that's exactly right. And then the question is, can you build the system with language models or not?

• 54:47Tim Scarfe?

  Yeah.

• 54:47Speaker 02

  And I I think you can build that with language models. And I think the fact that we're slowly climbing our ability to synthesize new information is a testament to kind of what I'm saying, which is when reinforcement learning and reinforcement learning, you know, with verifiable rewards is is fundamentally ensuring whatever circuits led to the right answer, they must be consistent with the deductive tree. So it's basically like, can you replace all of your pre-trained weights with weights that are coherent from the environment? The problem is all, you know, there are so many weights, there are so many weights for pre-training that it's very difficult. So I actually think a part of, you know, one of my, I guess, hot takes is that

• 55:29Speaker 02

  pre-training in many ways slows down reasoning. It makes it harder to reason because I I think I think the analogy I draw is, you know, you have consultants that know The names for things, but couldn't deduce the thing. And then you have Feynman, right? Who can deduce anything? And reinforcement learning is turning your consultant into Feynman. And this is, yeah, this is what I'm most interested in. And I think it's interesting because you get to play at both sides. You get to play with the pre-training of, okay, well, maybe we shouldn't include these things and let the model figure it out in reinforcement learning. Well, there's no guarantee if we pre-train it like this, you know, it's going to have the proper deductive circuitry. So maybe this is best, you know, left for post-training.

• 56:10Speaker 02

  And I think this is not, this is, this is a hot, this is not what people currently think. I think people think let's jam as much information as we can in pre-training and then we'll reinforcement learn when need, when we need. But I think this, this could be incorrect.

• 56:23Tim Scarfe?

  Yeah, and I pretty much agree with you, just with the caveats we discussed previously, that if we could do that on the basis of representations that are actually like grounded in the world, rather than things that just, you know, happen to give you the right answer for the wrong reasons. That's, that's absolutely true. Just a little bit of a curve ball. So I think in the first version of the article, you said you were inspired by Jan LeCun's JEP, these joint embedding prediction architectures. And he's also a big advocate of energy-based models, which are really cool. I don't know if you've seen the recent couple of papers that are applying it to transformers, you know, where essentially it's kind of a step towards probabilistic models where you actually have this uncertainty quantification. You can do counterfactuals and you actually have to solve an optimization problem at inference time.

• 57:08Tim Scarfe?

  So you can do adaptive computation and it's all very exciting. But I still have some reservations, but do you think like architectures like that are exciting?

• 57:17Speaker 02

  I think they are exciting. I'm a bit less excited about them, not because of their merits, but because I think I was underrating transformers when I wrote that. So this was really before I started. Do it being an actual researcher and, you know, building transformers, actually coding with them. Since then, I've had a new appreciation for language models. I think where I was coming from it was, it seems like language models are overfitting to the next token. Right? Like, and JEPHA is so interesting because, oh, all of a sudden you have them predicting concepts. And fundamentally, it's, we care about concepts. The words don't really matter. It's the concepts that matter. But I think that language models do operate on the conceptual level in the hidden layers.

• 58:02Speaker 02

  And so that was something that I slowly came to the realization of. So I think there's a lot of potential in JEPHA frameworks. I think they're really cool. I hope people keep pulling on them. But I think most of the benefits I thought that came from JEPHA exist in language models. I just didn't see it at the time.

• 58:18Tim Scarfe?

  You know, a lot of people get an earworm where they just get obsessed about an idea. And you're kind of thinking about it all the time. Like, what is that thing for you?

• 58:25Speaker 02

  If we do have these language models that have their weights that are aligned in this, I guess, tree of deduction.

• 58:35Speaker 02

  It seems like we're missing still one more thing, which is creativity, which we touched on, right? So you can have the correct deductive tree. But then how do you search through all of the possible premises you can add to this tree? How do you find the right ones? And I think there are a few experiments that I'm looking forward to doing. One of them is a bleating at will pre-training data. And then basically, unfortunately, building an environment to have the model regenerate that information. So for example, let's say you have the ability to oblate special relativity and all of the physics that came from special relativity from your pre-training data. That is a goldmine of an environment, right? Because now you can prompt the model, you can do everything. You can really try and get it to deduce special relativity.

• 59:20Speaker 02

  And my hunch is that part of the reason why models are not yet great at coming up with novel solutions and information is because they don't have that.

• 59:30Speaker 02

  The circuitry of invention. And I think that is actually a circuit that needs to be developed. And we don't have the environments to develop that circuit yet. And I actually just saw today, I think, OpenAI released this math. I skimmed it. This math paper, right, where they're coming up with, they almost came up with novel conjectures, something like that. And I think that's exactly what was in my head, which is I, I want to be able to build environments where the model's never seen something. And basically, it tries to deduce these new things that, that are, you know, outside of the distribution. And over time, it learns and practices and builds this invention circuit. So I think it's like, it's two things. It's reinforcement learning to make sure that the knowledge tree is consistent.

• 1:00:15Speaker 02

  And then making sure that the circuitry, to be able to pull from, you know, its entire corpus of, of understanding of the world and, and bring, bring that in to kind of, to fuel the innovation engine.

• 1:00:28Tim Scarfe?

  Yeah, I think we're so close. I think the only like slight disagreement is whether there could be such a thing as an invention circuit. It seems to me, just like that L L N biology paper and, and whatnot, that it would be very kind of patented in the weights. It would be templated. It would be kind of, you know, specific to certain domains. And I feel that we can do it, but we would need to build a controller on the top. I also feel, by the way, that creativity is very domain specific. And what I mean by that is just with this Kenneth Stanley view, that there's this big phylogeny of knowledge. And I've noted that when I hire creative professionals, they, like an editor, for example, they can't edit my show. And even if they're really good at editing other people's shows, and it's because they

• 1:01:13Tim Scarfe?

  simply don't know anything about machine learning. So, you know, like I've, I've just discovered the crazy degree to which creativity is domain specific. And I just wonder whether there is an algorithm for creativity. Cause I've got this epistemic lens of creativity, which is that, you know, it's simply about, okay, well, I can access all of the ancestors in my tree and I can do some composition. It's like building Lego and so on. And if I want to use something from the tree over here, this other branch, you know, maybe it's compatible, maybe I can bring it in, or maybe I just need to start a new branch, or maybe I need to jump over to the other branch. And I'm not sure whether I'm applying the same algorithm when I'm doing that.

• 1:01:48Speaker 02

  I think you, I think you actually are. And that it's.

• 1:01:53Speaker 02

  It's not sufficient to just be creative in this case. You need to be creative and you need to be knowledgeable. Otherwise, you're creative, right? You can't build the tree. I think your editors can't build the tree because they don't have the deductive footing. That would be my best guess, but that's an interesting perspective. I want to think about that. I mean, fundamentally, creativity is knowing which axioms to include in the next branch of the tree, right? So you're at level five of the tree. How do you get to level six? It's knowing which assumptions to pile in to get to level six, right?

• 1:02:26Tim Scarfe?

  It's so beautiful. We're writing an article about creativity at the moment, and I believe that in order to be creative, the depth of understanding of the tree is very important. And you know, just per Sholay, intelligence is the efficiency of how much of the history you can acquire. So like a university professor understands the tree very deeply, and that actually makes them unintelligible to a normal person. Like when Stephen Wolfram talks about the Rulie ad all the time, people have got no idea what he's talking about. He's actually being very expressive. He's talking about things at a level of abstraction which, you know, can refer to anything, but it's beyond most people's cognitive horizon. But there is something to be said for that. When you can respect the history deep down into

• 1:03:09Tim Scarfe?

  the epistemic tree, the creative stepping stones you take, because they respect the history, they actually have more evolvability, right? Because you're still grounded to the real world. You're not becoming incoherent. So there's something there about really knowing things deeply is important.

• 1:03:24Speaker 02

  Yes, yes. For the record, I would define understanding as being able to... I mean, I think understanding is a spectrum. I think on one end, it's memorization, which is zero understanding, and then on the other end, it's ability to deduce and also to deduce correctly. I agree with what you said. It's not enough to just have the right proof. It's you actually have to have understood the tree. And that is like, this is understanding, and then intelligence is just how many things you understand. And so it's really just how...

• 1:03:57Speaker 02

  How wide and and high is your garden of trees?

• 1:04:01Tim Scarfe?

  Yeah, we're pretty similar. I would say intelligence is the efficiency that you can acquire the tree. And reasoning is building the tree. Discursive reasoning is executing the tree. And and and understanding is simply just possession of the tree.

• 1:04:14Speaker 02

  So intelligence to you is the speed at which you can build the tree, not how many trees you have.

• 1:04:19Tim Scarfe?

  Yeah.

• 1:04:20Speaker 02

  Or not how large your tree is.

• 1:04:21Tim Scarfe?

  Yeah, yeah, yeah. I think understanding is how much of the tree do I have?

• 1:04:26Speaker 02

  I think that's correct. Yes, yes, because you like you could have a very intelligent child who doesn't know a lot about the world, but has the ability. It's like potential to build a tree.

• 1:04:37Tim Scarfe?

  Yes.

• 1:04:38Speaker 02

  It's a yes, I think that's correct.

• 1:04:40Tim Scarfe?

  There's a spectrum of understanding. So language models famously, they don't understand the tree very deeply. So they only understand the tree a few levels down. And so when language models are doing autonomous generation, the reason why we have to do so many different generations and select the best one is because obviously like it's not grounded. It doesn't understand the tree very deeply.

• 1:05:01Speaker 02

  Exactly.

• 1:05:01Tim Scarfe?

  We can overcome that, but because we understand the tree deeply. So we can put a prompt in there that constrains their generation. And now we can make them act as if they understood the tree when they didn't. But, you know, we just need to build the models that do understand the tree deeply. And then we can just trust them to generate autonomously.

• 1:05:16Speaker 00

  Hey, Spotify listeners. When you travel with American Airlines, you'll enjoy comfort and convenience at every altitude. Before takeoff, unwind in the Admiral's Club Lounge with curated self-serve offerings of hearty bites and refreshments. In flight, stay connected with high-speed Wi-Fi, sponsored by AT&T. Your journey is designed to feel easy from the moment you check in to the moment you arrive. No matter where you're going, American gets you there. Book now at aa.com.

• 1:05:43Speaker 01

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required. Compatibility and availability varies. 18 plus.

• 1:06:09Speaker 02

  Yes, that is, that is a very good way of, of, of saying what, what I think. And that is what I'm focused on. That's a really good way of putting it right, which is forcing the language models to develop these deep trees from the ground up. You can only develop it from the ground up, I think. And so you need to be, we need to come up with new techniques, new environments to, to grow the trees instead of pre-training, which is pre-filling random, you know, it's not random, but it's a web, it's not a, it's not a tree.

• 1:06:39Tim Scarfe?

  Yes, there's also the vexed issue of what happens at the bottom of the tree. So, Chollet argues that this spell key core knowledge, you know, these are knowledge primitives that are so fundamental that inside the deductive closure of those primitives, we can talk about anything. And, you know, you can still get lost in different parts of the tree, I suppose, and there's some issues of intelligibility going between them. But if you understood the tree deeply enough, you can go anywhere. But maybe there are different trees, maybe they're, you know, like in physics, there are different levels of, you know, levels of description of understanding the universe. So do you, do you think it's like one big tree or lots of trees?

• 1:07:14Speaker 02

  I think the, the, you have, it's kind of gated in the laws of whatever you're doing. And I know, you know, I, I looked into this a few times. I know there are some axioms that you need to take for granted and let's say deducing some forms of math, right? And they would be their own tree because you can't get to one part without the other. You know, if you take this for granted, then you can't, you can't deduce this, but you can deduce this. And so I'd say basically they have to, you know, they have to be logical chain. And so of course there might be, there might be multiple chains and maybe everything is grounded in logic, right? Like logic is the fundamental, I guess that is true, actually. Logic must be the fundamental block of a tree, right? Like the, everything comes from logic because if we didn't have logic, we couldn't have trees.

• 1:07:57Speaker 02

  So I guess that, yeah, I guess it would be all one tree and it's a logic tree. And I guess there are people that subscribe to logic trees and then there are people that are illogical and they, they don't have their trees at all.

• 1:08:06Tim Scarfe?

  Jeremy, it's been an absolute honour to have you on the show. Just before we go, are you hiring or anything you want to say to the audience?

• 1:08:13Speaker 02

  For sure, yeah. So at Reflection, we're building open intelligence models. So we're hiring across the stack, pre-training, post-training, large language models. We have a lot of GPU. So if you're interested in pre-training, post-training, we're in SF, we're in New York, and we're in London. So definitely, apply on our site, or you can just hit me up on Twitter.

• 1:08:35Tim Scarfe?

  Amazing. Jeremy, I've really enjoyed this. Thank you so much.

• 1:08:38Speaker 02

  Awesome, thank you.

• 1:08:39Tim Scarfe?

  Just keep doing what you're doing, man. I just, I really, really think that you're onto something here. I mean, obviously, like with the minor, you know, discussion about, you know, whether, how we're actually going to do this. But I think the direction is quite clearly set. I know, it's such a vexed issue, though. I mean, I'm interviewing a bunch of cognitive scientists in Japan next week, and you can really go down the rabbit hole on this. So for example, I'm a big fan of, you know, like, externalism, you know, like inactive cognition. And there are also all these philosophical views where, you know, like, consciousness is basically a property of certain types of physical material. So, because you

• 1:09:22Tim Scarfe?

  know what we're talking about here with understanding, like, if you actually abstract it into physics, we're talking about certain types of causal graph. And you can argue that certain types of cognition actually require certain types of physical instantiation, you know, where in the middle of that graph, you have material which is capable of producing consciousness. So some components of understanding are phenomenal, they're conscious, right? And like, when you start taking it to this philosophical level, there's almost like no, there's no end to it, you know, because you'll always have people that argue against functionalism, that say that cognition must be physically instantiated in a certain way. And I'm not sure where I am on that, because even the cognitive scientists are saying, okay, well, guys, we have to admit that, you know, even though we can make all of these

• 1:10:06Tim Scarfe?

  arguments, I mean, fuck me, these LLMs are, they're doing so well. But, um, yeah, I think it'd be a fascinating discussion, because a lot of neuroscientists are internalists. They basically, like Jeff Hawkins, they think that all of the shit happens in the brain. And, you know, we have these sensory motor circuits, and we've got this master algorithm in our neocortex, and that does all of the things. And I think there's something to be said for that. But, you know, I think it does ignore quite a lot of the other field. Yeah, well, it all comes down to if we could build machines that actually have a deep grounded understanding of the world. And let's assume that that doesn't actually have to be physically grounded, right? If if the representation is like, you know, grounded in the sense that it's a faithful description of what is happening out there.

• 1:10:49Tim Scarfe?

  And we can do this creative reasoning on that understanding, then what's to stop us from inventing new things? I mean, that's basically the thesis, right? Yeah, beautiful stuff. Jeremy, thank you so much, man. I really, really appreciate this.

• 1:11:03Speaker 04

  Stonefire naan is made with real buttermilk and ghee. It's the perfect flatbread for any occasion.

• 1:11:08Speaker 05

  I put out Stonefire naan with dips at every dinner party.

• 1:11:11Speaker 04

  I make naan grilled cheese when my daughter has friends over.

• 1:11:14Speaker 06

  I use Stonefire naan to reward the humans who do my bidding.

• 1:11:18Speaker 05

  Right, it's perfect for camping trips. Dance for it, feeble mortal. Sorry, is he talking to me or? Dance. Ha, ha, ha, ha, ha.

• 1:11:28Speaker 04

  From our tradition to yours, Stonefire. Together, we naan.

• 1:11:33Speaker 07

  Now at Kroger, your points add up to more ways to save every trip. Use them your way to take dollars off groceries at checkout or save at the pump. You can even save up to thirty five dollars on your next fill-up. Same points, more choices. And with low prices on fresh, quality food, plus digital and personalized offers tailored to how you shop, your dollar goes further. Redeem your points online or in the app. Kroger, fresh for everyone. Restrictions apply. See cider store for details.