Francois Chollet — Why the biggest AI models can't solve simple puzzles

Here is my conversation with Francois Chollet and Mike Knoop on the $1 million ARC-AGI Prize they're launching today https://arcprize.org/. I did a bunch of socratic grilling throughout, but Francois’s arguments about why LLMs won’t lead to AGI are very interesting and worth thinking through. It was really fun discussing/debating the cruxes. Enjoy! Watch on YouTube https://youtu.be/UakqL6Pj9xo. Listen on Apple Podcasts https://podcasts.apple.com/us/podcast/francois-chollet-mike-knoop-llms-wont-lead-to-agi-%241/id1516093381?i=1000658672649, Spotify https://open.spotify.com/episode/7bmeJQOvXGy4LYl6YoiYYP?si=obUSUEwjSA6tkB8EBcb18w, or any other podcast platform. Read the full transcript here https://www.dwarkeshpatel.com/p/francois-chollet. Timestamps (00:00:00) – The ARC benchmark (00:11:10) – Why LLMs struggle with ARC (00:19:00) – Skill vs intelligence (00:27:55) - Do we need “AGI” to automate most jobs? (00:48:28) – Future of AI progress: deep learning + program synthesis (01:00:40) – How Mike Knoop got nerd-sniped by ARC (01:08:37) – Million $ ARC Prize (01:10:33) – Resisting benchmark saturation (01:18:08) – ARC scores on frontier vs open source models (01:26:19) – Possible solutions to ARC Prize This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com https://www.dwarkesh.com?utm_medium=podcast&utm_campaign=CTA_1

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Dwarkesh Patel?

  Okay, today I have the pleasure to speak with Francois Chollet, who is an AI researcher at Google and creator of Keras. And he's launching a prize in collaboration with Mike Canuff, the co-founder of Zephyr, who we'll also be talking to in a second, a million-dollar prize to solve the ARC benchmark that he created. So first question, what is the ARC benchmark? And why do you even need this prize? Why won't the biggest LLM we have in a year be able to just saturate it?

• 0:27Speaker 01

  Sure, so ARC is intended as a kind of IQ test for machine intelligence. And what makes it different from most LLM benchmarks out there is that it's designed to be resistant to memorization. So if you look at the way LLMs work, they are basically this big interpretative memory. And the way you scale up their capabilities is by trying to cram as much knowledge and patterns as possible into them. And by contrast, ARC does not require a lot of knowledge at all. It's designed to only require what's known as core knowledge, which is basic knowledge about things like elementary physics, objectness, counting, that sort of thing.

• 1:10Speaker 01

  The sort of knowledge that any four-year-old or five-year-old possesses, right? But what's interesting is that each puzzle in ARC is novel, is something that you've probably not encountered before, even if you've memorized the entire internet. And that's what makes it, so that's what makes ARC challenging for LLMs. And so far, LLMs have not been doing very well on it. In fact, the approaches that are working well are more towards discrete program search, program synthesis.

• 1:41Dwarkesh Patel?

  Hmm, so first of all, I'll make a comment that I'm glad that as a skeptic of LLM, you have put out yourself a benchmark that... Is it accurate to say that suppose that the biggest model we have in a year is able to get 80% on this? Then your view would be we are on track to AGI with LLMs? How would you think about that?

• 2:01Speaker 01

  Right.

• 2:03Speaker 01

  I'm pretty skeptical that we're going to see LLM do 80% in a year. That said, if we do see it, you would also have to look at how this was achieved. If you just trained the model and millions or billions of puzzles similar to ARC, so that you're relying on the ability to have some overlap between the tasks that you train on and the task that you're going to see at test time, then you're still using memorization, right? And maybe it can work. Hopefully ARC is going to be good enough that it's going to be resistant to this sort of attempt and brute forcing. But you know, you never know, maybe it could happen. I'm not saying it's not going to happen. ARC is not a perfect benchmark. Maybe it has flaws, maybe it could be hacked in that way.

• 2:48Dwarkesh Patel?

  So I guess I'm curious about what would GPT-5 have to do that you're very confident that it's on the path to AGI?

• 2:58Speaker 01

  What would make me change my mind about LLM is basically if I start seeing a critical mass of cases where you show the model with something it has not seen before, a task that's actually novel from the perspective of its training data, something that's not in training data, and if it can actually adapt on the fly.

• 3:19Speaker 01

  And this is true for LLMs, but really this would catch my attention with any AI technique out there. If I can see the ability to adapt to novelty on the fly, to pick up new skills efficiently, then I would be extremely interested. I would think this is on the path to AGI.

• 3:37Dwarkesh Patel?

  So the advantage they have is that they do get to see everything. Maybe I'll take issue with how much they are relying on that. But let's suppose that they are relying, obviously they're relying on that more than humans do. To the extent that they do have so much in distribution, to the extent that we have trouble distinguishing whether an example is in distribution or not. Well, if they have everything in distribution, then they can do everything that we can do. Maybe it's not in distribution for us. Why is it so crucial that it has to be out of distribution for them? You know, why can't we just leverage the fact that they do get to see everything?

• 4:10Speaker 01

  Right, you're asking basically what's the difference between actual intelligence, which is the ability to adapt to things you've not been prepared for, and pure memorization, like reciting what you've seen before. And it's not just some semantic difference. The big difference is that you can never pre-train on everything that you might see at test time, right? Because the world changes all the time. So it's not just the fact that the space of possible tasks is infinite. And even if you're trained on millions of them, you've only seen zero person of the total space. It's also the fact that the world is changing every day, right?

• 4:50Speaker 01

  This is why we, the human species, has developed intelligence in the first place. If there was such a thing as a distribution for the world, for the universe, for our lives, then we would not need intelligence at all. In fact, many creatures, many insects, for instance, do not have intelligence. Instead, what they have is they have in their connectome, in their genes, hardcoded programs, behavioral programs, that map some stimuli to appropriate response. And they can actually navigate their lives, their environments, in a way that's very evolutionary fit, that way, without needing to learn anything. And well, if our environment was static enough, predictable enough,

• 5:35Speaker 01

  what would have happened is that evolution would have found the perfect behavioral program, a hardcoded, static behavioral program, would have written it into our genes. We would have a hardcoded brain connectome, and that's what we would be running on. But no, that's not what happened. Instead, we have general intelligence. So we are born with extremely little knowledge about the world, but we are born with the ability to learn very efficiently and to adapt in the face of things that we've never seen before. And that's what makes us unique. And that's what is really, really challenging to recreate in machines.

• 6:10Dwarkesh Patel?

  I want to rabbit hole on that a little bit, but before I do that... Maybe I'm going to overlay some examples of what an arc-like challenge looks like for the YouTube audience. But maybe for people listening on audio, can you just describe what would a sample arc challenge look like?

• 6:23Speaker 01

  Sure. So one arc puzzle, it looks kind of like an IQ test puzzle. You've got a number of demonstration input-output pairs. So one pair is made of two grids. So one grid shows you an input, and the second grid shows you what you should produce as a response to that input. And you get a couple pairs like this to demonstrate the nature of the task, to demonstrate what you're supposed to do with your inputs. And then you get a new test input. And your job is to produce the corresponding test outputs. You look at the demonstration pairs, and from that, you figure out what you're supposed to do, and you show that you've understood it on this new test pair.

• 7:08Speaker 01

  And importantly, in order to the sort of like knowledge basis that you need in order to approach these challenges, is you just need core knowledge. And core knowledge is, it's basically the knowledge of what makes an object, basic counting, basic geometry, topology, symmetries, that sort of thing. So extremely basic knowledge, LLMs for sure possess such knowledge. Any child possesses such knowledge. And what's really interesting is that each puzzle is new. So it's not something that you're going to find elsewhere on the internet, for instance. And that means that whether it's as a human or as a machine, every puzzle, you have to approach it from scratch.

• 7:53Speaker 01

  You have to actually reason your way through it. You cannot just fetch the response from your memory.

• 7:59Dwarkesh Patel?

  So the core knowledge. One contention here is, we are only now getting multimodal models who, because of the data they're trained on, are trained to do spatial reasoning. Whereas, obviously, not only humans, but for billions of years of evolution, we've had, our ancestors have had to learn how to understand abstract, physical, and spatial properties and recognize the patterns there. And so one view would be in the next year, as we gain models that are multimodal native, that isn't just a sort of second-class set as an add-on, but the multimodal capability is a priority, that it will understand these kinds of patterns, because that's something

• 8:43Dwarkesh Patel?

  it would see natively. Whereas right now, what ARC sees is some JSON string of 100100, and it's supposed to recognize the pattern there. And even if you showed a human such a, like a, just a sequence of these kinds of numbers, it would have a challenge making, making sense of what, what kind of question you're asking it. So why I want it to be the case that as soon as we get multimodal models, which we're on the path to unlock right now, they're going to be so much better at ARC-type spatial reasoning?

• 9:07Speaker 01

  That's an empirical question. So I guess we're gonna see the answer within a few months. But my answer to that is, you know, our grids, they're just discrete 2D grids of symbols. They're pretty, pretty small. Like, it's not like, if you flatten an image as a sequence of pixels, for instance, then you get something that's actually very, very difficult to parse. But that's not true for ARC, because the grids are very small. You only have 10 possible symbols. So there's these 2D grids that are actually very easy to flatten as sequences. And transformers, LLMs, they're very good at parsing the sequences. In fact, you can show that LLMs do fine with processing ARC-like data by simply fine-tuning

• 9:49Speaker 01

  LLM on some subsets of the tasks, and then trying to test it on small variations of these tasks. And you see that, yeah, the LLM can encode just fine solution programs for tasks that it has seen before. So it does not really have a problem parsing the input or figuring out the program. The reason why LLMs don't do well on ARC is really just the unfamiliarity aspect. The fact that each new task is different from every other task. You cannot basically, you cannot memorize the solution programs in advance. You have to synthesize a new solution program on the fly for each new task. And that's really

• 10:34Speaker 01

  what LLMs are struggling with.

• 10:36Dwarkesh Patel?

  So before I do more devil's advocate, I just want to step back and explain why I'm especially interested in having this conversation. And obviously, the million-dollar ARC prize, I'm excited to actually play out with it myself, and hopefully...

• 10:50Dwarkesh Patel?

  The Vesuvius Challenge, which was Nat Friedman's prize for solving decoding scrolls, the winner of that decoding the scrolls from that were buried in the volcanoes in the Herculeanum Library that was solved by a 22-year-old who was listening to the podcast, Luke Faratore. So hopefully, somebody listening will find this challenge intriguing and find a solution. So I'm, and the reason I I've had on recently, a lot of people who are bullish on LLMs, and I've had discussions with them before interviewing you about how do we explain the fact that LLMs don't seem to be natively performing that well on ARC? And I found their explanations somewhat contrived, and I'll try out some of the reasons on you. But it is actually

• 11:32Dwarkesh Patel?

  an intriguing fact that they actually, these are, some of these problems are relatively straightforward for humans to understand. And they do struggle with them if you just input them natively.

• 11:42Speaker 01

  All of them are very easy for humans. Like, any, any smart human should be able to do 90%, 95% on ARC.

• 11:48Dwarkesh Patel?

  Smart human?

• 11:49Speaker 01

  A smart human, but even a five-year-old, so with very, very little knowledge, they could, they could definitely do over 50%.

• 11:55Dwarkesh Patel?

  Hmm. So let's talk about that, because you, I agree that smart humans will do very well on this test. But the average human will probably do, you know, mediocre.

• 12:09Speaker 01

  Not, not really. So we actually tried with average humans. The score about 85.

• 12:14Dwarkesh Patel?

  That was with Amazon Mechanical Turk Workers, right?

• 12:16Speaker 01

  That's right.

• 12:16Dwarkesh Patel?

  I honestly don't know the demographic profile of Amazon Mechanical Turk Workers, but I imagine just interacting with the platform that Amazon has set up to do remote work, that's not the median human across the planet, I'm guessing. I mean, the broader point here being that, so we see the spectrum in humans where humans obviously have AGI. But even within humans, you see a spectrum where some people are relatively dumber, and they'll do, perform work on IQ-like tests, for example, Raven's progressive matrices. If you look at how the average person performs on that and you look at the kind of questions that is sort of mid or miss, half of people will get it right, half of people will get it wrong. Some of them are like pretty trivial. For us, we might think like this is kind of trivial.

• 12:56Dwarkesh Patel?

  And so humans have AGI, but from relatively small tweaks, you can go from somebody who misses these kinds of basic IQ test questions to somebody who gets them all right, which suggests that actually if these models are doing natively, we'll talk about some of the previous performances of people who tried with these models. But somebody with a Jack Cole with a 240 million parameter model got 35%. Doesn't that suggest that they're on this spectrum that clearly exists within humans and they're going to be saturated at pretty soon?

• 13:24Speaker 01

  Yeah, so that's a bunch of interesting points here. So there is indeed a branch of LLM approaches, suspected by Jack Cole, that are doing quite well, that are in fact state of the art. But you have to look at what's going on there. So there are two things. The first thing is that to get these numbers, you need to pre-train your LLM on millions of generated R tasks. And of course, if you compare that to a five-year-old child looking at R for the first time, the child has never done IQ tests before, has never seen something like an R task before. The only overlap between what they know and what they have to do in the test is core knowledge, is knowing about like counting and objects and symmetries and things like that.

• 14:09Speaker 01

  And still, they're going to do really well and they're going to do much better than the LLM trained on millions of similar tasks. And the second thing that's something to note about the Jack Cole approach is one thing that's really critical to making the model work at all is test time fine-tuning. And that's something that's really missing, by the way, from LLM approaches. Right now is that, you know, most of the time when you're using LLM, it's just doing static inference. The model is frozen and you're just prompting it and then you're getting an answer. So the model is not actually learning anything on the fly. Its state is not adapting to the task at hand.

• 14:51Speaker 01

  And what JACO is actually doing is that for every test problem is on the fly, is fine-tuning a version of the LLM for that task. And that's really what's unlocking performance. If you don't do that, you get like 1%, 2%. So basically something completely negligible. And if you do test time fine-tuning and you add a bunch of tricks on top, then you end up with interesting performance numbers. So I think what it's doing is trying to address one of the key limitations of LLMs today, which is the lack of active inference, is actually adding active inference to LLMs. And that's working extremely well, actually. So that's fascinating to me.

• 15:30Dwarkesh Patel?

  That there's so many interesting rabbit holes there. Should I take them in sequence or deal with them all at once? Let me, let me just start. So the point you made about the fact that you need to unlock the adaptive compute slash test time compute. A lot of the scale maximalist, I think this will be an interesting rabbit hole to explore with you, because a lot of the scaling maximalist have your broader perspective in the sense that they think that in addition to scaling, you need these kinds of things like unlocking adaptive compute or doing some sort of RL to get the system to working. And their perspective is that this is a relatively straightforward thing that will be added atop the representations that a scaled up model has greater access to.

• 16:14Speaker 01

  No, it's not, it's not just a technical detail. It's not a straightforward thing. It is everything. It is the important part. And the scale maximalist argument, it boils down to.

• 16:28Speaker 01

  You know, these people, they refer to scaling laws, which is this empirical relationship that you can draw between how much compute you spend on training a model and the performance you're getting on benchmarks, right? And the key question here, of course, is, well, how do you measure performance? What it is that you're actually improving by adding more compute and more data? And well, it's benchmark performance, right? And the thing is, the way you measure performance is not a technical detail. It's not an afterthought, because it's gonna narrow down the set of questions that you're asking. And so accordingly, it's gonna narrow down the set of answers that you're looking for.

• 17:09Speaker 01

  If you look at the benchmarks we're using for LLMs, they are all memorization-based benchmarks. Like, sometimes they are literally just knowledge-based, like a school test. And even if you look at the ones that are, you know, explicitly about reasoning, you realize, if you look closely, that it's, in order to solve them, it's enough to memorize a finite set of reasoning patterns. And then you just reapply them. They're like static programs. LLMs are very good at memorizing static programs, small static programs. And they've got this sort of like bank of solution programs. And when you give them a new puzzle, they can just fetch the appropriate program, apply it,

• 17:53Speaker 01

  and it's looking like it's reasoning. But really, it's not doing any sort of on-the-fly program synthesis. All it's doing is program fetching. So you can actually solve all these benchmarks with memorization. And so what you're scaling up here, like if you look at the models, they are a big parametric curves fitted to a data distribution, like on descent. So they are basically these big interpolative databases, interpolative memories. And of course, if you scale up the size of your database and you cram into it more knowledge, more patterns, and so on, you are gonna be increasing its performance as measured by a memorization benchmark. That's kind of obvious.

• 18:37Speaker 01

  But as you're doing it, you are not increasing the intelligence of the system one bit. You are increasing the skill of the system. You are increasing its usefulness, its scope of applicability, but not its intelligence, because skill is not intelligence. And that's the fundamental confusion. That people run into is that they're confusing skill and intelligence.

• 19:00Dwarkesh Patel?

  Yeah, there's a lot of fascinating things to talk about here. So, skill, intelligence, interpolation. I mean, okay, so the thing about they're fitting some manifold into that maps the input data. There's a reductionist way to talk about what happens in the human brain that says that it's just axons firing at each other. But we don't care about the reductionist explanation of what's happening. We care about what the sort of meta at the macroscopic level, what happens when these things combine. As far as the interpolation goes, so, okay, let's look at one of the benchmarks here. There's one benchmark that does grade school math. And these are problems that like a smart high schooler would be able to solve. It's called GSM 8K.

• 19:45Dwarkesh Patel?

  And these models get 95% on these, like basically they always nail it.

• 19:49Speaker 01

  That's memorization benchmark.

• 19:50Dwarkesh Patel?

  Okay, let's talk about what that means. So here's one question about from that benchmark. So 30 students are in a class. One fifth of them are 12 year olds, one third are 13 year old, one tenth are 11 year olds. How many of them are not 11, 12, or 13 years old? So I agree, like this is not rocket science, right? You can write down on paper how you go through this problem. And a high school kid, at least a smart high school kid, should be able to solve it. Now, when you say memorization, it still has to reason through how to think about fractions and what is the context of the whole problem? And then combining the different calculations is doing.

• 20:24Speaker 01

  It depends how you want to define reasoning. But there are two definitions you can use. So one is I have available a set of program templates. It's like the structure of the puzzle, which can also generate its solution. And I'm just going to identify the right template, which is in my memory. I'm going to input the new values into the template, run the program, get the solution. And you could say this is reasoning. And I say, yeah, sure, okay. But another definition you can use is reasoning is the ability to, when you're faced with a puzzle, given that you don't have already a program in memory to solve it, you must synthesize on the fly a new program. Based on bits of pieces of existing programs that you have,

• 21:09Speaker 01

  you have to do on the fly program synthesis. And it's actually dramatically harder than just fetching the right memorized program and replying it.

• 21:17Dwarkesh Patel?

  So I think maybe we are overestimating the extent to which humans are so sample efficient. They also don't need training in this way where they have to drill in these kinds of pathways of reasoning through certain kinds of problems. So let's take math, for example.

• 21:34Speaker 01

  Yeah.

• 21:35Dwarkesh Patel?

  It's not like you can just show a baby the axioms of set theory and now they know math, right? So they, when they're growing up, you had to do years of teaching them pre-algebra. Then you got to do a year of teaching them, doing drills and going through the same kind of problem in algebra, then geometry, pre-calculus, calculus.

• 21:51Speaker 01

  Absolutely. So training.

• 21:52Dwarkesh Patel?

  Yeah, but isn't that like the same kind of thing where you, you can't just see one example and now you have the program or whatever. You actually had to drill it. These models also had to drill with a bunch of fruit training data.

• 22:01Speaker 01

  Sure. I mean, in order to do on the fly program synthesis, you actually need building blocks to work from. So knowledge and memory are actually tremendously important in the process. I'm not, I'm not saying it's memory versus reasoning. In order to do effective reasoning, you need memory.

• 22:20Dwarkesh Patel?

  But it sounds like it's compatible with your story that through seeing a lot of different kinds of examples, these things can learn to reason within the context of those examples. And we can also see within bigger and bigger models. So that was an example of a high school level math problem.

• 22:36Dwarkesh Patel?

  Let's say a model that's like smaller than GPT three couldn't do that at all. As these models get bigger, they can, they seem to be able to pick up bigger and bigger.

• 22:42Speaker 01

  It's not really a size issue. It's more like a training data issue in this case.

• 22:46Dwarkesh Patel?

  Well, bigger models can pick up these kinds of circuits, which smaller models apparently don't do a good job of doing this, even if you were to train them on this kind of data. Doesn't that just suggest that as you have bigger and bigger models, they can pick up bigger and bigger pathways or more general ways of reasoning?

• 23:00Speaker 01

  Absolutely.

• 23:01Dwarkesh Patel?

  But then isn't that intelligence?

• 23:02Speaker 01

  No, no, it's not. If you scale up your database and you keep adding to it more knowledge, more program templates, then sure, it becomes more and more skillful. You can apply to more and more tasks, but general intelligence is not. Task-specific skills scaled up to many skills. It is because there is an infinite space of possible skills. General intelligence is the ability to approach any problem, any skill, and very quickly master it using very little data. Because this is what makes you able to face anything you might ever encounter. This is what makes, uh, this, this is the definition of generality. Like generality is not specificity scaled up. It is, uh, the ability to apply your mind to anything at all, to arbitrary things.

• 23:47Speaker 01

  And this requires fundamentally, this requires the ability to adapt, to learn on the fly efficiently.

• 23:53Dwarkesh Patel?

  So I, my claim is that by doing this pre-training on bigger and bigger models, you are gaining that capacity to then generalize very efficiently. Let me give you an example. Let me, let me give you an example. So your own company, Google, in their paper on Gemini 1.5, they had this very interesting example where the, um, they would give in context, they would give the model the grammar book and the dictionary of a language that has less than 200 living speakers. So it's not in the pre-training data. And you just give them the, the dictionary and it basically is able to speak this language and translate to it, including the complex and organic ways in which languages are structured. So a human, if you showed me a dictionary from like English to Spanish,

• 24:38Dwarkesh Patel?

  I'm not going to be able to pick up the how to structure sentences and how to say things in Spanish. The fact that because of the representations that it has gained through this pre-training, it is able to now extremely efficiently learn a new language. Doesn't that show that, uh, this kind of pre-training actually does increase your ability to learn new tasks?

• 24:57Speaker 01

  If you're right, if you were right, LLMs would do really well on arc puzzles because arc puzzles are not complex. Each one of them requires very little knowledge. Each one of them is, is very low on complexity. You don't need to think very hard about it. Uh, they're actually extremely obvious for humans, like even children can do them. But LLMs cannot, uh, even if, uh, even LLMs that have, you know, 100,000 times more knowledge than you do, they still cannot. And the only thing that makes art special is that it was designed with this intent to resist memorization. This is the only thing. And this is the huge blocker for LLM performance. Right? And so, uh, you know.

• 25:39Speaker 01

  I think if you look at LLMs closely, it's very obvious that they're not really synthesizing new programs on the fly to solve the task that they're faced with. They're very much reapplying things that they've stored in memory. For instance, one thing that's very striking is LLMs can solve Caesar cipher, you know, like a Caesar cipher, like transposing letters to code a message. And well, that's a very complex algorithm, right? But it comes up quite a bit on the internet. So they've basically memorized it. And what's really interesting is that they can do it for a transposition length of like three or five, because there are very, very common numbers in examples provided on the internet.

• 26:21Speaker 01

  But if you try to do it with an arbitrary number like nine, it's going to fail because it does not encode the generalized form of the algorithm, but only specific cases. It has memorized specific cases of the algorithm, right? And if it could actually synthesize on the fly the solver algorithm, then the value of n would not matter at all because it does not increase the problem complexity.

• 26:46Dwarkesh Patel?

  I think this is true of humans as well, where what was the study that...

• 26:50Speaker 01

  Humans use memorization and pattern matching all the time, of course, but humans are not limited to memorization and pattern matching. They have this very unique ability to adapt to new situations on the fly. This is exactly what enables you to navigate every new day in your life.

• 27:06Dwarkesh Patel?

  I'm forgetting the details, but there was some study that chess grandmasters will perform very well within the context of the moves that...

• 27:12Speaker 01

  Excellent example because chess at the highest level is all about memorization, chess memorization.

• 27:18Dwarkesh Patel?

  Okay, sure, we can leave that aside. What is your explanation for the original question of why can, why in context the GPT-1, sorry, Gemini 1.5 was able to learn a language, including the complex grammar structure? Doesn't that show that they can pick up new knowledge?

• 27:33Speaker 01

  I would assume that it has simply mined from its extremely extensive and immeasurably vast training data, it has mined the required template and then it's just reusing it. We know that LLMs have a very poor ability to synthesize new program templates like this on the fly or even adapt existing ones. They're very much limited to fetching.

• 27:54Dwarkesh Patel?

  Suppose there's a programmer at Google. They go into the office in the morning. At what point are they doing something that 100% cannot be due to fetching some template? That even if they suppose they were an LLM, they could not do if they had fetched some template from their program. Like, at what point do they have to use this so-called extreme generalization capability?

• 28:11Speaker 01

  Forget about Google's software developers. Every human, every day of their lives is full of novel things that they've not been prepared for. You cannot navigate your life based on memorization alone. It's impossible.

• 28:24Dwarkesh Patel?

  I'm sort of denying the premise that you're also agree they're not doing like quote unquote memorization. It seems like you're saying they're less capable of generalization. But I'm just curious of like, the kind of generalization they do. If you get into the office and you try to do this kind of generalization, you're going to fail at your job. What is the first point? You're a programmer. What is the first point when you try to do that generalization? You would lose your job because you can't do the extreme generalization.

• 28:50Speaker 01

  I don't have any specific examples, but literally, like, take this situation for instance. You've never been here in this room. Maybe you've been in this city a few times, I don't know. But there's a fair amount of novelty. You've never been interviewing me. There's a fair amount of novelty in every hour of every day in your life. It's in fact, by and large, more novelty than any LLM could handle. Like, if you just put a LLM in a robot, it could not be doing all the things that you've been doing today. Right? Or take, I don't know, like self-driving cars, for instance. You take a self-driving car operating in the Bay area. Do you think you could just drop it in New York City?

• 29:34Speaker 01

  Or drop it in London, where people drive on the left? No, it's going to fail. So not only can you drop, not like, make it generalize to a change of rules, of driving rules, but you cannot even make it generalize to a new city. It needs to be trained on each specific environment.

• 29:54Dwarkesh Patel?

  I mean, I agree that self-driving cars aren't AGI.

• 29:57Speaker 01

  But it's the same type of model. They're transformers as well.

• 30:00Dwarkesh Patel?

  I mean, I don't know. Apes also have brains with neurons in them, but they're less intelligent because they're small.

• 30:05Speaker 01

  It's not the same architecture.

• 30:06Dwarkesh Patel?

  We can get into that, but so. I still don't understand like a concrete thing of we also need training. That's why education exists. That's why we had to spend the first 18 years of our life doing drills.

• 30:19Speaker 01

  We have a memory, but we are not a memory. We are not limited to just a memory.

• 30:24Dwarkesh Patel?

  But I'm denying the premise that that's necessarily the only thing these models are doing. And I'm still not sure what is the, what is the task that a remote worker would be do, have to like, suppose you just stepped out of a remote work with an LLM and they're a programmer. What is the first point that you realize this is not a human, this is an LLM?

• 30:39Speaker 01

  What about they just send them an arc puzzle and see how they do?

• 30:43Dwarkesh Patel?

  No, like part of their job, you know?

• 30:45Speaker 01

  But you have to deal with novelty all the time.

• 30:48Dwarkesh Patel?

  Okay, so if you, is there a world in which all the programmers are replaced? And then we're still saying, ah, but they're only doing memorization laden programming tasks, but they're still producing a trillion dollars of worth of, you know, output in the form of code.

• 31:02Speaker 01

  Software development is actually a pretty good example of a job where you're dealing with novelty all the time. Or if you're not, well, I'm not sure what you're doing. So I personally use GPT very little in my software development job. And before, before LLM, so I think I was also using Stack Overflow very little. You know, some people maybe are just copy pasting stuff from Stack Overflow or nowadays copy pasting stuff from, from an LLM.

• 31:28Speaker 01

  Personally, I try to focus on problem solving. The syntax is just a technical detail. What's really important is the problem solving. Like the essence of programming is engineering mental models, like mental representations of the problem you're trying to solve.

• 31:45Dwarkesh Patel?

  But you can, you know, we have many, people can interact with these systems themselves. And you can go to chat GPT and say, here's a specification of the kind of program I want. They'll build it for you.

• 31:55Speaker 01

  As long as there are many examples of this program on like GitHub and Stack Overflow and so on. Sure, they will fetch the program for you from their memory.

• 32:03Dwarkesh Patel?

  But you can change arbitrary details.

• 32:05Speaker 01

  No, it doesn't work.

• 32:06Dwarkesh Patel?

  You can say, I needed to work on this different kind of server.

• 32:08Speaker 01

  If that were true, there would be no software engineers today.

• 32:12Dwarkesh Patel?

  I agree, we're not at a full AGI yet in the sense that these models have, let's say, less than a trillion parameters. A human brain has somewhere around the order of 10 to 30 trillion synapses. I mean, if you were just doing some naive math, you're like at least 10x under parameterized. So I agree, we're not there yet, but I'm sort of confused on why we're not on the spectrum, where, yes, I agree that there's many kinds of generalization they can't do. But it seems like they're on this kind of smooth spectrum that we see even within humans, where some humans would have a hard time doing an ARC-type test. We see that based on their performance on progressive Ravens matrices type IQ tests.

• 32:47Speaker 01

  I'm not a fan of IQ tests because for the most part, you can train on IQ tests and get better at them. So they're very much memorization-based. And this is actually the main pitfall that ARC tries not to fall for.

• 33:02Dwarkesh Patel?

  I'm still not, so if all remote jobs are automated in the next five years, let's say, at least that don't require you to be like sort of a service. It's not like a salesperson where you need, you want the human to be talking, but like programming, whatever, in that world, would you say that that's not possible? Because a lot of what a programmer needs to do definitely requires things that would not be in any free-training corpus.

• 33:24Speaker 01

  Sure. I mean, in five years, there will be more software engineers than there are today. Not sure.

• 33:29Dwarkesh Patel?

  But I just want to understand. So I'm still not sure. I mean, I know how to, I studied computer science. If I had become a code monkey out of college, like what would I be doing? I go to my job. What is the first thing, my boss tells me something to do? When does he realize I'm an LLM, if I was an LLM?

• 33:46Speaker 01

  Probably on the first day, you know? Again, if it were true that LLMs could generalize to novel problems like this and can actually develop software to solve a problem they've never seen before, you would not need software engineers anymore. In practice, if I look at how people are using LLMs in their software engineering job today, they're using it as a stack overflow replacement. So they're using As a way to copy paste code snippets to perform very common actions. And this is what they actually need is a database of code snippets. They don't actually need any of the abilities that actually make them software engineer.

• 34:27Dwarkesh Patel?

  I mean, when we talk about interpolating between stack overflow databases, if you look at the kinds of math problems or coding problems, maybe to say that there, maybe let's step back on interpolation and let me ask the question this way. Why can't creativity, why isn't creativity just interpolation in a higher dimension? Where if a bigger model can learn a more complex manifold, if we're going to use the ML language. And if you look at read a biography of a scientist, right? It doesn't feel like they're not zero shotting new scientific theories. They're playing with existing ideas. They're trying to juxtapose them in their head. They try out some like slightly ever in the tree of evolution, intellectual descendants. They try out a different evolutionary path.

• 35:09Dwarkesh Patel?

  We you you sort of run the experiment there in terms of publishing the paper, whatever. It seems like a similar kind of thing humans are doing. There's like at a higher level of generalization. And what you see across bigger and bigger models is they can, they seem to be approaching higher and higher levels of generalization. Where GPT two couldn't do a great school level math problem that requires more generalization than it has capability for, even that skill. Then GPT three and four can.

• 35:31Speaker 01

  So not quite. So GPT four has a higher degree of skill and a higher range of skills.

• 35:37Dwarkesh Patel?

  All right, I don't want to get into semantics here, but.

• 35:39Speaker 01

  Same degree of generalization.

• 35:40Dwarkesh Patel?

  I don't want to get into semantics here, but the question of why, why can't creativity be just interpolation on a higher, higher dimension?

• 35:49Speaker 01

  I think interpolation can be creative, absolutely. And you know, to your point, I do think that on some level, humans also do a lot of memorization, a lot of reciting, a lot of pattern matching, a lot of interpolation as well. So it's, it's very much a spectrum.

• 36:05Speaker 01

  Between pattern matching and true reasoning, it's a spectrum. And humans are never really at one end of the spectrum. They are never really doing pure pattern matching or pure reasoning. They're usually doing some mixture of both. Even if you're doing something that seems very reasoning heavy, like proving a mathematical theorem. As you're doing it, sure, you're doing quite a bit of discrete search in your mind, quite a bit of actual reasoning. But you're also very much guided by intuition, guided by pattern matching, guided by the shape of proofs that you've seen before, by your knowledge of mathematics. So it's never really, you know, all of our thoughts, everything we do is a mixture of this sort of like interpolative memorization based thinking,

• 36:50Speaker 01

  this sort of like type one thinking and type two thinking.

• 36:56Dwarkesh Patel?

  Why are bigger models more sample efficient?

• 36:59Speaker 01

  Because they have more reusable building blocks that they can lean on to pick up new patterns in their training data.

• 37:09Dwarkesh Patel?

  And does that pattern keep continuing as you keep getting bigger and bigger?

• 37:12Speaker 01

  To the extent that the new patterns you're giving the model to learn are good match for what it has learned before. If you present something that is actually novel, that is not in a steady distribution, like an ARC puzzle, for instance, it will fail.

• 37:25Dwarkesh Patel?

  Let me make this claim. The program synthesis, I think, is a very, very useful intuition pump. Why can't it be the case that what's happening in the transformer is the early layers are doing the figuring out how to represent the inputting tokens? And what the middle layers do is this kind of program search, program synthesis, where they combine the inputs to the, to the, you know, all the circuits in the model, where they go from the low level representation to a higher level of representation near the middle of the model. They use these programs and they do, they combine these concepts. Then what comes out the other end is the reasoning based on that high level intelligence.

• 38:01Speaker 01

  Possibly, why not? But you know, if, if these models were actually capable of synthesizing novel programs, however simple, they should be able to do ARC. Because for any ARC task, if you write down the solution program in Python, it's not a complex program. It's extremely simple and, and humans can figure it out. So why, why can LLMs not do it?

• 38:26Dwarkesh Patel?

  Okay, I think that's a fair point. And if I turn the question around to you, so suppose that it's the case that in a year, a multimodal model can solve ARC, let's say get 80%, whatever the average human would get, then AGI?

• 38:43Speaker 01

  Quite possibly, yes. I think if you start so honestly, what I would like to see is an LLM type model solving ARC at like 80%, but after having only been trained on core knowledge related stuff.

• 38:59Dwarkesh Patel?

  But human kids, I don't think we're necessarily just trained on. It's not just that we have in our genes, object permanence.

• 39:05Speaker 01

  Let me rephrase that.

• 39:05Dwarkesh Patel?

  Yeah.

• 39:06Speaker 01

  Only trained on information that is not explicitly trying to anticipate what's going to be in the ARC test set.

• 39:15Dwarkesh Patel?

  But isn't the whole point of ARC that you can't sort of, it's a new type of intelligence test every single time?

• 39:20Speaker 01

  Yes, that is the point. So if ARC were a perfect flawless benchmark, it would be impossible to anticipate what's in the test set. And you know, ARC was released more than four years ago, and so far it's been resistant to memorization. So I think it has to some extent passed the test of time. But I don't think it's perfect. I think if you try to make by hand hundreds of thousands of ARC tasks, and then you try to multiply them by programmatically generating variations, and then you end up with maybe hundreds of millions of tasks, just by brute forcing the task space, there will be enough overlap between what you train on and what's in the test set that you can actually score very highly. So you know, with enough scale, you can always cheat.

• 40:05Dwarkesh Patel?

  If you can do this for every single thing that supposedly requires intelligence, then what good is intelligence? Apparently you can just brute force intelligence.

• 40:11Speaker 01

  If the world, if your life were a static distribution, then sure, you could just brute force the space of possible behaviors. Could like, you know, the way I would think about intelligence. There are several metaphors I like to use, but one of them is you can think of intelligence as a pathfinding algorithm in future situation space. Like, I don't know if you're familiar with game development, like RTS game development, but you have a map, right? And you have, it's like a 2D map. And you have partial information about it. Like, there is some fog of war on your map. There are areas that you haven't explored yet. You know nothing about them. And then there are areas that you've explored, but you only know how they were like in the past.

• 40:56Speaker 01

  You don't know how they are like today. And now, instead of thinking about a 2D map, I think about the space of possible future situations that you might encounter and how they are connected to each other. Intelligence is a pathfinding algorithm. So, once you set a goal, it will tell you how to get there optimally. But of course, it's constrained by the information you have. It cannot pathfind in an area that you know nothing about. It cannot also anticipate changes. And the thing is, if you had complete information about the map, then you could solve the pathfinding problem by simply memorizing every possible path,

• 41:41Speaker 01

  every mapping from point A to point B. You could solve the problem with pure memory. But the reason you cannot do that in real life is because you don't actually know what's going to happen in the future. Life is ever-changing.

• 41:56Dwarkesh Patel?

  I feel like you're using words like memorization, which we would never use for a human children. If you're like, your kid learns to do algebra and then like now learns to do calculus, you wouldn't say they've memorized calculus. If they can just solve any arbitrary algebraic problem, you wouldn't say like they've memorized algebra. They'd say they've learned algebra.

• 42:11Speaker 01

  Humans are never really doing pure memorization or pure reasoning.

• 42:14Dwarkesh Patel?

  But that's only because you're so matically labeling when the human does a skill, it's a memorization. When the exact same skill is done by the LLM, as you can measure by these benchmarks, and you can just like plug in any sort of math problem.

• 42:22Speaker 01

  Sometimes humans are doing the exact same as the LLM is doing, which is just, for instance, I don't know, if you learn to add numbers, you're memorizing an algorithm, you're memorizing a program, and then you can reapply it. You are not synthesizing on the fly the addition program.

• 42:37Dwarkesh Patel?

  So obviously, at some point, some human had to figure out how to do addition.

• 42:40Speaker 01

  Yes.

• 42:41Dwarkesh Patel?

  But like, the way a kid learns it is not that they sort of figure out from the actions of that theory how to do addition.

• 42:46Speaker 01

  I think what you learn in school is mostly memorization.

• 42:49Dwarkesh Patel?

  Right. So my claim is that, listen, these models are vastly under parameterized relative to how many flops or how many parameters you have in the human brain. And so, yeah, they're not going to be like coming up with new theorems like the smartest humans can. But most humans can't do that either. What most humans do, it sounds like is similar to what you are calling memorization, which is memorizing skills or memorizing, you know, techniques that you've learned. And so it sounds like it's compatible in your, tell me if this is wrong. Is it compatible in your world if like all the remote workers are gone, but they're doing skills which we can potentially make synthetic data of. So we record everybody's screen and every single remote worker's screen. We sort of understand the skills they're performing there.

• 43:34Dwarkesh Patel?

  And now we've trained a model that can do all this. All the remote workers are unemployed. We're generating trillions of dollars of economic activity from AI remote workers. In that world, are we still in the memorization regime?

• 43:45Speaker 01

  So sure, with memorization, you can automate almost anything. As long as it's a static distribution, as long as you don't have to deal with change.

• 43:53Dwarkesh Patel?

  Are most jobs part of such a static distribution?

• 43:57Speaker 01

  Potentially, there are lots of things that you can automate.

• 44:00Dwarkesh Patel?

  Yeah.

• 44:00Speaker 01

  And LLMs are an excellent tool for automation. And I think that's, but you have to understand that automation is not the same as intelligence. I'm not saying that LLMs are useless. I've been a huge proponent of deep learning for many years. And, you know, for many years, I've been saying two things. I've been saying that if you keep scaling up deep learning, it will keep paying off. And at the same time, I've been saying, if you keep scaling up deep learning, this will not lead to a GI. So we can automate more and more things. And yes, this is economically valuable. And yes, potentially, there are many jobs you could automate away like this, and that would be economically valuable. But you're not, still not going to have intelligence. So you can ask, you know, okay, so what does it matter if we can generate all this economic value? Maybe you don't need intelligence after all.

• 44:42Speaker 01

  Well, you need intelligence the moment you have to deal with change, with novelty, with uncertainty. As long as you're in a space that can be exactly described in advance, you can just.

• 44:55Speaker 01

  You can just automate your pure memorization, right? In fact, you can always solve any problem. You can always display arbitrary levels of skills on any task without leveraging any intelligence whatsoever, as long as it is possible to describe the problem and its solution very, very precisely.

• 45:17Dwarkesh Patel?

  But when they do deal with novelty, then you just call it interpolation, right?

• 45:21Speaker 01

  No, interpolation is not enough to deal with all kinds of novelty. If it were, then LLMs would be AGI.

• 45:29Dwarkesh Patel?

  Well, I agree they're not AGI. I'm just trying to figure out, how do we figure out we're on the path to AGI? And I think sort of Crux here is maybe that it seems to me that these things are on a spectrum and we're clearly covering the earliest part of the spectrum with LLMs.

• 45:43Speaker 01

  I think so.

• 45:43Dwarkesh Patel?

  And, oh, okay, interesting. But here's another sort of thing that I think is evidence for this. Grokking, right? So clearly, even within deep learning, there's a difference between the memorization regime and the generalization regime, where at first they'll just memorize the data set of, you know, if you're doing modular addition, how to add digits. And then, at some point, if you keep training on that, they'll learn the skill. So the fact that there is that distinction suggests that the generalized circuit, the deep learning can learn, there is a regime it enters where it generalizes. If you have an overparameterized model, which you don't have in comparison to all the tasks we want these models to do right now.

• 46:20Speaker 01

  Grokking is a very, very old phenomenon. We've been observing it for decades. It's basically an instance of the minimum description length principle, where, sure, you can, given a problem, you can just memorize a pointwise input to output mapping, which is completely overfit. So it does not generalize at all, but it solves the problem on the training data. And from there, you can actually keep pruning it, keep making your mapping simpler and simpler and more compressed. And at some point, it will start generalizing. And so that's something called the minimum description expression principle. It's decided that the program that will generalize best is the shortest.

• 47:06Speaker 01

  Right? And it doesn't mean that you're doing anything other than memorization, but you're doing memorization plus regularization. Right?

• 47:15Dwarkesh Patel?

  AKA generalization.

• 47:16Speaker 01

  Yeah, and at least absolutely, at least to generalization.

• 47:19Dwarkesh Patel?

  Right, and then so you do that within one skill. But then the pattern you see here of meta learning is that it's more efficient to store a program that can perform many skills rather than one skill, which is what we might call fluid intelligence. And so as you get bigger and bigger models, you would expect it to go up this hierarchy of generalization, where it generalizes to a skill, then it generalizes across multiple skills.

• 47:38Speaker 01

  That's correct. That's correct. And you know, LLMs, they are not infinitely large. They have only a fixed number of parameters. And so they have to compress their knowledge as much as possible. And in practice, so LLMs are mostly storing reusable bits of programs, like vector programs. And because they have this need for compression, it means that every time they're learning a new program, they're going to try to express it in terms of existing bits and pieces of programs that they've already learned before.

• 48:07Dwarkesh Patel?

  And isn't this the generalization?

• 48:10Speaker 01

  Absolutely.

• 48:11Dwarkesh Patel?

  Oh, wait, so.

• 48:13Speaker 01

  This is what, you know, clearly LLMs have some degree of generalization. And this is precisely why, is because they have to compress.

• 48:20Dwarkesh Patel?

  And why is that intrinsically limited? Why can't you just go, at some point, it has to learn a higher level of generalization, a higher level, and then the highest level is the fluid intelligence?

• 48:28Speaker 01

  It's intrinsically limited because the substrate of your model is a big parametric curve. And all you can do with this is local generalization. If you want to go beyond this towards broader or an extreme generalization, you have to move to a different type of model. And my paradigm of choice is discrete program search, program synthesis. So, and if you want to understand that, you can sort of like. Compare, compare it, contrast it with deep learning. So in deep learning, your model is a parameter, a differentiable parameter curve. In program synthesis, your model is a discrete graph of operators. You've got like a set of logical operators, like a domain-specific language.

• 49:13Speaker 01

  You're picking instances of it. You're structuring that into a graph. That's a program. And that's actually very similar to like a program you might, you might write in, in, in Python or C++ and so on. And in deep learning, your learning engine, because we are doing machine learning here, like we're trying to automatically learn these models. In deep learning, your learning engine is gradient descent, right? And gradient descent is very compute efficient because you have this very strong informative feedback signal about where the solution is, so you can get to the solution very quickly. But it is very data inefficient, meaning that in order to make it work, you need a dense sampling of the operating space. You need a dense sampling of the data distribution.

• 49:57Speaker 01

  And then you're, you're limited to only generalizing within that data distribution. And the reason why you have this limitation is because your model is a curve. And meanwhile, if you look at discrete program search, the learning engine is combinatoral search. You're just trying a bunch of programs until you find one that actually meets your spec. This process is extremely data efficient. You can learn a generalizable program from just one example, two examples, which is why it works so well on ARC, by the way. But the big limitation is that it's extremely compute inefficient because you're running into combinatorial explosion, of course. And so you can, you can sort of see here how deep learning and discrete program search,

• 50:41Speaker 01

  they have very complementary strengths and limitations as well. Like every limitation of deep learning has a strength, a corresponding strength in, in, in program synthesis and, and inversely. And I think the path forward is going to be too much to, to basically start doing. So another way you can think about it is, so this, this parameter curve, a train with gradient descent. They're great fit for everything that's a system one type thinking, like pattern recognition, intuition, memorization and so on. And discrete program search is a great fit for a type two thinking, system two thinking, for instance, planning, reasoning,

• 51:25Speaker 01

  quickly figuring out a generalizable model that matches just one or two examples, like for an arc puzzle, for instance. And I think humans are never doing pure system one or pure system two. They're always mixing and matching both. And right now, we have all the tools for system one. We have almost nothing for system two. The way forward is to create a hybrid system. And I think the format it's going to take is it's going to be mostly system two. So the outer structure is going to be a discrete program search system. But you're going to fix the fundamental limitation of discrete program search, which is coming to explosion. You're going to fix it with deep learning. You're going to leverage deep learning to guide,

• 52:08Speaker 01

  to provide intuition in program space to guide the program search. And I think that's very similar to what you see, for instance, when you're playing chess or when you're trying to prove a theorem, is that it's mostly a reasoning thing, but you start out with some intuition about the shape of the solution. And that's very much something you can get via a deep learning model. Via deep learning models, they're very much like intuition machines. They're pattern matching machines. So you start from this shape of the solution. And then you're going to do actual, explicit, discrete program search.

• 52:51Speaker 01

  But you're not going to do it via brute force. You're not going to try things kind of like randomly. You're actually going to ask another deep learning model for suggestions. Like, here's the best likely next step. Here's where in the graph you should be going. And you can also use yet another deep learning model for feedback. But, well, here's what I have so far. Is it looking good? Should I just backtrack and try something new? So.

• 53:19Speaker 01

  I think this grid program search is going to be the key, but you want to make it dramatically better, or does it make it more efficient by leveraging deep learning? And by the way, another thing that you can use deep learning for is of course things like common sense knowledge and knowledge in general. And I think you're going to end up with this sort of system where you have this on the fly synthesis engine that can adapt to new situations. But the way it adapts is that it's going to fetch from a bank of patterns, modules that could be themselves curves, that could be differentiable modules, and some others that could be algorithmic in nature.

• 54:00Speaker 01

  It's going to assemble them via this process that's intuition guided. And it's going to give you for every new situation you might be faced with, it's going to give you with a generalizable model that was synthesized using very, very little data. And something like this would solve ARC.

• 54:18Dwarkesh Patel?

  That's actually a really interesting prompt because I think an interesting crux here is when I talk to my friends who are extremely optimistic about LLMs and expect AGI within the next couple of years, they also in some sense agree that scaling is not all you need, but that the rest of the progress is undergirded and enabled by scaling. And but still, you need to add the system to the test time compute atop these models. And their perspective is that it's relatively straightforward to do that because you have this library of representations that you built up from free training, but it's almost talking like, you know,

• 55:02Dwarkesh Patel?

  it just like skimming through textbooks. You need some more deliberate way in which it engages with the material it learns. In context, learning is extremely sample efficient, but to actually distill that into the weights, you need the model to like talk through the things it sees and then add it back to the weights. As far as the system too goes, they talk about adding some kind of RL setup so that it is encouraged to proceed on the reasoning traces that end up being correct. They think this is relatively straightforward stuff that will be added within the next couple of years.

• 55:32Speaker 01

  That's an empirical question. So I think we'll see.

• 55:35Dwarkesh Patel?

  Your intuition, I assume, is not that. I'm curious.

• 55:37Speaker 01

  My intuition is, in fact, this whole like system to architecture is the hard part, is the very hard and obvious part. Scaling up the interpretative memory is the easy part. All you need is, like, it's literally just a big curve. All you need is more data. It's a representation of a data set, an interpretative representation of a data set. That's the easy part. The hard part is the architecture of intelligence. Memory and intelligence are separate components. We have the memory. We don't have the intelligence yet. And I agree with you that, well, having the memory is actually very useful. And if you just had the intelligence, but it was not hooked up to an extensive memory, it would not be that useful because it would not have enough material to work from.

• 56:20Dwarkesh Patel?

  Yeah. The alternative hypothesis here that a former guest, Trenton Brooken, advanced is that intelligence is just hierarchically associative memory where higher level patterns, when Sherlock Holmes goes into a crime scene and he's extremely sample efficient, he can just like look at a few clues and figure out who was a murderer. And the way he's able to do that is he has learned higher, higher level sort of associations. It's memory in some fundamental sense, but...

• 56:48Dwarkesh Patel?

  So here's one way to ask the question. In the brain, supposedly we do program synthesis, but it is just synapses connected to one, each other. And so physically, it's gotta be that you just query the right circuit, right?

• 57:01Speaker 01

  You are. Yeah, yeah, yeah. It's a matter of degree.

• 57:04Dwarkesh Patel?

  But if you can learn it, if, you know, training in the environment that the human ancestors were trained in means you learn that those circuits, training on the same kinds of outfits that humans produce, which to replicate require these kinds of circuits, wouldn't that train the same kind of whatever humans have?

• 57:19Speaker 01

  You know, it's a matter of degree.

• 57:23Speaker 01

  If you have a system that has a memory and is only capable of doing local generalization from that, it's not going to be very adaptable. To be really general, you need the memory plus the ability to search to quite some depth to achieve broader, even extreme, generalization. You know, like, one of my favorite psychologists, Jean Piaget was the founder of developmental psychology. He had a very good quote about intelligence. He said, "Intelligence is what you use when you don't know what to do." And it's like, as a human living your life, in most situations, you already know what to do because you've been in this situation before.

• 58:06Speaker 01

  You already have the answer, right? And you're only going to need to use intelligence when you're faced with novelty, with something you didn't expect, with something that you weren't prepared for, either by your own experience, your own life experience, or by your evolutionary history. Like, this day that you're living right now is different in some important ways from every day you've lived before. It's also different from any day ever lived by any of your ancestors. And still, you're capable of being functional, right? How is that possible?

• 58:39Dwarkesh Patel?

  I'm not denying that generalization is extremely important and is the basis for intelligence. The crux is like, how much of that is happening in the models? But, okay, let me ask a separate question. We might keep going in the circle here. The differences in intelligence between humans, maybe the intelligence tests, because of reasons you mentioned, are not measuring it well. But clearly, there's differences in intelligence between different humans.

• 59:03Speaker 01

  Sure.

• 59:03Dwarkesh Patel?

  What is your explanation for what's going on there? Because I think that's sort of compatible with my story, that there's a spectrum of generality and that these models are climbing up to a human level. And even some humans haven't even climbed up to the Einstein level or the Francois level, but...

• 59:18Speaker 01

  That's a great question. You know, there is extensive evidence that intelligence, differences in intelligence are mostly genetic in nature, right? Meaning that if you take someone who is not very intelligent, there is no amount of training, of like, training data you can expose that person to that would make them become Einstein. And this kind of points to the fact that you really need a better architecture. You need a better algorithm. And more training data is not, in fact, all you need.

• 59:50Dwarkesh Patel?

  I think I agree with that. I think maybe a way I might phrase it is that the people who are smarter have in ML language better initializations. It just the neural wiring, if you just look at it's more efficient. They have maybe greater density of firing. And so some part of the story is scaling. There is some correlation between brain size and intelligence. And we also see within the context of quote unquote scaling that people talk about within the context of LLMs, architectural improvements where a model like Gemini 1.5 flash is performs as well as GPT-4 did when GPT-4 was released a year ago, but is 57 times cheaper on output. So the part of the scaling story is that

• 1:00:33Dwarkesh Patel?

  the architectural improvements are we're in like extremely low hanging fruit territory when it comes to those. Okay, we're back now with the co-founder of Zapier, Mike Canuff. We had to restart a few times there. And you're funding this prize and you're running this prize with Francois. And so tell me about how this came together. What prompted you guys to launch this prize?

• 1:00:57Speaker 02

  Yeah, I guess I've been sort of like AI curious for 13 years. I've been I co-founded Zapier, been running it for the last 13 years. And I think I first got introduced to your work during COVID. I kind of went down the rabbit hole. I had a lot of free time. And it was right after you published your On Measure of Intelligence paper, you sort of introduced the concept of AGI, this like efficiency of skill acquisition is like the right definition and the arc puzzles. But I don't think the first Kaggle contest was done yet. I think it was still running. And so I kind of it was interesting, but I just parked the idea. And I had bigger fish to fry at Zapier. We're in this middle of this big turnaround of trying to get to our second product. And then it was January 2022 when

• 1:01:40Speaker 02

  the Chain of Thought paper came out that really like awoke me to sort of the progress. I gave a whole presentation to the Zapier on like the GPT-3 paper even. So I sort of felt like I had priced in everything that Elms could do. And that paper was really shocking to me in terms of all these latent capabilities that Elms have that I didn't expect that they had. And so I actually gave up my exec team role at Zapier. I was running half the company at that point. I went back to be an individual contributor and just do to go do AI research alongside Brian, my co-founder. And ultimately, that led me to back towards ARC. I was looking into it again, and I had sort of expected to see this saturation effect that MMLU has, that GMSK 8K has. And when I looked at the scores and the progress since the last four years,

• 1:02:25Speaker 02

  I was really, again, shocked to see, actually, we've made very little objective progress towards it. And it felt very, it felt like a really, really important eval. And as I sort of spent the last year asking people, quizzing people about it in sort of my networking community, very few people even knew it existed. And that felt like, okay, if it's right that this is a really, really globally singularly unique AGI eval, and it's different from every other eval that exists that are more narrowly measures AI skill, like more people should know about this thing. I had my own ideas on how to beat the ARC as well. So I was working on nights and weekends on that, and I flew up to meet Francois earlier this year to sort of quiz him, show him my ideas.

• 1:03:07Speaker 02

  And ultimately, I was like, well, you know, why don't you think more people know about ARC? I think you should actually answer that. I think it's a really interesting question. Like, why don't you think more people know about ARC?

• 1:03:17Speaker 01

  Sure. You know, I think benchmarks that gain traction in the research community are benchmarks that are already fairly tractable. Because the dynamic that you see is that some research group is going to make some initial breakthrough, and then this is going to catch the attention of everyone else. And so you're going to get follow-up papers with people trying to beat the first team and so on. And for ARC, this has not really happened because ARC is actually very hard for existing AI techniques. Kind of ARC requires you to try new ideas. And that's very much the point, by the way. Like, the point is not that, yeah, you should just be able to apply existing technology and solve ARC. The point is that existing technology has reached a plateau.

• 1:03:59Speaker 01

  And if you want to go beyond that, if you want to start being able to tackle problems that you haven't memorized, that you haven't seen before, you need to try new ideas. And ARC is not just meant to be this sort of like measure of how close we are to AGI. It's also meant to be A source of inspiration. Like, I want researchers to look at these puzzles and be like, hey, it's really strange that these puzzles are so simple and most humans can just do them very quickly. Why is it so hard for existing AI systems? Why is it so hard for LLMs and so on? And this is true for LLMs, but ARC was actually released before LLMs were really a thing.

• 1:04:41Speaker 01

  And the only thing that made it special at the time was that it was designed to be a resistance to memorization. And the fact that it has survived LLMs and GNI in general so well kind of shows that, yes, it is actually resistant to memorization.

• 1:04:56Speaker 02

  This is what nerds like me. Because I went and took a bunch of the puzzles myself. I've showed it to all my friends and family too. And they're all like, oh, yeah, this is like super easy. Are you sure AI can't solve this? Like, that's the reaction and the same one for me as well. And the more you dig in, you're like, okay, yep, there's not just empirical evidence over the last four years that it's unbeaten, but there's theoretical like concepts behind why. And I completely agree at this point that like new ideas basically are needed to be dark. And there's a lot of current trends in the world that are actually, I think, working against that happening. Basically, I think we're actually less likely to generate new ideas right now.

• 1:05:32Speaker 02

  You know, I think one of the kind of trends is the closing up frontier research, right? The GPT-4 paper from OpenAI had no technical detail shared. The Gemini paper had no technical detail shared and like the longer context part of that work. And yet, that open innovation, that open progress and sharing is what got us to transformers in the first place. It's what got us to LLMs in the first place. So it's kind of disappointing a little bit, actually, that like so much frontier work has gone closed. It's really making a bet that like these individual labs are going to be the have the breakthrough and not the ecosystem is going to have the breakthrough. And I think sort of the internet open source has shown that that's like the most powerful innovation ecosystem that's ever existed, probably in the entire world.

• 1:06:08Speaker 01

  I think that's actually really sad that frontier research is no longer being published. If you look back, you know, four years ago, well, everything was just openly shared, like all the state of the art results were published. And this is no longer the case. And it's very much, you know, OpenAI single-handedly changed the game. And I think. Opening a basically set back progress towards AGI by quite a few years, probably like five to 10 years for two reasons. And one is that, well, they caused this complete closing down of research, frontier research publishing. But also they trigger this initial burst of hype around LLMs.

• 1:06:51Speaker 01

  And now LLMs have sucked the oxygen out of the room. Like everything, everyone is just doing LLMs. And I see LLMs as more of an off-ramp on the, on the path to AGI, actually. And all these new resources, they're actually going to LLMs instead of everything else they could be, they could be going to. And you know, if you look further into the past to like 2015, 2016, there were like a thousand times fewer people doing AI back then. And yet I feel like the rate of progress was higher because people were exploring more directions. The world felt, felt more open-ended, like you could just go and try,

• 1:07:34Speaker 01

  like have a cool idea of a launch and try it and get some interesting results. So there was, there was this, this energy. And now everyone is very much doing some variation of the same thing. And the big labs also tried their handle Arc, but because they got bad results, they didn't publish anything. Like, you know, people only publish positive results.

• 1:07:55Dwarkesh Patel?

  I wonder how much effort people have put into trying to prompt or scaffold, do some sort of maybe Devon-type approach into getting the frontier models and frontier models of today, not just a year ago, because a lot of post-training has gone into making them better. So Claude 3 Opus or GPT-4O into getting good solutions on Arc.

• 1:08:21Dwarkesh Patel?

  I hope that one of the things this episode does is get people to try out this open competition where they have to put in an open source model to compete, but also to like figure out if there maybe the capability is latent in Claude Opus and just see if you can show that. I think that would be super interesting. So let's talk about the prize. How much do you win if you solve it? You know, get whatever percent on ARC. How much do you get if you get the best submission, but don't crack it?

• 1:08:46Speaker 02

  So you got a million dollar, actually a little over a million dollars of the prize pool. Running the contest on an annual basis, we're gonna, we're starting it today through the middle of November and the goal is to get 85%. That's the lower bounded human average that you guys talked about earlier. And there's a $500,000 prize for the first team that can get to the 85% benchmark. We're also gonna run, we don't expect that to happen this year, actually. One of the early statisticians at Zapier gave me this line that has always stuck with me, that the longer it takes, the longer it takes. So my prior is that like ARC is gonna take years to solve. And so we're gonna keep, we're also gonna break down and do a progress prize this year. So there's a $100,000 progress prize, which we will pay out to the top scores.

• 1:09:30Speaker 02

  So $50,000 is gonna go to the top objective scores this year on the Kaggle leaderboard, which is, we're hosting it on Kaggle. And then we're gonna have a $50,000 pot set for a paper award for the best paper that explains conceptually the scores that they were able to achieve. And one of the, I think, interesting things we're also gonna be doing is we're gonna be requiring that in order to win the prize money, that you put the solution or your paper out into public domain. The reason for this is, you know, typically with contests, you see a lot of like closed up sharing. People are kind of private, secret, they wanna hold their alpha to themselves during the contest period. And because we expect it's gonna be multiple years, we wanna enter a game here. So the, the, the plan is, you know, at the end of November, we will award the $100,000

• 1:10:14Speaker 02

  prize money to the top progress prize and then use the downtime between December, January, February to share out all the knowledge from the top scores and the approaches folks were taking in order to re-baseline the community up to whatever the state of the art is and then run the contest again next year and keep doing that on a yearly basis until we get eighty five percent.

• 1:10:32Dwarkesh Patel?

  I'll give some people some context on why I think this prize is very interesting. I was having conversations with my friends who are very much believers in models as they exist today. And first of all, it was intriguing to me that they didn't know about ARC. These are experienced ML researchers. And so you showed them the, this is, this happened a couple of nights ago. We went to dinner and I showed them an example problem. And they said, of course, an LLM wouldn't be able to solve something like this. And then we take a screenshot of it. We just put it into our chat GPT app and it doesn't get the pattern. And so I think it's very interesting, like it is a notable fact. I was sort of playing devil's advocate against you on these kinds of questions. But this is a very intriguing fact and I'm extreme. I think this is prizes extremely interesting because we're going to learn. We're going to learn something fascinating one way or another.

• 1:11:17Dwarkesh Patel?

  So with regards to the eighty five percent, separate from this prize, I'd be very curious if somebody could replicate that result. Because obviously in psychology and other kinds of fields, which this result seems to be analogous to when you run test on some small sample of people, often they're hard to replicate. So I'd be very curious if you try to replicate this, how, what does the average human perform on ARC? Ask for the difficulty and how long it will take to crack this benchmark. It's very interesting because the other benchmarks that are now fully saturated like MMLU math. Actually, the people who made them, Dan Hendricks and Colin Burns, who did MMLU in math. I think they were grad students or college students when they made it.

• 1:11:59Dwarkesh Patel?

  And the goal when they made it just a couple of years ago was that this will be a test of AGI. And of course it got totally saturated. And I know you argue that these are tests of memorization. But I think the pattern we've seen, in fact, Epoch AI has a very interesting graph that I'll sort of overlay for the YouTube version here. Where you see this almost exponential where it gets, you know, five percent, 10%, 30%, 40% as you increase the compute across models. And then it just shoots up. And in the GPT four technical report, they had this interesting graph of the human eval problem set, which was twenty two coding problems. And they had to graph it on the mean log pass curve, basically because.

• 1:12:41Dwarkesh Patel?

  Early on in training, or even smaller models can have the right idea of how to solve this problem, but it takes a lot of reliability to make sure they stay on track to solve the whole problem. And so you really want to upweight the signal where they get it right at least some of the time, maybe one in 100 times, one in 1,000. And then so they go from like one in 1,000, one in 100, one in 10, and then they just like totally saturate it. I guess the question I have, this is all leading up to, is why won't the same thing happen with ARC? Where people had to try really hard, bigger models, and now they've figured out these techniques that Jack Colas figured out with only a 240 million parameter language model that can get 35%. Shouldn't we see the same pattern we saw across all these other benchmarks where you're just like,

• 1:13:23Dwarkesh Patel?

  sort of eke out, and then once you get the general idea, then you just go all the way to 100?

• 1:13:27Speaker 01

  That's an impeccable question. So we'll see in practice what happens. But what Jack Colas is doing is actually very unique. It's not just pre-training an LLM and then prompting it. He's actually trying to do active inference.

• 1:13:40Dwarkesh Patel?

  He's doing test time, right? He's doing like test time fine-tuning.

• 1:13:42Speaker 01

  Test time fine-tuning. And this is actually trying to lift one of the key limitations of LLMs, which is that at inference time, they cannot learn anything. They cannot adapt on the fly to what they're seeing. And he's actually trying to learn. So what he's doing is effectively a form of program synthesis. Because the LLM contains a lot of useful building blocks, like programming building blocks. And by fine-tuning it on the task at test time, you are trying to assemble these building blocks into the right pattern that matches the task. This is exactly what program synthesis is about. And the way we'd contrast this approach with discrete program search is that in discrete program search,

• 1:14:25Speaker 01

  so you're trying to assemble a program from a set of primitives. You have very few primitives. So people working on discrete program search on ARC, for instance, they tend to work with DSLs that have like 100 to 200 primitive programs. So very small DSL, but then they're trying to combine these primitives into very complex programs. So there's a very deep depths of search.

• 1:14:51Speaker 01

  On the other hand, if you look at what Jackul is doing with LLMs is that he's got this sort of like vector program database DSL of millions of building blocks in the LLM that are mined by pre-training the LLM, not just on a ton of programming problems, but also on millions of generated ARC-like tasks. So you have an extraordinarily large DSL. And then the fine-tuning is very, very shallow recombination of these primitives. So discrete program search, very deep recombination, very small set of primitive programs. And the LLM approach is the same, but on the complete opposite end of that spectrum,

• 1:15:35Speaker 01

  where you scale up the memorization by a massive factor and you're doing very, very shallow search. But they are the same thing, just different ends of the spectrum. And I think where you're going to get the most value for your compute cycles is going to be somewhere in between. You want to leverage memorization to build up a richer, more useful bank of primitive programs. And you don't want them to be hardcoded, like what we saw for the typical ARC DSL. You want them to be learned from examples. But then you also want to do some degree of deep search. As long as you're only doing very shallow search, you are limited to local generalization.

• 1:16:18Speaker 01

  If you want to generalize further, more broadly, this depth of search is going to be critical.

• 1:16:25Dwarkesh Patel?

  I might argue that the reason that he had to rely so heavily on the synthetic data was because he used a 240 million parameter model because the Kaggle competition at the time required him to use a P100 GPU, which has like a tenth or something of the flops of an H100. And so obviously, he can't use.

• 1:16:48Dwarkesh Patel?

  If you believe that sort of scaling will solve this kind of reasoning, then there you can just rely on the generalization. Whereas if you're using a much smaller model, for context for the listeners, by the way, the frontier models today are literally 1,000x bigger than that. And so for your competition, from what I remember, you the submission you have to submit can't make any API calls, can't go online and has to run on Nvidia Tesla T4.

• 1:17:15Speaker 01

  P100, P100.

• 1:17:17Dwarkesh Patel?

  Oh, it's a P100?

• 1:17:17Speaker 01

  Yeah.

• 1:17:18Dwarkesh Patel?

  Okay, so again, it's like significantly less.

• 1:17:20Speaker 02

  There's a 12-hour runtime limit, basically. There's a forcing function of efficiency in the eval.

• 1:17:24Speaker 01

  But here's the thing, you only have 100 test tasks. So the amount of compute you're available for each task is actually quite a bit, especially if you contrast that with the simplicity of each task.

• 1:17:34Dwarkesh Patel?

  So it would be seven minutes per task, basically, which for, you know, people have tried to do these estimates of how many flaws does a human brain have? And you can take them with a grain of salt, but as a sort of anchor, it's basically the amount of flaws that H100 has. And I guess maybe you would argue that, well, a human brain can solve this question in faster than 7.2 minutes. So even with a 10th of the compute, you should be able to do it in seven minutes. Obviously, we have less memory than, you know, like petabytes of fast access memory in the brain. And with these, you know, 29 or whatever gigabytes in this H100. Anyway, I guess the broader question I'm asking is, I wish there's a way to also test this prize with some sort of scaffolding on the biggest models

• 1:18:19Dwarkesh Patel?

  as a way to test whether scaling is the path to get to solving ARC.

• 1:18:26Speaker 01

  Absolutely. So in the context of the competition, we want to see how much progress we can do with limited resources. But you're entirely right that it's a super interesting open question. What could the biggest model out there actually do on ARC? So we want to actually also make available a private, sort of like one-off track, where you can submit to us a VM. And so you can put on it any model you want. Like you can take one of the largest open source models out there, find you need, do whatever you want, and you just give us an image. And then we run it on an H100 for ARC, 24 hours or something, and you see what you get.

• 1:19:02Speaker 02

  I think it's worth pointing out that there's two different test sets. There is a public test set that's in the public GitHub repository that anyone can use to train. You know, put it in an open API call, whatever you'd like to do. And then there's the private test set, which is the hundred that is actually measuring the state of the art. So I think it is pretty open and interesting to have folks attempt to at least use the public test set and go try it. Now, there is an asterisk on any score that's reported on against the public test set. Because it is public, it could have leaked into into the training data somewhere.

• 1:19:31Speaker 01

  And this is actually what people are already doing. Like, you can already try to prompt one of the best models, like the latest Gemini, the latest GPT-4, with tasks from the public evaluation set. And, you know, again, the problem is that these tasks are available as JSON files on GitHub. These models are also trained on GitHub, so they're actually trained on these tasks. And yeah, that kind of creates uncertainty about if they can actually solve some of the tasks. Is that because they memorized the answer or not? You know, maybe you would be better off trying to create your own private ARC-like, very novel test set. Don't make the task difficult, don't make them complex, make them very obvious for humans.

• 1:20:15Speaker 01

  But make sure to make them original as much as possible, make them unique, different, and see how much your GPT-4 and so on, or GPT-5 does on them.

• 1:20:24Dwarkesh Patel?

  Well, they're having tests on whether these models are being overtrained on these benchmarks. Scale recently did this, where on the GSM.

• 1:20:32Speaker 02

  That's really interesting.

• 1:20:33Dwarkesh Patel?

  AK, they basically replicated the benchmark with different questions. And so some of the models actually were extremely overfit on the benchmark, like Mistral and so forth. And but the frontier models, Claude and GPT, actually did as well on their novel benchmark or that they did on the specific questions that were in the existing public benchmark. So I would be relatively optimistic about them just sort of training on the JSON. I was joking with Mike that you should allow API access, but sort of keep an even more private validation set of these ARC questions. And so allow API access, people can sort of play with GPT-4 scaffolding to enter into this contest. And if it turns out, maybe later on, you run the validation set on the API.

• 1:21:18Dwarkesh Patel?

  And if it performs worse than the test set that you allowed the API access to originally, that means that OpenAI is training on your API calls and you like go public with this and show them like, oh, my God, they're, you know, they've like leaked your data.

• 1:21:31Speaker 02

  We do want to make, we want to evolve the ARC dataset. Like, that is, that is a goal that we want to do. I think Francois, you mentioned, you know, it's not perfect.

• 1:21:37Speaker 01

  Yeah, no, ARC is not perfect, the perfect benchmark. I mean, I made it like four years ago, over four years ago, almost five now. This was in a time before LLMs. And I think we learned a lot, actually, since about what potential flaws there might be. I think there is some redundancy in the set of tasks, which is, of course, against the goals of the benchmark. Every task is supposed to be unique in practice. That's not quite true. I think there's also, every task is supposed to be very novel, but in practice, they might not be. They might be structurally similar to something that you might find online somewhere. So we want to keep iterating and release an ARC 2 version later this year.

• 1:22:18Speaker 01

  And I think when we do that, we're going to want to make the old private test set available. So maybe we won't be releasing it publicly, but what we could do is just create a test server where you can query, get a task, you submit a solution, and of course, you can use whatever frontier model you want there. So that way, because you actually have to query this API, you're making sure that no one is going to by accident train on this data. It's unlike like the current public ARC data, which is literally on GitHub. So there's no question about whether the models are actually trained on it. Yes, they are, because they're trained on GitHub. So by sort of like getting access to querying this API, we'd avoid this issue.

• 1:23:00Speaker 01

  And then we would see, you know, for people who actually want to try whatever technique they have in mind, using whatever resources they want, that would be a way for them to get an answer.

• 1:23:10Dwarkesh Patel?

  I wonder what might happen. I'm not sure. One answer is that they've come up with a whole new algorithm for AI with something, some explicit program synthesis that now we're on a new track. And another is they did something hacky with the existing models in a way that actually is valid, which reveals that maybe intelligence is more of getting things to the right part of the distribution, but then it can reason. And in that world, I guess that will be interesting. And maybe that'll indicate that, you know, you had to do something hacky with current models. As they get better, you won't have to do something hacky.

• 1:23:45Dwarkesh Patel?

  I'm also going to be very curious to see how these multimodal models, if they will perform natively much better at ARC-like tests.

• 1:23:52Speaker 02

  If ARC survives three months from here, we'll up the price. I think we're about to make a really important moment of like contact with reality by blowing up the price, putting a much big price pool against it. We're going to learn really quickly if there's like low-hanging fruit of ideas. Again, I think new ideas are needed. I think anyone listening to this might have the idea in their head. And I'd encourage everyone to like, give it a try. And I think as time goes on, that adds strength to the argument that like we've sort of stalled out in progress and that new ideas are necessary to beat ARC.

• 1:24:19Speaker 01

  Yeah, that's the point of having a money prize is that you attract more people, you get them to try to solve it. And if there's an easy way to hack the benchmark that reveals that the benchmark is flawed, then you're going to know about it. In fact, that was the point of the original KARL competition back in 2020 for ARC. I was running this competition because I had released this dataset and I wanted to know if it was hackable, if you could cheat. So there was a small money prize at the time, that was like 20K. And this was right around the same time as GPT-3 was released. So people, of course, tried GPT-3 on the public data. It scored zero.

• 1:24:57Speaker 01

  But I think what the first context, the first contest told us is that there is no obvious shortcut. And well, now there's more money, there's going to be more people looking into it. Well, we're going to find out, we're going to see if the benchmark is going to survive. And you know, if we end up with a solution that is not like trying to brute force the space of possible ARC tasks that's just trained on core knowledge, I don't think it's necessarily going to be in and by itself AGI, but it's probably going to be a huge milestone on the way to AGI.

• 1:25:36Speaker 01

  Because what it represents is the ability to synthesize task, a problem-solving program from just two or three examples. And that alone is a new way to program. It's an entirely new paradigm for software development where you can start Programming potentially quite complex programs that will generalize very well. And instead of programming them by coming up with the shape of the program in your mind and then typing it up, you're actually just showing the computer what add, what you want, and you let the computer figure it out. I think that alone is extremely powerful.

• 1:26:18Dwarkesh Patel?

  I want to riff a little bit on what kinds of solutions might be possible here and which you would consider defeating the purpose of ARC and which are valid. Here's one I'll mention, which is my friends Ryan and Buck stayed up last night because I told them about this and they were like, oh, of course. Good, thank you for spreading the word. Of course, I'll solve this. And then so they were trying to prompt, I think Claude Obis on this and they say they got 25% on the public ARC test. And what they did was have other examples of some of the ARC tests and in context, explain the reasoning of why you went from one output to another output and then now you have the current problem.

• 1:26:59Dwarkesh Patel?

  And I think also maybe expressing the JSON in a way that is more amenable to the tokenizer. And another thing was using the code interpreter. So I'm curious, actually, if you think the code interpreter, which keeps getting better as these models get smarter, is just the program synthesis right there. Because what they were able to do was the actual output of the cells, the JSON output, they got through the code interpreter, like write the Python program that gets right output here. Do you think that the program synthesis kind of research you're talking about will look like just using the code interpreter in large language models?

• 1:27:35Speaker 01

  I think whatever solution we see that will score well is going to probably need to leverage some aspects from deep learning models and LLMs in particular. We've shown already that LLMs can do quite well. That's basically the Jack Call approach. We've also shown that pure discrete program search from a small DSL does very, very well. Before Jack Call, this was the state of the art. In fact, it's still extremely close to the state of the art and there's no deep learning involved at all in these models. So we have two approaches that have basically no overlap that are doing quite And they're very much at two opposite ends of one spectrum. Where on one end, you have these extremely large banks of millions of vector programs, but very, very shallow recombinations, like simplistic recombinations.

• 1:28:20Speaker 01

  And on the other end, you have very simplistic DSLs, very simple, like 100 or 200 primitives, but very deep, very sophisticated program search. The solution is going to be somewhere in between, right? So the people who are going to be winning the ARC competition and who are going to be making the most progress towards near term are going to be those that manage to merge the deep learning paradigm and the discrete run search paradigm into one elegant way. You know, you asked like, what would be legitimate and what would be cheating, for instance? So I think you want to add a code interpreter to the system. I think that's great, that's so legitimate.

• 1:29:00Speaker 01

  The part that would be cheating is try to anticipate what might be in the test set, like brute force the space of possible tasks and then train a memorization system on it. And then rely on the fact that you're generating so many tasks, like millions and millions and millions, that inevitably there's going to be some overlap between what you're generating and what's in the test set. I think that's defeating the purpose of Benchmark, because then you can just solve it with that and you need to adapt just by fetching a memorized solution. So hopefully ARC will resist to that, but you know, nothing, no Benchmark is necessarily perfect. So maybe, maybe there's a way to hack it. And I guess we're going to get an answer very soon.

• 1:29:40Dwarkesh Patel?

  Although I think some modified tuning is valid because these models don't natively think in terms of, especially the language models alone, which the open source models that they would have to use to be competitive here, compete here, they're, you know, they're like natively language. So they'd like, need to be able to think in the, in this kind of, the ARC type way.

• 1:29:57Speaker 01

  You want to input core knowledge, like ARC-like core knowledge into the model, but surely you don't need tens of millions of tasks to do this. Like cornerless is extremely basic.

• 1:30:08Dwarkesh Patel?

  If you look at some of these ARC-type questions. I actually do think they rely a little bit on things I have seen throughout my life. And for the same, like, for example, like, something bounces off a wall and comes back and you see that pattern. It's like I played arcade games and I've seen like Pong or something. And I think, for example, when you see the Flynn effect and people's intelligence is measured on Rovens progressive matrices increasing on these kinds of questions. It's probably a similar story where since now since childhood, we actually see these sorts of patterns in TV and whatever spatial patterns. And so I don't think this is sort of core knowledge. I think actually this is also part of the quote unquote fine tuning that humans have as they grow up of seeing different kinds of spatial patterns and trying to pattern match to them.

• 1:30:53Speaker 01

  I would definitely file that under core knowledge. Like, uh, core knowledge includes basic physics, for instance, bouncing or trajectories. That would be included. But yeah, I think, I think you're entirely right. The reason why as a human, you're able to quickly figure out the solution is because you have this set of building blocks, this set of patterns in your mind that you can recombine.

• 1:31:12Dwarkesh Patel?

  Is core knowledge required to attain intelligence? Any, any algorithm you have, does the core knowledge have to be in some sense hardcoded? Or can even the core knowledge be learned through intelligence?

• 1:31:21Speaker 01

  Core knowledge can be learned. And I think in, in the case of humans, uh, some amount of core knowledge, uh, is something that you're born with. Like, we're actually born with a small amount of, uh, knowledge about the world we're going to live in. Uh, we are not blank slates. Uh, but most core knowledge is acquired through experience. But the, the thing with core knowledge is that it's not going to be, uh, acquired like, for instance, in school. It's actually acquired very, very early in the first, like, three to four years of your life. And by age four, you have all the core knowledge you're going to need as an adult.

• 1:31:52Dwarkesh Patel?

  Okay, interesting. Uh, so I mean, on, on the prize itself, I'm super excited to see both the open source versions of maybe with a llama, uh, 70B or something where people can score in the competition itself than if to sort of. Test specifically the scaling hypothesis. I'm very curious to see if you can prompt on the public version of ARC, which I guess you won't be able to submit to this competition itself. But I'd be very curious to see if people can sort of crack that and get ARC working there. And if that would update your views on AGI.

• 1:32:22Speaker 02

  It's only be motivating. We're going to keep running the contest until somebody puts a reproducible open source version into public domain. So even if somebody privately beats the ARC eval, we're going to still keep the price money until someone can reproduce it and put the public reproducible version out there.

• 1:32:36Speaker 01

  Yeah, exactly. Like, the goal is to accelerate progress towards AGI. And a key part of that is that any sort of meaningful bits of progress needs to be shared, needs to be public. So everyone can know about it and can try to iterate on it. If there's no sharing, there's no progress.

• 1:32:51Dwarkesh Patel?

  What I'm especially curious about is sort of disaggregating the bets of like, can we make an open version of this versus is this a thing that's just possible with scaling? And we can, I guess, test both of them based on the public and the private version.

• 1:33:06Speaker 02

  We're making contact with reality as well with this, right? We're going to learn a lot, I think, about what the actual limits of the compute were. If someone showed up and said, hey, here's a closed source model that like I'm getting 50% plus on, I think that would probably update us on like, okay, perhaps we should increase the amount of compute that we give on the private test set in order to balance. You know, some of the decisions initially are somewhat arbitrary in order to learn about, okay, what do people want? What does progress look like? And I think both of us are sort of committed to evolving it over time in order to be the best or the closest to perfection that we can get it.

• 1:33:32Dwarkesh Patel?

  Awesome. And where can people go to learn more about the prize and maybe give their hand at it?

• 1:33:36Speaker 02

  ARCprize.org, which goes live today. So It's live now.

• 1:33:39Dwarkesh Patel?

  $1 million is on the line, people.

• 1:33:41Speaker 02

  Good luck.

• 1:33:42Dwarkesh Patel?

  Thank you, guys, for coming on the podcast. It's super fun to go through all the cruxes on intelligence and get a different perspective. And also to announce a prize here. So this is awesome.

• 1:33:49Speaker 02

  Thank you for helping break the news.

• 1:33:51Speaker 01

  Thank you, Funniness.