How François Chollet Is Building A New Path To AGI

François Chollet has spent years asking a different question than most of the AI world. Instead of scaling what already works, he’s trying to understand what intelligence actually is—and how to build it from first principles. In this episode of Lightcone, he traces that path from his early work on deep learning to the creation of the ARC prize, and the launch of ARC V3, a new benchmark designed to measure something deeper than performance: the ability to learn, adapt, and reason efficiently in entirely new environments. He explains why today’s systems may be hitting limits, what recent breakthroughs really mean, and why reaching true general intelligence may require a fundamentally different approach.00:00 - AGI by 2030?00:31 - Introducing Ndea: A New Path Beyond Deep Learning01:08 - A New ML Paradigm 01:30 - Replacing neural nets with compact symbolic programs03:04 - Why Ndea Isn’t Competing With Coding Agents05:20 - Why Everyone Might Be Wrong About Scaling LLMs07:22 - Why Coding Agents Suddenly Work So Well08:50 - The Limits of LLMs in Non-Verifiable Domains10:48 - What AGI Actually Means (And Why Most Definitions Are Wrong)13:30 - Why Deep Learning Hits a Wall 14:00 - ARC’s Origin Story18:20 - ARC Benchmarks Explained: From V1 to V322:49 - The RL Loop Powering Coding Agents Today27:03 - ARC-AGI V3: Measuring “Agentic Intelligence”31:14 - Inside the ARC Game Studio35:31 - Could AGI Fit in 10,000 Lines of Code?44:01 - Building Ndea: From Idea to Compounding Research Stack46:46 - The Future of ARC: Benchmarks That Evolve With AI47:21 - Why There’s Still Huge Opportunity for New AI Paradigms53:37 - How to Build a Breakout Open Source Project - Lessons From Kera56:39 - Advice For How To Think About AIApply to Y Combinator: https://www.ycombinator.com/applyWork at a startup: https://www.ycombinator.com/jobs

• 0:00Speaker 00

  I think we're probably looking at AGI 2030. Around the time the two are going to be releasing like maybe ARC 6 or ARC 7. You're not going to stop AI progress. I think it's too late for that. And so the next question is, OK, like AI progress is here. It's actually going to keep accelerating. How do you make use of it? How do you leverage? How do you ride the wave? That's the question to ask.

• 0:31Speaker 01

  Today, we're lucky to be joined by François Chollet, founder of the ARC Prize, a global competition to solve the ARC AGI benchmark. His latest project is NDA, a lab exploring a new paradigm in frontier AI research. François is one of the best people in the world to help us understand the current AI moment and where all of this is going. François, thank you so much for joining us today and congrats on the launch of ARC AGI V3.

• 0:58Speaker 00

  Thanks so much for having me. I'm super excited to be here. Super exciting time to talk about AI. So François, tell us a little bit about NDA. So what exactly is it and what are you guys trying to achieve? Right, so NDA is this new AGI research lab. And we are trying some very different ideas. And so our goal is basically to build this new branch of machine learning that will be much closer to optimal, unlike deep learning.

• 1:23Speaker 01

  All of us right now are sort of taken by what's going on with code. I have sort of this viral moment right now where I got to 40,000 stars this morning on GStack. So it's like, oh, this is an open source project that now is one of the biggest ones. And I have more than 100 PRs from contributors to deal with. I guess you're, you know, one of the best people to talk to about this because you're actually literally coming up with something that is a totally different pathway.

• 1:51Speaker 00

  That's right, that's right. So what we're doing at NDA is we're doing programs in this research. And when I talk about program synthesis, often people ask me, oh, so are you doing like code gen, are you building an alternative to coding agents? And it's actually not at all what we are doing. We are working at a much, much more, much lower level than that. What you're actually doing is that you are trying to build a new branch of machine learning, an alternative to deep learning itself, rather than like coding agents. Coding agents are like this very, very high level, last layer piece of the stack. And we're actually trying to rebuild the whole stack on top of different foundations. So we're building a new learning substrate that's very different from, you know, parametric learning, deep learning.

• 2:35Speaker 00

  So if you go back to the problem of machine learning, you have some input data, some target data, and you're trying to find a function that will map the inputs to the targets that will hopefully generalize to new inputs. And if you're doing deep learning, what you're doing is that you have this parametric curve that serves as your, as your function, as your model, and you're trying to fit the parameters of the curve via gradient descent. And this is basically what we are doing, except we are replacing the parametric curve with a symbolic model that is meant to be as small as possible. It's like the simplest possible model to explain the data, to model what's going on.

• 3:17Speaker 00

  And of course, if you're doing that, you cannot apply gradient descent anymore. So we are building something that we call symbolic descent, which is like the symbolic space equivalent of gradient descent. The idea is to build this new machine learning engine that's giving you extremely concise symbolic models of the data you're feeding into it, and then we're going to make it scale. And so everything you're doing with machine learning today, with parametric curves, we should be able to do it with symbolic models in the future in a, in a way that will be much, much closer to optimality.

• 3:55Speaker 00

  Much closer to optimality in the sense that you're going to need much less data to obtain the models. The models are going to run much more efficiently at inference time because they're going to be so small. And because they are so small, they will also generalize much better and compose much better. You know, the the minimum description length principle, that the model of the data that is most likely to generalize is the shortest. And I think you cannot find a model like this if you're doing pattern learning. You need to you need to try somebody. That's fascinating.

• 4:24Speaker 02

  So the rest of the industry is just pouring more and more billions of dollars down an approach that was set years ago. Can you like help make the case for why you think that it's the right thing to explore alternate approaches instead of just to keep putting more money into the current approach?

• 4:39Speaker 00

  I mean, everybody is is, you know, building on the L L M stack these days, which makes sense because, you know, the the returns are there, like it's actually working. So it would seem very sensible for everybody to just be doing what seems to be the the currently most proactive path. But our take is actually it's counterproductive to have everybody working on the same thing. Like, I personally don't think that machine learning or A I in fifty years is still going to be built on this stack. I think this is a stack that is very nice, maybe it even gets us to H I. But it's not as efficient as it should be. I think it's inevitable that the world of A I will trend over time towards optimality.

• 5:21Speaker 00

  And so I'm trying to sort of like leapfrog directly to optimality, like to build to build the foundations of optimal A I today. But in general, you know, our vision is very ambitious. And I'm not saying that we're going to be successful, like we have maybe a ten or fifteen percent chance of success. But that is enough that it's worth trying, right? And I think in general, like among among listeners, if you have a big idea and it has very low chance of success, but if it works, it's going to be big and no one else is going to be working on it, right? It's it's not something popular. It's not something if you don't do it, no one else will do it. And this is basically our situation. If you're in this situation, then you should try a chance. You know, should go and work on it.

• 6:04Speaker 02

  I mean, that's almost like the mission statement of Y Combinator, the thing that you just said.

• 6:08Speaker 00

  Yeah. Yeah, the reason it's important is that, again, if we don't do it, no one else will do it. Right. So it's worth trying, even if we don't succeed, it's worth trying.

• 6:15Speaker 03

  Has the success, well, very specifically of the coding agents, I guess, built on top of the L M stack. Like, has their success surprised you at all? And in particular, like, say, over the last six months or so?

• 6:26Speaker 00

  Yeah, absolutely. I think it has surprised many people. It definitely did surprise me. If you look at why everything is starting to work so well with coding agents, it's really because code provides you with a verifiable reward signal. And I think right now we're in this situation where any problem where the solutions you propose can be formally verified and you can actually trust a reward signal. It's not just some guess made by a model. Any domain like this can be fully automated with current technology, with the L M based stack. And code is sort of like the first domain to fall, but there will be many others in the future. I think mathematics is also primed to see a revolution in the next few years for the same reasons. Again, because the domain just gives you verifiable rewards.

• 7:11Speaker 04

  I guess the challenge for a formally verified domain is you have to

• 7:17Speaker 04

  somehow take a domain and make it verifiable, which is the trick. I mean, code is very natural. You could test there's bugs, compiles, et cetera, and mathematics as well, whether all the theorems and proofs work out. I guess it becomes more nebulous when you go a couple of degrees off where there are fields that are not naturally formally verified. And you need to come with a, again, with some sort of a function to come up with that reward that makes it verifiable.

• 7:46Speaker 00

  Yeah.

• 7:47Speaker 04

  With very fuzzy things like, let's say, English language and composing the perfect essay. How do you make that formally verifiable?

• 7:55Speaker 00

  Yeah, yeah, absolutely. I mean, writing SS is, you know, the typical example of a domain that's not verifiable. And so what you're going to see is that progress of reasoning models and base LLMs on this type of domain is, you know, is going to be very slow. Because the stack we're using, like the LLM stack, is very, very reliant on its train data. It's basically just operationalizing the train data. And for writing SS, the train data is coming from human experts, like annotating answers. And that's costly. So you're going to see this very, very slow progress, maybe, maybe it's even going to stall. But for any, any verifiable domain, like take code, for instance, what was the big unlock is

• 8:37Speaker 00

  when, when people started creating this code-based, like, training environment for, for post-training, where the, the, the reward signal, the verification signal is provided by things like unit tests and so on. And so that means that the model was not just working from human-generated annotations. It was actually trying some things, verifying the answer and generating a lot, lot more train data in the process, a much denser coverage of the problem space. And not just coverage in terms of like, is, is the answer right or wrong? But also starting to build models of the execution traces, right? So that the models could start incorporating a, an execution model, very much the way that

• 9:22Speaker 00

  human programmers, you know, when they look at code, they're, they're sort of like executing the code in their minds, they, they keep track of value variables and so on. It's also what the models are trying to do now, and this is why it's working so well. And it's possible because you're working with this very formal, fully verifiable environment. You cannot do that with SS, you cannot do that with, you know, LAR or many other problems.

• 9:41Speaker 04

  I think I really like how you define intelligence and how to measure it, which brings to the question of also sharing, having you share. The history of AGI.

• 9:52Speaker 00

  Yeah, so my definition of general intelligence. You know, many people around the industry these days, they say AGI is going to be a system that can automate most economically economically valuable tasks. And to me, that definition is it's about automation. It's not about intelligence, it's not about general intelligence. So my definition is AGI is basically going to be a system that can approach any new problem, any new task in a new domain and make sense of it, like model it, become competent at it with the same degree of efficiency as a human could. So meaning it's going to need basically the same amount of training data and training

• 10:35Speaker 00

  compute as a human would, which is which is very little, like humans are really, really data efficient. So general intelligence is human level skill acquisition efficiency on the on the same scope of tasks that humans could potentially learn to do.

• 10:52Speaker 02

  Do you think it's possible that we will accomplish the first definition of AGI, the automate most economically useful work, before we accomplish your definition?

• 11:01Speaker 00

  Absolutely, I think that's that's a trajectory that we're on right now. And I think it's already true that in principle, current technology can fully automate at human level or beyond any domain where you have very favorable rewards, right? And code code being the first one. And I think if you're going at AGI, if you're going at like human level, you know, learning efficiency over arbitrary tasks, that's probably going to take a different sort of technology, a different mindset, a different approach.

• 11:29Speaker 02

  Do you think that L M's can be bent to have the same sample efficiency as humans? Or do you think it's like fundamentally just impossible and we need a new approach? And that's that's the thing that you're hoping hoping to solve?

• 11:41Speaker 00

  With enough compute, everything starts looking like everything else. Every like computers are great equalizer, every approach starts looking the same. And I think it's possible in principle to build something that looks a lot like AGI on top of the L L M stack. But it's not going to be LLMs per se. It's going to be this new layer. Perhaps it's going to be even a few layers above, not just one layer above, but a few layers above. But you can build it on top of LLMs because LLMs are a kind of computer, right? I do believe, however, this would be the wrong thing to do because it would be very inefficient. I think AI research will have to trend towards not just efficiency, but in fact, optimality over time. And for this reason, future AI in a few decades,

• 12:26Speaker 00

  it's not going to be this harness on top of reasoning model, on top of a Bayesian LLM. It's going to be much, much lower than that.

• 12:35Speaker 02

  To Diana's question, do you want to talk about how you actually designed ArcAGI and why it's a good barometer of that?

• 12:40Speaker 00

  I mean, I've been doing deep learning for a very, very long time. And initially, my tech, my mindset was that deep learning was going to be able to do everything.

• 12:50Speaker 04

  You were the creative at Keras before even all the other frameworks became very popular?

• 12:55Speaker 00

  That's right. That's right. I was training the planning model for natural language processing, in fact, in 2014. And from that work, you know, I actually started developing this open source library, which I released, in fact, exactly 11 years ago, March, March 2015. So it was Keras. And then it got popular and then I ended up sort of like doing less of the research that I started Keras for and more working on the framework itself just because it had really, really good product market fit. And so my tech, you know, around that time, around like 2015, 2016, was that deep learning was extremely general, that you could do everything with deep learning, that you didn't need anything else. It was train complete.

• 13:40Speaker 00

  So my tech was basically a deep learning was differentiable programming. So anything you would do with software, you could in principle, train a deep learning model on the right inputs and outputs to do the same thing. And in 2016, I was doing research at Google Brain on trying to train deep learning models to help with reasoning problems. And in particular, first order logic problems, theorem proving, and so on. And I started finding that you could not really get gradient descent to encode sort of like reasoning style algorithms. It was not because the models could not represent these algorithms.

• 14:25Speaker 00

  It was because gradient descent could not find them, right? So the problem was that it wasn't about deep learning not being trained completely or anything like that. Like, that was not the problem. The problem was gradient descent, right? Gradient descent would not find generalizable programs. It would instead end up doing overfit pattern matching, right? Over sequences of input tokens.

• 14:47Speaker 01

  I guess people could argue like that's what's happening.

• 14:50Speaker 00

  I mean, this is what's happening today in a slightly it's a slightly higher level version of it.

• 14:57Speaker 04

  Just with a lot of data. So it doesn't feel like overfitting because the data has a lot more distribution.

• 15:01Speaker 00

  Yeah, with a lot more data. And also, I think models today, they're a lot more compressive after that. Which is why they generalize better.

• 15:08Speaker 01

  All models are wrong, but some models are useful. And then I guess what I'm hearing is like, your method might find the right model.

• 15:16Speaker 00

  That's right. That's where the idea came from. And I was like, you know, at the time, back in 2016, 2017, I was like, okay, we're going to need a benchmark to capture these ideas. We're going to need a program synthesis benchmark. And my mental model for that was ImageNet. I was like, oh, I'm going to make the ImageNet of reasoning. So I started brainstorming a few ideas around like 20s, 2017. I explored many different things. I tried working with, in particular, Sylar Automata, like a setup where you show a model Sylar Automata outputs and it must recreate the program that generated them, like that sort of thing.

• 15:57Speaker 00

  And eventually I settled on the Arc CGI format around like early 2018. You know, I was doing this on the side. It was a side project, like my main project was developing Keras at Google. I wasn't moving very, very fast on that. So summer 2018, I wrote the Arc Task Editor.

• 16:18Speaker 00

  And then I started just making lots of tasks by hand. And about one year later, I had made one thousand tasks. And so I wrote up the paper that was explaining what this was about, what the big idea was, like intelligence as a scale acquisition efficiency. And I published all of that in twenty nineteen.

• 16:36Speaker 04

  In parallel, G P T three twenty twenty was coming out and starting to show signs. Until the chat G P T moment around twenty twenty two, end of the year. And the industry took off with that. And this was one of the benchmark that was really performing really badly. And it was very obscure. I don't think many people knew about it. It was mostly niche research communities that maybe read your paper.

• 17:00Speaker 00

  Yeah, people who worked on programs knew about it. But a lot of people who worked on on deep learning and scaling up L L M stayed very careful. And part of the reason why is because L L M did not work well at all on the benchmark. For a benchmark to capture the attention of the research community, it needs to start working a little. If it's too hard, people are just gonna dismiss it.

• 17:24Speaker 01

  You're just ahead of your time, clearly, because we're not on Arc A G I one anymore. And then two is reaching saturation. And then

• 17:33Speaker 00

  That's right.

• 17:33Speaker 01

  three is out now.

• 17:35Speaker 00

  Yes.

• 17:36Speaker 04

  And I think the cool thing about Arc A G I, it has been a very good parameter for the industry of the big changes that happen. Because V one was not working at all for a long time until twenty twenty five, when reasoning models came out, right?

• 17:55Speaker 00

  Yeah, absolutely. If you look at frontier performance on Arc V one first and then V two. So base L L M's were scoring extremely low on V one, like sub ten percent, basically. And I mean, it was true of the original lag G P T three, I was just scoring zero. But that's even true of the latest base L L M's today, you know, as of as of March

• 18:18Speaker 01

  Without reasoning.

• 18:19Speaker 04

  Without reasoning.

• 18:19Speaker 00

  Without reasoning, yeah. So the base models. So performance of base L L M's on on V one stayed very, very low, even though in the meantime, you know, we had scaled up these models by fifty thousand X, right? So it was retaining you that, you know, more scale, scaling up pre-training alone was not gonna crack the benchmark. This was not enough. To demonstrate that the model had fluid intelligence. And then the moment models starting performing well on Arc One was with the first reasoning models, in particular, the OpenAI O1 and then O3 models, which by the way, they were demonstrated by OpenAI on Arc, because it was the one unsaturated reasoning benchmark

• 19:02Speaker 00

  that was really showing that this model was different, that had new capabilities that we had not seen before. And so with reasoning models, you start seeing this sudden like step function change on Arc One. And so Arc One was really the benchmark that signaled that at this moment in time, something was happening.

• 19:20Speaker 04

  Something big.

• 19:21Speaker 00

  Yeah, something big, like new capabilities were emerging, like reasoning was new and different. And it was actually not obvious at the time, like, you know, I don't know if you remember when when O3 preview was was announced by OpenAI.

• 19:36Speaker 04

  That was end of 2024, actually.

• 19:37Speaker 00

  Yeah, December 2024. And like, sure, it was like a huge step function progress on Arc, but it was very expensive. It didn't actually have product market fit effectively. But if you looked at Arc results, you knew that this was big and important. And then we released Arc Two, which was the same format, but more difficult, like with more composition at the level of the reasoning chains. And what happened is that so the the earliest reasoning model started very, very low on Arc Two. And then around the same time, as coding agents started working, you saw this.

• 20:16Speaker 00

  Yeah, so very, very recent, just a few months ago, you saw this very, very fast like saturation of Arc Two. And so again, like Arc Two signaled that, yes, there was this this new set of capabilities emerging. So I think the benchmark did a really good job at capturing the advent of reasoning models and then the advent of agentic coding, like this this new paradigm where if you have very valuable rewards, then you can basically fully automate the domain, which by the way, is true of Arc, like Arc does provide a very valuable reward.

• 20:48Speaker 01

  I guess for V2, what what caused the, so one was clearly reasoning, two, a benchmark doesn't care how you solve it, I guess, embedded in what you said, like. Were people using code gen to then solve?

• 21:01Speaker 00

  That's right. So not necessarily code gen per se, but the frontier labs have been targeting Arc V2. And the progress you saw on Arc V2 is actually a result of this very, very large scale targeting. So what you can do to solve Arc V2 is you ask your reasoning model to make more tasks, like those in the benchmark. And then you try to solve them using, let's say, let's say, program induction, for instance, still using your reasoning model. Then you verify the solution. Again, it's very favorable. So you can, you can trust the answer. And then you fine tune the model on the successful reasoning chains. And then you keep repeating, like you generate new tasks, you solve them, you verify the

• 21:45Speaker 00

  solution, you fine tune the model on the reasoning chains. And you can keep doing this millions of times, right? Like you just need to spend more money.

• 21:54Speaker 01

  This is the RL loop that is just happening.

• 21:56Speaker 00

  Yeah, exactly. And the, the new paradigm in AI is basically that any domain where this is true, where you have the ability to generate these, these true verification signals, you, you can run this, this kind of loop, right? If you can run this kind of loop, you can mine, you can brute force mine effectively the entire space and get extremely high performance. This is basically the, the process through which Arc 2 was saturated. So what it tells you is that it's not so much that the models have higher fluid intelligence than, than they did with the, with the first reasoning models. It's just that you have this new paradigm of post-training. And this is exactly what led to agency coding. So it does matter. It is, it is valuable. It is useful.

• 22:36Speaker 01

  It's not that the models are smarter. It's that they're suddenly more useful. And it's possible to be more useful in particular domains without being smarter.

• 22:46Speaker 00

  Yeah.

• 22:47Speaker 01

  Clearly, because that's, means good things for me. I'm not getting any smarter right now, like at, at, you know, age 45, but you know, I can learn how to do things. And that's sort of what's happening with the models as of like late.

• 23:00Speaker 00

  Yeah, absolutely. When, when it comes to a Competency, there's always a trade out between intelligence and knowledge. If you have more knowledge, if you have better training, you need less intelligence to be competent. And that's exactly what happened with the rise of coding agents, right? The models don't have higher fluid intelligence per se, they don't have like a higher IQ, so to speak. It's just that they're way better trained and they're way better trained in two ways. So they're not just trying to complete code anymore. They're actually trained via trial and error in these high-end posturing environments with true rewards signals. And also they're trained to embed this model of code execution, right?

• 23:43Speaker 00

  Where they learn to keep track of the value of variables over an execution cycle. And that's what's leading to this extremely strong product market fit of urgent coding today. And it's completely changing software engineering.

• 23:57Speaker 04

  This happened not too long ago, the saturation. We actually had the founder of Poetic that came and spoke about the approach, which is really, it sounds like this new way of getting LMs to perform is building this agent harness, right? And the harness is basically structuring a problem domain into something that can be formally verified. And they did that basically for ARC V2, which when they released it, they were at the top of the benchmark. But then the crazy thing is I actually worked with a company in the winter 26 batch not too long ago, called Confluence Lab, which actually ended up saturating the V2 results with 97%. And I think their task cost was a lot more efficient too.

• 24:41Speaker 04

  And the approach they basically took is similar to this. I think they built the harnesses on top of it in order to get the LMs to go and build different tasks and. Programmed through it.

• 24:55Speaker 00

  Yeah.

• 24:55Speaker 04

  Which then, for me, I was like, wow, is this batch? During the batch, they only worked on it for a couple of months and they were able to saturate this benchmark that has been around for a long time. It's like something special is happening.

• 25:07Speaker 00

  Yeah, yeah, there's a lot of progress right now. It's driven by custom harnesses around the task. And the harness is basically a way for the human programmer to input into the model or like a higher level like solution strategies, basically. I mean, to me, the fact that you need humans to engineer these harnesses is also a sign that we're short of AGI today because if we had AGI, you know, AGI would just make its own harness. It would not need to be told how to solve a problem. It would just figure it out. But it is very effective, like harnesses. I don't think they get us closer to AGI in any sense, but it's a very valuable area for search because that can lead to task automation at scale.

• 25:48Speaker 01

  YC's next batch is now taking applications. Got a startup in you? Apply at ycombinator.com/apply. It's never too early and filling out the app will level up your idea. Okay, back to the video.

• 26:02Speaker 04

  Can you tell us about then what V3 is going to measure? That's just got released?

• 26:08Speaker 00

  Yeah, absolutely. So if you look at V1, V2, it was really focusing on your ability to produce like causal models of a pattern that was just given to you, like the data was given to you. So it was static, it was passive and really focused on modeling. And V3 is completely different. We are trying to measure agentic intelligence. So it's interactive, it's active, like the data is not provided to you, you must go get it. The idea is that your agent is dropped into a new environment, which is kind of like a mini video game. And it's not provided any instructions, it's not told what to do, it's not told what the goal even is or what the controls even are.

• 26:54Speaker 00

  And it must figure out everything on its own via trial and error. So we are not just measuring, you know, the E I's ability to model its environment. We're also looking at its exploration efficiency, its ability to acquire goals on its own, like goal setting, and of course, its ability to plan through the model of the environment that's created and to execute the plan. And so together, you know, all of these abilities, we call that agentic intelligence. And we are looking for AI systems that could learn to play these games and, you know, crack them with the same degree of action efficiency as a human. If you look at the human, they are dropped into this new environment.

• 27:37Speaker 00

  They try a few things, they start understanding how things work. They can, they can solve the environment, you know, in a few hundreds to thousands of factions. We're trying to look for AI systems that could match this efficiency. And by the way, we know that all of these test environments in Arc3 are solvable by humans with no prior training, because we actually tested them on, on regular people. Yeah, at first you just see this screen and you, you know, you have these keys available, but you don't know what they do and you must figure out everything from scratch. And humans are really good at that, by the way. They're really good at exploring efficiently, at making sense of something new and eventually cracking the game. And frontier models today, they are not very good at it.

• 28:21Speaker 02

  If the reasoning models cracked V1 and the like reinforcement learning environments cracked V2, do, do we need a new advance to crack V3? Do the, do even the best techniques currently like not work?

• 28:34Speaker 00

  Yeah, I mean, I'm very curious to see how Frontier Labs are going to react to V3 and how they're going to start targeting it. It is designed to be more resistant to the same kind of targeting strategy as what we saw for V2 in particular. Like, of course, you can try to just make more Arc3-like games and then train your agents in them. But the thing is, we've deliberately tried to create a private set of environments that is significantly different from the public set. Like, you can look at the public set, it's not actually giving you that much information about what's in the private set. In the private set, you will have very different games with very different concepts.

• 29:16Speaker 00

  And also, the public set is meant to be substantially easier. Your performance on the public set is not actually, it's not representative of how well the system we do in practice. So for these reasons, it's going to be harder to target. And that makes it a better test of fluid intelligence as opposed to a test of how much effort you put into cracking it.

• 29:35Speaker 02

  I'm so curious, how do you come up with these games? They're so creative.

• 29:38Speaker 00

  Yeah, we set up an entire video game studio, right, to create them. So we got over 250 games. And, you know, they're pretty quick to play, like each game takes you maybe 10 minutes or a bit less to play from scratch, like up on first contact. And we have like 250 plus. And we set up this very productive game studio where we had, any given week, we had multiple games in progress, where like this pipeline, including, you know, design, implementation, review, human testing, and many iteration cycles to make sure that the game comes out right.

• 30:18Speaker 02

  Who's working in the studio?

• 30:20Speaker 00

  Right.

• 30:21Speaker 02

  Who are the creators?

• 30:22Speaker 00

  Yeah, we hired a team of game developers and we built our own game engine.

• 30:26Speaker 02

  Wow, so it's actually people who like previously worked in the game, in the, in the video game industry?

• 30:32Speaker 00

  That's right, that's right. So one thing to be in mind though is that the games in Axiom are unique, right? They're trying to not borrow elements, concepts from previous video games. They're built entirely on top of core knowledge prior, like things like, just, you know, elementary knowledge, like basic physics, understanding of objects, understanding of the notion of agents, for instance, like an agent and objects with goals and intentions. But we are not incorporating any language, any like cultural symbols, like, you know, arrows, for instance. Or the color green meaning go and color red meaning star, that sort of thing. There's no external knowledge that's involved in these games.

• 31:17Speaker 04

  It's like one of those IQ tests that are just pattern matching, but now it has time series.

• 31:20Speaker 00

  Yeah, it's not just time series, it's interactive. You must create your own path through game space, right? You must, you know, in, in, in an IQ test, like, problem, like, you know, what arc one and two is, the data that you must model is provided to you. You already have the data, you just, you just need to find the causal rule to explain it. With R3, you actually must gather the data and you must do so efficiently. Like, of course, you could say, well, I'm just gonna, you know, brute force mine the space of every possible game state. And then I find the solution. You cannot do that because if you try to do that, you would score extremely low, even if you manage to solve the level.

• 32:02Speaker 00

  Because you're scored on your efficiency, you must match human level efficiency.

• 32:06Speaker 01

  It's funny, it's like almost coming full circle. This level of AGI with games sort of is the match pair to OpenAI writing. I mean, you know, Tom Brown, one of the co-founders of Anthropic, had to write like the harness code to allow like the, you know, pre-GPT AI at OpenAI to play Starcraft.

• 32:27Speaker 00

  Yeah, yeah, OpenAI worked on, on, in particular on Dota two. They had the OpenAI five model, which was, if I recall correctly, so this was like not just pre-GPT, but also mostly pre-transformers because they were working with a stack of LSTM. Layers, if I recall correctly. And even before OpenAI, DeepMind worked a lot on video game, you know, solving video games via Deep Isle. And they were the first to do Atari games, right? Back in 2013, that, you know, they were very, very early, very visionary in that sense to work on on this problem so early with these methods, which are still very modern methods. So the big difference is that if you look at.

• 33:11Speaker 00

  At Atari games, for instance, even Dota, you're training on the same environment as what you use for testing. So effectively, you're just trying to memorize the best strategies. You're trying to, at training time, explore the full space of possible game states and productionize, operationalize the knowledge into the model. And then at inference time, you're basically just recalling that knowledge. And that's explicitly what we're trying to avoid with ARC 3. You're not playing games that you've seen before. You're not playing games that you've been trained on like for millions of hours,

• 33:54Speaker 00

  like the OpenAI 5 model, for instance, was playing a restricted version of Dota 2, and it was trained on like tens of thousands of hours of gameplay, effectively. I think maybe in millions, but it's just an insane amount of trained data. With ARC 3, you're being evaluated on games that you're seeing for the very first time. And every action you spend exploring is counted towards your efficiency score, right? So you're really focused on measuring fluid intelligence, your ability to efficiently explore, efficiently produce a world model of the environment, and then use this model to infer goals, plan towards these goals, and eventually crack the game.

• 34:35Speaker 01

  One of the arguments for your NDA is that you're able to do all of the intelligent tasks for, you know, an ARC task might be like 0.3 cents for an ARC task, but for the same task on a foundation model with LLMs, it's a dollar to $10. And then there's this other aspect that we've been tracking where it seems like more and more intelligence, at least on the LLM side, can be distilled down into smaller and smaller models. And so on the one hand, like they're scaling up, but then they're like distilling smarter and smarter small models. I guess your approach might indicate that it's not billions of parameters,

• 35:19Speaker 01

  like the, you know, NDA achieving AGI might not be. It's sort of inherently a scale thing at all. There's a platonic ideal of the NDA model that achieves AGI.

• 35:31Speaker 00

  Yeah.

• 35:31Speaker 01

  Do you ever think about it in terms of like, well, it would fit on a floppy disk?

• 35:35Speaker 00

  Well, okay, there are two things to separate. There's the sort of like fluid intelligence engine. I think it's going to be a very, very small code base and a very small set of models that's suited with it. And it's probably going to be on the order of megabytes, right? And then you have the knowledge base, so to speak, that's going to be layered below this fluid intelligence engine. Like, you know, fluid intelligence has to draw on some knowledge and that knowledge is going to take up a lot more space. I think it's important to differentiate the two. I do believe that, you know, when you create AGI, retrospectively, it will turn out that it's a code base that's less than 10,000 lines of code.

• 36:18Speaker 00

  And that if you had, if you had known about it back in the, in the 1980s, you could have done AGI back then. Using the computer resources available.

• 36:27Speaker 02

  Wow, that's a crazy prediction.

• 36:29Speaker 00

  That's, I think retrospectively, this will turn out to be, to be true.

• 36:33Speaker 02

  Wow, so it was just like hiding under our noses in plain sight for like 40 years. It took us like 40 years to figure it out.

• 36:38Speaker 00

  That's right, that's right.

• 36:39Speaker 01

  Well, that second thing sounds like Douglas Lenat's like psych project. Or is that the wrong way to think about it? It's like there's sort of knowledge about the world. And then there's methods, like the program, what I hear is like, the program might be 10,000 lines and then it operates on like.

• 36:55Speaker 00

  On knowledge base, it's very large. So the problem with psych, I mean, there were many issues with it, but one of the big issues is that there was no learning involved.

• 37:04Speaker 01

  Yeah, it's just the knowledge.

• 37:06Speaker 00

  Like the knowledge was uncrafted.

• 37:07Speaker 01

  It was like purely symbolic knowledge and it was probably inaccurate.

• 37:10Speaker 00

  The way you want to be building AGI is that you want to be removing humans from, from the improvement loop as much as possible. You don't want a system where every improvement in system capability has to involve a human engineer doing something. And it's actually the strength of deep learning and foundation models is that you can just scale up the knowledge base. Like an LLM is effectively a knowledge base. It's a bank of, of, you know, modular vector programs that map. Patterns of input tokens, patterns of output tokens. And you can scale up that knowledge base by just adding training data and training compute with no further human involvement. I mean, of course, there's still a little bit of human involvement in making sure the

• 37:53Speaker 00

  training job completes, but it's minor. You've managed to remove humans from this improvement as much as possible. And that's also what we want for our system. We want a system that's self-improving, where the improvements are compounding, meaning that every time the system increases its capabilities, it's also increasing the rate at which it increases its capabilities.

• 38:15Speaker 01

  I think this is a PGism. It's like, I'm sorry, the essay is so long. If I had more time, I would make it shorter.

• 38:22Speaker 00

  Yeah. When you're looking at a hard problem, it's actually harder to produce a short, elegant, concise solution than a messy, over-engineered solution.

• 38:31Speaker 01

  Yeah, you can brute force it, but you know, the more elegant version is very, very short. And that's kind of like what you said with how this might come about.

• 38:39Speaker 00

  This is, yeah, this is literally the shape of the type of AI approach we are creating. And I think this is also the shape of science itself. Like, science is fundamentally a symbolic compression process where you're looking at a big mess of observations, like, you know, the position of planets in the sky or something like that. And you're compressing that down to a very simple symbolic rule. You're saying like, yeah, like, all this, you know, thousands of observations, actually just all this one simple equation. That's symbolic compression. And to do this, by the way, you need the model to be symbolic.

• 39:20Speaker 00

  Like, you could not fit a curve and say, well, you know, that curve is my model. That would never be optimal. It would never be concise or elegant enough. And that's not what science is doing. Science is not about curve fitting. Science is about finding the equation, finding the most compressive symbolic model of your pile of observation. And that's the process that you're trying to recreate in software form. Like you could say that the NDI approach to program synthesis is that we are building science incarnate, science, the scientific method in algorithmic form.

• 39:51Speaker 01

  I'm curious if you compare it to.

• 39:54Speaker 02

  Biology. Clearly, L L M's don't learn the way that humans do because no baby reads the whole internet. Do you think program synthesis is closer to the way that humans learn? Or do you think that's yet a third branch where even if program synthesis is correct, there will be some yet as undiscovered third way to do it, which is the thing that we do?

• 40:13Speaker 00

  I think so. I do think humans do some amount of program synthesis. I think the way humans learn and the way the human mind works is very messy. It's not like there's one simple elegant principle behind it all. It's an implementation of fundamental principles, the fundamental principles of intelligence, which, you know, I think we can identify these principles and re-implement intelligence from scratch, from first principles, in a way it will be much more efficient than the human brain. I think the human brain is messy and it's, it can be a good source of inspiration for AI, but I think it would be counterproductive to just try to, you know, observe it and

• 40:54Speaker 00

  re-implement it like, and, and, and make it biologically plausible. I think that's counterproductive. It's not what I'm trying to do at N D A. We're only trying to find what are the first principles of intelligence and what is the system that would best implement them. But yeah, I do believe the human mind does at the highest level, something that looks a lot like program synthesis, like we're currently building causal models of our surroundings. Like we're, we're describing our surroundings in our mind as, you know, a sort of objects and agents and relations between objects that are fundamentally symbolic and causal in nature. This is exactly the process that lets us generalize so well and adapt so well to novelty

• 41:39Speaker 00

  on the fly.

• 41:40Speaker 03

  I'm curious about N D A, the company, and as you, as you're building it. We've all here heard of the OpenAI founding story. And something that's always struck with me is just like both Sam and Greg say that it was a little odd in the early days because they didn't actually know what to do. It's sort of just like a bunch of people like hanging out in an apartment. I would love to hear kind of what's that been like for India? Like, what did like the day one look like? And just maybe for just people who are interested in starting these alternative approaches who don't have sort of a researchy background. How should they think about that?

• 42:11Speaker 00

  Yeah, so we started on day one with the symbolic learning vision. Like, we basically knew that we wanted to do a symbolic program synthesis that we wanted to create a new approach to machine learning, where you replace parameter curves with the shortest possible symbolic models. And then the big question was, okay, so how do we find these models? We started from the base idea, which is still the idea that we're following today, which is that we are going to do deep learning guided program search that you have a symbolic search space to explore. And it's big, it's in fact commutative all. You're not going to make progress if you just use brute force. It's not going to scale. You have to break the commutative wall.

• 42:56Speaker 00

  And the way to do it is to add is to add deep learning guidance. It's actually very similar to the principles that underlies something like AlphaGo or AlphaZero. That was our starting point. We also, you know, didn't have very clear ideas about how to build it. So we tried many different things. We tried many, many different ideas. And it took us half a year roughly to get to good foundations, where we could start building a system that compounds. And I think that's what's really important when doing a lab like this, that you don't want to be in a situation where you're constantly trying something new. It's not reusing any learnings, any findings from the previous approaches. You want to you want a compounding stack.

• 43:40Speaker 00

  You want to build reusable foundations and then the next layer and then the next layer. And of course, you want to be building on to the right foundation. So don't commit to the foundation layer too early, but also make sure that at some point you're building this compounding structure. And that's the situation that we're in now.

• 43:59Speaker 03

  Is Arc three the end or will there be an Arc four, five, six? Can you keep making it harder?

• 44:04Speaker 00

  Yeah, yeah, I think there will absolutely be Arc four and Arc five. I mean, we're currently planning Arc five. The point of the Arc AI Benchmark series is not to say that, well, you know, here's this test. If you pass it, this is a GI. Instead, what you're trying to do is we are targeting the residual gap of their capabilities. Like Frontier is advancing and we're saying, well, if you compare it to human abilities, there's all these tasks, all these things, it's not doing well. So we're going to create a benchmark to target that. And so it's a moving target, right? It's not fixed points, moving targets. There will be Arc 4, which will be in the spirit

• 44:46Speaker 00

  of Arc 3, but more focused on continual learning and curriculum learning at longer time scales. So you're going to have fewer games, but they're going to have way more levels and the levels are going to be compounding, meaning that for each level you need to reuse stuff that you've learned before. And then that's going to be Arc 5. And I'm actually really, really excited with Arc 5. It's very, very new and different. And it's all about invention. And I mean, you will see, you will see what that means. Eventually, I expect we will run out of things to test. Like as, as we get closer to AGI, eventually there will be no measurable difference between human capabilities and part of our human

• 45:27Speaker 00

  learning efficiency and Frontier AI. And when that happens, when it becomes effectively impossible to measure the gap, this is the AGI moment.

• 45:36Speaker 01

  Well, then the machines will take over and then they will create Arc ASI 1.

• 45:41Speaker 00

  Yes.

• 45:41Speaker 01

  And then it will continue from there.

• 45:43Speaker 00

  Yeah, yeah.

• 45:43Speaker 01

  Yeah. If you had to put a guess, I mean, years, decades, months.

• 45:50Speaker 00

  My timeline to AGI, you know, if you, if you just try to extrapolate from the current rate of progress and the amount of investment that's going into not just DLM stack, but also like side ideas, side bets that might work out, like, you know, NDA, for instance. I think we're probably looking at AGI 2030. Early 2030s, most likely. So around the time that you are going to be releasing like maybe ARC six or ARC seven, that's probably going to be a GI.

• 46:25Speaker 02

  You guys are doing a different approach to LLMs. Do you think there's room for more startups to explore other new approaches? And are there any other ones that you think are promising but don't have time to explore yourself?

• 46:37Speaker 00

  Yeah, absolutely. I mean, there are many different approaches that you could try. I've said like compute is a great equalizer. I think if you look at the amount of compute and resources that we've thrown at deep learning and gradient descent and scaling that up. If you had thrown the same amount of investment into almost anything else, you would also have seen extremely exciting results, like genetic algorithms, for instance. If you try to scale up genetic algorithms, I mean, I'm sure you can do incredible things with that. You could, in fact, probably do new science because that's based on search. And search is the best fit for automating the scientific method.

• 47:18Speaker 00

  I think so right now, there's also like approaches that build on top of the current stack with their slightly alternative like state space models, for instance. That's the XLSIM architecture. Like you can basically, you know, current frontier AI is it's a stack of things. And you can take any layer in the stack and try to propose an alternative. Like if you propose an alternative architecture, you can be doing, for instance, like, yeah, like more like recurrent models instead of transformers for the architecture. Or you can do even lower level, you're going to be like, okay, we're still going to be training parameter curves, but you're going to get rid of gradient descent. Right? We're going to use like search. Maybe you're going to do new evolution. That's the lower level.

• 48:03Speaker 00

  And the lowest level is the level where we're operating, where we're saying, well, actually, forget about curves. Forget about parameter learning, forget about gradient descent. We're just going to do something completely different. And I think if you want to build optimal AI, you're kind of forced to go back to the foundation of the stack. It cannot be like one layer added on top of the pile.

• 48:28Speaker 04

  So do you think for aspiring researchers to want to do a new Neolab with a different approach, they should be reading research papers from the 70s or 80s? Go deeply in those with approaches that were not as invested nowadays.

• 48:41Speaker 00

  That is actually a great idea because earlier in the in the history of the AI research timeline, people were exploring more things and very different things. You've had this sort of like collapse of everything into one approach. It's actually kind of a bad idea. Like, consider that not too long ago, like about about twenty years ago.

• 49:03Speaker 04

  We had the collapse into SVMs too.

• 49:05Speaker 00

  Yeah, I mean, it wasn't, I wouldn't describe it as a collapse because there weren't that many people doing SVMs. And AI was a much, much smaller field back then. But there was this widespread understanding that neural networks were were a failed approach, that neural networks didn't work. And it was a waste of time to to to keep trying.

• 49:25Speaker 04

  In the nineties, right?

• 49:26Speaker 00

  Yeah, no, even even in the in the in the late two thousands, this was this was this sort of things. Basically, when I got into into AI, people are telling me like, hey, neural networks, don't try that. I was like, yeah, but it it looks a lot like what the brain is doing. Like, I'm interested in that. If everybody is working on something, you are discarding ideas that will actually turn out to be very proactive ideas. Right? And, yeah, like, back in the seventies, back in the eighties, people are trying more things. And I think genetic algorithms is actually a very good example of that. I think this is an approach that has a tremendous amount of potential, but there's there's not too many people are looking into scaling it up deeply.

• 50:07Speaker 01

  Are there any characteristics that you would be looking for? I mean, is it as simple as like if there's a scaling law that could happen, then even if it's a different? Or is it is that too like, you know, thinking by analogy?

• 50:22Speaker 00

  I think you are looking for approaches that scale.

• 50:26Speaker 01

  Yeah.

• 50:26Speaker 00

  I think it's it's a nonstarter. If you're working on something, but the only way to increase the capability of the system is to have human engineers and researchers spend time on it. It will not work because even if the idea is very clever and very elegant and works really well, capabilities are going to be bounded, they're going to be bounded by human investment, right? You want to be in a setup where the system can improve its capabilities with no human in the loop, with no human bottleneck.

• 50:54Speaker 01

  So you would say like, don't just do it the way we did it like 10 years ago. Do it with the idea that recursive self-improvement is baked in at the beginning.

• 51:02Speaker 00

  Yeah, not necessarily recursive self-improvement because deep learning, for instance, is not recursively self-improving, but with the idea of scaling up with no human bottlenecks, you want to remove the human from the improvement loop. The great strength of deep learning is that the models get better and better simply by adding training, training compute and training data. I mean, it's a little bit of caricature because of course, just adding these factors requires a lot of human involvement. But basically, that's the idea, that you have this decoupling from the improvement curve and the amount of human effort that's needed to be injected into the system.

• 51:39Speaker 02

  Yes, or human effort that's already happened because the L M's do actually require an enormous amount of human effort. It's just the human effort to build the internet and we'd already built it.

• 51:47Speaker 00

  Yeah, actually, less and less now that we are doing training in interactive, very valuable environments. Because then you only need a small amount of human effort to create the environment. And from that small amount of effort, you're creating exponentially more training data. But at first, I think to sort of like prime the machine, you need this tremendous amount of human-generated abstractions and coalescing text data. And if you don't start from that, you cannot get the system into this loop.

• 52:21Speaker 01

  Do you have any advice for me, starting a open source project, things to do, things not to do in the AI space? Because I am not sure how I signed up for this in the last 14 days, but I think I have, I don't know, on the order of like 10 to 30,000 people using Gstack every day.

• 52:41Speaker 00

  That's wild.

• 52:43Speaker 01

  Yeah. And I don't know, like, I have a job.

• 52:48Speaker 01

  I guess like, you know, what was it like to start Keras? And how did you keep maintaining it? What's a good maintainer? Like, what did you learn from that? I don't know, this might be a whole hour.

• 52:57Speaker 00

  Yeah, I mean, lots of learnings from

• 53:00Speaker 01

  Too many things.

• 53:01Speaker 00

  From growing Keras. So right now, I'm less involved with it. There's a big team at Google that's working on it and they're doing an amazing job.

• 53:09Speaker 01

  So it is possible to not, you know, to put people together to like

• 53:12Speaker 00

  It is possible to start something. It is possible to start something.

• 53:16Speaker 01

  That's a relief.

• 53:16Speaker 00

  And then get more people involved and at some point it becomes its own thing. And just, you know, it used to be your baby, but now it's all grown up. It's all adult and going on with its own life. So if you ask me, the factors that we made Keras successful. And first of all is that there was this big focus on making the API simple. And in Twitter, there was this big focus on usability. And this was inspired by scikit-learn. Like scikit-learn was sort of like the OG machine learning library for Python. And what made it successful was that it was so easy to get started with it. So at first I was like, okay, I'm going to package all this functionality I've created.

• 53:58Speaker 00

  And there are really, really simple APIs going to be like the scikit-learn API. That was like the big idea. The focus on usability is not just making sure the API is simple. It's also making sure the entire onboarding experience is nice and easy. Like the docs should be very informative. You should, you know, the docs should be not just telling you about how to use this thing. They should actually be teaching you about the domain in the first place. Because the folks who land on your website, they're not going to be already deep learning experts. They're going to be people looking to maybe start using deep learning. And so you have to teach them not just how to use the tool, but what the tool is good for. And the entire field around it.

• 54:41Speaker 00

  And then, you know, you have to put a lot of investment into community building. One thing we did a bit at Google, in fact, you know, Google made it kind of difficult. And I was sad about that is hire your power users, like hire your fans. This is a really, really good idea. Like find, find the most enthusiastic users from your community and just hire them on your team.

• 55:06Speaker 01

  Amazing.

• 55:06Speaker 00

  Yeah. And these are the always the best people, right?

• 55:11Speaker 01

  All right, time to start Gstack.org, put in a bunch of my own money and then hire a bunch of people to work on it. That sounds good. I think you've been a leader and pioneer and we're so lucky to have you sit with us. There are people watching who are at the beginning of their, you know, adulthood even like they're certainly their professional careers. Or actually like people just around the world, they're like trying to understand like, what does this mean? As intelligence becomes broadly applicable, like what would you tell, you know, if you were 18 right now, what would you tell them?

• 55:44Speaker 00

  Yeah. I mean, there's a lot of people today who are very pessimistic, very negative takes about the rise in the capabilities. They say, oh, you know, I'm going to be out of a job soon. There's going to be mass unemployment. AI is just going to take over completely. And my, my take is actually, you know, the more you know, the more expertise you have about things like programming, for instance, the better you're able to use and leverage these tools for your own benefit. And with the right kind of expertise, all this AI progress is actually empowerment. Like it's something that you can leverage for yourself. I mean, that's, that's exactly what you did with your project, right?

• 56:25Speaker 00

  And yeah, more people should have this mindset of trying to learn as much as possible, not just about AI, but about the domain that they want to apply AI to, right? So that they should, they should seek to. Turn this new development into an opportunity, into a tool they can use for themselves to improve their own lives. I think that's the right mindset because you're not going to stop AI progress. I think it's too late for that. And so the next question is, OK, like AI progress is here. It's actually going to keep accelerating. How do you make use of it? How do you leverage? How do you ride the wave? That's the question to ask.

• 57:03Speaker 01

  I wish we could keep going for a couple hours because I'm sure we could. Francois, thank you so much for spending time with us.

• 57:09Speaker 00

  Thanks so much for having me.