ARC Prize v2 Launch! (Francois Chollet and Mike Knoop)

We are joined by Francois Chollet and Mike Knoop, to launch the new version of the ARC prize! In version 2, the challenges have been calibrated with humans such that at least 2 humans could solve each task in a reasonable task, but also adversarially selected so that frontier reasoning models can't solve them. The best LLMs today get negligible performance on this challenge. https://arcprize.org/ SPONSOR MESSAGES: *** Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/ *** TRANSCRIPT: https://www.dropbox.com/scl/fi/0v9o8xcpppdwnkntj59oi/ARCv2.pdf?rlkey=luqb6f141976vra6zdtptv5uj&dl=0 TOC: 1. ARC v2 Core Design & Objectives [00:00:00] 1.1 ARC v2 Launch and Benchmark Architecture [00:03:16] 1.2 Test-Time Optimization and AGI Assessment [00:06:24] 1.3 Human-AI Capability Analysis [00:13:02] 1.4 OpenAI o3 Initial Performance Results 2. ARC Technical Evolution [00:17:20] 2.1 ARC-v1 to ARC-v2 Design Improvements [00:21:12] 2.2 Human Validation Methodology [00:26:05] 2.3 Task Design and Gaming Prevention [00:29:11] 2.4 Intelligence Measurement Framework 3. O3 Performance & Future Challenges [00:38:50] 3.1 O3 Comprehensive Performance Analysis [00:43:40] 3.2 System Limitations and Failure Modes [00:49:30] 3.3 Program Synthesis Applications [00:53:00] 3.4 Future Development Roadmap REFS: [00:00:15] On the Measure of Intelligence, François Chollet https://arxiv.org/abs/1911.01547 [00:06:45] ARC Prize Foundation, François Chollet, Mike Knoop https://arcprize.org/ [00:12:50] OpenAI o3 model performance on ARC v1, ARC Prize Team https://arcprize.org/blog/oai-o3-pub-breakthrough [00:18:30] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, Jason Wei et al. https://arxiv.org/abs/2201.11903 [00:21:45] ARC-v2 benchmark tasks, Mike Knoop https://arcprize.org/blog/introducing-arc-agi-public-leaderboard [00:26:05] ARC Prize 2024: Technical Report, Francois Chollet et al. https://arxiv.org/html/2412.04604v2 [00:32:45] ARC Prize 2024 Technical Report, Francois Chollet, Mike Knoop, Gregory Kamradt https://arxiv.org/abs/2412.04604 [00:48:55] The Bitter Lesson, Rich Sutton http://www.incompleteideas.net/IncIdeas/BitterLesson.html [00:53:30] Decoding strategies in neural text generation, Sina Zarrieß https://www.mdpi.com/2078-2489/12/9/355/pdf

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  This is not just more of the same that we've seen in the past. We have now an existence proof that computers are able to do something that they've never been able to do before in history of humanity.

• 0:10Tim Scarfe?

  ARC version 2 has just been released, and even the frontier foundation models are failing spectacularly.

• 0:18Speaker 02

  Today, we are releasing ARC G I 2, the next version of the benchmark. ARC G I and now ARC G I 2 is pretty much the only unsaturated benchmark that is feasible by real people. And so it's a very good yardstick to measure how much fluid intelligence these models have, how close we are to true it here.

• 0:36Speaker 00

  And alongside that, we're really excited to be welcoming everyone to ARC Prize 2025. Contest kicks off officially now, it's going to run all the way through the end of 2025. The structure of the contest is very similar to last year. We're going to have the Kaggle leaderboard running. We're going to be testing this year on the semi-private data set and wrapping up and testing the final leaderboard on the private data set. Still got the big prize, it's unclaimed. In order to get the big prize, you have to open source your solution, have high degree of efficiency running on Kaggle. And yeah, we're really, really excited to see all the new ideas. I think there was a lot that came out last year in 2024 that really pushed the frontier.

• 1:11Speaker 02

  The next version of the benchmark, it's more challenging, it's extremely unsaturated. All frontier models are scoring effectively within single legit percentage. And it's the first time where we've calibrated the human-facing difficulty of the tasks. So we actually hired roughly 400 people. We tested every single task and every single task has been solved by at least two people. So we know it's very feasible for humans, it's extremely out of reach for any system today. This is the frontier.

• 1:43Tim Scarfe?

  The ARC benchmark forces us to confront an uncomfortable truth about our pursuit of artificial general intelligence, that the field has been overlooking.

• 1:51Speaker 02

  Intelligence is not just about capabilities, it's also about the efficiency with which you acquire and deploy these capabilities. Intelligence is about finding that program in very few hops, using actually very little compute. Like, look at the amount of energy that a human expends to solve one hard task over, you know, two, three, four minutes. It's almost zero, right? And compare that to a model like Austrian High Compute Sourcing, for instance, which is going to use like over 3,000 bytes of compute. So it's never just an economics problem. Efficiency is actually the question we're asking, right? Efficiency is a problem statement, it's not capability.

• 2:31Speaker 00

  The goalpost is AGI. That's like what we're here to do. That was the whole point of launching ARC Prize in the first place was to raise awareness that there was this really important benchmark that I thought shared, showed something important that like the sort of research community in AI was missing about the sort of nature of artificial intelligence. You know, this is like one of the things I think makes ARC special and very unique and important, I would argue. You know, there's a lot of benchmarks in the world today. And to my understanding and knowledge, pretty much every other benchmark, all the frontier benchmarks basically are trying to test for these like superhuman capabilities, right? These like PhD plus plus type skills that you need to have in order to succeed at the benchmarks.

• 3:16Speaker 02

  It's not just compute, it's not just scale. You have to be scaling the right thing. You have to be scaling the right ideas and maybe you have them.

• 3:22Speaker 00

  I personally just keep getting like surprised and impressed by ARC Prize community, how much folks are pushing the frontier. And I think it was really exciting too, because it means that individual people and individual teams can actually make a difference. If we're kind of in an innovation constrained world, an idea constrained world, which I think ARC AGI shows, that means you out there could actually make a significant contribution to the frontier of AGI. So if you're going to enter the contest, go to arcprize.org and good luck.

• 3:49Speaker 02

  Good luck to see you on the leaderboard.

• 4:01Speaker 02

  This sort of like test time optimization techniques or test time search techniques, that's the current frontier of AGI, right? And there are many ways to approach this. Of course, you can do, you can do just test time training or in this case, you know, test time.

• 4:17Speaker 02

  You can, you can do search, you can do search over symbolic space, you can do search over Chinothard space, over token space, or you can do search in latent space as well, right? So you have, you have many fun ways to do it, but really the frontier is how do you adapt to novelty at test time by recombining what you know into some novel structure?

• 4:35Tim Scarfe?

  MLST is sponsored by Two for AI Labs. Now, they are the DeepSeek, based in Switzerland. They have an amazing team. You've seen many of the folks on the team. They acquired Minds AI, of course. They did a lot of great work on ARC. They're now working on O1-style models and reasoning and thinking and test time computation. The reason you want to work for them is you get loads of autonomy, you get visibility, you can publish your research. And also, they are hiring, as well as ML engineers, they're hiring a chief scientist. They really, really want to find the best possible person for this role. And they're prepared to pay top dollar as a joining bonus. So if you're interested in working for them as an ML engineer or their chief scientist, get in touch with Benjamin Cruzier. Go to twoforlabs.ai and see what happens.

• 5:19Tim Scarfe?

  Well, Mike, it's amazing to have you on MLST. Welcome.

• 5:22Speaker 00

  Yeah, thank you so much. We're very excited to be here today.

• 5:24Tim Scarfe?

  Mike, I hear that you guys have got some very exciting news today. Tell me about it.

• 5:28Speaker 00

  Yeah, we're super excited today. We're back. We're really excited to be launching both ARC AGI 2 alongside an updated ARC Prize 2025 contest. Both are going to be launching today. And you can go to arcprize.org to learn more and enter the contest.

• 5:42Tim Scarfe?

  Okay. And in a nutshell, what is V2 and how is it different from V1?

• 5:45Speaker 00

  The way I have to sort of think about it is ARC AGI 1 was a benchmark that was designed to sort of challenge deep learning. And ARC AGI 2, in contrast, is really a benchmark that's designed to challenge these new AI reasoning systems that we're starting to see from pretty much all of the frontier labs. And one of the really cool things about ARC AGI 2 is we're basically seeing, you know, models, AI systems that are purely based on pre-training effectively scoring 0%. And some of the frontier AI reasoning systems, we're in progress of testing them right now. And we're sort of expecting single-digit performance. So a really big update in terms of over ARC AGI 1 from 2024.

• 6:24Tim Scarfe?

  So the original version of ARC was very much based at these kind of foundation models that didn't do reasoning. Version 2 is tuned for the reasoning models. But what would you say to the charge that you're moving the goalposts? I mean, how is ARC V2 sort of like meaningfully an evolution of the benchmark?

• 6:42Speaker 00

  Yeah, I mean, I think the way that I think about it is that the goalpost is AGI. That's like what we're here to do. That was the whole point of launching ARC Prize in the first place was to raise awareness that there was this really important benchmark that I thought showed something important that like the sort of research community in AI was missing about the sort of nature of artificial intelligence. So that's kind of our goalpost. And you know, the definition that I use for AGI and the one that the foundation adopts is assessing this capability gap between humans and computers. And ARC Prize's foundation is really to drive that gap to zero. I think it would be hard to argue that we don't have AGI if you look around and you can't find any more tasks that are very straightforward, simple and easy for humans that computers can do as well. And the fact is that we were

• 7:27Speaker 00

  able to find still lots of those tasks. In fact, all of the tasks in ARC AGI 2 data sets sort of fit into this category of things that are relatively easy and simple and straightforward for humans and comparatively very, very difficult and hard for AI today.

• 7:40Tim Scarfe?

  Okay, cool. So I know you guys have done loads of human calibration and we'll talk about that in a minute. But the fundamental philosophy of the ARC challenge is focusing on human gaps. But at the same time, AI models are becoming superhuman in so many respects. So is the big story the human gaps or is the big story the expansion of capabilities that are superhuman?

• 8:01Speaker 00

  You know, this is like one of the things I think makes ARC special and very unique and important, I would argue. You know, there's a lot of benchmarks in the world today. And to my understanding and knowledge, pretty much every other benchmark, you know, all the frontier benchmarks basically are trying to test for these like superhuman capabilities, right? These like PhD plus plus type skills that you need to have in order to succeed at the benchmark. Humans like can't solve the problems that are in these benchmarks. So you have to be very, very like have a lot of experience, a lot of education, a lot of training in order to be able to sort of even get close to sort of solving the benchmarks as a human. And I think those are important. Those are useful. But I think it's actually

• 8:43Speaker 00

  more illustrating about something that like we're missing from the nature in the sense of like artificial intelligence by looking at the gaps, the remaining gaps between what's simple and easy for humans and what's hard for AI. I think that's much more of an inspiring Story. I think it's one where it's actually necessary to target this in order to actually get AGI that is capable of innovation. You know, I think this is one of the main reasons I got into AI and AGI in the first place was being really inspired and excited about trying to build these systems that would be capable of like compressing science timelines. And if all we have is AI that looks like we had at the beginning of 2024, right? Based on pre-training, based on a memorization regime.

• 9:26Speaker 00

  You're never going to get to that because these are systems that are fairly going to reflect back the experience and the knowledge that humanity has sort of gained over the last, you know, 10,000 generations as opposed to being ones that are capable of like producing new knowledge, new technology, adding to sort of humanities like colossus of sort of knowledge and technology. If we want systems that can actually do that, we need AGI. And this definition that we've sort of used for this foundation of easy for humans and hard for AI. I think is if we can close that gap, we'll actually get technology that's capable of doing that.

• 9:54Tim Scarfe?

  I wonder whether you think we are just about five discoveries away from AGI. Because there's going to be version three of the art challenge. They'll presumably there'll be a version four. Intelligence is multi-dimensional and I can see this both ways, right? Because, you know, many critics of AI, they are almost gaslighting us. They're saying that this amazing technology that you're using, it doesn't work. And I'm like, well, yeah, it does work. And would it be the case that the criticisms will become more and more kind of philosophical? And they'll say, oh, you know, because it's not biological or whatever, it's not the same thing. Or do you think meaningfully we're about five steps away from AGI?

• 10:29Speaker 00

  I think this is why benchmarks are important. And I had a similar question, actually, you know, when I was starting to get back into AI in 2022. And trying to understand the world, like, are we on track for AGI or not? How far off are we? And I find that it's really, really hard to get a sense of understanding of the capabilities of all these systems purely by using them. You can certainly get a sense by just interacting with them. But if you really want to understand what are they capable and not capable of, you really need a benchmark to discern this fact. This is, you know, one of the interesting things that I picked up from building AI products at Zapier as well. It's very different. With AI than it is classic software, right? One of the big differences is when you're building classic software, you know, you can build and test with five users and know, okay, hey, this is going to be like,

• 11:14Speaker 00

  you know, this product can scale to millions. It's going to work the exact same way. And that's fundamentally just not the case with AI technology. You really have to like deploy it to a large scale in order to assess how it works. You need a benchmark alongside that scaling in order to tell you, hey, is this system working or not?

• 11:27Tim Scarfe?

  What were the main lessons that you learned from version one that you moved into version two?

• 11:31Speaker 00

  I think so. ArcAGI two has been in the works actually for several years. Franz was started working on it, crowdsourcing some tasks for it years and years ago. There was a bunch of sort of inherent flaws we ran into with it that we learned as we sort of started popularizing the benchmark over the last year or so. You know, one of the things we learned was that a lot of the tasks were very susceptible to brute force search. So that's something that has zero intelligence at all. And we wanted to minimize the sort of incident rate of tasks that were sort of susceptible to that. We hadn't human calibrated it. We anecdotally, we relied on some anecdotes to say that, hey, ArcAGI one is easy for humans. We had, you know, a couple sort of STEM, two STEM folks who had taken the whole data set, including the private set, and were able to solve, you know, ninety eight, ninety nine percent.

• 12:14Speaker 00

  But, you know, relying on anecdote, we didn't have that calibrated across the sort of three different data sets that we had. And then we had all these AI frontier reasoning systems come out over the last, you know, three, four months. And we've got a chance to study these and learn what are the sort of qualities of Arc tasks that are remain very, very challenging for these AI reasoning systems, which we can get into if you're curious. And so those are the main sort of insights and learnings that we took from ArcAGI one to try and produce an ArcAGI two benchmark that I think will be a useful sort of signal for development this year in artificial intelligence.

• 12:46Tim Scarfe?

  Can we quickly touch on the OpenAI situation? So in December, they, they look, well, they didn't launch, but they gave you access to O three. And it got incredible performance on ArcV one, human level performance, something that we just didn't think really would be possible so quickly.

• 13:02Speaker 00

  Yeah, it surprised me.

• 13:03Tim Scarfe?

  It came out of nowhere. I mean, can you just tell me the story behind that?

• 13:06Speaker 00

  Yeah, yeah, I'll tell you, this is, you know, one of the reasons why I I'm always hesitant to make predictions in AI about timelines. You know, I think it's very easy to sort of make predictions along smooth scaling curves. But the nature of innovation is it's a step function, right? And step functions are really, really hard to predict when when they're going to come out. I think the the best thing that I can say, having spent some time with O3 and looking at how it performs at ARC, is that systems like O3 demand serious study. This is not just, you know, more of the same that we've seen in the past. This is, you know, we we that we have now an existence proof that computers are able to do something that they've never been able to do before in in in the history of humanity, which is, I think, really, really exciting.

• 13:50Speaker 00

  I think there's still a long way to go to get to AGI. But I do think that these things are important to understand and sort of even discern how they work from a capability standpoint in order to make sure that future AI systems that we're developing, building, look more like this and not like the sort of pre-training pure scaling regime that we've had in the past. So I I still remember the, to sort of give you the anecdote, I still remember the two-week period, the sprint we had on testing O3. It was right at the end of the contest. We had wrapped up ARC 2024 in, I think, early November last year. And we had a three-week, four-week period where we were really, really busy on judging all the final submissions, the papers, getting together the technical report. And we were dropping it on all the sort of results on a Friday. And, you know, I was I was really hoping and anticipating that, you know, I was going

• 14:34Speaker 00

  to have a nice, relaxing, you know, you know, holiday period in December. And the day that we dropped the technical report, we had a reach out from one of the folks at OpenAI who said, hey, we'd really love you to test this new thing that we're working on. We think we've got some impressive results on on on ARC AGI 1. And so that kicked off a very, very like hectic, fast, frantic, two-week period to try and understand, okay, what what is this system? Like, does it reproduce the claims that OpenAI had on testing it? And what does this mean for the benchmark? What does this mean for for AGI? And, you know, I think we were able to show, the final result was that O3 on its sort of high-efficiency setting, which fit within the sort of budget constraints that we'd set out for Public leaderboard got about seventy five percent or so.

• 15:19Speaker 00

  And then they had a high compute version, which used, I think, like maybe two hundred X more compute than the low compute setting, which was able to score even eighty five percent. And these are, this is really impressive. And I think this shows a system like O three has this, you know, sort of binary switch. It had, we've gone from a regime where, you know, these AI models have no ability to adapt to novelty to something like O three, where in its existence, proof of now, an AI system that can adapt to novelty in a, in a small way.

• 15:47Tim Scarfe?

  Breaking this down a little bit, there were, there were some interesting caveats that you just alluded to. So first of all, they did some kind of fine tuning. And people at the time joked that, you know, isn't it scandalous that they were training on the training set? So based.

• 16:00Speaker 00

  Yeah, this is, this is like a very bad description. This is a very poor critique. I think it misses the point of the benchmark. I think the folks who feel this way, it's just because they're so used to thinking about benchmarks and AI from the pre-training scaling regime, where like, hey, if I trained on the data, you know, that's cheating them to test on the data. Right? And that's true in the pre-training regime. But ARC is a very, very special, different benchmark, where it explicitly makes a training set available with the intention to train on it. This is very explicit. This is like what the benchmark expects you to do. We expect AI researchers to use the training set in order to teach their AI systems about the domain of ARC. And then what's special is we've got a private data set that, you know, very few humans have ever seen. The private data set, it does not look like the training set.

• 16:42Speaker 00

  It requires you to generalize and abstract the core knowledge concepts that you learn through the training set at test time. Fundamentally, you cannot solve like the ARC AGI one or two private data sets purely by memorizing what's in the pre-training set. This would be like, you know, maybe a crude analogy would be, you know, if I was going to teach an AI system on grade school math and then test it on calculus. This is very similar to the type of thing that we do with, with ARC, where, you know, the training set is much simpler, easy curriculum to learn on. And then the test is a much more difficult one, where you actually have to express true intelligence, you have to have an actual capability of adapting to novelty at test time to solve it.

• 17:20Tim Scarfe?

  Okay, okay, all of, all of that is fine, but that, you know, there's a couple of things, right? They, two and a half thousand dollars per task or. Or that means they were probably doing samplings, they were doing a ridiculous amount of completions. They were doing a solution space prediction, which is very interesting. But but the main thing, Mike, just deepen your bones, deepen your bones. Do you think that they were training on API data or surely they were training on a whole bunch of data to do that well? And the extension of the question is, when they released the vanilla version of it, what performance would it get compared to their their tweaked version?

• 17:52Speaker 00

  We will test that as soon as it comes out and I would love to report the results on that. They told us all they did was train it on the training set and I believe that's what they did.

• 17:58Tim Scarfe?

  Okay, very interesting. And just comment on the solution space prediction. I mean, I was amazed that just predicting the output space directly, they could do so well. I mean, doesn't doesn't that almost take away from the idea that we need to have discrete code DSL type approaches if you can just predict the solution space so well?

• 18:17Speaker 00

  Effectively, what O three is doing is it's able to use its pre-trained experience and recombine it on the fly in face of a novel task. It does this through like a regime called chain of thought. This is all informed speculation, by the way, we don't have confirmed details. This is just, you know, my sort of personal assessment of how these systems work, particularly things like O one Pro and O three. If you compare them with systems like O R one or O one, these are systems that basically sped out a single chain of thought and then use that chain of thought in order to ground a final answer. This is distinct from how systems like O one Pro and O three work, where they actually have the ability to do multi-sampling and recomposition at test time of that chain of thought.

• 18:59Speaker 00

  This allows them to build novel COTs that don't show up anywhere in the pre-training, not in existing experience, and allows these systems to reach more sort of effect over more situation space effectively, based on what was in the original pre-training. Fundamentally, these systems are a combination of like a deep learning model and a synthesis engine that is put on top, and I think the right way to think of them is these are really AI systems, not single models anymore.

• 19:21Tim Scarfe?

  Yeah, and I agree with you. It's really funny, though, how you see the critique in the community, because, you know, Gary Marcus is now saying, oh, it can't draw pictures of bicycles and labels and label the parts. Whereas we see O one Pro and O three, and it really does seem like a dramatic. move towards, you know, intelligent systems. But Mike, can we just quickly talk about the testing methodology? So the real, the real work that you guys did was you got a whole bunch of human subjects. And you had the, I think you had, was it 400 test subjects? And they all needed, like, at least two people needed to solve every single task. And you had to do this experiment design and you had to balance complexity of the tasks and so on. How did you do all of that?

• 20:01Speaker 00

  Yeah, this was, so this is one of the biggest things we wanted to fix with Arc G I one. We didn't, we never had a formal human calibration study on how do humans actually do on these things? You know, we relied on anecdote. So we had to set up a testing center down in San Diego. We recruited tons of just folks from the local community, all the way from, you know, Uber drivers to single moms, to UCSD students. And brought these folks out to go take arc puzzles. It was a really cool, like, we'll have to share some of the photos, like, these, like, testing shots where you have, like, you know, dozens and hundreds of people, like, taking our tasks on laptops. Our goal with, originally with the data set was to ensure that every single task that we put in Arc G I two is solvable by at least one human.

• 20:38Speaker 00

  And what we actually found was something even a higher standard, I think, which was that we found that every single task in the new V two data set is solvable by at least two humans under two attempts. And these are the same rules that we give to AI systems on the benchmark, both on the contest as well as the the public leaderboards. I think this is a pretty good sort of assertion of the sort of, you know, straightforward comparison we can actually use now between, hey, are these tasks easy and straightforward for humans? Yes, are they hard for AI? Yes, like I said before, frontier systems generally are getting close to zero or single digit percentages on on these tasks now.

• 21:12Tim Scarfe?

  Okay, but the the idea, though, is this more of X paradox, right? Which is that, you know, basically, while while we can select problems that are easy for humans and hard for AIs, we haven't got AGI yet. But I was looking through some of your challenges and I felt that some of them were very difficult. Like, it would have taken me five or six minutes of deep thought to to get it. Like, are you are you finding that it's still easy to, you know, to find these things that are easy for humans and hard for AIs? Or are you kind of scraping the barrel a little bit?

• 21:41Speaker 00

  So I think easy for humans, hard for AI is a relative statement. The fact is, these V2 tasks were solvable by human on a $5 per task solve rate budget. They were solvable in five minutes or so, and AI cannot solve these at all today. And so, yes, I do think if you look at tasks, you have to think about them. There's like, you know, some thought you need to put in to sort of ascertain the rule. But the sort of data, I think, speaks for itself that, you know, we've got every single task now in the V2 dataset from the public eval set to the semi-private set to the private eval set. Every single one is solvable by at least two humans under two attempts. And yeah, these frontier systems can't solve these things at all. Or if they can, with a really, really expensive budget, to your earlier point, thousands of dollars per task.

• 22:25Tim Scarfe?

  So you guys have been cooking. You're already working on version three of ARC. What can you tell us about that?

• 22:32Speaker 00

  So the way I kind of think about the multivirgents here, ARC AGI-1, again, was designed to challenge deep learning as a paradigm. ARC AGI-2 is designed to challenge these AI reasoning systems. I don't expect that ARC AGI-2 is going to be as durable, right? ARC AGI-1 lasted for five years. I don't expect ARC AGI-2 is going to be quite as durable as that. I hope there will continue to be a very useful signal for researchers over the next year or two. But yeah, we've been working on ARC AGI-3, and I think the pithy way to talk about ARC AGI-3 is it's going to challenge AGI systems that don't even exist in the world yet today.

• 23:08Tim Scarfe?

  Can you tell me about the foundation that you're setting up?

• 23:10Speaker 00

  Yeah, so this was one of the big cool things, I think, from ARC Prize 2024. When we launched it, it was very much an experiment. You know, our ambitions were not quite what they were. I think when we went into 2024, our main goals were just to raise awareness of the fact that this benchmark exists. And I think what we found was, or what I personally found, I just kept getting surprised by the community around ARC. You know, I remember this really specific moment when O1 preview came out, and there was thousands of people on like Twitter, like demanding that we test this like new model on ARC. And that was not my mental model of like what this benchmark was or what the community was, and that was so cool. And that moment happened again when we ended the contest. That moment again happened when we launched the results. On O three.

• 23:53Speaker 00

  And this kind of showed, I think, hey, there's a real demand for what ARC is providing. There's a real demand for benchmarks that look like this, that ascertain these like capability gaps between humans and computers. And so we set up this foundation in order to basically be the North Star for AGI and continue to produce useful, interesting, durable benchmarks in the sort of spirit of trying to discern like what are the things that are simple, straightforward and easy for humans and still remain impossible or very, very difficult for AI. And we're going to carry that torch all the way until we get to AGI.

• 24:25Tim Scarfe?

  As you can see now, all of the large AI labs, they're focusing on reasoning and I'd like to think that ARC was at least a small part of that. And you folks are very focused on open source as well.

• 24:36Speaker 00

  Mark Chen said specifically on the OpenAI podcast that it was that they'd been thinking about ARC V one for years.

• 24:40Tim Scarfe?

  There you go. Well, yeah, exactly. But just tell me a little about that. So there's the industry impact, but you guys are really focused on open source as well. So how do you see those two things?

• 24:51Speaker 00

  So my sort of overriding philosophy at this point is AGI is the most important technology that humanity is going to develop. And if it is true that we are in an idea constrained environment, we still need new ideas to get to AGI, which I think our AGI two shows is true. If that's true about the world, then I think we should be designing the like most innovative sort of ecosystem and environment across the world that we possibly can. This is one of the reasons why we launched ARC Prize originally internationally to reach solo researchers, to inspire researchers again to go try and work on these new ideas, to get past this pre-training regime, try something that we knew that it would needed to be something beyond this and even beyond what we what we have today. And I think if you look at like a really healthy, strong innovation and ecosystem,

• 25:34Speaker 00

  you're going to look at one that is very open and there's a lot of sharing and there's a lot of diversity of approach. And this is in contrast to, you know, an ecosystem that would be very closed, very secretive, very dogmatic, very monocultural. And so those values of openness, those values of sharing are what the ARC Prize Foundation stands for in order to sort of increase the chance that we can get AGI soon.

• 25:53Tim Scarfe?

  So talking about the the version two of the ARC challenge, can you just give us the elevator pitch of that?

• 25:59Speaker 02

  Sure, so ARC 2 is basically a new version of ARC that keeps the same format, but tries to address the main flaws that we saw in ARC 1. So for instance, in ARC 1, we knew that there was a little bit of redundancy across tasks. So we saw that actually very early on, as early as the 2020 KAL competition. And also ARC 1 was way too brute forceable. So back in 2020, what we did after the KAL competition is that we tried to look at all tasks that were solved at least once by one entry in the competition. And we found that half of the private data sets could be solved, in fact, just via the sort of basic brute force program search methods that were deployed during the first competition.

• 26:40Speaker 02

  So that means half the data set actually doesn't give you very good signal about AGI at all. So the other half was actually good enough, required enough generalization that the benchmark overall was still useful and it still lasted quite a few years after that. But it told you from the start that there were some pretty significant flaws, which is expected, by the way. Like when I started creating ARC back in 2018, 2019, I was flying blind. I was trying to capture my own thoughts, my own intuition about what does it mean to generalize, what is abstraction of this reasoning? And that turned into this benchmark, but I could not anticipate what kind of AI techniques would be used against it.

• 27:22Speaker 02

  And so, yeah, as it turns out, a lot of it could be brute forced. So ARC 2 completely addresses that. You cannot score higher than 1% or 2% at most using brute force techniques on ARC 2. So that's good news. And other than that, we generally try to make it a little bit harder. So what we saw with ARC 1 is that it was very easy to saturate for humans. Like if you're like a STEM grad, for instance, you could very easily get 100% or within noise range of 100%, like something like 97, 98. And so that means that you were not getting a lot of useful bandwidths to compare AI capabilities with the capabilities of smart humans.

• 28:05Speaker 02

  And if you just make it a little bit harder, then you get more range. Where if you're not very intelligent, you'll score lower. If you're very intelligent, you'll score higher. And you're not super likely to completely saturate it until you are at the very top end of the distribution. So that's what Arc2 is. Same format, same basic rules. So we're only using core knowledge. You have these input-output pairs of grids that are at most thirty by thirty. But the content is very different. You're not going to find tasks where you only have to apply one basic rule that could be anticipated in advance, like some kind of gravity, things falling task or symmetry task.

• 28:49Speaker 02

  All the tasks are very compositional. So you have multiple rules, you have more objects, the grids are generally bigger. The rules can be chained together or can be interacting together. And that makes it completely out of reach for brute force methods. And as it turns out, it also makes it out of reach for the base L M pre-training paradigm.

• 29:11Tim Scarfe?

  You're saying that you've made them more compositional and iterative and harder for humans?

• 29:17Speaker 02

  That's right.

• 29:18Tim Scarfe?

  Could you give me a little bit more detail on that? I mean, if you think about it, there are different dimensions of things that A I models can do. And there are different dimensions of things that humans can do. Have you sort of quite diversely explored that? Or I mean, could you just give me a bit of a breakdown of the task characteristics?

• 29:34Speaker 02

  So in Arc1, you had many tasks that were very basic, where you just had one rule. Let's say, for instance, you have a few objects and you have to flip them, right? So this is an example of a task that's easy to brute force. Because flipping is something that you can acquire via pre-training as a concept or that you could just hard code in a brute force program search system. So if that's the only rule you have to apply and just apply it once, that's not compositional. That's actually pretty easy to anticipate. It's easy to brute force. So a compositional task is going to be a task where you have more than one concept. And typically, they're going to be interacting together. Like an example of a very simple compositional task is, let's say, you have object flipping, but also the objects are falling, right? So you have

• 30:19Speaker 02

  Two rules to apply to each object at once. But that again is a kind of task that could still be found via brute force program search. If you have as key elements in your D S L, gravity and flipping, for instance. And so you want to create tasks that are where the rules are chained to a sufficient level of depth. That there's no way you could you could find a chain by just trying every possible chain where it will become too expensive. Of course, humans can still do it because humans are just trying every possible combination of things they know on every problem that they see. They just have this very efficient, very intuitive way of searching for a theory that explains what they see.

• 31:02Tim Scarfe?

  You know, my co host Keith, he had this idea of doing a recursive version of of ARC. But the thought occurred to me is that even though we do this systematic compositional reasoning. We still have some kind of cognitive limit. So if we nested, let's say, four levels of of ARC challenges within the same problem. Wouldn't you find very quickly that humans just can't solve it?

• 31:24Speaker 02

  Right, so if you just like concatenate two ARC tasks, for instance. You get something that's much less brute forcible, that's much harder because there are more rules going on. It's not quite what I would call compositional though. Because even though you have two rules at once, they're not interacting with each other, right? You can solve them separately and they concatenate the solutions. And I think it's not it's not a bad idea at all, like it will work as a way to make ARC more difficult. With again, this caveat that you're not actually testing for depth of compositionality.

• 32:00Speaker 02

  One one issue, though, is that it will only really work once. Because as soon as the person developing the AI system notices that the task can actually be decomposed into subtasks. Then it's game over, you know. So I think it's actually more interesting to have multiple rules at once, but they're actually being chained together, or they're interacting together in some way, where for instance, one rule might be writing some information on the grid that needs to be read by the second rule, right?

• 32:27Tim Scarfe?

  What performance do the frontier models get on Arc V two?

• 32:30Speaker 02

  So what we saw was a big gap between models that don't do any kind of test time adaptation, like any kind of test time search or test time training, and models that do. And the base L L M's, even models like G P T four point five, they're basically scoring zero. I think one of them, I think it was R one, maybe, score like slightly above zero, something like one percent. But you know, it's within noise range of zero. So any model that cannot do a test time adaptation, that is, that does not possess a fluid intelligence does effectively zero. So in that sense, Arc two is actually a very strong sign that you have fluid intelligence, better than Arc one. I think Arc one could already tell you that, but less perfectly.

• 33:13Speaker 02

  So on Arc one, if you do not do test time adaptation, you can still do up to roughly ten percent, right? So on Arc two, that's actually zero. So it's better test. Now, when it comes to model that do test time adaptation, so we tried, for instance, some of the top entries from the K R competition last year, in the models that were doing test time training in particular, or some kind of program search. And the best model, the model that actually won the K R competition, can do, I believe, three percent on Arc two. And if you take an ensemble of the top entries from the competition, you get to four percent, right? So that's not very high. We also estimate that, so O three would be the current state of the art in terms of

• 33:58Speaker 02

  an A I model that does exhibit fluid intelligence. And so we haven't been able to test O three on low compute settings on all of the tasks that you wanted to test on, but we've tested it on a subset. And so we can extrapolate what it would score on the entire, on the entire set. And it sounds like it's going to be about four percent, right? So not, not, not super high, there's a lot, there's a lot of room to go higher than And we haven't been able to test O3 on high compute settings at all. So you know, the model that was scoring eighty eight percent on on Arc One. So I can I can make a guess. I guess based on what we saw from O3 LoK and and other models.

• 34:41Speaker 02

  I think you might get up to like fifteen, maybe even twenty percent. If you were really maxing out the compute setting and spending like, you know, ten K per task, for instance. But we will still be like far below average human performance, which will be more like sixty percent.

• 34:59Tim Scarfe?

  So that four percent that O3 gets on R G V two, do you think of that as fluid intelligence? Or do you think of that as a potential gap in? I mean, presumably you could have designed R G V two. If you if you selected the correct sets of human calibrated challenges, you could have found a set which was still zero one O three.

• 35:18Speaker 02

  Yeah, absolutely. No, you you could have, you know, adversely selected against O3 and then O3 would do would do zero. It would be very, very easy to go from four percent to zero percent, right? It's just a few few a few tiles that you need to change. So we not we not actually try to do that. Yes, I do believe that four percent does show that you have non zero fluid intelligence, which is also something that you could you could get as a signal from Arc One. And I think the the the sign that you see fluid intelligence in these models is the performance gap between the huge pre-trained only models that don't do test adaptation, that score effectively zero, maybe one. And you could say that that one percent is in fact a flood, that is, that's true. It should in practice be zero.

• 36:03Speaker 02

  And the models that do test adaptation and do non zero, that do you know, three percent, four percent, maybe five percent, right? And that means that there's something like ninety five percent of the data set that will actually give you this useful bandwidth for measuring how much fluid intelligence the model has. And that's something you are not getting with Arc One. Arc One was more binary, where if you don't have fluid intelligence, you're going to do very, very low, like below ten percent roughly. If you do, you're going to score significantly higher and getting About fifty percent would be very easy. But because the the the the measure would saturate very quickly. As as soon as you started in non-zero fluid intelligence, you did not get that that useful benefits that you're getting with the ARC two.

• 36:46Speaker 02

  So I think ARC two should allow for, you know, answering the question. Is this model actually as fluidly intelligent as the average human? Which is something you could not get out of ARC one.

• 36:59Tim Scarfe?

  I guess it's just an economics thing at this point. So if you spent, let's say, a billion dollars or half a billion dollars, you could saturate ARC B two. I'm not sure if you would agree with that. But if if that isn't the case, I mean, what do you think are the specific things that are missing from O three that that are stopping it from doing it doing better?

• 37:16Speaker 02

  So it's never just an economics question, because intelligence is not just about capabilities. It's also about the efficiency with which you acquire and deploy these capabilities. And sure, if you spend billions and billions of dollars, maybe you can saturate ARC two. But that would already have been true back in twenty twenty using like extremely crude brute force program search. If you have a DSL that's actually train complete, then you know that for every ARC task, there exists a program that may not in fact be all that long, that will solve the task. And all you need to do to find it is iterate over all possible programs in order of length. And then the first one you find is the one that's going to that's going to generalize, right? Because it's it's the shortest, it's the most the most parsimonious.

• 38:00Speaker 02

  So if you spend unlimited resources, you already have a HGR in that sense, just just in the pure skill sense. You can always just try every possible program until you find one that works. But that's not what intelligence is. Intelligence is about finding that program in very few hops, using actually very little compute. Like, look at the the amount of energy that a a human expends to solve one ARC task over, you know, two, three, four minutes. It's it's it's almost zero, right? And compare that to a model like O three on high compute settings, for instance, which is going to use like over three thousand bucks of compute. So it's never just an economics problem. Efficiency is actually the question we're asking, right?

• 38:45Speaker 02

  Efficiency is a problem statement, it's not capability.

• 38:48Tim Scarfe?

  So intelligence is knowledge acquisition efficiency. O three did very well on ARC V one. And now that it does so badly on ARC V two. The whole point of your definition of intelligence is that, given some basis knowledge, you efficiently recombine, you produce new skill programs. You're saying that in the absence of the base knowledge in V two, there is no intelligence. Therefore, is O three not actually as intelligent as we thought it was?

• 39:13Speaker 02

  I think O three is one of the first models, perhaps arguably the first model that does show fluid intelligence. So now, what the results on ARC two are telling you is that it's not human level fluid intelligence, right? But still, I would consider O three as a kind of proto AGI with two big flaws, two big caveats. So one is of course efficiency. Efficiency is part of the problem statements, in fact, the central point. So as long as you're not as efficient in terms of, for instance, you know,

• 39:46Speaker 02

  data efficiency, compute efficiency, energy efficiency, then it's only a temporary solution, we'll find a better solution in the future.

• 39:55Speaker 02

  And also, it's not quite human level. If it were human level, you'd expect it to score, you know, something like O three should score like over sixty percent on ARC two. And we don't know what the exact number is going to be, but you know, probably around like forty five percent, right?

• 40:10Tim Scarfe?

  Do you think that general intelligence is a category or a spectrum?

• 40:15Speaker 02

  So general fluid intelligence, it's, I would say it's both. Because there's a huge difference between just having memorized a bunch of skill programs that are static and knowledge factors versus being able to adapt novelty to a non-zero extent. So that is a binary distinction. Either you have fluid intelligence or you don't, right? And ARC one couldn't answer that question for any system. But once you have non-zero fluid intelligence, then the question is how much do you actually have and how high it compares to humans? And that's related to the notion of

• 40:54Speaker 02

  recombination of the skill programs that you have, the knowledge that you have, and depths of recombination. So if you do no recombination at all, you don't have fluid intelligence. If you do some recombination, you do, but then the question is how deeply can you recombine? Like, for instance, if you're using a program synthesis analogy,

• 41:18Speaker 02

  the question is how big of a program can you write on the fly to adapt to a new problem?

• 41:27Speaker 02

  And of course, as well, how efficiently, how fast and how efficiently you can write it? So it is a binary, but it's also a spectrum. And ARC one was trying to ask the binary question, does this system has any fluid intelligence at all? And ARC two is more on the side of trying to measure how much fluid intelligence you actually have compared to humans.

• 41:49Tim Scarfe?

  How long do you think it will take for V two to be saturated? And do you think it will survive until V three comes up?

• 41:55Speaker 02

  So that's a question where you have to take into account resource efficiency. So if you're asking how long it will take before we have a system that can score higher than, let's say, eighty percent on ARC two, using less than ten thousand dollars of compute, for instance. I think probably around a couple of years. So it's very difficult to make predictions here. I think if you're just looking at current techniques and scaling up current techniques, I think it could take a while. I think that ARC two is actually way out of reach of current techniques. But of course, we are not limited to current techniques. In 2025, we're probably going to see new breakthroughs, in the same way if we saw new breakthroughs last year.

• 42:38Speaker 02

  And these breakthroughs are actually very difficult to predict. I was personally very surprised with the performance that O three could get on ARC one last year. That came as a surprise. So maybe we'll have new surprises this year. But I would be extremely surprised if we see an efficient solution that's human level on ARC two by The end of 2025, I would basically rule that out. By the end of 2026, maybe, right? Which is why we have the Arc 3 coming, of course.

• 43:09Tim Scarfe?

  So on analysis of failure modes, I'm sure you saw the blog post that I read, where it went through all of the different failure modes of O3. And of course, it was solution space prediction, which made it more surprising to me. My take on it was, I was really impressed that even when it failed, it was because the solution space got too big, or it was just getting minor mistakes. But broadly, it got the direction of many of the problems quite well. You know, similarly, tell me about the failure modes on V2.

• 43:37Speaker 02

  Right, so we were unable to test O3 as much on V2. But I can tell you about failure modes based on what we saw on V1. And well, there are many, but generally, this is a model where reasoning abilities can decrease exponentially with problem size. If you have more objects in the scene, if you have more rules, more concepts interacting, you see this exponential decrease in capabilities. It's also, you know, because it's a model that needs to, it works by writing a kind of natural language program that describes what it's seeing, that describes the problem and the sequence of steps to solve it.

• 44:20Speaker 02

  So in that sense, it's 100% a natural language program. And that means that in order to solve a problem, it has to talk about it using words. And as a result, if you have a task where the rule is very simple to draw up for a human, but in a non-verbal way, but it's very difficult to put it into words, it has no verbal analogy, that's actually much harder to solve for this direction of that model. So other than that, we saw that just one of the big challenges is, you know, compositionality, having multiple rules interact. There's also, it seems there's a bit of locality bias going on as well, where if you have to combine together bits of information

• 45:04Speaker 02

  that are spatially co-located together on the grid, that's easier for the model than if you have to do the exact same thing, but the two bits of information you have to synthesize are pretty distant. So having to combine together bits of information that are separate. Having to... So it seems as well that the model has a trouble simulating the execution of a rule and then reading the result. Like for instance, if you're solving an R task and you grok a certain rule and then you start applying it, let's say it's like you're continuing, you're doing line continuation or something. And then you have to take another rule and use that rule to read a bit of information

• 45:49Speaker 02

  that you have written in the process of executing the first rule. That sort of thing is completely out of reach for the chain of thought models.

• 45:58Tim Scarfe?

  How multidimensional do you think intelligence is? You know, one school of thought, and I think you might subscribe to this, is that the universe is kind of almost made up of platonic rules that are disconnected from the world that we live in. And then there's this kaleidoscope idea you talk about, and they get combined together, and that's what we see. But another school of thought is that there'll always be another dimension of intelligence. We'll always need ARC V4, V5, V6, and there'll always be something missing.

• 46:27Speaker 02

  Each step of generality that you cross, you're gaining a nonlinear amount of capabilities, right? And so after a few steps, you are so overwhelmingly superhuman across every possible dimension that, yeah, you can say without a doubt that you have AGI, in fact, you have superintelligence. But yeah, intelligence is, in a sense, multidimensional. And what ARC is trying to capture is really just this fluid intelligence aspect, this ability to recombine core knowledge, building blocks. So in my definition of intelligence, intelligence is about efficiently acquiring skills and

• 47:08Speaker 02

  knowledge and recombining them to, well, again, efficiently recombining them to adapt to novel tasks, to novel situations that you cannot prepare for explicitly. And... Purely the ability to take a bunch of building blocks and recombining them during kind of program synthesis, that's one aspect of that. That's probably the most central aspect, which is why, you know, this is why we are focusing on with ARC. But it's not the only aspect, because this is assuming that you already have this pile of knowledge available. So it's not, it's overlooking the acquisition of these abstractions. It's also overlooking the acquisition of information about the task.

• 47:50Speaker 02

  In ARC, you're provided all the information about the task at once. But in the real world, you have to collect that information, you have to take actions, set goals, to discover what your environment is even about, what you can do within it. And you have to do these things efficiently, of course. And that efficiency aspect is very important, because, you know, intelligence was developed by evolution, it's in evolutionary adaptation. And when you're exploring the world, you are taking on some risk. You know, you might get killed by a predator, for instance. And so you want to be, you want to gain the maximum amount of information and thereby

• 48:32Speaker 02

  power over your environment by taking on a minimum amount of risk and expending a minimum amount of energy. That's not something you can measure, that's not something you can capture with ARC, V1 or V2 alone.

• 48:46Tim Scarfe?

  Can you just expand on the significance of the solution space prediction with O3? Because that rather suggests to me that it's almost this rich Sutton idea where it's nearly a blank slate and it's very empiricist and we just take the data in and the neural network does all of the things. I always imagined that we would need to have some kind of structured approach which took, you know, the core knowledge into account. Do you think that, you know, it's actually simpler than we thought?

• 49:15Speaker 02

  Trying to directly predict the output versus trying to write down the steps to get the output. They are not entirely separate things because of course, once you've written down the steps, you can do what looks like transaction. And O3 is not actually a real transaction model because it's much closer to a program synthesis model, where it's searching for the right chain of thought to describe the task and list the sequence of steps to solve it. And once you have the chain of thought, you can just use the model to execute it, and it gives you the output. So from the outside, if you treat the entire system as a black box, it looks like transaction, but the same would be true of any program search system.

• 50:00Speaker 02

  What it's actually doing, and the reason why it's able to adapt novelties so well, is because it's synthesizing this chain of thought, which serves as a recombinational artifact for the knowledge and the skills that the model has. That recombinational artifact is adapted to the particular task at hand. So it's much closer to a program synthesis model.

• 50:23Tim Scarfe?

  This is something that the community found very confusing because in the last interview, you were describing, I think it was O1 Pro, as being a kind of explicit search process.

• 50:32Speaker 02

  Yeah.

• 50:32Tim Scarfe?

  And what seems to be the case is that it's, you know, there is some kind of reinforcement learning thing in the pre-training, and then it maybe does some sampling inference time. So, you know, we're doing a whole bunch of completions. And are you saying it's as if it's doing a program search? Or are you saying it's somehow explicitly doing a chain of thought, you know, program search?

• 50:52Speaker 02

  It's searching over the space of possible chain of thoughts and finding the one that seems most appropriate. So in that case, it's entirely analogous to a program search system, where the program you're synthesizing is a natural language program, right? A program written in English.

• 51:08Tim Scarfe?

  Okay, it just seems a bit strange, doing auto regression on a language model, how that could be characterized as a search process.

• 51:15Speaker 02

  It's, so a model like O1 Pro, for instance, or O3, is not just auto-regressive. It has actually this test-time search step, which is why. It can adapt to novelty much, much better than the base models are purely auto aggressive. And that's why, again, you see on, on, so in general, ARC, even ARC one, has completely resisted the pre-training, the purely auto aggressive pre-training scaling paradigm. Like from 2019 to 2025, we scaled up these models by like 50,000 X, like from, from GPT two to GPT four point five. And even on ARC one, you went from a zero percent to something like 10%. And on ARC two, you're, you're going from zero percent to zero percent, right?

• 51:59Speaker 02

  And meanwhile, if you have any system that's actually capable of doing test time adaptation, like test time search, like O one Pro or O three, then you're getting, you're getting much, much better performance than this huge performance gap. So in general, you can tell the difference between a model that does not do test time adaptation and the model that does by looking at this performance gap, this generalization gap on ARC, also by looking at latency and by looking at cost. So of course, a model that does test time search is going to give you your answer. It's going to take much longer, like if you look at O one Pro, for instance, it's taking 10 minutes to answer your queries and it's going to cost you much more as well because of all this work it's doing.

• 52:39Tim Scarfe?

  So I could download the Deep C car one model and I'm running it on my machine. And as far as my machine is concerned, it's just a normal L L M, it's doing greedy sampling.

• 52:48Speaker 02

  That's right, that's right, which is why, which is why it does not adapt to novelty and it scores basically zero on ARC or like one percent, maybe.

• 52:55Tim Scarfe?

  Oh, so, so, so you're saying there is something different about?

• 52:57Speaker 02

  O three is qualitatively different, that's, that's correct. It is qualitatively different from all the other models that came before. It is actually a model that has fluid intelligence, it has a non-zero amount of fluid intelligence. And, and, and R one, for instance, is not.

• 53:12Tim Scarfe?

  Okay, so categorically, it's doing some kind of active search processor inference.

• 53:16Speaker 02

  That's what it looks like. So of course, I don't actually know how it works, but that's, that's what I would speculate it looks like, yes. And you see it in the latency, in the cost, and of course, the ARC performance.

• 53:25Tim Scarfe?

  Would you be shocked and surprised if you, if it came to light that it was just doing like auto aggressive greedy sampling?

• 53:32Speaker 02

  Honestly, I think it's very, very likely because it's completely incompatible with the, the characteristics of the system that we know of, that, that we were exposed to when we tested O three.

• 53:41Tim Scarfe?

  Awesome, and do you think that there will always be human gaps?

• 53:46Speaker 02

  Probably not always. Today, there are very clear, very significant gaps, right? Like, we're not actually that close to a G I. right now. But eventually, you know, as we get closer and closer, there will be fewer and fewer gaps. And at some point, you know, we're going to have AI system that are just overwhelmingly superhuman along every possible axis you choose to look at. So, you know, I don't think that there will be gaps forever.

• 54:11Tim Scarfe?

  Mike Tim, thank you so much for doing this.

• 54:12Speaker 02

  Thanks, Tim.

• 54:13Tim Scarfe?

  I really appreciate it. Looking forward to seeing it in a couple of weeks.