R1, OpenAI’s o3, and the ARC-AGI Benchmark: Insights from Mike Knoop

In this episode of Gradient Dissent, host Lukas Biewald sits down with Mike Knoop, Co-founder and CEO of Ndea, a cutting-edge AI research lab. Mike shares his journey from building Zapier into a major automation platform to diving into the frontiers of AI research. They discuss DeepSeek’s R1, OpenAI’s O-series models, and the ARC Prize, a competition aimed at advancing AI’s reasoning capabilities. Mike explains how program synthesis and deep learning must merge to create true AGI, and why he believes AI reliability is the biggest hurdle for automation adoption. This conversation covers AGI timelines, research breakthroughs, and the future of intelligent systems, making it essential listening for AI enthusiasts, researchers, and entrepreneurs. Mentioned Show Notes: https://ndea.com https://arcprize.org/blog/r1-zero-r1-results-analysis https://arcprize.org/blog/oai-o3-pub-breakthrough 🎙 Get our podcasts on these platforms: Apple Podcasts: http://wandb.me/apple-podcasts Spotify: http://wandb.me/spotify Google: http://wandb.me/gd_google YouTube: http://wandb.me/youtube Connect with Mike Knoop" @mikeknoop Follow Weights & Biases: https://twitter.com/weights_biases https://www.linkedin.com/company/wandb Join the Weights & Biases Discord Server: https://discord.gg/CkZKRNnaf3

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:02Lukas Biewald?

  You're listening to Graded Descent, a show about making machine learning work in the real world, and I'm your host, Lukas Biewald. This is a conversation with Mike Newp, who is both an AI researcher and an incredibly successful entrepreneur. He started a company called Zapier about 15 years ago with a very small amount of funding and grew it into a very large business. He then got up to speed on the frontier of AI research and recently started an organization called NDA, which is one of the new research labs working on the forefront of AI. This is a really interesting conversation. We go into business and how AI fits into business, and then also some of the details

• 0:47Lukas Biewald?

  on how the new R1 and R1.0 models work, especially on the AGI prize, which he funded and made popular. I really hope you enjoy this conversation. Why don't we start with R1 and work backwards?

• 1:02Speaker 01

  Okay, sounds good.

• 1:04Lukas Biewald?

  Let's do it. So R1, obviously this suddenly famous model coming out of China, do you want to maybe just describe that briefly and then talk about R1.0 as well and pontificate a little bit?

• 1:19Speaker 01

  So, yeah, these, basically two actual models got dropped, R1.0 and R1. They are very similar in sort of nature to what I would call OpenAI's O1 model. They're, you know, these reasoning models, and they were trained in a similar way. We know for sure how R1 and R1.0 trained because DeepSeek chose to open source their sort of training methodology for R1 and R1.0. They really did move the science forward here. I did see a public comment from, I think, Mark Chen, who leads research at OpenAI on this stuff, that, you know, he shared that like he thinks that it's similar in spirit to how O1 was trained as well. So I think there's some, probably a pretty good agreement here between sort of the ideas that have gone into both of these, both of these systems.

• 2:05Speaker 01

  And fundamentally, the whole, like, the O series from OpenAI, this like R series from DeepSeek are fundamentally a paradigm shift from the type of AI systems that we've seen in the past. For example, from OpenAI, like GPT series, you know, from three to three and a half to four to four O. These are all the same sort of like broad paradigm of scaling up pre-training, where we're trying to make these sort of models more intelligent by feeding them, you know, more data, making the models bigger. Now, it is also the case that four was actually a little bit like this smaller, you know, there's probably some distillation that went into four to make it more efficient. But roughly, we're sort of in the broad paradigm of making models smarter by trying to like give them more human data. And they're effectively memorizing answers.

• 2:50Speaker 01

  This means that they really have no ability to adapt to novelty. May have heard of, you know, the ARC Prize, the ARC AGI benchmark. What this is, is it's a benchmark that tries to assert or assess an AI system's ability to solve problems that it hasn't seen before. The data set's highly resistant to just like being able to memorize the answers. Even if you're given the training set for ARC, it's, you can't just memorize the training set and solve the test set. This is kind of another underappreciated sort of public point. It legitimately is very hard. That's why it went unbeaten, V1 went unbeaten for sort of five years. And, you know, in December, we had this big news moment with OpenAI. It was O3, you know, they had the 75% score on ARC V1.

• 3:34Speaker 01

  They had this 85% score with this really expensive, high compute performance version of it. And this was showing that this kind of like new reasonings type of system is, has a fundamental new capability that we've not had in computers before, that they have the ability to adapt to novelty. And this actually has some implications in, you know, beyond just that quirky fact. This will actually lead to more sort of robust and reliable AI systems too, which we can, we can touch on, it matters a lot for this agent stuff.

• 4:01Lukas Biewald?

  Wait, before we go down that path, they're like immediately off track and I love it. But you know, when you talk about like novelty, like, you know, how do you define that, right? Because I feel like there's a lot of debate on like, are these systems just memorizing or not? And they're obviously not just like memorizing text and then like regurgitating exactly the same text. I mean, it's quite clear there's like some, you know, adaptation. So.

• 4:26Speaker 01

  Yeah, there's some form of generalization that comes from like compression, right? Because like, it's literally, it's not literally a database, which would be like a lookup table.

• 4:32Lukas Biewald?

  Right, right, exactly.

• 4:33Speaker 01

  If you had one parameter for every fact, okay, you could just have a database. You know, these like GPT-style class systems do compression. That is where you get some of the like interesting generalization coming from. The sort of claim that I would have is that the amount of generalization they're capable of doing is fixed because the architecture is fixed. Effectively, you've got the transformer underlying architecture in every language model up until, you know, last year. And that means that your sort of level of intelligence is fixed. Yeah, you can memorize more, but the amount of generalization you have, the amount of adaptability you can do from your training data to a new novel situation, it has been fixed and that has not changed up until 01.

• 5:12Speaker 01

  01 was the first prototype we saw last September of a system that actually had a legitimately increased amount of intelligence in its ability to be given a fixed amount of input information, be able to do more things accurately. They're further away from its training data.

• 5:28Lukas Biewald?

  But wait, so like, what would you, would you say that, like, for example, AlphaGo couldn't generalize? It seems like in the domain of Go, it could reason pretty well and adapt to novel situations.

• 5:41Speaker 01

  This one's true for Arc2, right? And the Arc's been around for five years. There's a lot of, you know, state of the art in Arc from a pure solver standpoint. If you disregard the, you know, front-end LMs, was like something like 50% coming out of the Arc Price 2024 contest. But they're very, very domain-specific solvers. You know, they required researchers to have a, they basically.

• 6:04Speaker 01

  You know, these systems have all of their G, the G factor, the generality is being put into the sort of system that solved ARC during the contest. They came from the researcher's brain, right? Because the researcher is thinking about the problem. They're trying to model the problem and say, okay, here's what I'm trying to get this computer to do. I'm going to like encode my own understanding of the problem into the system, into the architecture in order to get this thing to work. And it ends up being fairly sort of, you know, constrained and domain. And it doesn't generalize well. And this is true for AlphaGo 2. You know, we've had AI systems for years and years and years that can be superhuman at humans at games. But like the fact is historically remained that, you know, what can you do that these AI systems can't? It's the fact that I could like sit you down, look at this and teach you a new

• 6:49Speaker 01

  card game, new board game in like a couple hours and get you up to human level proficiency. You know, I could take you in a totally new game. I could go teach you how to drive a car you've never driven before and get you up to proficiency probably in a couple of days. This like ability to adapt on the fly to a situation or a type of problem that you've never seen before, but never trained on is what has been historically very unique and special to you as like a human relative to the AI systems that we have today.

• 7:11Lukas Biewald?

  And it's probably worth saying, like, I don't know how you, ARC is, I think, better looked at than described in words, but you know, I think what's

• 7:19Speaker 01

  We should like overlay like a puzzle somewhere here.

• 7:21Lukas Biewald?

  Totally, yeah, it's for sure to overlay a puzzle. But you know, for someone just like listening, you know, I think what's astonishing about ARC is maybe how easy these puzzles seem to be and how these systems do actually fail on these easy looking puzzles. But maybe you could describe a little bit more of what it is.

• 7:38Speaker 01

  It looks like an IQ test. It's a grid of colors. It's a, you know, a 2D grid of colors. We're given some examples of input and outputs. And your goal is to find the rule, to find the pattern between what's the common consistent rule between these inputs and outputs? And then apply it on a test that you're given. And yeah, you're right. It's easier to visually describe or visually see than sort of describe. But ARKs basically challenges you to like on the fly recompose knowledge that you've acquired throughout your life, basically on what we call these core knowledge priors. These are things like symmetry and rotation and object detection and tracking and basic understandings of physics.

• 8:19Speaker 01

  ARK requires you to kind of abstract and compose those core knowledge priors on the fly to a task that you've never seen before. And this is what the sort of historical LM scaling paradigm is never even good at. Like GPT-4 scored 4%, for example, on the ARK dataset. In contrast to the really impressive performance we started to see from these O systems.

• 8:40Lukas Biewald?

  Okay, so what changed from like 4O to these O systems that are doing well? Like what, what do they do differently to make it work better?

• 8:48Speaker 01

  Yeah, I mean fundamentally, like the strongest thing I can say is from my capability, I can make a very strong capability assertion and then I can have some informed speculation on how. You know, the capability assertion, you see this in just the score. If you look at ARK V1, right, for over a five-year period from when it was first introduced in 2019 up through basically last fall, the best sort of LM, GPT-4 got 4% on it. And then O1 came out, O1 Pro came out, O3 came out, and you saw the score rapidly go from 4% all the way up to 75, 85% on this really extreme high-end performance version of O3. It looks like a straight line, like it's pretty, pretty nuanced. And this is actually a really good thing to see in a benchmark, actually. This means there's more signal in the benchmark and it truly is doing a capability assertion.

• 9:32Speaker 01

  You know, it's a little tougher to understand capabilities by looking at these like kind of monotonically smoothly increasing benchmark scores that go slowly over time. When you see something have a sharp bend, you know that something has distinctly changed. And I think that's the case with these reasoning systems. And specifically, the thing that they have done is they have added the ability to recompose knowledge that they've been trained on in their foundation models at test time. OpenAI calls this like test time compute. It's kind of the broad paradigm of we want to use more test time to, we want to use more compute at test time to like think before we just jump to an answer. You know, GPT-4, oh, you start splitting out tokens in 500 milliseconds and you know, that's its answer. Whereas, you know, the intuition here is we want to allow these systems to have more time thinking upfront before they have to render, render a view.

• 10:17Speaker 01

  And the way they think is by what's called this like chain of thought paradigm.

• 10:23Speaker 01

  You might have heard like, let's think step by step, right? This like CROMTAC that came out literally like three years ago. This paper actually is very special to me. It was one of the things that got me first into go all in on AI. But this like COT moment that happened in literally January 2022. Like, we're just sort of like downstream of that moment still. We're trying to figure out, how can we apply this paradigm of chain of thought about having the models basically think out loud to themselves in a sequence? Where you say, okay, hey, what's the next step to solve this problem? Okay, now take the next step. What's the next step? Okay, now what's the next step? And you do this sort of in a big, in a big, in a big chain. And then the model is able to use that entire sort of chain of thought trajectory to ground its final answer. And this is what O1 does. This is what R1 does.

• 11:07Speaker 01

  This is what R1.0 does. There is some differences between R1.0 and R1. Well, we should talk about that are very important, but roughly this is.

• 11:14Lukas Biewald?

  Before we go there, maybe let's take a moment to be like astonished that this works, right? Because you're saying that these like LLMs are compression algorithms that, you know, kind of can't reason. They can only sort of, you know, do some kind of like limited, you know, generalization. They're now like generating text and the chain of thought, you know, paper had them generating text in steps. So these, these kind of dumb models are generating text now in steps. And suddenly you claim they can reason like that. That doesn't seem obvious that that would.

• 11:48Speaker 01

  I don't think it is. And it's one of the reasons why I, like I said, I went all in on AI back in 2022. There was this like up until January 22, too. And I co-founded Zapier 15 years ago, but working on it, I was an exec at the time. I was like running all over product engineering org, like half the company, building our new product stuff. And I saw this and I had been paying attention to AI. And I saw this chain of thought paper that came out. And I thought I had a good perspective of like what LLMs could do and couldn't do up to that point. And then this paper dropped where it was like, oh, just by asking them all to think out loud, you saw these performance scores on the reasoning benchmarks at the time were showing really, really large spiking behavior and they've grown from 30% to like 70% or so. And that was like a smile, I kind of like, oh, shoot moment. Are we on track for AGI with this technology possibly?

• 12:32Speaker 01

  And I felt that was really important to know first, just from a Zapier standpoint, like, should we start using this in our products? And also, you know, just from our human, I just want to know, this is really, I think this is some of the most important technology in the history of the world. And so I kind of went all in to start understanding the sort of paradigm here. One interesting thing about that old school version of Chain of Thought, like Zapier was like proud stepping with Chain of Thought in fall 2022 as well. And with a very similar paradigm of like, ask model to iteratively prompt it out loud. It has a very low degree of ability to adapt. And we see this even with O1 and R1. You know, the scores on Arc are only about 15%. It is a big step up from the sort of GPT-4O class, you know, the 5% territory.

• 13:15Speaker 01

  But it's still really, it's a relatively weak amount of adaptation. The thing that has really made these things work with O1 Pro and O3, and this is getting informed speculation, is they've added search and sampling on top of the COT generation process. They're not just asking for a single COT and then say, okay, now give me the answer. They are sort of generating multiple COT steps in parallel and saying, okay, which one's the best one? Okay, use that one. Now go to the next step. Now ask me for a bunch more. Pick the best one. Go to the next one. They're effectively doing this like program synthesis or program search at inference time. And this insight has allowed a significantly higher degree of like adaptation as what's getting up to the 75, 85%.

• 13:59Speaker 01

  And I'll be like, you know, to your original point, this is astonishing. Like it's a big update. I think these systems demand like serious study. And it's also why I'm very excited to see R1 and R1.0 open source, because I think that's going to allow more people to do the science.

• 14:12Lukas Biewald?

  Totally. And before we get into the R1 and R1.0, what like, what's the human performance on Arc? And where are these best models at today?

• 14:26Lukas Biewald?

  Like when you say 85%, put that in context of of human level performance.

• 14:31Speaker 01

  Humans, like smart humans can get up basically 100% on Arc V1.

• 14:35Lukas Biewald?

  I see.

• 14:35Speaker 01

  Yeah, if you like, the data we have is like, if you take two, like, Let's call STEM graduate humans and put them in front, they'll get 100%. Like, I think that actual data we have is like 98, 99% across two. This has been one of the actual flaws of V1 is that we haven't had like strong assertions of human capability. Something we're fixing in V2. We've actually been working on V2 for years now, actually. We put a lot of effort into building it last summer. And we're putting the final touches on it right now and we're going to wash it with ARC Prize 2025 this year. One of the things we have with V2 is actually strong human study baseline testing to make confident assertions that every single puzzle in the benchmark is solvable by humans in order to justify the easy for humans. I do think that that is the spiritual guidance of like the ARC benchmark in the future is,

• 15:20Speaker 01

  I mean, this is true for V1, will be for V2 and V3 and all future versions, is that we want ARC to represent this concept of things that are easy for humans and hard for AI. And I think that's the gradient we're driving to zero. Because I think if you could get that gap to zero, you legitimately, I think it's going to be hard for anyone to claim we don't have AGI. If there's no, if you can't find a single thing that like humans find easy, but humans, computers find hard. You know, I think that's a reasonable like goal to set and a target to shoot for. And this is, this is in contrast to like how a lot of other frontier benchmarks work. You know, Dan Hendricks, like humanity's last exam or Epic AI's frontier math. Now they're investing in making like these ever more difficult benchmarks. And I, and I think that's a fine thing to do, by the way. I don't sort of discount that effort at all. I think it's useful.

• 16:03Speaker 01

  But I think it's becoming more important to understand from a capability assertion standpoint around like, well, what do, what do humans still find easy? I think that's more revealing of something we're missing in these systems still.

• 16:14Lukas Biewald?

  I totally agree, although it's funny. I mean, it's been such a long line in my lifetime and especially in the last year or two of like, you know, these benchmarks just being like, okay, this is human level performance. If we get there, we're going to have AGI and then like over and over and over and over, you know, beating these benchmarks. Like, do you, I guess it's really hard to make this claim with certainty, but do you feel like ARC could be like the last of this genre? Like, is there something else out there that, that, you know, AI solves this? But then there's something left as a benchmark that it can't do, that humans can do easily?

• 16:49Speaker 01

  What I can say more confidently is that we're going to run the gap of easy for humans hard for A to zero. You know, I think that is a up to the world to decide is that easy? I personally think it is. But that is our, that's the design philosophy we have for V2. That's going to be the design philosophy for V3 and future versions. I'll say this, like, you know, V2 is going to look probably pretty similar to V1 in terms of domain because we've been working on it forever. It is harder for computers, but it's still easy for AI. V3 will likely be something that looks very different from what we've done so far. I think it will look like Arc still, but like, I think it's going to test different capabilities that humans still find quite, quite easy that the sort of current benchmarks don't. And it will likely also include a sort of a formal version of measuring efficiency,

• 17:33Speaker 01

  which I think is also going to be a really important thing we care about getting up to G I.

• 17:36Lukas Biewald?

  Okay, so, so R1 and R1.0, can, I mean, can you describe what those are and how they do, like, why they do well on, on Arc?

• 17:47Speaker 01

  So R1.0, R1, both kind of in the spirit of being trained on COT, change of thought, and then generating them, basically a single COT and then, you know, giving us an answer. I think the really important thing to know about is how, what differs? Why is our, why are, why are there two models? Why is R1.0 and R1 different, separate? This is a very interesting thing, like, they didn't have to release R1.0. In my view, R1.0 is a more important system to understand than R1. And for this reason, what does the zero stand for? The zero stands for no human data in the training.

• 18:21Lukas Biewald?

  Like Alpha Zero, I guess, is homage to that, probably, right?

• 18:24Speaker 01

  Yeah, they're using it, they're training it purely on R L, reinforcement learning, they're using domains and math encoding to create verifiers where they can have the model DeepSeek V3 generate a COT and then they can use a formal domain of, like, literally just run a computer program and feedback, was that right or wrong?

• 18:41Lukas Biewald?

  But just to be clear, just to be clear, before we go down that path, so they're not, obviously, there's like a, an earlier training step where there is human data in it, right? I mean, they can't, I mean, how does it learn?

• 18:53Speaker 01

  There's a foundation model, DeepSeek V3. This is not a forever given, though. I think this is going to be an engineering try off that AI system developers have to make in the future, which is how much knowledge do we put in the foundation model that we start from? And how much do we have the system sort of, you know, generate itself at like test time, basically at runtime? I think likely what you'll sort of, my sort of expectation is you're going to use these reasoning style systems basically to generate new knowledge and add them back into some like ever growing corpus of knowledge that they have. And you'll probably use those as your starting points in future training runs and future inference systems that you get. So yes, it's, it is the case, obviously, Fora was trained on human data, DeepSeek V3

• 19:37Speaker 01

  was trained on human data. But this is not going to be likely the forever case in terms of the capabilities of these systems. I think you will see, and this is why R1.0 is important to look at, if it's able to bootstrap itself up in knowledge from literally first principles, like math, you know, give me the piano arithmetic operators and I'm going to bootstrap up Calc 3. Like that is something legitimately that a system like R1.0 could potentially do using RL with their humans in the loop.

• 20:05Lukas Biewald?

  Okay, so they train the system on, they train like a foundation model on human data, and then what's this next step that they're doing with R1.0, just like specifically? So they're, they're generating chains of thought?

• 20:16Speaker 01

  Yeah, yeah, it's a general single chain of thought, you know, step by step, taking the full chain of thought, and then the whole thing is being used to get the final grounding for a final response out from the same DeepSeek V3 model.

• 20:27Lukas Biewald?

  And how does the feedback get incorporated?

• 20:29Speaker 01

  That is on offline at training time. So what I just described is kind of like a test time, right? The inference time, how they work, right? It's like, okay, user inputs query, I'm going to like generate one COT very long and give you a final answer. All right, come here, yeah, let's kind of go forward. The feedback loops at, at training time, this happens offline, right? The developers who did this months ago, you know, where they took the DeepSeek V3 model and had it generate lots of COT, and they had another computer program looking at all those COTs and giving the sort of training loop feedback of like, was that a good thought or a bad thought? And that RL signal is basically being used to sort of fine tune the ultimate R1.0 model. And then in the case of R1 full, they actually also allow human experts to label those.

• 21:13Lukas Biewald?

  But in the case of the zero, how did they? How do they have the model that knows if it's a good thought or a bad thought? Like, how do you cold start that?

• 21:22Speaker 01

  This is what's different from 0.01. So 0.01 uses a greater model where they have like a neural model basically giving feedback and saying, like in text form, "Oh, good job, bad job," kind of like style stuff. 0.1, it's purely a symbolic verifier, right? So they are like saying, "I'm gonna run a computer, I'm gonna take this like, you know, potential code that you've just outputted, try to symbolically run it and use it to give a feedback symbol, like, is it good or bad?" And that's generally what you're seeing in like this RL space, right? It's like you wanna get a sort of final answer, and then you're gonna symbolically try to verify that and make a like 100% reliable assertion of whether that was right or wrong, which is you can do it in domains where you can run a computer program to verify, you

• 22:03Speaker 01

  know, a Python program, a math equation, a piece of code, does the code compile, does it get the answer I expect? You know, you've got benchmarks, we have all this stuff set. This is different from, yeah, some of the 0.1 stuff where they're actually using a process greater model, which is a separately neural trained one that we, again, this is all in full speculation, they haven't shared any of this, so this is all like details I'm just like trying to pick up and understand based on these systems, but that's what I understand is they have a separate actual LM model that's giving feedback in their training loop.

• 22:32Lukas Biewald?

  So in the case of 0.1.0, it's pretty cool they've been so transparent about how they do it. Do...

• 22:41Lukas Biewald?

  Like, is it possible that there's domains like computer programs that are really easy to do kind of chain of thought and then verify, but then other domains where it's harder? And so you end up with models that are specializing towards the domains where it's sort of easier to tell if the result is accurate or not?

• 22:57Speaker 01

  This is the bet, right? Like, can 0.1.0 scale up without having to add humans in the loop? The evidence we have is that 0.3, which does way better on ARC, like basically effectively beats the V1 dataset, required supervised fine-tuning from humans to get that to work at a high enough degree of efficiency to make it like tractable. We don't have any evidence yet that you can bootstrap a pure RL-based LM chain of thought system to get there. That's like what you're probably gonna see happen though in 2025, I would expect.

• 23:27Lukas Biewald?

  I mean, I think another obvious thing that's, you know, come up with these models that people have noticed is that they were a lot cheaper to train by orders of magnitude. Why, do you have a thought of why that is?

• 23:41Speaker 01

  I haven't looked deeply into this. The only commentary I probably can offer here that's interesting is the ticket price is what people often are comparing here between like, what's the commercial pricing rate of like O1 versus what the DeepSeek commercial rate is for their model they're hosting? And because R1, R1.0 are open, you can actually run them on your local infrastructure. You don't have to go pay the rate. Like, they're giving you basically an ad cost. Like, it's very close to the actual ad cost. And OpenAI has got a margin. They're a company. They got researchers to feed. They got future research to invest in. Like, they're building a business there. So like, my, again, informed speculation is that like, there is quite a bit of margin built into O1's inference costs right now

• 24:27Speaker 01

  in order to fund future R and D. And that the true costs are probably more comparable than most people expect. At least for R1 and R1.0 versus O1. I think there's this other debate of like, how much did DeepSeek V3 cost to train? You know, are we hiding the fact that like, this headline number of, oh, it costs $5 million to train R1, R1.0, upfusks the fact that there was like a bunch of money that went into training the foundation model? And that is accurate. I have no idea how much the foundation model trained. I haven't looked into this. I have no sense of what it cost. I don't know what a GPT-4 cost either. But I would maybe make the single comment on like, from an inference cost, I would sort of broadly put these like, our systems and our systems in somewhat of the same bucket

• 25:10Speaker 01

  from a cost basis, like broadly speaking. Got it.

• 25:13Lukas Biewald?

  Okay. All right. Well, thanks for jumping in. I feel like maybe we should take a step backwards. I think, you know, when I first met you is like maybe more than a decade ago. You know, you're the founder of a company called Zapier, which I've long admired, but maybe some people have not heard of. You want to talk about what Zapier does?

• 25:33Speaker 01

  Yeah, Zapier is an automation company. We try to deliver automation software that's very easy to use for non-technical folks in order to automate largely parts of their business. We're used predominantly by individual people, either their own life, on small teams, individual teams with enlarged organizations. This is very much easy, it's intended to be very, very easy to use. And that's in contrast with like historically, the sort of automation system that has existed out there. We're used today by, I think, several, I don't know, three, four million businesses in the US. We have a pretty large international presence as well from a customer base standpoint. And the core idea is you can connect apps to use.

• 26:16Speaker 01

  So you've got business software used like Gmail, or Slack, or Salesforce, or whatever. And businesses build internal workflows around these processes. They often have humans that have to shepherd data between the different steps, make decisions. And this is something Zapier can completely automate for you. And it's easy enough to use for a line of business type of folks that you don't have to get an engineer involved.

• 26:38Lukas Biewald?

  And I think some of the notable things about Zapier also is, I mean, this might be unique. I mean, you raised a tiny amount of money from VCs, and then you got off the VC train like almost nobody does, and got to incredible scale. I mean, it's been an amazing return for your initial investors, and I think you're one of the first companies-

• 26:56Speaker 01

  We debated even raising the money in the first place.

• 26:58Lukas Biewald?

  What's that?

• 26:59Speaker 01

  We debated even raising the money in the first place. You know, we, it was a very practical decision, frankly, to sort of do it. The three founders, me, Brian Wade, we're all from Midwest. I grew up in St. Louis. And there's no venture capital market in the Midwest, really. Maybe they're just a little bit, but like, you know, 2010, nothing. And so how do you build a business? Like, build a useful service product, you sell it, you use the proceeds to invest back into the business. And so that was kind of our bias, and how we kind of initially operated. And we went through YC, and we were really debating, should we, we've got this demo day moment, everyone else is going to raise money, should we? And we did office hours, actually funny enough with Sam Altman, who was the guy who

• 27:42Speaker 01

  we chatted with, and we're like, should we raise or not? And I think the question he asked us was like, well, look, what's the constraint of the business? Yeah, it was a good question because I think the legit concern of the business for Zapier at the time was Brian, Wade and I were all waking up and doing support until like noon every day. And so it was taking away time to make the product better to not have as much support in the first place. And we're like, oh, well, we should go hire a support person. And we didn't have enough cash on hand yet to go hire that person. So we decided to go raise a small round in order to like, you know, be able to hire someone ASAP to like, you know, give us more product engine time back. And the funny anecdote, sad anecdote is that by the time we like got the round raised, got the money in the bank, found the person we wanted to hire, got them on board, started payroll,

• 28:24Speaker 01

  and they had their first paycheck, revenue had actually caught up enough to just pay them directly. And so I'm pretty sure you could like trace the million dollars that we raised like in lineage all the way through like to today. Which, you know, it did, it did allow us to like activate though and start, which I think is honestly the biggest value we got out of Y C, which was just like the activation moment to like really go on and get and go full time on it.

• 28:47Lukas Biewald?

  Amazing. And you were also a remote first from the beginning, right?

• 28:51Speaker 01

  Yeah, another weird thing Zapier did. Yeah, we were globally remote team and have been since 2011. I think the only other company at the time that I knew of was like WordPress, Automattic was like fully remote.

• 29:00Lukas Biewald?

  Yeah.

• 29:01Speaker 01

  Thirty seven singles. I think those are the only two that we knew about at least.

• 29:06Lukas Biewald?

  And then I think kind of the third interesting thing about Zapier from, you know, my perspective is you were very early to use L L M's. Like I think you were like kind of the first company that I saw with real L L M use cases. Like, can you describe what those were and what that experience is like?

• 29:21Speaker 01

  Yeah, I think this comes back to the chain of thought paper that came out in January 2022. Like I said, I actually, my background is in engineering at college. I did like mechanical engineering. I did optimization research, which ends up being like the exact same math as all this like deep learning stuff. I didn't figure this out until 2017, but like as soon as I realized.

• 29:41Lukas Biewald?

  You're on the gradient descent podcast, actually.

• 29:44Speaker 01

  Yeah, exactly. As soon as I like figured this out, I was like, oh, okay.

• 29:50Speaker 01

  Like I should, I know, I know how this works. It's like a demystified it a lot. And so I started paying more attention to the research side, but like Zapier was growing. I had other priorities, you know, we had to like, you know, grow the company, introduce new products, yada, yada. And so I really wasn't like that much paying attention. You know, I read the G P T two paper or the release played with it. I've read the G P three paper, gave a whole presentation to the company on it. And, you know, and then it's like, okay, cool. We can do some maybe basic stuff around the margin. Cool tech, but that's it. And then this like January 2022 Chainof Thought Jason's newspaper came out and that's what really got me to say, oh, this might actually be extremely relevant now for what Zapier customers are trying to do.

• 30:30Speaker 01

  And I actually went to Wade, who's CEO, and I said, Wade, I need you to take all like, take back all my, my half the company. You need, you need to go run product engineering because I need to just go do AI research here at Zapier and figure out like, what does this mean for us, for our business, for our customers? And so for a good six, 12 months, me and Brian, CTO, just like coded all day long and tried to understand what can this tech do, what can it do, what were the limits? And that got us, and this is like summer 2022, so we're still four or five months before even ChatGPT came out. And we had built graph, like we had built like tree of thought prototypes, we had built a version of ChatGPT internally using the technology, like we'd pretty much prototyped all this kind of foundational pieces in probably like three or four months and identified

• 31:12Speaker 01

  that like probably the most obvious place Zapier could start playing first was this like concept of tool use. Could we equip LNs that are frozen weights, right, that don't have the ability to take action in the real world? Could we equip them with the tools on Zapier's platform, all the actions, all the search endpoints that we have and allow them to do more? And that, that kind of activated us really early on to start building and delivering AI products. And I think that's the reason why we're so early and it also gave me a pretty, you know, this leads into the later story, but it was also the, I think what allowed me to start seeing the limitations of this paradigm really early as well because, you know, I spent, I talked to hundreds of Zapier customers trying to deploy this like AI technology in the middle of their automations to do stuff and, you know, these like Zapier has been deploying AI agents now for two years.

• 31:57Speaker 01

  So I've just gotten to hear like, what, what, what do people want from this tech? What, and what, where does it not work? And the number one problem that they all tell me, and it's consistent across the board, is like, the promise is there, I get what it can do for my business, but I just don't trust it enough yet to go hands off. And again, Zapier's an automation prize. It's different from ChatGPT, right? Where it's like you're typing on a keyboard and you get a response, you can audit it. Zapier is running on a server offline, you're not watching or monitoring it. And that was the feedback. I just don't trust it because the reliability is not high enough in how it's operating yet to not put a human sort of in the loop. And this was kind of like, this feedback was so loud and unchanging from GPT like three and a half to four to 4.0. And it was in the same contrast of like all this, you know, dogma around scaling hype

• 32:42Speaker 01

  that was crazy for 2023 into 2024. And just wasn't matching up with my lived reality. And I was like, okay, how the heck do I explain this? You know, I have two sets of facts that like are incongruent. And that's when I rediscovered Francois's podcast on Lex back for during COVID, which is when I think I first listened to him and learned about the ARC benchmark more. I'd actually like been thinking about it a little bit. But really dug into read the On Measure of Intelligence paper that he published in 2019. And that was kind of my aha moment because I thought that paper did a really good job of articulating the third promise of the technology that we've seen and why it's important and impressive, but also where it's fundamental limits on just scaling a pre-trained memorization we're going to hit to.

• 33:24Speaker 01

  It was leading to all the facts that I was seeing from customers. And once I got to that conclusion, I was like, well, clearly the ARC benchmark is the number one most important benchmark in the world. More people should know about it. And going into last summer, at least, it was a relatively obscure benchmark at that point.

• 33:41Lukas Biewald?

  And then you made a prize for the benchmark.

• 33:45Speaker 01

  Yeah.

• 33:46Lukas Biewald?

  Do you want to talk about that? Like that?

• 33:47Speaker 01

  I think the Zapier was where I kind of, in this phase is where you and I, like, you made the introduction to Francois, which thank you for doing that. I think that's hopefully been a very helpful thing for the world. I think you've got hopefully high leverage out of that introduction here.

• 34:02Lukas Biewald?

  Well, I got you on the podcast, so that's great.

• 34:04Speaker 01

  The world, the world thanks you, Lucas, for doing that. And yeah, so I'd been surveying, researching the Bay area, at least for a couple, maybe like a year or six months at that point, I'm like, hey, I think this is the most important benchmark. Have you ever heard of it? And the awareness rate on ARC was relatively low. It was like 10, 20, 20% of people I met had heard about it. Most of the people who thought they'd heard about it actually had heard about the old, bad version from like the Allen Institute that had been like long beaten biolinguage models. And it was just like this, you know, relatively obscure thing. You know, it had a Francois and this other sort of Swiss lab, like Lab 42, been running the small version of a contest for several years going to that point. So there was some evidence that it was like a robust benchmark. It wasn't like, you know, completely obscure, but certainly in the AI industry, it was

• 34:49Speaker 01

  a under, you know, the awareness was quite low. And, and I was like, well, this is direct, concrete evidence. It's probably the only major concrete public evidence we have that like, there are fundamental limits to prescaling. Every other benchmark in the world was like saturating up faster and faster, except for ARC. And I was like, awareness is clearly the problem. And so after your introduction, I flew up to Seattle and I got lunch with him and I was like, you know, I was pitching him all my ideas of like how to beat ARC, I had some ideas and it was a fun chat. But I also had some like pretty critical questions of like, well, why don't, why do you think awareness is so low? Why aren't you, you know, why aren't you working on it more? And he had really good answers for all, so the key questions I had. And, you know, I walked out of that, I had one question at the very end of my notes when I was flying up that I put together, which is like, you know, pitch, pitch ARC prize.

• 35:34Speaker 01

  And I was like, and I was like, okay, I think that, I think that it is true that this is just like, the reason awareness is low is because it's legitimately hard. You know, every frontier lab has given it a try. And that increased my confidence it was a really important benchmark that we should grow awareness around. So I was like, well, one way I know we could probably do that is with a prize. I had just seen Nat Freeman and Dan run the Viscous Challenge the previous year, which was super successful at growing awareness around this like kind of obscure problem and growing the status, growing the interest in it to get people to kind of shift gears and work on it. And so it's like, I bet we could do something similar. And that's where ultimately ARC Prize came from.

• 36:11Lukas Biewald?

  And so what happened when you launched the ARC prize? Was it successful in getting engagement with the, like, new engagement with the?

• 36:18Speaker 01

  Well, I think going into June 2024, maybe, again, like 10% of AI researchers had heard about ARC. I think coming out of 2024 of the year, like, as of December 31st, everyone in tech has probably heard about ARC. So like, I think we solved it. Honestly, my true response is like, I have been continually surprised at how much energy there has been around the benchmark. Yeah, we, like, I'll give you a concrete example of this. We, we were going into the end of the contest, like, early November. And, you know, we had a bunch of teams on the leaderboard. One of their requirements for winning the cash prize was that you open source your progress. This is an idea of trying to rebase line the progress in the community each year.

• 37:03Speaker 01

  And the number one team, you know, who was, you know, it was kind of the runaway winner. They had been on the top of the leaderboard all summer long. He had this like 55% score or something like this. And, you know, they emailed us the week before, like, four, five days before the contest. And I was like, hey, guys, we're not sure we want to open source our, our solution. And I was like, oh, man, shoot, is the like, you know, is this, is this gonna work? You know, did we get something fundamentally wrong about like how to structure this contest to like help make progress towards AGI go faster? And in the next 72 hours after that call happened at the close of the contest, there was like three other, two other teams that shot up the leaderboard from like, I don't know, 10th place up to second place.

• 37:47Speaker 01

  The, the number two team actually was like right at neck and neck with the number one team. There was like three different papers that got dropped within 24 hours of the contest. They just timed their ARC research papers to drop kind of like at the same time to like time it and enter the paper contest. And it was just this phenomenal amount of like energy right at the end of the contest that it was kind of just hidden from us. We just didn't see it all. And I think that happened again when we, when OpenAI's O1 model came out. There was just this incredible outpouring of demand for us to test O1 on ARC that I didn't expect. I mean, we had like thousands of people on Twitter like begging us to go test this thing. And so we did like, and it was brilliant, important. I'm glad we did it.

• 38:27Speaker 01

  But it was just like, there's been these moments of surprise, I guess, for me in terms of its relevancy. And I think how much like awareness we've been able to carry on it.

• 38:36Lukas Biewald?

  What kind of insights do you think have come out of wrestling with the prize?

• 38:42Lukas Biewald?

  Or with the challenge?

• 38:43Speaker 01

  In terms of like structuring contests like this?

• 38:45Lukas Biewald?

  No, no, actually, I mean, in terms of in AI, like, you know, like, what? What approaches have worked? Have we learned about how to build intelligent systems?

• 38:54Speaker 01

  So

• 38:58Speaker 01

  the classic way that people have been trying to beat ARC for four years was with program synthesis, pure program synthesis. The idea is to, you know, you build a domain-specific language, think about it as like little Python functions, Python transformations, and you build this DSL typically by human, like, looking at the puzzles and trying to make sort of guesses about what the transformations could be. And then you have a brute force search effectively over the space of all possible combinations of those transforms to look for ones that match your input and output, and then you apply them on the test. This is the kind of the classic way that you tried, people have tried to beat it up till, you know, this year, basically. It doesn't really work well. It's just very slow. It's very inefficient.

• 39:37Speaker 01

  But like, and it's very brittle, right? Because it's like all the sort of, you know, generalities built into the DSL that the human is sort of going after. And ARC Race 2024, there was a couple of brand-new approaches that were really interesting to to see that I think the sort of contest popularized. You know, I think the the first major one was early on about using, I guess, like, it's like an induction-based approach where you're trying to generate lots and lots of like Python programs using language models. And then searching over in like kind of hinting and sort of informing the language model and how they generate a Python program based on sort of the input from the puzzle in order to kind of guide the language model program generation.

• 40:18Speaker 01

  And this is like Ryan Greenblatt's thing. He got a really, really early score, like around 40% or so using a technique like this. It was very inefficient, like you'd you'd journey hundreds of millions of Python programs to do it, but it it kind of like showed some promise of this like program induction-style method.

• 40:31Lukas Biewald?

  How do you then like pick among, like, you generate those programs, how do you pick which one looks promising?

• 40:37Speaker 01

  You run them.

• 40:39Lukas Biewald?

  But.

• 40:39Speaker 01

  This is RL, right? It's like symbolically verified.

• 40:42Lukas Biewald?

  Is the ARC.

• 40:42Speaker 01

  When you generate a program, you can run the program.

• 40:44Lukas Biewald?

  But then how do you evaluate if it's? Is it easier to evaluate if an answer is right than?

• 40:49Speaker 01

  Yeah. Yeah, the data set has like, has the answer, you know? And so it's like, and this is like true of all, um, you know, math coding. This is like where R01, R1, R3 all really dominate, right? As in these sort of easy to verify domains where you have an answer and you can check it quickly. If a computer can check the answer quickly and with strong, you know, with exact incorrectness, then this, these types of things will, will work. And the other big one was test time training. This is the other really big kind of novel approach that we saw come out where, um, you know, both these basically are trying to adapt the novelty. It's like what they're trying to get at. Uh, and the way test time training kind of works is the ARC dataset has a public dataset and there's a private dataset. And this is the one that Kaggle contests, the actual money prizes attached to.

• 41:34Speaker 01

  Private dataset, no one has seen, or very few humans have ever seen in the world. And you're not allowed to see it when your test gets it in order to have strong guarantees around, um, ability to adapt. You're, you know, we're trying to reduce the chance of like cheating or we're fitting on the private dataset. And so what people figured out is that they can take the private dataset inside Kaggle and use that data as a starting spot to generate lots and lots of, uh, similar data using data augmentation. They might like change the colors, mirror the grid, things that don't change the semantic rule, but things that generate lots of like nearby permutations. They then fine tune a model using these like, you know, several thousand, tens of thousands of like, you know, uh, locally trained things, and then they inference it. And that actually was working.

• 42:17Speaker 01

  I think that's what got one of the top 50% scores, was this paradigm of, of test time training. And I do think that like, you've kind of got two really broad points of evidence now, even, even with O3 beating ARC V1, two broad sets of ways that you can, that we know about how to use language models to adapt to novelty. One is COT search, which is, which is like what O1 Pro and O3 do. They're doing lots of sampling, lots of search in their per COT step. And then you've got this test training paradigm, test time training paradigm where you want to take the, you know, situation. And try to create data augmentation around that in order to feed that back into the manifold and do inference on it. This is a form of knowledge recomposition, I guess,

• 42:58Speaker 01

  is a way to think about it. And ARC shows that both these approaches are pretty promising in terms of getting computers to not just strictly memorize what they've been working on before, what you had out before.

• 43:10Lukas Biewald?

  Do you think if ARC was broader and not just sort of in this domain of sort of like pixels and changing colors, if that would kind of break that approach? Like, it seems like you have sort of contained the domain of ARC by just the format of it.

• 43:25Lukas Biewald?

  Like, I remember, I remember reading years ago.

• 43:27Speaker 01

  There's pretty obvious ways you'd...

• 43:28Lukas Biewald?

  Okay. Or I just like, I remember reading Douglas Hofstadter's book as a kid, I think, where he like makes these sequences of numbers and you just try to like guess what the next number would be in the sequence. Like, that sort of seems like a variant of like an ARC-like reasoning challenge, but in a different structure.

• 43:44Speaker 01

  Yeah. This is something that's underappreciated about ARC, I think, or a misconception, like a common misconception about ARC, is that it is a visual benchmark. You know, I get why this misconception exists, because like we render it visually for humans to take. See, the intuition is like, ah, you know, our intelligence systems aren't good at ARC because they just aren't like good at dealing with visual domains yet. ARC is, should more be thought of as a program synthesis benchmark than it is thought of as a visual benchmark. And the intuition here is that like, classic program synthesis is exactly what you just said. It is given a sequence of integers in, in a sequence out, figure out what's the rule. How do you map the 1D line of numbers into another sequence of 1D

• 44:26Speaker 01

  array of numbers? ARC extends that into 2D for the first dataset. It's just like, it's a matrix instead of an array. And, but it's still at its heart, a, it's a program synthesis like challenge of like, okay, given a, given a matrix of numbers, now instead of a vector of numbers, what's the transformation rule? The exact same problem statement is still still surrounds. Now, to your, to your, I think to your kind of maybe curiosity here is like, isn't this kind of like domain, like constrained? Could, could you get into like non-formal domains with this kind of stuff? I think this is where... Like pure program synthesis is not going to be able to do that. You are going to have to, in order to like, kind of solve ARC the hard way and build a system

• 45:09Speaker 01

  that has a lot of domain generality, you're going to need to merge deep learning and program synthesis together into the system. Although, PPS is just going to be sort of too, I think fundamentally brutal, in order to like, represent everything there. You're going to need some degree of being able to like, create model descriptions and, you know, be a bit fuzzier on the edges around how to sort of capture that model. But ultimately, yeah, I think ARC is today, at least, a better articulated, at least V1, as a program synthesis benchmark that we haven't been part of.

• 45:42Lukas Biewald?

  Okay, so that actually, so that makes me think of a question, which is, is the 2D nature of ARC like essential to it at all?

• 45:53Speaker 01

  I think it's an important design principle, because it's trying to make us, I think one of the magic, this is all credit to Francois. One of the beautiful things about ARC is its ability to be, to capture your curiosity as a human. Right? You look at the puzzles, and I've given them a lot to a lot of people, my friends and family and everyone. You look at the puzzles, and based on all the hype you hear about AI, and all, you know, coming last year, and all of this, or even individual intuition of working with these systems, you look at them, you're like, you take the puzzle, you're like, oh, okay, that was super easy. And then you're told, like, AI can't do that yet. And you're like, wait, what? Like, are you, are you sure? Like, what if I just, what if I just like pasted it? Like, I'll just take a screenshot and throw it to you. Are you really, really?

• 46:36Lukas Biewald?

  Totally.

• 46:37Speaker 01

  It just like invites this curiosity moment of like, what's going on here? And that, that I think is, it's like, I think it's a really standout thing. And why we render the benchmark visually is to like, make a strong, not just to claim about something that like, is intellectually interesting about, you know, capability we don't have yet. But its goal is to inspire people to work on it. You know, benchmarks aren't useful if no one actually cares about them. And, and I think ARC has done a really good job, Francois did a really good job from the design standpoint, of making it something that captures your imagination. Makes you ask why and it pulls you down in the funnel of like, all right, let me just try, I have some ideas, let me just go try it. And it's goal is to inspire folks, right?

• 47:19Speaker 01

  Like, this is, this is the whole point of launching Arc Prize was, you know, we launched it to raise awareness on this important unbeaten benchmark and raise awareness of this like definition of AGI, this design principle we have around that benchmark, and to inspire more AI researchers to go try new ideas. You know, my sort of macro philosophy at this point in my life is basically, AGI is the most important technology in the history of humanity. We should be doing everything possible. Anybody who has any unique idea of how to go beat this thing should be trying to create AGI, should be trying to do it, myself included. And this is like, I was, I just got really frustrated around how much, when I met researchers, talk of venture investors, like how much funding, energy, mind share, just going into this of one paradigm. And like, even if it's the right one,

• 48:00Speaker 01

  the world deserves like a few counterbets just to make sure to be increase our overall chance that we're going to get there. And so I stand by that. I think we want the strongest global innovation environment we can get in order to get to AGI as quickly as possible. So we can live in that future and use to actually accelerate the future that like we all want.

• 48:19Lukas Biewald?

  Okay, so your new organization, is that, is that, is the idea that you're going to make some counterbets?

• 48:26Speaker 01

  Yeah, yeah. So India, so Francois and I, we launched a few weeks ago, officially announced it. Francois and I started to do, I'll call it like a, you know, intelligent science lab, basically. And our view is that the way to get to AGI, that it removes most of the bottlenecks that we currently see even from systems like O3, is a combination of deep learning and progress synthesis. We've actually been talking about this for, like, since we started launching Arc Prize. Like, we gave, like, we went on the university tour and sort of espoused these viewpoints. And you can go find those talks if you're curious to like get the gist of the idea.

• 49:00Speaker 01

  This makes the idea that we want to, you know, deep learning and program synthesis are two completely different paradigms. And program synthesis, by the way, program search is what's making O3 work. So I think we have very strong evidence now that this new, this is the new paradigm that is actually unlocking this new set of capabilities. And my belief is that I think in retrospect, you know, we're going to look at this O3 kind of, you know, beating R21 in December moment as the sort of starting point of another five to 10 year scaling journey on program synthesis. In a similar way that we look back in 2012 at like, you know, AlexNet beating the ImageNet contest. Funny enough, it was literally a contest they're running. And that's kind of viewed as one of the shotgun kickoff moments of, of deep learning. And I think we're kind of at that point today where we're looking ahead at the next five to 10 years and, you know, we, but what don't we have yet?

• 49:45Speaker 01

  Like, we don't have, you know, and this is sort of a, sort of colorful rhetoric, but like, we don't have the, you know, the transformer for program synthesis yet. Like, there's, there's a lot of technology that has not been invented. If you even just look at the fields here, you've got like, probably a million deep learning experts now in the world today, or at least engineers in the world today. You've probably got in comparison like a few hundred folks on the program synthesis side. And if this is truly the path to AGI, we're going to need to grow that field. We're right at the starting spot. And that's, that's our basic view is that we're going to need to merge these two paradigms in order to get extremely efficient AGI technology that doesn't have human bottlenecks in our terms with its learning process.

• 50:22Lukas Biewald?

  Okay, so what is program synthesis?

• 50:24Speaker 01

  The best way I can articulate program synthesis is probably by example. So it's actually in a really old field. It's older than deep learning is. It goes back to, you know, 70s, 80s and 90s and that stuff. And classic program synthesis is dealing with trying to figure out a program that maps a sort of integer sequence of numbers to another integer sequence of numbers. This is typically the benchmark that you've like, researchers have like cared a lot. There's even this really cool like thing called the like online database of like integer sequences. It's like literally researchers have like made hundreds of thousands of these sequences that you can use for this kind of, this kind of research. And you're given an input sequence and output sequence and your goal as a, as a, as a engineer is to

• 51:09Speaker 01

  Create a computer program that can automatically figure out what that what that sequence is. You might be surprised, or yeah, this is actually a very hard challenge. And the reason is because, you know, depending on the complexity of the program and how much hidden state there is, how many hidden variables there are, programs can be quite long or quite complex. And the longer the program gets, you get into this problem of trying to basically enumerate. This is kind of how people classically have tried to do it, is they try to brute force it. They try to just search over every possible program that could potentially that could potentially exist. And they just, you know, check it, like, let me plug in the input and see if I get the output. It's extremely inefficient, it's exponential, it's scaling, it's O of X to the end, it's like nuts. It's just like, you're never really going to be able to like make this work in a reasonable amount of time.

• 51:53Speaker 01

  But that's the that's the rough form of the rough form of the problem, I'll say. Is that you're trying to create programs. And I'll map this into O3, because I think like how O1 and O3 work, because you can actually use these systems, or at least you can in O1. You can kind of get a better feel for it, you know, with these chains of thought, right? Oh, I give O1 a question, it thinks for a while, it builds a big chain of thought, and then it gives you an answer. A way to think about that chain of thought is like a program. It's a natural language program, but it is a program. It's got like individual steps, and each step is kind of like a transformation of the latent space from, you know, the step, the thought before to the step after it. And you're constructing it, you're sort of constructing this program. And what O1, Pro, and O3 do is actually program recombinations, where they actually search over the space of all possible programs,

• 52:37Speaker 01

  very similar to how program synthesis, classic program synthesis, where you're trying to like enumerate through options to sort of find it. Again, the big challenge of program synthesis is, and by contrast, deep learning at PS, is that program synthesis can learn out of distribution, or you can find programs that generalize out of distribution. This is in contrast to deep learning, which cannot, right? Deep learning is like a paradigm where you have, you need to give it a lot of data embedded on a high dimensional manifold, and you can make quick, approximate judgment calls, intuitions for inferencing new data off that manifold. But they're not going to be exact, there's no guarantees of exactness, and you need a lot of data. And you're going to get like in distribution accuracy. Programs in this issue, you're looking for a program. You only need probably a couple of examples. You don't need a million. You don't need 100,000.

• 53:22Speaker 01

  You need like three in order to like find the rule contours of a program that does that. And once you find the rule and you have the program, it's going to work for any input, no matter what input you give it. So it requires very little data. It generalizes out of domain, but counter explosion is your problem. You just can't. It's just in traction to the search for all possible programs there. Or in a one's case, search for all possible reasoning chains. So how do you, so like, how do you, how do you do this? Well, the insight is to use the sort of pros and cons for both sides to merge them together. You want to use, you know, the upside of deep learning, which is to make quick approximate sketches, you know, to inform a search process in order to make sure the search process is not just brute forced. You don't want to brute force it. That's not how humans work. Humans don't sit here and think there are every, you know, 1,000 Python programs to solve an art puzzle.

• 54:07Speaker 01

  We use our intuition to generate, like to do the sort of, you know, sketch of a couple of possible answers to a solving puzzle. And then we symbolically in our head verify it. We run through the steps. We say, is this right? Is this right? Is this right? And if it's not, well, we go back to our, you know, deep learning part of our brain and say like, okay, give me some more ideas. And there's a smooth back and forth between the two, two systems. And we think that's sort of the fundamental substrate here that we can, we can construct around.

• 54:32Lukas Biewald?

  It's funny, you know, my, my co-founder Sean has been doing a lot of work on SWE bench. Like he recently, you got the highest score there and it's a, it's funny. He talks a lot about the same stuff you're saying. Like, I think, you know, he's, it's a very different domain, but he's using, you know, O3 and O1 to generate programs and then run them and then try to figure out like which is the best one. I guess the runtimes are expensive here, but.

• 54:56Speaker 01

  Efficiency is going to be a big factor. I think this is something the field of AI research has not reckoned with fully yet. You know, when we launched the O3 news, like we had to report that on a 2D graph where the X axis was effectively like token cost or like cost per task. Because you're, we're getting this paradigm now in AI where you can spend more money to get higher accuracy and higher reliability. Now it's still on a logarithmic curve, like, so, you know, it's not like it just goes linearly up here, but like you can spend more money to get a better answer in a lot of domains. And this means that.

• 55:31Speaker 01

  You can't report a single benchmark number anymore. In fact, ARK, we still need to fix this for ARK for next year. Like, we need to, like, have our leaderboard with, like, kind of an efficiency dimension in it somehow. Because, like, you can't just say, "Oh, well, okay, that system got 75% on the benchmark." Well, okay, well, how long did it take them to get that? How much did it cost them to get that? These are types of questions I think we're gonna have to, like, be able to answer and make assertions around in order to inspire and guide research attention to continue to drive efficiency. Because I think that's, like, there's a couple of human bottlenecks remaining in these types of reasoning systems. Once we get past those, efficiency is really gonna be the main key thing we have to, we have to figure out.

• 56:10Lukas Biewald?

  Although, it's, I mean, if you can, you know, use compute to get better answers, if we compute cost reliably, it goes down, right? So the shape of that, that curve really matters too, doesn't it?

• 56:24Speaker 01

  I'm not sure I follow.

• 56:25Lukas Biewald?

  Sorry, if you can, like, the slope even in log scale of the trade-off between, like, cost and performance, I think also, also really matters. Like, I would love to spend, you know, more money to get better answers in lots of domains.

• 56:39Speaker 01

  Yeah, there is definitely a lot out there. One of the major ones that's gonna start working this year is agents. Coming back to the Zapier anecdote, you know, the number one blocker for deploying automation with agents right now is reliability. Trust isn't high enough. This is an underappreciated point that I don't think most people have figured out quite yet, is that, what does increasing adapt, what does the ability to adapt the novelty actually mean in practice? Like, oh, sure, it's a key to the benchmark, it beats that. What it really means in practice is you can more consistently get the same answer, not even necessarily the right answer, just more consistently get to an answer from these systems. And that now gives humans the ability to steer them and control their behavior more precisely than we had before.

• 57:21Speaker 01

  This is gonna raise the reliability bar, and I think a lot of use cases where people wanna use agents, and it was not a cost problem, it was just like, they just don't work enough. I'm willing to pay up the human labor rate for this system to work. Now we'll start having those use cases get unlocked as a result of plugging things like O1, O1 Pro, O3, R1 into some of the planning steps for how these agent systems work.

• 57:43Lukas Biewald?

  I guess one of the things that's been in the back of my head as I do this interview with you is we're sort of taking for granted that, you know, this kind of like reasoning is important. And I kind of wonder if you're, we're like losing people by showing these sort of toy like problems in the sense of like, what does that actually translate to in the real world?

• 58:00Speaker 01

  Yeah.

• 58:00Lukas Biewald?

  You kind of said it, but I guess, do you have an intuition like, what, like, where does like the Ark price failure like show up?

• 58:07Speaker 01

  Can I give you a concrete example?

• 58:08Lukas Biewald?

  Perfect. Fantastic. Yeah.

• 58:09Speaker 01

  I'll give you a funny story, uh, which is what kind of led me down a lot of this like AI benchmarking path a couple of years ago.

• 58:15Lukas Biewald?

  Love it.

• 58:15Speaker 01

  So when we were first building some of our AI agents at Zapier, um, we had one that I had set up. Uh, Zapier has been a, a long time strong partner with, with OpenAI. Uh, we've been a part, I've been a part three now, like major OpenAI launches, been, been sort of fun. And, uh, so we, we have, we, both of us use Slack. We have a shared Slack channel. And when we were building some of our early AI agent prototypes, um, we had built the system where we wanted the AI agent to like, uh, automatically send a message into Slack from some, uh, like lead management stuff coming in from HubSpot, or maybe like our sales team was doing transcriptions and feeding into this process. And when we were testing it, we said, hey, like, let's have the agent, like, it's got two main functions it needs to fill in to like, uh, do this automation.

• 58:59Speaker 01

  It needs to pick a Slack channel to send the customer information to, and it needs to send a, you know, write the message to the body of the Slack message. And we had the agent guessing both, um, from the inbound message. And we would, we, our system allowed us to like, and this is how Zapier works. So you can give hints to the agents on how to fill in those two fields. Like you could say, oh, for the, for the Slack channel, use the testing, hashtag testing channel. And this is a plain text description. And for the message, you know, grab the lead, first name, last name, phone number, email, likely to buy, and build a, build a lead, like, you know, little widget and put it in. Yeah, that'll be the body of the message. And, uh, when we first turned it on, Like we had a Slack channel called hashtag testing.

• 59:39Speaker 01

  The name of the channel we had with OpenAI was called like OpenAI dash like partner dash testing. And the agent just like picked one a couple of times that partnership channel to like start sending, you know, like information to. And it was like, oh, shoot, that's not good. This is like a production system. You know, it's a good, it's a strike, it's an important partnership. We don't want, and like, we don't want customer information being like shared like that. Totally. So immediately turned it off, scrubbed it, fixed the situation. But it was my first realization moment that like, oh, reliability and overrides and control are going to be extremely important things for building trust. Because like the first time you see that, you're like, and it's like, it's a user who saw that. It's like, nope, turn this thing off 10, like get that thing 10 feet away from me.

• 1:00:23Speaker 01

  I can't, I can't deal with this. This is, you know, it's putting my business in jeopardy. And it was very real, it made us realize early on that in order to start deploying AI agents, like a year and a half ago when we started, we needed to offer users hardcoded control over constraining guessing. So in this new case, when you use Apples AI agents, you can allow it to guess if you want to, but the default is just choose a channel. And that allows you to have some amount of certainty and guarantees around, well, okay, it's not going to go off the rails too much. Or you can, you can say, here's the three channels I want you to guess from. And it allows you to build in that sort of hard guarantee. Again, all this was just early insight that like reliability matters a ton. It's probably the number one thing that matters for deploying agents that can automate tasks

• 1:01:08Speaker 01

  that businesses and users care about. And that is the type of thing we will now start to see get reliably solved using O1O3. You're not going to see this sort of as much stochastic randomness in doing the wrong thing. If you, if you like build an agent, say, send it to the testing channel and it works the first three times, you're going to expect it's going to keep working and that's going to be the truth. Whereas in the past, with the sort of very, you know, stochastic nature of how these LLM peer systems have worked, you really couldn't make that same guarantee without like, you know, some extra guardrails on top that developers created.

• 1:01:43Lukas Biewald?

  And so why did you start India as its own organization versus work with OpenAI if you have this close relationship with them? Like, why do you feel like they needed a new, a new organization?

• 1:01:58Speaker 01

  I think the reality is that if you look at the sort of frontier of what most of these companies are working on, I don't think they share the view that we do about the importance of program synthesis. I saw a lot of Tom Fong folks, Frontier Labs, OpenAI included, and even showing them like, you've got these over three results now. And I think there is a view that this is still, we're still in a deep learning paradigm. This is deep learning with a little bit of special search stuff on top, but that's not really that important. I do think there are people, by the way, in these orgs that get it. But I think the broad view still in industry today is that like, this is, you know, deep

• 1:02:43Speaker 01

  learning is still scaling. And I, we fundamentally disagree with that perspective. We, my view is that program synthesis is at least 50% of the equation. Maybe not from a compute budget standpoint, but from like, if you were to measure like, where are the ideas in the future, highly efficient, no human model, like AGI system? It's gonna be, half of that is gonna be coming out of the discipline of like, program synthesis in some way. And like I said before, I think AGI is such an important technology, anyone who has like a unique differentiated idea, I think should be going, trying it. I think it's important for the world, like, this is why we launched Dark Prize. We want to inspire more people just to like, go try new stuff again, you know, try to

• 1:03:25Speaker 01

  get back to the AI industry research model we had in 2018, 2019, where everyone was trying new and different ideas. And I do think Dark Prize has helped shift the Overton window from a narrative standpoint. I'm starting to see it now, I'm very excited. We're starting to see more fundamental innovation now coming out of a lot of the like, small startups. In fact, one of the biggest things that surprised me about Dark Prize, you know, I sort of expected that we were gonna get a lot of individual researchers and maybe some big lab attention to like, work on the benchmark. The biggest surprise I probably had coming out was about seven or eight startups came up to be some point throughout the contest or afterwards and told us that they had changed their, like, these are like AI companies, they had like, changed, they had pivoted, they changed their research roadmap to go work on ARC.

• 1:04:10Speaker 01

  And I thought that was a really exciting thing to hear because I was like, okay, I think it's starting to have the impact that we hoped it would, which is to get people to at least explore more and increase the sort of overall probability that like we figure this thing out. And, you know, I think we're, we've got exciting long-term ambitions about what we want to use with the technology that I think also differ quite a bit from any of the other sort of major players and companies that exist out there. But purely if you're just looking at the technology of like, why did we start India? I think it is the fact that we just had a, you know, I think we have a differentiated view that I think has a high degree of chance of success that increases the probability of getting to AGI quickly for the world. And we're going to put our full effort and attention into trying to make that.

• 1:04:54Lukas Biewald?

  And are you also oriented towards making your products and making money and things like that?

• 1:05:00Speaker 01

  I have somewhat of an interesting view here. So to ask you a quick question, like, no, this is a research lab. Like, we are, like, there's no off-road for products in the near term. We're like, this is get AGI by my definition, our definition, you know, to reduce the gap to zero between easy for computers, hard, easy for humans, hard for computers. You shouldn't expect to see us really make products ahead of that. Now, we may start using this, some of the prototypes of the technology to start trying to advance some of the frontiers of science. This is actually one of the reasons I mentioned, besides just building AGI, building AGI is step one for us. The thing that gets us really excited, me really excited, is to start leveraging the technology to accelerate the sort of pace of innovation we see at the frontier of a lot of different fields of science.

• 1:05:43Speaker 01

  Like, AGI is going to solve a lot of problems. No doubt about it. It's already solving problems with our customers. Like, it's going to, like, and it's not even AGI yet. Like, we're, like, people are going to use this technology to solve a lot of problems. And that's great. I'm fully supportive of that. The thing that gets me, like, more excited about the technology is it's sort of like accelerating into this, like, unknown, unknown future.

• 1:06:08Speaker 01

  And I'll make this like kind of illustrative story, you know, around like the printing press, you know, back in the 1400s, when it was first introduced, you know, was it 600 years ago now? You know, like, what was the reaction? It's like, obviously, lots of fear and skepticism, but also a lot of excitement around sort of this knowledge proliferation of, you know, sharing knowledge globally, being able to exchange freely ideas. I think if you went to anyone, though, and like asked them, hey, make some predictions about what the year 2025 looks like, you'd probably be pretty hard pressed to get anyone to imagine a future of like Wikipedia and AI that got trained on Wikipedia and like, I can talk to computers. What's a computer? Like, you know, we're just so far down the

• 1:06:50Speaker 01

  technology like tree from these genesis moments of like the catalyzed everything. And I think that's what really gets me. It's almost more like an adventure than it is problem solving motivated. You know, there's going to be really exciting, cool things that come in the future of technology that I can't even tell you what they are yet. But I think we're, in order to get there faster, the main constraint is like, we need computers that can work in autonomous ways to do innovation. And that is bottlenecked on creating AGI. And that's what I'm really excited to sort of help be a part of and help accelerate if we can.

• 1:07:23Lukas Biewald?

  Do you have a point of view on timeline for AGI?

• 1:07:30Speaker 01

  To the definition I shared before of like, you've gotten rid of all the gaps from, you know, easy for computers, hard for AI?

• 1:07:41Speaker 01

  One, one other thing I think it's actually to add to that definition that is worth considering is the efficiency. You know, I think we'll probably reach that definition before we reach it at a human efficiency level, would be a guess. So let me, let me go ahead and just like add a little asterisk on the end and say like, we're able to sort of, you know, there's no remaining task that reason for humans are for AI and like, we're doing it at human level efficiency, you know.

• 1:08:05Lukas Biewald?

  Well, what's, do you, can you put a dollar in human level efficiency? Is that like your salary?

• 1:08:10Speaker 01

  This is literally something we're debating right now, how to measure efficiency. And I don't think the industry knows that ARCA is still like, ARCA, we're trying to figure this out too. You kind of want something that you can contrast between humans and computers. And like flops aren't really great. You can maybe do dollars because you could like pay a computer to do X, you know, X work for X dollars. You could pay a human to do X work for X dollars. Time is kind of interesting from a wall clock perspective because you can, you can make time faster with computers by paralyzing. And that's something that they're good at, right? Honestly, I don't know the answer. I'd probably make the avocation that dollars are probably the best one today, based on what I know, in order to make sort of comparisons. Because dollars, kind of, everything fits into dollars. You know, how much you pay a human for a library versus how much compute costs. Dollars will track the price performance of compute getting more efficient as well,

• 1:08:55Speaker 01

  which is kind of nice. We don't know. So let's say, like, yeah, level human level efficiency. We're paying humans the same we're paying computers to do these, like, easy tasks that humans find hard. Something like that. I'll add that sort of asterisk on. My expectation right now, if I just had to make some wild guesses, you know, we got ArcV2 we're working on for this year. I expect it probably will be durable for, let's call it 12 to 18 months. That's my best guess today, based on how it's doing on all the frontier systems we've been testing it on.

• 1:09:27Speaker 01

  It's not going to be another five-year benchmark, though, you know, like the V1 was. Because ArcV2, it's kind of in the same domain. It's just like, it's raising the bar of difficulty for computers without raising the bar of difficulty for humans. And this should be an interesting capability. I think it will still be interesting. I think it will still be a good gradient of tool that will point research in a certain direction. But the thing I'm really excited about is V3, that we've started designing and kind of prototyping for next year. And our design goal for V3 is that it would be durable for three years. That's our hope and goal. We'll have to see, obviously, make contact with reality and all that stuff. But, you know, my sort of anticipation is like, we will not have, you know, easy for humans, hard for AI solved at that level of efficiency, you know, based on today for the next, at least three to four years. Honestly, I'll tell you this.

• 1:10:10Speaker 01

  One of the hardest things about making predictions is it's very easy to make predictions around smooth scaling curves. It's extremely hard to make predictions around step function changes in capability. Because, obviously, these curves you can make, you can sort of track out, right? Make future guesses around capabilities just joints. Like, if you were looking at the Arc, if you were trying to make progress about when is ArcV1 going to reach 85% from a general-purpose system? It's like going for five years, it went from 0 to 4%. And then in two months, it went from 4% to 85. That is an extremely hard thing to predict because two, two things about this. One, you don't know if the technology exists in the world to do it yet. You don't know if the ideas are in the world yet to do it yet. And then if they, and then if they are in the world, you just don't know if someone hasn't like,

• 1:10:55Speaker 01

  put them together into a system to demonstrate it to you. So you have a lot of unknown variables in trying to make guesses about when do you, when do these step function capability jumps sort of happen? I am, I'm not a, I guess to make my concrete views, I do not believe we're in a pure sort of scaling regime of where you can make a nice smooth curve over these like system sizes and cast a weight coin out into the future and say, that's where we're going to have all this stuff solved, we'll have a job at that point. I treat it a lot more empirically and I think, you know, you can make informed guesses about when you think step functions might happen, but I think it's in the public's interest to understand that like this is actually the reality. We're still looking for step function capability leaps and we don't know when those are going to show up.

• 1:11:36Lukas Biewald?

  Well, awesome. I think that's a good stopping point for this interview. Is there anything?

• 1:11:41Speaker 01

  You got to do a lot of content.

• 1:11:42Lukas Biewald?

  A lot of content, a lot of wide-ranging content. Thanks so much for listening to this episode of Grading Descent. Please stay tuned for future episodes.