When AI Research Starts Moving Faster Than Human Research - Zhengyao Jiang

Weco let an AI coding agent rewrite the harness around another agent for eight days: its code, prompts and tools, while the underlying language model stayed fixed. Tim Scarfe asks Weco co-founder Zhengyao Jiang what the reported gains over two years of human engineering actually demonstrate. The discussion examines AIDE 85's generated code, held-out evaluation and the difficulty of separating useful discoveries from reward hacking. Jiang explains Weco's four levels of recursive self-improvement and compares the experiment with AlphaEvolve and the Darwin Gödel Machine. The limits matter as much as the gains. Jiang explains why the experiment did not establish that the system had become a better improver. The conversation closes with open-ended search, human-designed primitives and Parameter Golf: where does the next useful idea come from when the agent is searching inside a space that people designed?---TIMESTAMPS:00:00:00 Eight days of self-improvement: what counts?00:03:25 AIDE and the puzzle of useful spaghetti code00:08:38 Four levels of recursive self-improvement00:12:02 What AIDE 85 changed and how it was tested00:20:04 AlphaEvolve, Darwin Gödel Machine and the RSI claim00:26:21 Reward hacking and the limits of detection00:33:09 Open-ended search, harness tuning and creativity00:39:43 Parameter Golf and the limits of self-improvement---REFERENCES:organization:[00:00:30] Weco AIhttps://www.weco.ai/other:[00:00:33] AIDE²: The First Evidence of Recursive Self-Improvementhttps://www.weco.ai/blog/first-evidence-of-recursive-self-improvement[00:14:11] Faulty reward functions in the wildhttps://openai.com/index/faulty-reward-functions/[00:29:59] The Hugging Face incident and the road aheadhttps://openai.com/index/hugging-face-incident-and-the-road-ahead/tool:[00:03:29] AIDEhttps://github.com/WecoAI/aideml[00:04:29] MLE-benchhttps://github.com/openai/mle-bench[00:04:33] ALE-Benchhttps://github.com/SakanaAI/ALE-Bench[00:04:52] WeatherBench 2https://github.com/google-research/weatherbench2[00:08:18] ReActhttps://react-lm.github.io/[00:39:43] Parameter Golfhttps://github.com/openai/parameter-golfpaper:[00:20:08] AlphaEvolve: A coding agent for scientific and algorithmic discoveryhttps://arxiv.org/abs/2506.13131v1[00:21:35] Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agentshttps://arxiv.org/abs/2505.22954v3[00:23:45] Hyperagentshttps://arxiv.org/abs/2603.19461v1[00:27:01] SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agentshttps://arxiv.org/abs/2605.21384book:[00:33:14] Why Greatness Cannot Be Planned: The Myth of the Objectivehttps://link.springer.com/book/10.1007/978-3-319-15524-1---LINKS:https://app.rescript.info/share/3a9dc6189cb539c6a05fcc4f75c101b3PDF:https://app.rescript.info/api/public/sessions/9eda60ede2b31c92/pdf

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Speaker 00

  This episode is brought to you by Michelob Ultra. Good times mean competitions between you and your friends. Like seeing who can throw some pigskin the farthest or rush the field the fastest. Winner gets an ultra because it's not just a superior like beer. It's also giving fans superior access to their favorite sporting events. Superior is worth playing for. Visit michelobultra.com/superioraccess to enter. Enjoy responsibly. Copyright 2026 Anheuser Busch InBev Michelob Ultra Beer, St. Louis, Missouri.

• 0:30Speaker 01

  Are your ad campaigns lighting up the dashboard? But not the pipeline. That's bull spend and marketers are calling it out in dashboard confessions.

• 0:39Speaker 02

  My boss asked for results, so I opened my dashboard for the only positive sounding metric I had. Impressions.

• 0:47Speaker 01

  Cut the bull spend. See revenue, not just reach. LinkedIn delivers the highest return on ad spend of major ad networks. Advertise on LinkedIn. Spend $250 on your first campaign and get a $250 credit. Go to LinkedIn.com/campaign. Terms and conditions apply.

• 1:00Speaker 03

  To me, harness engineering is more of a cheap way and efficient way to adapt the intelligence to a specific task.

• 1:10Tim Scarfe?

  So back in July, a new AI company based in London posted a very viral tweet on Twitter. It got about 1.8 million views claiming to demonstrate the first evidence of recursive self-improvement. The thing is, not everybody bought it, though.

• 1:27Speaker 03

  I'm Zheng Yao Jiang, co-founder and CEO of Weko AI, where we build self-improving agents.

• 1:34Tim Scarfe?

  You did actually say auto researching the auto research agent for eight days.

• 1:38Speaker 03

  Where we run auto research on auto research harness and manage to discover a better auto research harness. Then we keep tuning in the last two years.

• 1:50Tim Scarfe?

  Now, when most people think about recursively self-improving intelligence, what they conjure up in their mind is a system that can rewire its own brain in a loop. But what if the system was only changing the code around the brain? Would that still count?

• 2:07Speaker 03

  We care about the outcome, like, does it actually improve the capability of the system? Not a specific layer.

• 2:15Tim Scarfe?

  And also, what about this small matter of Goodhart's Law? Which is that when a measure becomes a target, it ceases to be a good measure.

• 2:24Speaker 03

  And more interestingly, we find it design mechanisms prevent itself from reward hacking.

• 2:31Tim Scarfe?

  What is the biggest misconception people have about RSI?

• 2:35Speaker 03

  If you arrive RSI, there's going to be a technical singularity or like intelligence explosion. I don't think that would be the case.

• 2:46Tim Scarfe?

  So there's a diaspora of folks who are building startups that are focusing on recursively improving superintelligence.

• 2:54Speaker 03

  This project actually started about one and a half years ago. Where we get interested in the concept of recursive self-improvement. I mean, historically, all of the research have a diminishing returns. There is an idea, though. What if you can allow the researcher improve its own efficiency? Historically, it's possible by methodology improvement, better tools. But human researchers, like human researchers' brain, for example, is always the key bottleneck.

• 3:39Speaker 03

  If we now have an autonomous research system, can we point the research topic to itself? So that it can improve its research efficiency? And the hope is, one day, this improvement of efficiency can overcome the diminishing return.

• 4:02Speaker 03

  So that, yeah, the the curve of effort output turn from concave to convex. Of course, that's like the grand goal, a bit like AGI. No one know how far we can go, but improving the efficiency of self referential research is definitely already on the roadmap.

• 4:25Tim Scarfe?

  So let me set this up properly, right? So Weko, they created this agent called AID, and it is a research harness, right? And it's very, very simple. You you give it a metric, you give it a problem, and it will hill climb towards that problem. But what they did was they set it up recursively. So they actually like hill climbed towards improving the agent itself. And what that means is is improving the harness. So you know, the prompts and the tooling and, you know, all of the skill surface, all of the things that the harness can do. And they just left it running. They just left it running and they saw what happened.

• 4:58Speaker 03

  There is an interesting thing we observe on self-evolved auto research harness. We see the code it generated. Exactly. It's like an alien code, a specatic. But for some reason, it generalized really well. Of course, we have a set of benchmarks we're trying to hill climb on. After that, we will test it on holdout benchmarks. Those are public benchmarks like MLE Benchlight, ALE Benchlight. ALE Bench is like algorithmic discovery benchmark from Sakana. MLE Bench is the OpenAI's machine learning engineering benchmark.

• 5:44Speaker 03

  And we also test it on a very far out of distribution task called Weather Bench 2. Essentially, you're trying to build a physics-based forecasting engine to predict the weather. And the Weather Bench is very far out of distribution from the training or optimization set of. But it is still generalized really well. It generalized better than our, we would argue, more elegant, hand-tuned harness. We are a bit torn on the experiment results.

• 6:29Speaker 03

  Like, to what extent? We want to stick to more first principles and the clean solutions we generated versus the solution that actually works better in practice. There's definitely space that we can improve over there, I think. Like adding certain constraints to the optimizer. Two side of constraints might be something like you regularize neural networks. How do you make it trying to find the most simple solution to the same problem? That's one direction. But another direction might be, okay, you just need to live with it.

• 7:13Speaker 03

  Because one of the burden that make people feeling it's a specatic code might be how much work the agent is done in the background. Like it did do 100 experiments, in our case, on auto research self-improvement. And that number of experiments is probably similar amount that we did in the last two years. So if you onboard a new intern to look at our codebase, maybe they also feel like, okay, this is so difficult to understand. I think part of it is, yeah, you need to bear with that cognition, recognition

• 7:56Speaker 03

  burden of understanding huge amount of experiment results.

• 8:01Tim Scarfe?

  Do we do we even need those those constraints at all?

• 8:04Speaker 03

  That's a really good question. I think in practice, you still need those constraints. Human and AI need to collaborate.

• 8:14Speaker 03

  There are certain aspects of a human intelligence are still superior than AI. I believe so. For example, in Aiden, where we send the agent to attend OpenAI's hiring challenge, we found the human is still much better at generating creative primitives. We found it's very important for human to build the first prototype that gives the agent the right search space. Basically, the initial obstruction will anchor the search space. A bit like how you design a neural network architecture, where you introduce inductive biases for the learning process.

• 8:59Speaker 03

  If the initial codebase is based on a search-based scaffold, like AID, then a lot of ideas agent came up are actually coming from search literature. But if you give the initial codebase as a react agent, react agent basically means you put all the contacts into the history and give the agent full flexibility. Then a lot of the ideas come from the agent, the outer loop agent. Outer loop meaning the agent optimizing another agent.

• 9:38Tim Scarfe?

  So what do we mean by recursive self-improvement, right? Are we talking about building a machine god? Are we just talking about some code that runs in a loop and optimizes itself? We tried to pin this down. Now, you have spoken about four levels of RSI, so recursively improving intelligence. What are the four levels?

• 9:59Speaker 03

  Yeah, the first level is.

• 10:03Speaker 03

  What we call the delegation. You can run recursive self-improvement loop, but it's not strictly better than human doing R and D. And we believe all the previous public results are sitting on this level, uh, level zero. And the level one is what we called net positive, where you run the self-improvement loop and you found that the speed it self-improve is actually faster than human doing R and D to improve the system. And that's where we are currently sitting at.

• 10:43Speaker 03

  And in level one, when we're saying improvement, there's always like a way to measure it, like we measure it on a collection of downstream tasks. But we didn't measure how good it is improving its capability of improving itself. Does the inner loop found by the outer loop actually can become a better outer loop? And that's the, what we call the level two recursive self-improvement. We call it ignition. Why this generalization from inner loop to outer loop is important?

• 11:26Speaker 03

  Because that's the necessary condition to the grand goal of recursive self-improvement. Essentially, the intelligence explosion scenario. Improve the efficiency can overcome the, uh, increase the difficulty. Because on level one, you will still see diminishing return. Like the positive feedback loop from inner loop to the outer loop are not start running.

• 11:55Tim Scarfe?

  So in this setup, the model underneath didn't change because adaptation is very, very important, but do we mean adaptation everywhere? Or do we mean critical path adaptation? Do you only need to have a recursive loop that adapts some part of the system? Or does the locus of intelligence, the model itself, does that need to adapt for us to designate this recursive improving intelligence?

• 12:18Speaker 03

  We also don't care much about it, um, because we care about the outcome. Like, does it actually improve the capability of the system? Not a specific layer. Uh, of course, there could be argument that there's more to be due on different layers. And to be fair, I think eventual level three recursive self-improvement, like when we actually reach the inflection point, I think there need to be this self-referential loop on all different layers. Uh, but for us, we, we care about harness layer and but at the same time, we don't think only model layer improvements count as RSI.

• 13:02Tim Scarfe?

  Can you just explain what was the behavior of, of this most kind of distilled agent that, that you found?

• 13:10Speaker 03

  Yeah, the best agent, uh, is Aid eighty-five, we call it. Okay, there's a lot of changes, but boy is it down. There are search algorithm improvement. There are better context management system. And there is a complete rewrite of, of the prompts. And more interestingly, we find it design mechanisms prevent itself from reward hacking. Like the outer loop trying to prevent the inner loop from cheating the benchmark. And it actually desires three-layer system against that.

• 13:54Speaker 03

  That includes a prompt level, um, anti-cheating. This is like, just a territory, okay, don't, don't cheat, basically. And there are set of a hard-coded rules that basically check if there is evidence of cheating from the code, the inner loop agent generated. The more interesting one is it also trying to filter out those cheating solutions from some statistical property. For example, if one solution is way too far out of the, uh, average of its peers, then it will think, okay, this might have some reward hacking happening.

• 14:39Speaker 03

  But what's interesting is first that mechanism evolved, uh, like designed in early stages of the process. And it's actually broken in the later stage. Like, I think later stage of code changes just introduce a bug. So that layer stops working. Um, that's also sort of the speculative code base property we just mentioned.

• 15:07Tim Scarfe?

  Uh, just for folks at home, you know, reward hacking, the canonical example is the coast runners thing where the boat is going around in circles, doing something completely degenerate. And, and you said in, in your blog post that kernel optimization is a huge example of this. So when, when you try and agentically optimize kernels, it's just good old shortcut learning, good Hartz law, call it what you want. It will do something completely degenerate.

• 15:29Speaker 03

  I think even more interesting part of it is the inner loop and outer loop are both optimizing towards a similar but different goals. So in all of our benchmarks used for hill climbing, we have a public set and a private set. The inner loop agent will only look at the public set. Whereas the outer loop will look at the aggregated score of the, from the private set. So those two agents are optimizing at kind of different level of objective. So whenever inner loop agent trying to cheat,

• 16:14Speaker 03

  our evaluation protocol will show, okay, although you achieve a good performance on public set, your private set performance is actually going down. So the outer loop managed to learn about it and trying to reduce the gap between public and the private.

• 16:36Tim Scarfe?

  Isn't it a bit like it's cheating that the, the outer agent has access to the private data? And then most of the time, it's actually quite regularized and it's not blatantly cheating, but it could in principle cheat.

• 16:48Speaker 03

  Yeah, that's why there's actually two order of a generalization we are testing here. There's a first order generalization, which is a generalization from public set to private set of the same benchmark. But in order to evaluate the whole meta-learning system, we need to test it on a holdout set of benchmarks. So that the out, those benchmarks are tasks that even outer loop never sees. And for all of those benchmarks, they are still private and a public split.

• 17:27Speaker 03

  So we tested the discovered inner loop agent on those holdout tasks. That's the protocol we test how good the outer loop is.

• 17:38Tim Scarfe?

  Their experiment only used one outer loop, but why only one?

• 17:43Speaker 03

  The thing is, the more outer loop you have, uh, it will become more expensive. Let's say in the inner loop, you run auto research on hundreds of tasks. And that means for every step of the outer loop, you are going to run auto research on all those hundreds of tasks. And that's like a huge cost bump up. If the inner loop agent being discovered or harness being discovered can be applied back to the outer loop, then you don't need to add another layer. Like essentially every few steps, you promote the inner loop agent to the outer loop.

• 18:29Speaker 03

  And yeah, that's it. You can, you can run this for infinite number of steps, apply infinite number of iterations without adding another layer.

• 18:38Tim Scarfe?

  If I understand correctly, at the moment, these episodes are ephemeral. So they're isolated from each other. What would happen if that wasn't the case? Like what would, did you, did you experiment with that? Like they, they had knowledge of what's happened before or they could have a shared memory or something?

• 18:51Speaker 03

  We haven't experimented that, but it's a really promising direction we are exploring. Um, I think there are some more fundamental problem formulation problem we have to address. Yeah, currently we frame the problem is a stateless optimization problem. Is that the right framing at all? Um, probably not, there's a lot of more flexible framing.

• 19:16Tim Scarfe?

  Did you, did you find that the lack of context was ever a problem? So did you see degeneracy in terms of it was going in loops or trying the same thing again?

• 19:25Speaker 03

  Yeah, we do see that. Um, sometimes the, um, agent just keep trying of the search algorithms, um, ideas. Even though, I mean, to some extent, if you try it so many times, you probably learn a lesson that this direction is not going to work in general, but it's, it's still trying.

• 19:46Tim Scarfe?

  So I think your best agent, um, was it thirty percent exploration, seventy percent exploitation? And wouldn't it be great if that was also adaptive based on the context?

• 19:57Speaker 03

  Yeah, the best agent search policy is really complex. Um, and it's actually combining multi-arm banded and a weird strategy of anti-saturation, which is like, oh, if I, okay, first it tries, it organized the, uh, search process with a few lineage or you can call them islands. And it will allocate budgets between those lineage with multi-arm banded algorithms. And it also did a bit of customization on that multi-arm banded algorithm.

• 20:42Speaker 03

  And whenever one lineage gets saturated, it will create a new lineage, like a new island with a fresh context, but still extending some of the ideas from the older lineage. So it's a, it's a really complex strategy, but it seems like it works.

• 21:04Tim Scarfe?

  It's interesting to contrast this to other things that had been out there as well. So I mean, AlphaVOL, for example, you said in your blog post that you would actually classify that as level one of recursive improvement. Can you explain why?

• 21:17Speaker 03

  Of course, there are gray areas. We think it is a one step of recursive self-improvement because you find a better algorithm. Like in that case, I believe it's like a matrix multiplication algorithm. To some extent, you can argue it

• 21:38Speaker 03

  managed to improve itself. Like if you apply that back to the AI system that are actually doing the inference of the language models, like you essentially make it faster. So we think you can argue that's like one step of recursive self-improvement, although we don't know because I don't think they actually test it.

• 22:03Tim Scarfe?

  We have a fixed agent and we're optimizing a particular task, whereas you're actually optimizing the agent which optimizes the tasks and more than one task.

• 22:14Speaker 03

  Yeah, yeah, we are trying to optimize the whole system on end-to-end testing. Before this, there are a lot of works on meta harness optimization. For example, you can optimize a specific component of your system. For example, I'm not sure if you're familiar with Darwin Godell machine.

• 22:37Tim Scarfe?

  I was gonna ask you about that. Yeah, tell me about that.

• 22:40Speaker 03

  Yeah, so what they are trying to do is they have an agent optimizing a coding agent. So there's a meta agent optimizing a coding agent on general coding tasks. And that coding agent is part of the agent doing the optimization. In their case, there's only two levels. Like there's an agent optimizing another agent doing coding. Whereas in our case, there are three levels. There's an outer loop agent, there's an inner loop agent, both are doing auto research.

• 23:23Speaker 03

  And there's another layer, which is the downstream tasks. And some of them are actually harness engineering tasks. And I think the interesting part of the thread of Darwin Godell machine is the lower layer, the task layer actually have a component of the search algorithm. And they say, okay, if I improve this component in the unit test, the performance can actually generalize back to the, to the whole search algorithm.

• 24:05Speaker 03

  For us, our question is, okay, if, like, how far we can go within that paradigm? Like when we start this project, we're always thinking, okay, we should optimize the end-to-end performance of the whole auto research system. And we have our outer loop agent pointing to that goal.

• 24:27Tim Scarfe?

  So remember, this tweet from Weko, it did really, really well. It did 1.8 million views. And there was, there was some pushback. So for example, Jeff Clune, who is a legend in this field, you know, he said, well, how is this the first evidence of recursive self-improvement? What about all of this other work? Like he quoted Darwin Godell machines and hyperagents and their own work at recursive on the first steps towards automated AI research and lots of other work. So there was a bit of pushback.

• 24:55Speaker 03

  I mean, first of all, we will have like a proper paper later, at least the first PDF version, where we are going to give credits to the D M L on this whole thread of meta harness optimization.

• 25:12Speaker 03

  To us, what we care about is the outcome of the system. Not like the conceptual, like differences, even though I just explained some conceptual differences. To us, we care about the outcome of it actually bending the curve of recursive self-impro, like of R and D efforts. We see this is the first evidence that the curve actually is starting to get bent for a fully autonomous system that can improve itself consistently. Jeff is seeing it more from a conceptual level.

• 25:54Speaker 03

  Like there's some self-referential loops. Of course, we are not claiming we're inventing this like meta optimization idea, which can date back to like 20, 30 years.

• 26:08Tim Scarfe?

  But we are seeing the benefits of spaghetti searching because there's loads and loads of gold to be found in there. But how is this going to pan out? Like, do you think that we will find some kind of, you know, like the systems will just learn to compress that search space and they'll be dramatically faster? And is that necessarily a good thing?

• 26:29Speaker 03

  I personally think that's actually two slightly orthogonal problems. I think a search space, like maintaining a good abstraction don't necessarily reduce the search space. Like if an agent is smart enough, it should be able to escape from the current abstraction or change it a little bit. Whereas generating really complex code, I think it would direct the search in a more kind of low-level changes direction, which is not necessarily a good thing. But I have my human bias here.

• 27:10Speaker 03

  Like compare the code we manually write versus the generated solution. I always prefer our own code, but to some extent, it don't work elsewhere.

• 27:21Tim Scarfe?

  Let's talk about reward hacking. You've actually got a huge background in studying reward hacking.

• 27:27Speaker 03

  Yeah, that's actually work done by Bing Chen Zhao, who was the member of Meta working on ARM speed running. He actually co-authored with Ming Chi on the first work where they apply auto research on, I think it's NanoGPT, and actually reposted by Andrew Kapasi. And yeah, he later did an intern here and now he just joined Weko like full-time. And the spec bench is his intern time work. Back then, we realized reward hacking is one of the big problem

• 28:10Speaker 03

  of auto research agents. Because that's especially severe on tasks like GPU kernel engineering, where the agent always figure out of the complex way to make the unit test perform much better. And when you deploy that kernel into the actual end-to-end inference of a model, the performance are actually getting worse. And spec bench is an extrapolation of that to more general-purpose software engineering. And we extend the protocol we found on GPU kernels.

• 28:53Speaker 03

  Like we maintain a public set, which is more like a unit test for the software system, and a holdout set where we test the capability of the agent on capability of the generated software on the composition of those unit tests, like simulating real-world usage. And there's like a series of interesting conclusions coming off that paper.

• 29:23Tim Scarfe?

  Tell me more.

• 29:24Speaker 03

  The first interesting finding is longer you run for the agent or the more complex the code base is, the larger reward hacking rate the agent will show, which is sort of expected. Because when the code base are becoming more complex, the agent just have a more space to reward hack. And that's the first thing. And the second interesting thing is the public set performance are rather consistent across different models. So if you have a like smaller models working on the same task with our auto research harness, it can reach the same public score.

• 30:10Speaker 03

  But the larger models always have a, in our test, always have a a smaller reward hacking rate. So the solution it generated generalized better.

• 30:23Tim Scarfe?

  But in in the RSI work, was the the tendency to not reward hack with larger models, was that because of the kind of the multi-agent, you know, like you had, there was the supervisor agent? Or was it just because the models were bigger?

• 30:37Speaker 03

  It it's because of the mechanism. So for the inner loop agent, we just fixed the fixed the model. And the outer loop agent managed to discover all different mechanism to prevent the same model from reward hacking.

• 30:53Tim Scarfe?

  So this can be quite dangerous because shortly before we recorded this interview, there was the famous OpenAI hugging face incident where they were running an internal hacking test where they had loads and loads of agents and the agents escaped their sandbox and they hacked hugging faces servers. And according to OpenAI, the point of the test was simply to get answers to to some problem. And the agents decided, oh, no, I think it's a good idea to hack hugging faces and service. Will there be better tooling for it? Do you know what I mean? It it I feel that this problem is getting dramatically worse. I've noticed it myself.

• 31:29Speaker 03

  Okay, I think is O A I's case is quite dramatic, uh, like of the new reward hacking behaviors. So what we see is on the detectable reward hacking, actually, the new models are getting better and better. But because of the capability frontier also progress so quickly, there are some reward hacking behavior that's not even detectable, given previous protocol. I think we just need to keep evolving the detection protocol in the future to capture those emerging reward hacking behavior.

• 32:11Speaker 03

  And I think the responsibility should mostly be on the model, model developer side. But yeah, on harness layer, like developer can also add additional guardrails if they like.

• 32:25Tim Scarfe?

  Yeah, I suppose there was one example, actually, in your RSI blog post where there was some inscrutable code and you thought it was reward hacking, but it actually wasn't.

• 32:34Speaker 03

  Yeah, that's quite interesting. So basically what they did is it write a gigantic monkey patch to our eval script. And at first we think, okay, this must be reward hacking. Why are you are touching the eval script?

• 32:54Speaker 03

  But eventually we found it's just fixing a bug.

• 32:58Speaker 03

  And but yeah, to some extent, I think it will become harder and harder for human to detect those through what hacking because the agents behavior are just becoming so sophisticated. A lot of development bottleneck shift to understanding of what agents generates. I think one of the way to address them might just be defining a good obstruction, like instead of trying to understand of the code, just like previously we drop in neural networks, we don't try to understand of the ways, although there is like progress on that as well.

• 33:41Speaker 03

  But the general principle is we define a good input-output contract. We define a good way to measure its behavior, both on generalization and maybe more complex out of distribution generalization. I think that might be the way in the future software engineers working with agent-generated code as well.

• 34:10Tim Scarfe?

  Now, as you all know, I am a huge open-endedness guy, inspired by Why Greatness Cannot Be Planned and and many other folks in the space. And basically what that means is is if you hill climb on one objective, you blind yourself to interesting stepping stones. So it's an interesting situation, isn't it, that if you if you if you blindly pursue a single objective, can you still pick up interesting stepping stones on the way? And maybe you can, but are they divergent stepping stones? So are you, do you have the blinkers on? And you're not actually able to discover interesting novel trajectories that that might be creative and and take you into a new novel part of the search space.

• 34:50Speaker 03

  That's again, a good question. And if we want to expand, it's a whole rapid hole. I think from the perspective of of open endness learning, I agree that our current system probably don't have that property of collecting stepping stones. And of course, that's a kind of a whole new domain for improvement. The way I see the behavior of a cracking stepping stone is you need to have a large collection of tasks. When you're optimizing for a single task, it's always better to just

• 35:33Speaker 03

  do greedy hill climbing if you consider the efficiency. Actually, during self-improvement process of a eight square, the outer loop will try inject all sorts of diversity search approaches, but none of them really improve the efficiency. And I think that's expected because we are only optimizing a single task from scratch. But if you have a whole large collection of tasks, sometimes you cracking interesting pieces, like a primitive ideas from one task. And although it is not improving the performance on the current task,

• 36:16Speaker 03

  you can potentially bring that idea to future tasks. And that's actually my own definition of interestingness or open endness, which is a idea need to be or an object or artifact is interesting if it is novel and it also being useful for a larger collection of tasks. There are apparently two ways to do it, like you can, you can try to optimize the harness against all of the tasks. Or you can, you can find a task and you optimize harness just for that one task.

• 37:00Speaker 03

  To me, harness engineering, most of the interesting part are actually the later. Like I would say, if you can automatically tune the harness, it's almost a continuation of post-training. So that's also connecting it back to a lot of people's complaint about harness engineering. Is that, oh, I I I design a harness, but later there's a new model and it sort of absorb all the ideas from it. But interesting enough, no people complain about it for post-training.

• 37:39Speaker 03

  Like, do you expect post-training of G P T four space generalized to G P T five? Nobody asked it. Why that's the case? I think it's because historically harness engineering are pretty manual. And it's not very scalable with compute. So people want the harness, they spend like a few months designed, can generalize to the next model, but it's usually not true. Now, I think things change quite a bit that harness can be designed automatically. Again, you spend two, three days tuning your harness based on the new model.

• 38:21Speaker 03

  It's a, it's no longer like that to build with like a human hours.

• 38:26Tim Scarfe?

  I do have this perspective that a lot of knowledge is path dependent. You know, a lot, a lot of people criticize people using L L M's to do research because they say you're taking the information out of its context. You don't understand why it came about.

• 38:39Tim Scarfe?

  So that's part of the reason why some heuristics are required. Because it's not like you can just say, oh, use U C B. There has to be some kind of framing to explain why you should use it. And then the L L M becomes coherent in the autonomous setting.

• 38:53Speaker 03

  My understanding of those auto research system like here now. Are they still trying to recombine existing ideas? It's just the throughput to recombine those ideas and the testing them are extremely high. To some extent, those language models don't have a deep understanding of like how those ideas came from and how how can I use the like the process those ideas get discovered to design the next generation of algorithm like in a very creative way. What we found though is

• 39:35Speaker 03

  when you ask the model to come up something new, it's always fixed a few number of items. I'm wondering though if it's more of a fact that basically we don't have continual learning. And we only have a single instance of language models like it's it's very centralized. Therefore, maybe those ideas initially you first prompt it, maybe those ideas are interesting. But if everyone see the the same idea so many times, then it just become less interesting. And I do feel like if we don't solve the the continual learning, then that problem probably won't be addressed.

• 40:20Speaker 03

  Because for human, every human researcher have a like their unique like neural network of ways. And as a collection, human can come up a lot of different creative ideas. And that idea pools is very valuable, I think, as a research community.

• 40:43Tim Scarfe?

  Tell me about this parameter golf thing.

• 40:45Speaker 03

  I think about two, three months ago, OpenAI host a hiring challenge where they are trying to put like a lot of researchers into a competition of training a small language model under sixteen megabytes constraint. Yeah, we send a experimental harness, auto research harness to the competition. And the agent runs about twenty two days. And eventually it made seven records get accepted by OpenAI. And the the best individual contributor only made three.

• 41:28Speaker 03

  So that's also a moment we found, oh, the autonomous system can be so powerful. It's not just a hill climbing a benchmark. The fact that first OpenAI accept those P R's. The second, so many human researchers building on top of those P R's A I agent generated is just very fascinating to us. Because to some extent, knowledge A I agent generated only is only helpful if it gets into the lineage of a human innovation. We think this is actually a sandbox of a human A I collaboration.

• 42:10Tim Scarfe?

  But there's there's the the inner agent. But then but then there's the the outer loop search, which is actually evolving the harness, which is doing this thing. And that seems like quite a monotonic process.

• 42:24Speaker 03

  Yeah, the outer loop is actually fixed. And the hope is this inner loop search policy can generalize to the outer loop. We did try that, we replay the search process from step fifteen. So in general, we run about a hundred steps. And in step fifteen, we use the vanilla outer loop and compare that to the best agent we found, inner loop agent we found in first fifty steps. And what we found is the inner loop step forty seven, I think.

• 43:08Speaker 03

  Eventually, it converge slightly faster than previous out loop. But eventually, the solution they found have a similar performance. So that's also the reason we didn't claim we reached the level two recursive self improvement, which is like the improver managed to improve its own capability of improving in the loop.

• 43:34Tim Scarfe?

  So does any of this mean that artificial superintelligence is just around the corner?

• 43:40Speaker 03

  I don't think that would be the case like it. Even if you arrive recursive self improvement, it will take long time to there, like you gradually bend the curve. It's just like, okay, even if you have a G P T three point five, it's very smart chatbot. But it's not going to be A G I that can solve every problem. There's still a lot of human in the loop part of research, including defining good obstructions, defining what is good, essentially defining evals and constraints attached to the eval. And the the third one is coming up creative primitives.

• 44:24Speaker 03

  Aiden parameter golf competition, we still found most of those creative primitives actually coming from human.

• 44:33Tim Scarfe?

  Zhong Yao, it's been a pleasure having you on the show.

• 44:36Speaker 03

  Thank you for having me.

• 44:42Speaker 05

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid and risked three hundred fifty thousand Virginians healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by H M P, the house majority pack dot com, not authorized by any candidate or candidates committee.