Karpathy’s autoresearch could make scientists of us all

Published in early March 2026, Andrej Karpathy's autoresearch AI tool makes autonomous scientific experimentation cheap and easy — but it was designed to solve machine learning problems. I wanted to see if I could apply its loop architecture to my own work: refining my worldview, testing arguments, solving business problems. In this video, I share how I adapted Karpathy’s autoresearch loops for problems that aren't easy to quantify, how to avoid the local minima trap, and the broader impact of these kinds of methods. I covered:

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Azeem Azhar?

  Today I wanted to talk about a little tool that I came across about a month ago that has really changed the way that I do some of my work. And that was a tool released by Andrej Karpathy. Now, what did Karpathy release? He released this thing called Auto Research. It's a piece of open source software. It's only 600 lines of Python code. It's already had 57,000 stars on GitHub. And what Auto Research does is it enables AI systems and you running an AI system to automatically conduct research. I said, look, this could probably work outside of the machine learning domain. Could we find some other way of defining the objective, right?

• 0:45Azeem Azhar?

  Why does it have to be the loss function in an ML product? Couldn't we define a business objective? So my new setup is this new version of Auto Research. And what that allows me to do is run this iterative loop, this scientific loop hypothesis driven testing automatically on all sorts of commercial, intellectual, academic, theoretical problems that are relevant for us. This is a reduction in the cost of the scientific method. You know, I'm applying this scientific method now to questions that benefit from it where it would have been too expensive previously. And science is the best method we have found for producing knowledge.

• 1:26Azeem Azhar?

  And we have given that method now to LLMs at very, very low cost. And I get to choose what they investigate. Well, I mean, for now.

• 1:43Azeem Azhar?

  I want you to zoom back and understand what this is about. This is about addressing. The best strategy that we as humanity have for producing knowledge, that strategy is called science. And science is the act of coming up with some kind of hypothesis and iterating through a series of experiments towards, against, beyond that hypothesis. And it's science that breaks us out of the Malthusian trap. You know, prior to having regularized science, we effectively hemmed and hawed with natural resources and population. And science allows us to get more from less.

• 2:25Azeem Azhar?

  The scientific method is the thing that allows us to do that really, really reliably. So that's the big picture. Now, Andrej Karpathy is one of the great AI researchers. He's much more than that. He's also an educator in AI. He is also somebody who builds practical tools. And a few weeks ago, he launched something called Auto Research. The thing that Auto Research does is it allows you as a human to conduct certain experiments. And what you do as a human is set a strategic direction. You set some constraints. You define what good looks like. And then you give that as a set of guardrails to your agent.

• 3:09Azeem Azhar?

  And the agent does lots of experimentation and has full autonomy as long as it stays within its lane. So there's this autonomous improvement loop within Auto Research. You change one thing. The agent picks a hypothesis. It sees how well that hypothesis performs. It measures that. And if it's better, it keeps that as the new baseline and it moves on to another hypothesis. And if it's not better, it discards it. You've solved there the principal agent problem, right? The human owns the objective, the function and the strategy, and the agent owns the execution. And the human can't get annoying and interrupt the agent and the agent can't ever get too big for its boots.

• 3:54Azeem Azhar?

  Now, the way that Karpathy designed this was that every experiment takes about five minutes. So you Have up to twelve experiments an hour, which is pretty fast. And you can run hundreds and hundreds of experiments in just a couple of days. Now, that happens to be a lot. You know, I don't know what your baseline is, but it turns out that seven hundred M L experiments in two days is a lot. And and what he discovered was twenty genuine improvements and an eleven percent speed up in the one thing that he was trying to do. Toby Lutke, who is the C E O and the founder of Shopify, which is that big e commerce platform, adapted this and he was able to develop as a consequence a small machine learning model that beat ones that were twice the size.

• 4:38Azeem Azhar?

  What we've done within auto research is solve these two really interesting problems. The first is this small automation in the production of knowledge. And the second is the agent control and harness problem. Because the way you design an auto research is you give a specific objective the agent has to optimize for, and it can't argue with you. It's a little bit like the human effectively determining where a car wants to go and the A I being the engine within the car that just churns away. Now, this is all very technical. How many of us are really trying to optimize machine learning? Probably not many, but I looked at this and I took this product and I said, look,

• 5:25Azeem Azhar?

  could we find some other way of defining the objective? Couldn't we define a business objective in some kind of way that was measurable and tangible? So if the objective was to come up with the headline for one of our essays that was more likely to get clicked, could we have three or four judges, synthetic judges that would judge that on three specific criteria and simply take the overall sum of their nine judgments, three judges, three criteria, three times three is nine, and try to maximize that over the course of an iteration process? So I constructed this version of auto research. It is not as.

• 6:09Azeem Azhar?

  Technically robust as what Carpathia had, but it had the same design principles. I, as a human, set the constraint and the experimentation takes place within the loop. And what I found was a really, really impressive piece of software. I found I was able to apply this to headlines for articles we were writing, but more importantly, I was able to find it for a thesis. And I was able to say, look, this is a thesis that I'm exploring. Let's have the oracle made up of synthetic judges and give some judging criteria to the oracle.

• 6:51Azeem Azhar?

  The way that I have built my version of auto research, this is version one, is that you can either specify what you want the criteria to be, or you can leave the AI system to come up with reasonable criteria. And then it will run through loop after loop after loop. And I will get an email that will show every iteration of the sharpening of a thesis and why certain decisions were made. And in one case, for an element in my new book where I wanted to kind of get a really strong argument in a particular area, we ran through about 19 iterations. And in fact, it was iteration 17 that was the one that I thought did the best and scored the best. Now, what are the oracles doing is the oracles are actually scoring all of this out of 10.

• 7:34Azeem Azhar?

  And you can see the quantitative leaps that are being made. We've gone from 4.6 to 5.2, from 5.2 to 5.9. We went from 5.9 to 5.7. So we will go back a step and we will do this process again. I have found that this version of auto research works across a whole range of business problems, provided that you're able to crunch down that problem into a single scoring metric, a single scaler. Now, you can hear that and you can say, well, listen, there are all sorts of problems with that approach, right? The first approach is that, can you really take a complex business problem and simply give it a single, a single number?

• 8:17Azeem Azhar?

  I mean, there are so many factors involved. And, and the answer is. You are of course simplifying as any reasoning or decision tool involves, it involves some degree of simplification. I mean, that's the nature of it because the decision you have to take is A or B, but the world is very multi-hued. There's a second problem. And that second problem is one that if you've ever worked in machine learning or in life, in fact, you you'll have come across, which is that what's expedient is not necessarily what is best. So in in machine learning or mathematical terms, that's about finding a local minima or a local maxima rather than the global minima.

• 9:02Azeem Azhar?

  And what do I mean by that? What I mean is that if you think about a problem and you think about what the very, very best possible practical solution might be, you might never see that solution because you find a good enough solution for it. If you imagine a a landscape which has got some valleys in it and what you're trying to do is find the deepest possible valley, you might be in quite a deep valley. Is that the lowest point of all the valleys in this landscape? Or have you just got to one point? That's called the local minima problem. And of course, an auto research and auto loop set up like this can run into that local minima problem. I found that I was running into that local minima problem, or rather, I was skeptical

• 9:46Azeem Azhar?

  of the quality of the results that I was getting back. And so I pushed quite hard to figure out how can I get out of those local minima? How can I give myself more confidence? So I took my version of auto research. Now, it has exactly the same process archetype as Karpathys. The human sets the objective, the human sets the guardrails. Slight caveat, I actually allow a separate AI process to set the objectives if I'm feeling a bit lazy, which, as those of you who know me will know, is a lot. And then the AI executes within those guardrails and it executes a loop. It executes little experiments. So if it is executing something around a thesis, it might say, let's change a particular word

• 10:30Azeem Azhar?

  from could to will, and does that strengthen the thesis or not? And then the oracle evaluators give that a score and we decide whether it's improved or not improved. So from that process.

• 10:43Azeem Azhar?

  I wanted to figure out how to avoid the local minima. If you're doing a commercial exploration, like what is the, what should the pricing of this partnership be? Or what's a value proposition we're going to take to this client? You can often get something that's bland and safe and over optimized. And so what I built was an escape harness. The escape harness is an additional layer that starts to throw in some random behaviour, just tries to push the system to a different part of the landscape. Because if we were already at the global minima in our explorations, we would eventually find our way there once again. So you sort of just kind of throw the problem further out.

• 11:28Azeem Azhar?

  If it converges, we were at the global minima. If it doesn't, we found a better place. It's a little bit like evolution's sudden mutations. And that works really well. It worked, it's worked well in arguments and speeches that I'm giving. If you read our piece on helium shortages at the start of this week, one part of the contribution to the reasoning was the work that I had done with my auto research on the question of helium shortages. So my new setup is this new version of auto research. I'm going to tell you what it's called in a second.

• 12:08Azeem Azhar?

  And what that allows me to do is run this iterative loop, this scientific loop, hypothesis-driven testing automatically on non-ML problems, but on all sorts of commercial, intellectual, academic, theoretical problems that are relevant for us. What we do with this escape harness is we avoid the problem of the local minima.

• 12:32Azeem Azhar?

  A quick note, if you want to support us in bringing more of these conversations to the world, please consider subscribing to the show.

• 12:43Azeem Azhar?

  Now, I've called this piece of software Auto Wolf, and I just want to explain why it's called Auto Wolf. In the 80s, I used to watch lots of terrible TV, and there was a really, really terrible show called Auto Man, which was about a man who somehow had superpowers and would turn into a car, which was slightly absurd, but it stuck in my mind. It ran at the same time as another terrible TV series called Air Wolf, which was about a very powerful helicopter. So I took Auto from Auto Man and Wolf from Air Wolf and put them together and called it Auto Wolf. And Auto Wolf now is a bit of code that my open claw agent, Arminy Arnold, or one of the open claw agents that work for it,

• 13:25Azeem Azhar?

  can call on if we need to do some reasoning in a particular area. And it has all of those looping capabilities, but also that escape harness in place to prevent the local minima problem. There are some practical learnings from all of this. You know, one is that the loop can find moves that you wouldn't make. Some are better, some are worse, and you have to judge. The cost is trivial. It's dollars, maybe tens of dollars, often on work that you've procrastinated on for a long time. You do need some kind of stopping criteria. So I tend to have this thing stop at 20 iterations.

• 14:08Azeem Azhar?

  You do want to have the discipline to check where the thing is going after five or six iterations. So normally, the way I have it set up is that every five iterations, there's like a bit of intervention that I would go in and make and say, this is going in the right direction, or this is so bland, I just need to stop it. The pattern works, but it doesn't work on everything, and it's part of a much larger architecture. So within Exponential View, we have a sort of a ladder of reasoning architecture, tools that we use as part of our overall reasoning architecture, which starts with single-shot expert panels, where we can draw personas in from our persona library

• 14:52Azeem Azhar?

  of hundreds and thousands of personas. We then have kind of the traditional one-shot auto research, which is built within large language models, top to toe, which is iterative and pretty good. There's also now auto wolf, which has a bit of code to make it a bit more reliable, but also has the escape harness. And then beyond auto wolf, there are more sophisticated, more expensive methods that we will use in order to make sure we're thinking things through. There are still kind of interesting challenges that come out of all of this. If the outcome is unmeasurable, you can't really use a process like auto wolf at all.

• 15:35Azeem Azhar?

  I mean, auto wolf will need that objective measure that gets syncretized by the oracle and the judges that you've created within the oracle. If those metrics are contested, you run into the same problem. If there's complexity and path dependence, again, you've got something that is too complex a problem to put into this. There's something else that emerges from all of that, and it's true within our overall reasoning architecture, from the expert panels to auto research to auto wolf and to the things that sit beyond, which is that the human interaction is still absolutely critical. I mean, look, is it garbage in, garbage out? Well, in to some extent it is, right?

• 16:17Azeem Azhar?

  If I kind of bark something that's incoherent, we're going to optimize on something that's incoherent. But you have to judge and your role moves from doing the work to judging the work. We're talking a lot about this in terms of how do humans fit into a world with this kind of machine cognition, machine intelligence, and people talk about it being to do with validation, or we talk about it to do with verification. This is judging. I mean, verification is one subset of judging and judging and validation are similar, but they're not identical. And by judging, what I mean is that I have to look into these thinking traces that are quite long documents. I mean, every round is maybe a thousand word markdown document that comes out and have

• 17:02Azeem Azhar?

  a look at it or a scan and say, that's heading in the right direction or that's not heading in the right direction. Now, in truth, I'm used to doing that anyway, right? These little grey hairs on my beard are a sign, not of too much sun exposure, but of the fact that I've been working for 30 years, which means that I'm used to getting people who are working for me to give me interim updates and for me to course correct them. Sometimes it's a nudge here, sometimes it's a nudge there, sometimes it is the escape harness. We've started down this path, actually it's clearly not the right one at all, let's go somewhere else. So I'm used to that method, it's just that I'm having to do it at a much higher cadence than I ever have before.

• 17:42Azeem Azhar?

  So if I run an auto research prompt on something that that matters, I will often find myself in the morning, the first 30 minutes of the day, going through things that would run overnight or that have backlogged from the day before to say, are these right and do we need to move them on in any way? And then you get to the next part of the bottleneck, which is the pace with which we're making decisions is much faster than the rate at which we are used to making decisions and therefore acting on those decisions that come through. And we are learning how to make use of all of that and to make sense of it. But that's what this iterative loop has enabled.

• 18:22Azeem Azhar?

  Because when you run a Carpathia auto research in this loopy economy, the loop economy, you get through something that was going to take a week in perhaps an hour. And I've certainly found that similar processes happening in the way, the types of problems that we would look at. What that means is that there's a whole class of decision making, which might have been made, to put it euphemistically, heuristically, like, oh, I just kind of made it up.

• 18:51Azeem Azhar?

  Now go through some kind of soak testing in a way that is robust and in a way where I have been forced to explicitly state what the objective is. That in of itself is a very, very useful piece of self-reflection to help me think about the problem to hand. I think this is a really interesting product capability that Carpathie released and certainly it's become part of my toolkit in the last three or four weeks. I do turn to it and think, okay, let's go off and push that through auto research. Not for everything, not everything benefits from something like that.

• 19:30Azeem Azhar?

  What I love though is that this is a reduction in the cost of the scientific method. You know, I'm applying this scientific method now to questions that benefit from it where it would have been too expensive previously. And science is the best method we have found for producing knowledge and we have given that method now to L L Ms at very, very low cost and I get to choose what they investigate. Well, I mean, for now, those are human decisions that I make. And of course, those of you who know me, who listen in on this, know that I'm always experimenting to see to what extent the agents can help me set up the right experiment to give them the right objective the next time I run that loop.

• 20:15Azeem Azhar?

  As I experiment more with Arminy Arnold and the agents that work for it, I will of course come back and report my findings to all of you. If you're interested in me putting in the effort to make Auto Wolf available on GitHub, then on the Sunday newsletter, when it comes out, please comment again, leave the specific thing that you're interested in looking at and that will give me motivation to find the time to do all of that.

• 20:52Azeem Azhar?

  Thanks for listening all the way to the end. If you want to know when the next conversation is released, just hit subscribe wherever you're listening. That's all for now and I'll catch you next time.