When AI Discovers The Next Transformer - Robert Lange (Sakana)

Robert Lange, founding researcher at Sakana AI, joins Tim to discuss *Shinka Evolve* — a framework that combines LLMs with evolutionary algorithms to do open-ended program search. The core claim: systems like AlphaEvolve can optimize solutions to fixed problems, but real scientific progress requires co-evolving the problems themselves. GTC is coming, the premier AI conference, great opportunity to learn about AI. NVIDIA and partners will showcase breakthroughs in physical AI, AI factories, agentic AI, and inference, exploring the next wave of AI innovation for developers and researchers. Register for virtual GTC for free, using my link and win NVIDIA DGX Spark (https://nvda.ws/4qQ0LMg) • Why AlphaEvolve gets stuck — it needs a human to hand it the right problem. Shinka tries to invent new problems automatically, drawing on ideas from POET, PowerPlay, and MAP-Elites quality-diversity search. • The *architecture* of Shinka: an archive of programs organized as islands, LLMs used as mutation operators, and a UCB bandit that adaptively selects between frontier models (GPT-5, Sonnet 4.5, Gemini) mid-run. The credit-assignment problem across models turns out to be genuinely hard. • Concrete results — state-of-the-art circle packing with dramatically fewer evaluations, second place in an AtCoder competitive programming challenge, evolved load-balancing loss functions for mixture-of-experts models, and agent scaffolds for AIME math benchmarks. • Are these systems actually thinking outside the box, or are they parasitic on their starting conditions? When LLMs run autonomously, "nothing interesting happens." Robert pushes back with the stepping-stone argument — evolution doesn't need to extrapolate, just recombine usefully. • The AI Scientist question: can automated research pipelines produce real science, or just workshop-level slop that passes surface-level review? Robert is honest that the current version is more co-pilot than autonomous researcher. • Where this lands in 5-20 years — Robert's prediction that scientific research will be fundamentally transformed, and Tim's thought experiment about alien mathematical artifacts that no human could have conceived. Robert Lange: https://roberttlange.com/ --- TIMESTAMPS: 00:00:00 Introduction: Robert Lange, Sakana AI and Shinka Evolve 00:04:15 AlphaEvolve's Blind Spot: Co-Evolving Problems with Solutions 00:09:05 Unknown Unknowns, POET, and Auto-Curricula for AI Science 00:14:20 MAP-Elites and Quality-Diversity: Shinka's Evolutionary Architecture 00:28:00 UCB Bandits, Mutations and the Vibe Research Vision 00:40:00 Scaling Shinka: Meta-Evolution, Democratisation and the Three-Axis Model 00:47:10 Applications, ARC-AGI and the Future of Work 00:57:00 The AI Scientist and the Human Co-Pilot: Who Steers the Search? 01:06:00 AI Scientist v2, Slop Critique and the Future of Scientific Publishing --- REFERENCES: paper: [00:03:30] ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution https://arxiv.org/abs/2509.19349 [00:04:15] AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery https://arxiv.org/abs/2506.13131 [00:06:30] Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents https://arxiv.org/abs/2505.22954 [00:09:05] Paired Open-Ended Trailblazer (POET) https://arxiv.org/abs/1901.01753 [00:10:00] PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem https://arxiv.org/abs/1112.5309 [00:10:40] Automated Capability Discovery via Foundation Model Self-Exploration https://arxiv.org/abs/2502.07577 [00:15:30] Illuminating Search Spaces by Mapping Elites (MAP-Elites) https://arxiv.org/abs/1504.04909 [00:47:10] Automated Design of Agentic Systems (ADAS) https://arxiv.org/abs/2408.08435 PDF : https://app.rescript.info/api/sessions/b8a9dcf60623657c/pdf/download Transcript: https://app.rescript.info/public/share/SDOD_3oXOcli3zTqcAtR8eibT5U3gam84oo4KRtI-Vk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  This episode is brought to you by Indeed. You're ready to level up your business, but the pressure to find the right person to make that happen is intense. Indeed sponsored jobs can help carry the load. It boosts your job post to help it reach quality talent fast. Cut through the chaos with Indeed sponsored jobs. Get a $75 sponsored job credit at indeed.com/podcast. Terms and conditions apply. Hiring now? Then this is a job for Indeed sponsored jobs.

• 0:31Speaker 01

  Low energy, unfocused, foggy? You might be dehydrated. Whether it's hot yoga, over 100 degree weather, or too much sun. Gator Light, with a specialized blend of five electrolytes, was scientifically designed to help move fluids into the body faster for rapid rehydration. Shop now at retailers nationwide. Gator Light hydrates faster than water. Is it in you? Symptoms noted are signs of mild to moderate dehydration. Consult a healthcare professional if symptoms persist.

• 0:59Speaker 02

  I think a lot of analogies from evolution transfer to scientific research, right? In the sense that we traverse a tree of different ideas or different experiments, and then in a paper, we report one path through that tree.

• 1:12Tim Scarfe?

  When we run LLMs autonomously, they tend to just kind of, like, nothing interesting happens.

• 1:20Speaker 02

  But oftentimes, innovation for a specific problem might require first inventing a different problem, right? Sort of automatically coming up with this reduction or like this, let's say, recursive nature of problem solving is something these systems right now not necessarily have built in intrinsically, right? Oftentimes, it's easier to generate a lot of solutions than to actually hard verify them, right?

• 1:42Tim Scarfe?

  The reason why I'm not that worried yet about labor market disruption is I still believe deeply that humans are the source of deep understanding and creativity in the world. If I didn't believe that, I would be very worried.

• 1:55Speaker 02

  So I think it's going to be an amplifier of these latent dimensions humans are great at, right?

• 2:01Tim Scarfe?

  And I think one of the Rubicon moments is when the new transformers architecture or something massive is discovered by AI, and we're all using it. Nvidia GTC starts Monday in San Jose and it's free to attend virtually online. There's already been a leak this week of something called Nemo Claw, which is an open source agent platform. And if it's real, it could be one of the bigger announcements this year. So it's definitely worth watching Jensen's keynote for that alone. I'm giving away a DGX spark. Nvidia just hiked the price $700. You probably heard about these memory shortages, right? So yeah, it's now $4,700, which is very, very expensive.

• 2:42Tim Scarfe?

  And Merv from Hugging Face, by the way, she got one for her birthday and she said she literally cried. So it's a really cool bit of kit. If you register through my link in the description and you attend at least one session, then you are in the draw. This is a massive conference. Physical AI and robotics are going to be the breakout theme. And Jensen does the keynote Monday at 11:00 AM Pacific. The link is in the description. Don't miss it. Robert Lange, it's amazing to have you on MLST.

• 3:09Speaker 02

  Thank you, Tim. It's a pleasure to be back.

• 3:11Tim Scarfe?

  So you're working for Sakana. Tell us about that.

• 3:13Speaker 02

  Sakana AI is a Japanese AI startup, working mostly on AI for Japan. And at the same time, sort of exploring, let's say, novel or ambitious ideas on the research side.

• 3:26Tim Scarfe?

  It's been around for over a year now. You're one of the founding researchers, right?

• 3:30Speaker 02

  Exactly. So Sakana has been around for now, like almost two years, like one and three quarters, I would say. And yeah, it's pretty fascinating to look back and to look at the early days and how much the company sort of organizationally has changed. But in spirit, like we're trying to sort of embrace Ken Stanley's open-endedness idea and sort of explore many different ideas, which might not get the resources right now in the ML community more generally.

• 3:56Tim Scarfe?

  And we've got a few interviews coming out with Sakana that we filmed here in Japan. So I won't spoil the surprise, but the CEO is David Ha. And David, you know, like there are these epic, you know, giants out there like, you know, Cloo and Stanley. David Ha is one of these people.

• 4:11Speaker 02

  David's work has had a lot of influence on my personal PhD, right? He did a lot of fascinating work on hyper networks and sort of modulation in neural networks, but also on evolutionary computation and evolutionary optimization. And yeah, that's sort of also painted my path during the PhD.

• 4:28Tim Scarfe?

  You've released a paper called Shinka evolve and we were just saying that that kind of means evolve evolve because in Japanese shinka is evolve but that's quite common. It's a common thing to do to have these like multilingual, you know, double, double namings in Japanese. Just before we get there, so we interviewed the Alpha evolve team and I also interviewed Jeremy Berman a few weeks ago. And your paper is very much like a more sophisticated version of those in the sense that it's using language models to generate programs and it's doing an evolutionary approach where we generate the program, we refine the generated program and we have an evaluator and we do this over several steps. And, and your, your approach does many things that the other ones don't do. Tell me about the paper.

• 5:12Speaker 02

  First off, of course, this was partially inspired by Alpha evolve. I think it's great work. I know Alex and Mate and I think they're doing incredible science. One thing that sort of is important about sort of using all of these evolutionary L L M driven methods is sample efficiency, right? So many of these systems sample like, let's say a thousand programs for a given task. And what we tried to do with Shinka evolve was try to essentially cut down costs as well as sort of computation evaluation time by introducing a set of sort of technical innovations to this evolutionary search. And we showed that it's possible with very few program evaluations to basically improve upon like, for example, the circle

• 5:53Speaker 02

  packing canonical result that they showed in their paper. And yeah, more generally speaking, I think we're right now at a point or like at an inflection point where these sort of, let's say evolutionary driven L L M systems can really revolutionize scientific discovery. And yeah, we hope to have made a step forward to making this more democratically accessible, right? So the code is open source available and yeah, by its sample efficient nature, we hope that many people can interact with the system and can make their own scientific discoveries as well.

• 6:28Tim Scarfe?

  Yeah, that's actually a really important point because I suppose we can use these foundation models. And first of all, isn't it just fascinating to reflect that we have these amazing models out there that we can access, so like G P T five and Grok four, and they are so much better when you get them to refine their solution in several steps. Why is that? I mean, I suppose a naive question would be, why aren't they just good out of the box?

• 6:53Speaker 02

  Potentially, like with enough random samples, right? It's sort of this monkey typing on the keyboard. They would potentially be able to get there, right? But in principle, it's sort of coming back to the principles of evolution, right? In the sense that you need to collect a bunch of stepping stones first and then build on top of them to really find innovations or to tune innovations down the line. And I think language models with the right sort of evolutionary harness are extremely powerful in terms of scaling up to make discoveries. And yeah, I think Jeremy, as well as the Alpha Evolve paper, as well as sort of work we've done on like the Davin-Godel machine, for example,

• 7:34Speaker 02

  shows that this sort of stepping stone accumulation plus iterative verification and collecting sort of information and evidence from the real world, real synthetic evaluator is really important for that.

• 7:48Tim Scarfe?

  Very cool. And stepping stone collection, so this is, it came from Kenneth Stanley. It's a wonderful paper, Why Greatness Cannot Be Planned. And he said that it's better to have systems that don't converge. So in natural evolution, we are just trying all of these different things. And greatness quite often follows a diverse path, which means you have to do things which initially seem quite stupid. And then later on, they turn out to be incredibly useful.

• 8:15Speaker 02

  Yeah.

• 8:15Tim Scarfe?

  We're trying to design algorithms that can kind of allow for a population of slightly weird things. And then we kind of lock in and converge a little bit. So we're still converging though. So we're still building systems that don't diverge forever. What are we losing?

• 8:33Speaker 02

  One thing I find extremely important after having done Nanchika evolve is sort of this problem, problem, right? So with all of these systems so far, maybe except for the AI scientist, which we can also talk about, the problem is given, right? So you have an evaluator, you have a correctness checker, and you sample programs only on that single problem, right? But oftentimes, innovation for a specific problem might require First inventing a different problem, right? So, for example, I think in the matrix multiplication result that the alpha evolve people show, you can recursively apply sort of the algorithm to larger matrices. So it's actually an important result, right? But sort of automatically coming up with this reduction or like this, let's say recursive

• 9:17Speaker 02

  nature of problem solving is something these systems right now not necessarily have built in intrinsically, right? So I think going forward, it's going to be really important to not only sort of do open-ended, let's say, optimization of solutions, but sort of do the co-evolution of problem and solution together in order to collect even more diverse stepping stones and to really kick off this this open-ended process. Because also to me, like one of the the big life goals or achievements I would want to see is really having a process that can run not only for, let's say, a week or many weeks, but like for years even potentially, right? Collecting even more diverse, interesting stepping stones.

• 9:57Tim Scarfe?

  Yeah, I spoke to Joel Lemon and he was talking about the knighting uncertainty, which is that machine learning algorithms aren't very good with unknown unknowns. And in a sense, the unknown unknown is talking about these these stepping stones that might be useful later. And when we run these algorithms at the moment, it's the same with L L M's and reasoning systems, is that they're very, very good when we give them a specific thing. And what you're pointing to is we might need to invent new unrelated problems and find the solutions which might then be related to what we're trying to do. So that feels like a bit of a catch-22 situation, right? So we're saying, you know, circle packing, here's my evaluation function, and I want

• 10:40Tim Scarfe?

  you to sort of diversify and then, you know, kind of, and then converge towards this solution. It's just, I had the same thought with genie, by the way, that it gives you exactly what you ask for. So you put a prompt in, you know, like a Swiss lake with, you know, with boats on the water and mountains on the side. And I was thinking, where are the birds? Oh, I forgot to put birds in the prompt. Right, so how can we meaningfully build systems that actually kind of bring in other unknown things that might be useful?

• 11:07Speaker 02

  I think one inspirational thing I would personally wanna sort of research are systems like outlined in Power Play or Poet by Schmitt-uber and by Jeff Clune and others. So where there is essentially like a set of tasks and a solution generator and both of them sort of co-evolve in this almost like art curriculum, play-like style, right? And I think sort of in Poet, the natural first application was sort of reinforcement learning, but I think this can now be broadened up to, yeah, science more generally, right? At least when there is a simulator available for running these evaluations.

• 11:44Speaker 02

  And by doing such a co-evolution, you always try to max out the capabilities of that generator while sort of increasing this convex hull or potentially even, yeah, more diverse problems while doing so.

• 12:03Tim Scarfe?

  I know that there's always the leading thought that even with Poet, which was this thing where you had like a load of environments and agents and the environments were complexified, so the agents would have a kind of effective curriculum to learn things in increasing complexity. But even then, isn't there a kind of design bias in the system where there's some code somewhere which complexifies the environment step by step? And wouldn't that also just be designed by the humans? So it would also just give you exactly what you ask for?

• 12:35Speaker 02

  Ultimately, this comes down to like the hypothesis that language models can potentially do extrapolation or interpolation, right? In the sense that even though these things might be in the end designed by humans, there are many unknown unknowns, right? That we humans didn't think of while designing them, right? So potentially, it is possible for an LLM to, yeah, find a novel discovery simply by us not having thought about it before, right?

• 13:02Tim Scarfe?

  When we run LLMs autonomously?

• 13:05Speaker 02

  Yeah.

• 13:06Tim Scarfe?

  They they tend to just kind of like nothing interesting happens. So depending on the prompt you give them, they'll kind of go a few steps in that direction. And then no new interesting novelty emerges. And I think even if you wire them ad freshly with environmental feedback, they they still seem quite parasitic on their starting conditions. With an L L M, could we build a system which actually adapted to novelty that could actually discover new things?

• 13:34Speaker 02

  I think it really kind of also depends on what do you give the L L M as a starting point, right? So for example, in Shinkai Wolf, we from time on time saw that if you give an initial solution program, which is already pretty optimized on the problem at hand, you still kind of get stuck in in local optimal, right? Where not a lot of novelty is introduced, right? While if you start off from like an impoverished solution, there's much more room for diversity. And I think this is sort of coming back to sort of what I did before in in my research, namely meta learning. Sort of this classical trade off where you can either start out from something very, let's say unconstrained, from like a very simple solution and give much more room for the optimization.

• 14:19Speaker 02

  But this might actually require open endedness and a long time to find a good solution. Or you start out from something that is already very constrained by inductive biases, let's say. And then you might be much more efficient in terms of convergence, let's say. But you don't have this sort of open ended, big novelty sort of benefit from it.

• 14:42Speaker 00

  This episode is brought to you by Indeed. You're ready to level up your business, but the pressure to find the right person to make that happen is intense. Indeed sponsored jobs can help carry the load. It boosts your job post to help it reach quality talent fast. Cut through the chaos with Indeed sponsored jobs. Get a $75 sponsored job credit at indeed.com/podcast. Terms and conditions apply. Hiring now? Then this is a job for Indeed sponsored jobs.

• 15:12Speaker 04

  Low energy, unfocused, foggy, you might be dehydrated. Gatorade Zero powders have been scientifically designed to help improve hydration balance in the body. With a clinically proven electrolyte blend and no artificial colors or flavors. Great tasting hydration, no matter where you're going. Shop now online or at retailers nationwide. Gatorade, hydrates better than water. Is it in you? Symptoms noted are signs of mild to moderate dehydration. Consult a healthcare professional if symptoms persist.

• 15:42Tim Scarfe?

  Yes, I suppose where we want to get to is building systems which are not designed by humans. So for example, if I'm leveraging my deep understanding, you know, LLMs are really good if you understand something deeply. And similarly, we could kick off a Chinkar revolve and we could, we could put a starting solution in there which leverages my understanding. We want to have AI systems that, that anyone could use. So just a non-expert could say, I want to solve this problem and it will solve the problem. We should talk about the evolutionary approach, right? So to maintain diversity, you had a population of programs and they were separated into islands. Tell me about that.

• 16:22Speaker 02

  The way how Shinkar evolved, similarly to Alpha evolved, works is you keep an archive, like a database of programs, and then you sample parent programs with a set of sort of inspiration programs. And then you ask an LLM to basically make an improvement to that program, right? So to provide code edits or rewrite an entire program or to potentially even cross over different programs. And then you, you basically, you query the LLM, you get a program out and you evaluate it on the problem it had, right? So for example, increasing the sum of the radii of a bunch of circle in a square. You run this basically each time collecting evidence from the evaluator, adding it to the database and then sort of repeating this process.

• 17:05Speaker 02

  And you don't do this sort of sequentially, but you do this in parallel for many different programs. And each time sort of a program is added, you essentially try to diffuse the knowledge that was collected by the program across the entire sort of database, right? So one way to think about this is you have a tree, a tree where each node in the tree represents a program and then you sort of branch off of it based on the parent nodes, right? And interestingly, like these approaches do tend to scale. But ideally, we can make the scaling happen at a faster rate, right? And this is something we tried in Schenka evolve by sort of doing a bunch of innovations, including sort of model ensembling.

• 17:50Speaker 02

  So we're not using just Gemini, but we're using basically all frontier model providers and figuring out a smart way how to use each model for a given parent, right? So if you have a certain program and some situations, it might be better to use a sort of GPT model. In other settings, it might be better to use Gemini model. And we sort of introduce a sort of adaptive prioritization scheme that can adapt sort of the evolutionary algorithm on the fly while running the algorithm. And this sort of also comes back to the naming, right? So Schenka evolve, evolve, evolve kind of means that this evolutionary algorithm that we apply using L L M's sort of also co-evolves at the same time while we optimize the programs.

• 18:35Tim Scarfe?

  And on this, while we're on this circle packing problem, so you had this plot showing how it converged and it seemed to converge quite quickly. So we'll show the plot on the screen now. So very quickly, the performance jumped up and then it slowly converged. And you said in the paper that it was using three, I think, three core innovations. And my thinking was, if you ran this 50 times, would it be the same every single time? And how, to what extent is it thinking outside the box? You know, Sebastian Bubeck is always posting on Twitter, talking about how G P T five has just, you know, discovered new things. And there's always the question of, well, is it just searching the internet? Is it just finding things that have been found before? And yeah, combining things together in a new way, but could it really think outside

• 19:20Tim Scarfe?

  the box?

• 19:21Speaker 02

  Yeah, I think this is almost like a subjective question, right? So first off, I don't know all problems on the internet that try doing circle packing, but what I can see in the tree that we also depict is there's, for example, like a crossover operation between two programs happening where sort of different concepts are combined, right? So one important part is, for example, the initialization of the circles. Another one is like the optimization. So basically, like a constrained optimization program is executed. And then the final part is basically like a reheating stage, right? Where noise is added and sort of more strategy squeezed out. And to me, like this sort of propagation of information through the tree is one that's really, really fascinating, right?

• 20:06Speaker 02

  Where in some sense, these stepping stones are actually used and so in a complementary fashion, right? And with regards to rerunning the program multiple times, right? Of course, there's some stochasticity in it, right? So we're using language models and sort of due to like the queuing device scheduling on their server side, basically, we can't get rid of all the noise. We've seen that, at least for the general quality of the solution, so what is around afterwards, it is possible to reobtain this. But sometimes with a different program, like or most of the times just by stochasticity, right? So it's not like there's for many problems, there's like not one solution that achieves that score, but there is like a spectrum or like a region, let's say, in the program

• 20:51Speaker 02

  space that, for example, is the same, right? I think one thing that was very interesting about the circle packing problem, sort of also coming back to the problem, problem that I discussed initially, was that originally we we used a formulation where the correctness is checked with like a very tiny amount of slack, right? So the circles could overlap a tiny little bit. And then afterwards, we sort of reduced the rate AI and the solution was exact, right? This didn't change the score by too much, so it's still state of the art. But it was essentially like a proxy problem. We then reran the Shinkai evolve on the exact setting. We found that it took a little bit longer to actually obtain the same quality of a solution.

• 21:36Speaker 02

  So I think this already points a little bit in this direction of what I discussed in the beginning. Like sometimes sort of surrogate problems might actually be extremely valuable in making such discoveries. And having an automated way for designing these surrogate problems in an efficient way might be something really important going forward.

• 21:55Tim Scarfe?

  Yeah, that's absolutely fascinating. It reminds me of support vector machines where we make the optimization tractable by introducing slack variables. And you can think of that as a kind of surrogate problem. But then I'm thinking, well, would Shinka evolve, would Alpha evolve? Would it know to introduce a surrogate problem? Because, you know, as designers who understand, you know, we can think outside the box and we can do stuff like that. Because presumably if the fitness function had the constraints that there were no circle intersections, then it wouldn't it wouldn't occur to the algorithm to come up with a surrogate problem.

• 22:25Speaker 02

  Exactly. Yeah, this is a big limitation right now, right? So at this current point in time, we take the problem to be fixed and we optimize for that problem. But when you think about humans, we're really, really good at sort of inventing our own problems, right? Or reformulating the problem so then we can actually sort of work with it, right? So I think a lot of sort of the innovations in, let's say, mathematics come from taking a very different perspective on a problem, right? So taking sort of number theory and applying it to linear algebra or the other way around. And I think right now these systems are not yet at the point of achieving such level of, let's say, transfer.

• 23:05Tim Scarfe?

  Yes, and it reminded me, I spoke to Leon about this. You've got this Sudoku bench and a lot of folks watch Cracking the Cryptic YouTube channel. And that's exactly what they do. They invent new problems based on abstractions that capture the essence or aspects of the problem you're solving. And then they do something which is similar to Shinka evolves. They do this kind of evolution where they take these different solutions and they kind of combine the best aspects of both of them. And they forge a divergent path to a new solution. And that seems to be the essence of what we need to do.

• 23:36Speaker 02

  Yeah, for sure. I mean, there is some work also by Jeff Clune, Shengren Hu and On automatic automated capability discovery. So there they look at language models that generate tasks, right? But it's in a, let's say, unstructured way in the sense that it's not done in order to enable the solution to one target problem, right? And I think sort of doing these connections is going to be very fruitful down the line.

• 24:02Tim Scarfe?

  Very cool. Now, the other thing, we'll show the graph on the screen, the evolutionary graph. So for the circle packing problem, I was looking at that. And first of all, it looked incredibly parsimonious, which is good. It looked like it had found an optimal path to the solution very quickly. And I was thinking in my mind, well, maybe there's some natural pattern. There's something about that that we could use in the abstract to guide the evolution in the future. But the other thing I'm thinking about is right now, the problem with machine learning is that we don't really have semantics baked in. So what we're doing is we have a verifier, we're looking at the reward, and we're sort of like doing pattern exploration and we're taking steps towards the, you know, towards the target.

• 24:42Tim Scarfe?

  And I love mechanistic forms of reasoning where we actually know something about what the program components mean. And the reason this is important is when we're merging together the best performing programs from two different islands. That's a kind of first order interaction. And it might not make sense to merge them together. It's wonderful that L L M's, you can give them any pairs of programs and it will find a way to merge them together. But wouldn't a more principled way be of there's there's some kind of semantic primitives here? And we know they fit together. So there's this Lego analogy that we're kind of building up based on principles rather than forging a path based on the performance.

• 25:22Speaker 02

  Yeah, that's a good point. So one thing we do in Shinkai Evolve as well is we keep essentially a scratch pad. So each program is being summarized. And then from the program summaries, we keep sort of a set of global insights, let's say, that were shared or like extracted from these programs. And then based off of the scratch pad, we construct sort of meta recommendations that then become part of the system prompt, right? So that way you can try to sort of semantically grasp some of the discoveries. But in general. Problem, which is again sort of task dependent is thereby you sort of diffuse that knowledge across the tree. Right. But sometimes you want things to be much more isolated. Right. It's always like a trade off where you somehow have to find for your problem.

• 26:07Speaker 02

  The right position on the spectrum of how much knowledge diffusion do you want to have? And how much sort of, let's say, hard islands of programs do you want to have? Right. And yeah, we're trying to make steps in the direction of sort of automatically adjusting. This in an optimal way, but again, it's very program sensitive. And then sort of, I think another point where you're already sort of going into is sort of Jeremy. Jeremy's solution to Arc AGI, right? And sort of doing solution evolution in the instruction space, right? Instead of the program space, I do think that this is something important. And we're, like I said, with like the construction of this meta scratch pad, trying to do sort of both at the same time.

• 26:50Speaker 02

  Again, it's problem dependent. Like I played around a little bit with Arc AGI one, AGI one and Arc AGI two. And I think on Arc AGI one, actually the the transform sort of program direction is actually quite effective. Right. It's like Jeremy said, it's deterministic and it's easier to sort of get clear signal to improve on during your evolution process. While on others like Arc AGI two, like this whole sort of semantic evolution seems to be more efficient. So I think ideally we we can get a system that can automatically, in some sense, decide whether or not it wants to take like a programmatic approach in settings where it's actually feasible and easier to bootstrap off. Or it takes the semantic approach of evolving instructions or like L L M driven input output mappings.

• 27:36Tim Scarfe?

  Yeah, it's it's so interesting because, you know, like a symbolic AI person would say, oh, I don't like connectionism because it doesn't, you know, the only semantics in connectionism is this notion of similarity. It doesn't really understand things. So so they would say, well, just just start with an entity relationship graph. And then just kind of build up using, you know, composition and first principles. That that that doesn't work, right? So we're using neural networks because they're incredibly flexible. They understand a lot of things about the world, but they don't have the kind of constraints that we want. So what we do is we use these tricks. So Jeremy evolved program descriptions. On your program selection, you had a semantic novelty detection, you know, using like a.

• 28:16Speaker 02

  Embedding based similarity, yeah.

• 28:17Tim Scarfe?

  Yeah, so you had like a kind of self similarity matrix and, you know, based on the the cosines. And indeed, you've got this meta scratch pad. So what we're seeing is this fascinating spectrum of possibilities. Where still using neural networks, you can imbue semantics in using all of these different tricks, but they all come with trade offs.

• 28:35Speaker 02

  Yeah, for sure. Like, I think it's it's kind of interesting. We we've had a long period of computer science where algorithms were sort of designed by humans, right? Then we had sort of this Andre Kapathie software two point one O paradigm where like we trained neural networks that then performed a certain function. And now we're sort of at this point where we're using L L Ms to design algorithms or solutions more generally, right? And I think actually, like, even though, like, large frontier language models are extreme, like, let's say, black boxes, or it's very hard to get a full mechanistic understanding of them, the outputs can be, right? The programs, the instructions and so on, right? So I think it opens up a very sort of new paradigm of

• 29:20Speaker 02

  doing research or basically doing anything, right? If you if you think about it. But I think we're we're just sort of at the starting point of figuring out the the right user interface for that.

• 29:30Tim Scarfe?

  So the other innovation in the paper was using U C B, which is upper confidence bound. It comes from the multi armed bandit literature, which is this problem where you can pull these these levers. And at the beginning, you don't know which levers to pull. And and over time, you kind of reduce your uncertainty and you can kind of pull the ones that work. But there's this exploration, exploitation dilemma. And you've implemented that for figuring out which L L M. So it could be Gemini, it could be like, you know, Groff or something, to figure out which one to use.

• 30:01Speaker 02

  We're we're using like a model ensemble, right, to propose program mutations and

• 30:08Speaker 02

  intuitively, one could say like the the best frontier model on on Sui Bench is always the best mutation proposal. Model, but that's actually in practice, not always the case. Right. And in general, it's extremely hard in this evolutionary setting to assign clear credit to a single model, right? So you have, for example, like one improvement is implemented by GPT five. And then the next one is implemented by Sonnet four point five. And it's unclear, basically, if the performance gain you get from the second mutation actually originated from GPT five, sort of collecting the first stepping stone or from Sonnet four point five. So instead of sort of uniformly sampling models, what we do is we implement this bandit

• 30:51Speaker 02

  based approach where each model is basically one arm of a bandit. And then we look at how often did this model improve performance of a sort of parent node by creating a mutation. And we then adjust sort of this posterior probability to sort of first explore all arms once, right? And then essentially change over the course of time in order to prefer models that sort of yielded improvements before for similar nodes.

• 31:21Speaker 00

  This episode is brought to you by Indeed. You're ready to level up your business, but the pressure to find the right person to make that happen is intense. Indeed sponsored jobs can help carry the load. It boosts your job post to help it reach quality talent fast. Cut through the chaos with Indeed sponsored jobs. Get a $75 sponsored job credit at indeed.com/podcast. Terms and conditions apply. Hiring now? Then this is a job for Indeed sponsored jobs.

• 31:51Speaker 04

  Low energy, unfocused, foggy? You might be dehydrated. Gatorade zero powders have been scientifically designed to help improve hydration balance in the body with a clinically proven electrolyte blend and no artificial colors or flavors. Great tasting hydration, no matter where you're going. Shop now online or at retailers nationwide. Gatorade hydrates better than water. Is it in you? Symptoms noted are signs of mild to moderate dehydration. Consult a healthcare professional if symptoms persist.

• 32:20Tim Scarfe?

  The great thing about using a UCB-like algorithm is it actually has a theoretical regret, which means it's like only log worse than the optimal switching path, if that makes sense. But if I understand correctly, UCB is based on a sort of like a global rating, like a mean score of every single LLM. And I think what we want is to have more of a contextual switching decision, which means we know for this particular program Gemini is better. And do I understand correctly at the moment that it might converge to a single frontier model? And then in a nuanced situation, we might still get the wrong model.

• 33:01Speaker 02

  So in general, like there is some amount of probability allocated to all models, right? So it's not like it can just pick on one model and then you stop using the others, right? So there's still a chance for open-endedness and serendipity, if you will. And we, in general, like for the problems we consider, we haven't seen that like one model clearly dominates all the others, right? We've seen then it really depends on the course of this evolutionary process, like which model is better. And UCB or like the bandit approach that we take dynamically adjusts this in an efficient way.

• 33:37Tim Scarfe?

  And would it be possible in the future to use an LLM to make this judgment?

• 33:40Speaker 02

  Potentially, in some sense, in that case, again, you think of the LLM as a surrogate model, right? In some sense, you can think of like a Gaussian process as a surrogate regression model. And there has been some work sort of showing that language models can act as surrogate models. And the real question to me is like, how do you represent the information to the LLM, right? In the sense that if you use like the raw programs and their fitness evaluations, you quickly run out of context, right? So you need some amount of compression in order to present the information the right way to the LLM in order to do this prioritization of the models.

• 34:15Tim Scarfe?

  I hadn't appreciated how long the context is. I mean, I was thinking, you know, could we use like an eight billion Llama model? And we're doing active fine tuning. So we're saying, I just ran it on, you know, I just ran this program on Grok and it got. This score.

• 34:30Speaker 02

  Yeah.

• 34:31Tim Scarfe?

  And then over time, that, you know, this thing for the given run of this evolution, it will kind of know that Grok is good at these problems.

• 34:39Speaker 02

  Yeah, potentially. I'm not sure like how efficient this fine-tuning is if if we're only evaluating like 150 programs. But in principle, one could imagine, I think it's on the engineering side, not necessarily like the prettiest to do. Yeah, it could, it could in fact happen. But I think like for all of these things, we started out sort of with the, let's say, most intuitive algorithmic component that we had and UCB was one that really did the job here. And yeah, much credit to Eduardo Setin, who introduced us to Shinkar.

• 35:11Tim Scarfe?

  So let's talk about the the diffs and the mutations. So we we generate programs and I think you folks were inspired a bit by Alpha Evolve. So they actually had this gating where where you kind of gate part of the code which is mutable. Tell me about all of that.

• 35:28Speaker 02

  A program is just, let's say, a long string, right? And in order to to make sure that certain parts, which are sort of essential to the evaluation, for example, into the imports and so on, were not sort of deleted by the LLM mutations. And there are so-called markers, which basically state which parts of the code are mutable and evolvable. And it's easy to like programmatically sort of make them actually mutable when you get a diff proposal. And these will not be changed. So only the the rest of the the code snippet will be changed. We sort of implement a type of rejection sampling with reflection approach, where if an LLM by chance, for example, tries to mutate this part, it's going to be rejected and you re-sample a new proposal.

• 36:13Speaker 02

  And yeah, thereby you you can somewhat mitigate certain security or safety problems. And yeah, get a robust sort of mutation. One of the sort of, I think, the the bigger questions is how can you turn this from a single file mutation setup to a multi-file mutation setup? So working on an entire code basis, in principle, you can Represent many code bases in a single file, right? But the hierarchical structure might be actually useful. And there are some ideas from, let's say, Ada, this coding tool, where you construct like a repository map and sort of have some level of abstraction, but they also come again with positive and negative trade-offs, basically.

• 36:57Tim Scarfe?

  I love Ada, by the way. It feels that in the future, the code generation systems will actually resemble Shinku evolve. And if you think about it, it'll be using some kind of Git repo. Maybe Curse already does this, because in Curse you can restore previous checkpoints. But it can be exploring different branches and merging checkpoints together. And, you know, obviously, you just say in natural language what you want to do. But we didn't talk about mutation, by the way. So we just spoke about diffs. And there's also an option to do the full file rewrite.

• 37:32Speaker 02

  Exactly.

• 37:32Tim Scarfe?

  But there's also this notion of crossover. So how does that work?

• 37:35Speaker 02

  A small innovation on top of Alpha evolve, where I believe they only use sort of diff-based mutations, is that here we wanted to have more flexibility to entirely rewrite the program, right? To come up with a completely different stepping stone, if you will. So again, there you can make parts of the code mutable. But instead of proposing, let's say, a patch to change certain parts of it, we essentially rewrite the entire program. And this sometimes is helpful, right? It's not always like a clear benefit, but it allows you to essentially get more diversity into the search, right? So this is one type of mutation next to sort of this diff-patch-based approach.

• 38:17Speaker 02

  And the other one is a crossover mutation where we sample basically not only a single parent program, but sort of two different ones. And we ask the system to sort of make a complementary improvement. And again, on some problems, this is really helpful, and on others, it's not. But generally, we found that sort of having a diversity in terms of operators is also helpful in discovering new things. And I wanted to sort of follow up on the point you made before about this sort of being a new parent. I think so too. I'm really convinced. I think right now we're sort of at the beginning where we we still think a lot about sort of this chat assistant interface. As the way how we interact with L L M's, but it's most of the times inherently single threaded, right?

• 39:00Speaker 02

  So we're sitting in front of the computer, we're interacting with the chat, we're seeing sort of changes as they occur in the editor, we accept them and so on. But I think this is sort of also just a stepping stone towards sort of a more, let's say, distributed way about thinking about research, optimization and so on. So I like to sort of think of Vibe coding, Vibe chatting. And on the other hand, we have sort of Vibe optimization and Vibe researching, where sort of my ideal future scenario is one in which you as a researcher, sort of during the day, co-work with like a system like Shinkka or the A I scientist, you sort of steer the ship like a shepherd in some sense.

• 39:41Speaker 02

  And then during the night, you you you you press play and you go to bed and in this in the background, you have multiple experiments running. And automatically new ones being proposed by L L M's, evidence being accumulated. And then in the morning, you come back and sort of you have a multi-threaded sort of system running in parallel. And you're more like the shepherd of this ship than the person actually. Executing experiments and analyzing, or you're still analyzing, but you're not executing, this is happening sort of by the system itself.

• 40:10Tim Scarfe?

  Yes, and increasingly, this might be semi-supervised or even proactive. I mean, you know, there's that new product from Open A I where it knows what you're interested in and while you sleep, it's going off and you know, yeah, pulse, that's right. And, you know, we're in the situation now where we're reasonably technical people, so you know, MATLAB and Mathematica, they're they're supremely powerful. But you need to know how to express problems precisely. Whereas I can imagine a future where we express problems just in natural language, or maybe just based on our interactions with language models. The platform knows what we're interested in and it can just go and find things on our behalf. Because this is about democratizing this technology to people who perhaps don't know exactly what. What they're looking for.

• 40:55Speaker 02

  I think one of the bigger problems there is sort of this verification aspect to it, right? In the sense that oftentimes it's easier to generate a lot of solutions than to actually like hard verify them, right? Language models are capable of doing sort of soft verification, looking at code and sort of uh litany running like a like a stack trace of execution, right? But it's not exact, right? And I think sort of these notions of reward hacking and sort of um not doing real discoveries, but sort of short cutting them is one where we need to put more time and effort into to figure out um yeah, how to make sure that this actually moves in the right direction, right? And I would hope that language models at some point can do this efficiently themselves, right?

• 41:38Speaker 02

  So either implementing in code or latently doing it. But this is also like part of the problem problem, right? It's not only coming up with a problem, but also with the automatic verification at the same point.

• 41:50Tim Scarfe?

  Yeah, isn't it a tantalizing idea that there are natural patterns in the world? And the building blocks to construct novel solutions are already there, right? And and maybe they're there for a reason, maybe they just reflect natural regularities in in the universe. Because there's always this question of, you know, intelligence is about adapting to novelty. So the world is always changing and the world tomorrow will have things that we can't explain, you know, with our with our knowledge today. But we do have like abstract knowledge that could be easily recombined to explain the future. And L L M's might already have those building blocks.

• 42:29Speaker 02

  Yeah, for sure. I think like in some sense, the more you think about sort of Occam's razor applying to everything in our world, like let it be language or let it be sort of science. Is is pretty interesting because like these artifacts now go into our language models of today. And potentially there is some amount of this being captured. I think though it might also be an interactive bias that leads to a local optimum at some point, right? And you need more complexity. But I do think like with systems that sort of do this evolutionary mutation sort of style approach, you might still sort of push. The system out of this local optima eventually.

• 43:03Tim Scarfe?

  Yes, and there's also the notion of the importance of adaptivity. So this is what Sholay says intelligence is. And since we've had these models that actually do adaptivity at inference time, so things like test time, active fine tuning, and the reasoning models and so on, they started getting non-trivial performance on R. Now, it's very, very expensive to have adapting huge foundation models. You know, it's just a practical concern why we haven't done that yet. But what we can do is build systems like Shrinker Evolve that leverage the best of both worlds. So they leverage frozen foundation models, but they give you adaptivity. And the purpose of adaptivity is to respond to novelties, to create new building blocks,

• 43:46Tim Scarfe?

  synthesize new building blocks in this principled tree-like structure that allow us to adapt to novelty. So we are having our cake and eating it.

• 43:55Speaker 02

  I have to say, I found it very interesting that Jeremy, basically in your podcast, when you asked him about Chinkai, was saying like, he doesn't believe that there are a lot of sort of percentage points to be gained by using a system like Chinkai. But you can make it much more efficient, right? That was sort of the gist of his answer. And to me, it's like, once you have made it much more efficient, you can scale it up again, right? So if you essentially have a cheaper system that can generate many more sort of instructions, I would expect that by the nature of open-endedness, you might get some amount of improvement out of it. Right now, I don't have any evidence for it. I would love to collect that evidence. It's again, like the magic of open-endedness that comes into play, that as long as sort of these training examples of RKGI give you a good signal for a final test submission, you should be able to progress.

• 44:40Tim Scarfe?

  Yes, and that is a great segue because certainly on the circle packing problem, it was so sample efficient that in less than 200, you know, interactions with an LLM, you converged on the solution. But I was thinking that Great, but it's still quite dependent on the starting conditions. You know, we talk about this design bias and so on. So what we put in is very important. But now what we could do is scale out. So we could run this a thousand times and we could have another process which prompts, generates, breeds the starting conditions. Because every time we run Shinka Evolve, what it's doing is it's searching parts of the epistemic tree. And what would happen if we just scaled that out massively?

• 45:22Speaker 02

  We haven't tried, but you could even start with like an empty program, right? Which would be basically the same, right? And then you would branch off of that empty program, I would expect. Yeah, we haven't done this simply out of sort of cost and time reasons. But I do think in many ways, sort of, this is the question that will push us towards like this true open-ended vision of running a system for like a month or so, right? Really trying to squeeze this out. Yeah, I'm not sure if we're entirely there yet, but I will do my best that we will.

• 45:51Tim Scarfe?

  And the reason this is interesting is we know as a practical matter that we can't start with nothing. If we were just sort of like starting from the most primitive building blocks, the search space would just be huge and there'd be no learning signal. So we know we need to start a little way up the stack, but we can massively parallelize that. So yeah, let's say we have a thousand different instantiations of Shinka Evolve. It doesn't have to be embarrassingly parallel. We could still have some sharing. So during their execution, we could still have a little bit of like crossover and maybe then we could run all the Shinka Evolve instantiations in a similar kind of meta-evolution loop. And my suspicion is contra Jeremy, I agree with you.

• 46:31Tim Scarfe?

  We know there are diverse stepping stones out there that could dramatically, dramatically improve many of these solutions. We simply haven't scaled it up yet.

• 46:40Speaker 02

  Yeah. I also believe that using a system like Shinka Evolve could be able to sort of automatically detect whether or not like an instruction-based optimization approach for a given problem or a transform-based approach is actually the right thing to do. And sometimes potentially it's like even the mixture, right? There's some things you can probably easier even articulate in Python than you can articulate in sort of language, right? So I would be really interested in sort of exploring that.

• 47:08Tim Scarfe?

  Yeah, I mean, you said earlier about Jeff's, what was Jeff Kloon's paper, the thing that generates?

• 47:13Speaker 02

  Automated Capability Discovery.

• 47:15Tim Scarfe?

  I did speak to him about this at NURIS, but something like that could be fascinating as well. You know, we're also generating the problems and solutions and then kind of moving the back end. But I think the way this will land commercially is there'll be a new type of GPT where everyone is solving different types of problems and the system, it'll be like a kind of chinker revolved but a massively distributed version where mathematicians are using the platform over here to solve this problem and it will see commonalities and it will kind of like link them together. Because you need to leverage like human creativity in this process as well, I think.

• 47:48Speaker 02

  Like a big challenge going forward is going to be like, how do we change our incentive system for this to actually scale, right? I think like, for example, some amount of economy will be needed or some amount of mechanism design in order to make sure that everyone is still happy to engage in it, right? So maybe we're going to have many more leaderboards for whatever is numerically sort of scorable. And I think this will be really, really interesting to see how sort of compute these automated agents, human shepherding and steering will ultimately sort of change and revolutionize science and I guess society more generally.

• 48:25Tim Scarfe?

  And Rob, looking at the future, we've got a load of people in San Francisco that want to scale language models and they are adding in implicit forms of adactivity and composition so that they're building controllers and they're doing reinforcement learning with verifiable feedback and so on. I think that you subscribe to the slightly different idea that we need to be far more open-ended and we need to be using evolutionary algorithms and so on. But do you think that they are on a path to nowhere? Do you think they might change tack? I mean, where is this going?

• 48:55Speaker 02

  So I actually think that these things can be complementary, right? In the sense like, let's say you find your model to be like a circle packing. Expert, right? So I do believe that mixing in sort of different sort of RL fine-tuned models into sort of the ensemble of models and then having a good way to adaptively select which model to use is not a bad idea, right? So to me, I just very fully subscribe to this philosophy of open-endedness and reading Ken's and Joel's book was really like a fundamental moment in my life and I want to see how far we can push this. And I think we're not yet at sort of convergence

• 49:39Speaker 02

  where either the capabilities of the models has converged or the way how we scaffold around them or the way how we humans interface with them. So to me, they're really like these three points, like model capability, model scaffolding and sort of the user interface. And I think we have a lot still to push on all three angles.

• 49:59Tim Scarfe?

  Beautiful. The only thing we didn't talk about was, we spoke about the circle packing problem, but you also applied it to a few other things. Can you tell us about that?

• 50:06Speaker 02

  So one thing we did was we sort of used a framework called ADAS, Automatic Design of Agentic Systems, where basically instead of manually writing an agent scaffold, you use an LLM to write agent scaffolds for a specific task, right? So what we did is we looked at mathematics tasks, so AMI, and we used Chinkha to evolve basically an agent, right? So using an agent to evolve an agent. And we found that there we could dramatically improve sort of the performance of very cheap models like GPT 4.1 Nano. But the agent scaffold was also able to either like generalize to other language models

• 50:46Speaker 02

  or to different years of AMI, right? That was one application. One important other application that we did was to AL-E Bench. AL-E Bench is basically work done by other folks at Sakana, including Yuki, who's also part of the paper, which is considering heuristic sort of programming contests, sort of previously done and executed by Adkoder, which is like this famous Japanese competitive programming. Um. Organization and we sort of showed that uh Shinkka can also work very well as a coscientist. So basically, we we took initial solutions obtained by um an A L E agent that was previously designed.

• 51:27Speaker 02

  And then we optimized on top of these initial solutions with Shinkka and showed that on one of these um sort of programming tasks, if the combination of this agent and Shinkka would have competed in the challenge, it would have um ranked second place, basically. So I think there's uh some evidence that Shinkka can work as a coscientist and um not only for L L M agents, but potentially even for humans like we discussed before. And then finally, the final application that we looked at was um designing sort of mixture of expert load balancing loss functions. So at Canna, we've done some uh previous work called Disco Pop. I think we discussed this during the last podcast we did, where we are using L L M's to design objective functions. And back then, we did it for preference optimization in post-training.

• 52:12Speaker 02

  And here we did it for load balancing of um mixtures of experts. Also there, uh we found that within, I think, like even only twenty sort of generations, uh we were able to sort of um explore, let's say, not only a single objective function, um but sort of uh, let's say, a convex hull, where there are different trade-offs between sort of performance and load balancing and so on. So I think this is another application of Shinkka, where it's not only basically about sort of finding the best solution, but essentially illuminating a program space where there are always potential trade-offs between, like, let's say, for example, runtime and the quality of the circle packing, right? And um having a system that can explore all of these is important as well.

• 52:55Tim Scarfe?

  I'm very excited to see you apply this to the ARC challenge. Like, what are what are your thoughts about that?

• 53:00Speaker 02

  I still need to collect results. So I I don't want to make any claims like or hard claims before having done this. Um. But I would hope that there is some chance of for sure improving sort of the the cost of these systems and then potentially even performance. But yeah, to be seen.

• 53:17Tim Scarfe?

  Oh, very exciting. So you've done some experiments. Exciting news is potentially coming.

• 53:21Speaker 02

  I've started looking into it.

• 53:23Tim Scarfe?

  Yeah, and I mean, what are your thoughts in general about about ARC, though?

• 53:26Speaker 02

  I think it's great. I think it's uh it's really important and I think um. It fills an important gap and I do really deeply respect Francois and sort of read the paper when it first came out and no one thought of actually being able to to get numbers above 10%, right? And it's also pretty fascinating on a society level how far we've come since then. And sometimes while you're sort of deep in the, let's say, battle mode or work mode, you can forget where you were one year ago and then just looking back, it's pretty amazing. Also, how far we've come since, oh, what?

• 54:05Tim Scarfe?

  It's insane. I think Francois doesn't get enough credit because it's such a good benchmark and not necessarily for reasons people think because Francois is always saying that we need to have a benchmark which is easier for humans and hard for AIs. And in a sense, that's not quite the case. I said when ARC V2 came out that it's actually very difficult for humans. That, you know, there was one task where Dogar was stumped for about 15 minutes. There was three of us looking at it and we just, and it's one of those things that depending on your perspective, you might get it straight away or you might not. So there's that criticism. And people have said that ARC V3 is even harder, you know, but I think that's rather missing the point. I think he's saying that with a lot of these competitive coding problems, the dataset is contaminated. These are problems

• 54:48Tim Scarfe?

  that have been solved before in part or in whole, which means when you look at the epistemic tree, many of the building blocks for solving them are very high up in the tree. He's looking at these problems that there is very little dataset contamination and they need to be solved from very abstract building blocks. So you're starting much lower down the tree and you're synthesizing a model by composing together very abstract building blocks, which is the essence of intelligence. And I think for that reason, ARC is really kind of pushing us to build adaptive systems which we could say are intelligent.

• 55:22Speaker 02

  Yeah, I agree. I mean, like, in many ways, I'm really looking forward to the next years and seeing how far we can push this and then also how much generalization we can get afterwards. Because I believe like when you look at sort of the more recent models, they're getting much better at the transform style code evolution or outputting for ARC than they are on the instruction based level. And I think this might already be like a small sign of some amount of overtraining on ARC AGI 1 at least, right? I do believe there are some aspects of work which Will be automated before it comes to sort of fully science automation and the type of work I'm doing, but I could imagine that certain parts of the dimensions that I deal

• 56:07Speaker 02

  with every day are for sure going to be hit by AI. And then the question is, are there going to be new dimensions opened up that we as humans will fill in, right? And I think what I said before about like shepherding and so on, I really hope that that's the way forward, right? In the sense that humans are the ones steering the ship while just being massively amplified in their productivity.

• 56:31Tim Scarfe?

  Right now, I am not really seeing the kind of job market disruption that was being predicted. I know from personal experience that in a sense, it's made it very difficult to hire people. You know, script writers use ChatGPT, I can spot it instantly. And writers and copy editors are actually in more demand than they were before fixing all of the crap that has been generated with ChatGPT. And there's the cloud analogy as well. So, you know, IT system administrators who were earning, you know, sixty thousand pounds a year in the UK, they rebranded as as cloud or DevOps engineers and they more than doubled their pay. And people are very adaptive. They see new trends, new bandwagons,

• 57:15Tim Scarfe?

  and they just adapt and they add value on top. And that has been the trend for, you know, for a very long time. Do you think that AI is going to be so transformative that it will transcend people's ability to adapt?

• 57:29Speaker 02

  I think it's just a question of speed, right? So I was talking about sort of cultural evolution and technological evolution. And it seems like we humans, we need more adaptation and more time to get used to the technology to carve out these niches where we can fill in and it's complementary, right? So first off, I think we're still not at the ceiling of the sort of technological progression, right? So maybe in a couple of years, we will need less of sort of slop editing, like you said. But I do think we need some more time to adapt to the different modalities of interacting with these systems, right? I think everyone can sort of interact with a chat assistant. But I think this is the

• 58:12Speaker 02

  most sort of naive form of interacting with AI agents, for example, right? So, yeah, I think. We need to get the pacing of all of this right and we need to do much more exploration in human machine interfaces, UI UX design and how to make sure that humans sort of feel or feel fulfilled during this experience.

• 58:36Tim Scarfe?

  This is particularly relevant because, you know, you were behind the AI scientist paper and there's now version two of that. Allow me to be a tiny bit skeptical, you know, we were talking about when we evolve systems to do a particular thing. And at the moment, it feels like as good as they are, they are still quite parasitic on the instructions and intentions of the human supervisor. So it's very much an exchange between the humans and the system. Because the implication is that in the future, we might have systems that are so autonomous and so open-endedness and can figure out valuable things to research that humans wouldn't be needed anymore.

• 59:18Tim Scarfe?

  And the reason why I'm not that worried yet about labour market disruption is I still believe deeply that humans are the source of deep understanding and creativity in the world. If I didn't believe that, I would be very worried.

• 59:32Speaker 02

  I agree. To me, like the AI scientists, like V1 and now V2 are sort of glimpses into a potential transformation. But I fully agree in order to make really big scientific breakthroughs, like multiple of them, like every day or whatever, you still need humans in the loop to sort of either seed or guide the direction in which to explore or to verify, check and actually, yeah, transfer these insights, right? So I think it's not going to be like all ML PhDs will be unemployed. It's more going to be a sort of core evolution of humans with this technology and potentially

• 1:00:14Speaker 02

  like in an ideal future for me, like it will allow humans to focus on what they're really, really great at, right? So I think it's going to be an amplifier of sort of these latent dimensions humans are great at, right? I think something that's critical is that we as humans try to interact with these systems as early as possible in order to actually like have influence and ownership over like this development process, right? It's ultimately collective intelligence that will shape all of these systems together.

• 1:00:43Tim Scarfe?

  And do you think these systems can become incredibly sophisticated such that they are, you know, somewhat detached from humans?

• 1:00:53Speaker 02

  Well, I mean, with the AI Scientist V2, we sort of released that one paper that we submitted to an ICLAR workshop was able to sort of pass the acceptance threshold before meta review. So I do think, at least for sort of workshop level contributions, we're getting there. While not every submission in an AI Scientist paper does, is, or is reaching that threshold. We're at the point where we can even talk about sort of noisy review processes and this actually being, yeah, something that as long as you have a large budget, you might get something out of it. I think going forward for the bigger innovations and so on, for now, you still need humans.

• 1:01:39Speaker 02

  But we're sort of at the GPT one moment of making this sort of a reality and potentially in 10 years, this is going to look very, very different once the sort of also the infrastructure for it has been built up, right? So there are places like periodic labs, right, which sort of now are building like real physical labs with robotic systems to automatically sort of execute experiments. This will take some time, but it is sort of imaginable for sure that as we sort of do RL on these types of systems and we actually also account for negative results and for actual like hypothesis testing, so getting these systems to be a real good hypothesis testers with verifiers in the loop, that we might be able to unlock many more capabilities.

• 1:02:24Tim Scarfe?

  Yeah, I mean, I suppose I don't want to sound like a Luddite. So it's entirely possible that this is just, you know, I don't have the imagination to think about the future. So it is possible that in the future that these systems might understand very deeply and be creative, you know? I think right now the problem is they only understand things a few levels down in the epistemic tree. So they can do some surface level recombination and they can discover new things in the basin of things that have already discovered, but, but we understand things very deep down in the epistemic tree, which means our, you know, our cone of creative potential is much wider. It's possible that that gap might be closed. What would happen then?

• 1:03:06Speaker 02

  The way how I kind of think about the scientific process is like a tree search ultimately, right? So I think a lot of sort of analogies from evolution transfer to scientific research, right? In the sense that we traverse a tree of different ideas or different experiments. And then in the paper, we report one path through that tree. And I think what I kind of alluded to before, we need much more like full tree data sets for training these L L M. Systems to actually learn how to do this exploration and this foraging, basically. At the same time, I feel like evolution will also take place on a cultural level like for us, right? We will get better

• 1:03:46Speaker 02

  at sort of steering the ship. And I can imagine that in a future world, sort of the way how we do research will be completely different. And I'm pretty sure that right now, already 99% of machine learning research is done with sort of AI assistance, right? Think about chat G P T brainstorming, cursor coding, Claude code, et cetera. In the long run, we're going to move on that spectrum from sort of with AI closer to by AI and then sort of more high level sort of orchestration and overseeing by humans.

• 1:04:19Tim Scarfe?

  There's also the notion of how intrinsically coupled to humans is the value function. So one school of thought is that AI will develop a mind of its own and it will, you know, basically transcend humanity and it will just have agency, which is not parasitic on ours. I personally don't subscribe to that view, but the other view is that it is like, let's say the AI scientist, you know, like version 10, it's going to be continually epistemic, you know, epistemic foraging, it's going to be finding new things that are useful. And they kind of have to be useful to us. Because if it finds things that are not useful to us, then we just won't use them. And then nothing will happen. So do you think there'll always be a kind of coupled value function to humans?

• 1:05:04Speaker 02

  Jeff Clune had this work on Omni, right? And using L L M. S. As sort of amortized notions of interestingness for humans, right? And I think ultimately, the way how we train these systems is coupled in in human data, right? And going forward, it will also be coupled with human data that is collected using verifiers, right? So I have a hard time believing that. In the long run, when you run this open-endedness sort of paradigm with AI scientist agents, it's going to completely divert to something that's either fully non-interpretable or unrelated to problems we as humans care about, right? And then again, like, humans can steer to a certain degree where, like, the search happens, right?

• 1:05:48Speaker 02

  So you can tell a system, okay, try to do cancer research, right? And sort of work on problems that we care about. And ultimately, like, we are the ones who control how much flops are being pushed into this.

• 1:05:59Tim Scarfe?

  Yeah, because as a thought experiment, I can imagine, let's say, in the world of mathematics, what if an AI scientist could come up with entirely new problem formulations and then solve them? And these are things that humans had never conceived of before. And maybe they would be less interested in the answer because humans hadn't spent time thinking about it. And if you think about it, we could just explore the phylogeny of mathematics just to the nth degree. And at some point, maybe we just wouldn't care anymore. Maybe we can just carve out that space just forever and ever.

• 1:06:28Speaker 02

  Yeah, but maybe down the road, there is a stepping stone that enables a new innovation in a different field that we actually care about, right? So it's very hard to say a priori whether or not something is interesting or not, right?

• 1:06:41Tim Scarfe?

  Yes, and there's also the notion of, I love this idea of diverse intelligences and diverse minds. And maybe we could just create artifacts in a space which is completely alien to us. And we might even ascribe moral value to them. And we might not want to turn off, you know, the power because we want these alien artifacts to stay alive.

• 1:07:03Speaker 02

  Maybe, like, I'm, I read a lot of science fiction, but I would sort of shy away from speculating about all of this. But I do think one thing I'm extremely certain of is that the way how we conduct research and science is going to fundamentally change in the next five years, 10 years and 20 years. And I hope that we're going to be able to sort of tackle some of the biggest problems, which are still. Sort of seemingly unreachable right now with and by AI.

• 1:07:32Tim Scarfe?

  So Terrence Talm has posted that he's been using G P T five to and and it's it's been speeding him up. It's it's taking away a lot of the the drudgery. But the cynical take is that and Scott Aronson posted something similar as well. The cynical take is that maybe laziness is is stepping in and in some pernicious way. Using AI models is actually stopping us from thinking outside the box. So it's it's encouraging us to kind of search in the neighborhood of things that are known. And that is very useful. It's very useful to have an artifact that knows all of the experiments, all of the things that are ever done by people twenty years ago.

• 1:08:10Tim Scarfe?

  But now we don't have people really kind of applying their their brilliance, their talent in completely new areas.

• 1:08:19Speaker 02

  So first off, it's great that these experts are already using the technology in their day to day work, right? And I think it's also important that really, really top level scientists try to push what's capable with these systems or squeeze out where there might be sort of black spots or stuff where these systems can't do. Um. Second off, I think it comes down sort of to discipline and how we raise sort of the next generation, right? So discipline on a personal level, like how much do you just sort of tap accept everything that's being proposed by these systems? And um responsibility in terms of educating the next generation in the sense that we need to sort of teach our kids that ultimately what comes out of these systems might not always be be true.

• 1:09:02Speaker 02

  That uh facts can be a sort of uh subjective, if you will, and that there needs to be more research about uh what's being given to you. And um I think this will be, like I said, this cultural evolution that we have to step through and um try to make the best out of.

• 1:09:18Tim Scarfe?

  Yeah, the autopilot thing is very interesting because there is a tendency using cursor just to, you know. At at some point, the models are getting so quickly that you can't even read the tokens coming at you, and then you just press accept and you press accept. It's the same thing in cars that as soon as you have too strong of an autopilot, you just completely switch off and. And then you see a divergence because there's something about thinking that it must be grounded on your path. There's this path dependence. And when you start kind of becoming parasitized by this other train of thought, then you stop thinking about your path and then you're not in the driver's seat anymore.

• 1:09:54Speaker 02

  This is now like a bit of a harsh statement, but sometimes I wonder if these systems, like these coding assistants, are almost like drugs, right? In the sense that you become addicted, you you use up all your sort of budget and then you need to load up again. And once you you fully reached sort of the the budget limit, you feel like, okay, what am I going to do now? And I think once that happens to you, you should really sort of rethink the way how you work, right? And to me, right now, there are certain parts where like sort of auto-accepting is acceptable. And there are certain parts where it's definitely not, and you really need to go deep into it. And I think right now we're sort of in this weird non-equilibrium state where things are moving constantly, right?

• 1:10:39Speaker 02

  So the systems or the models are changing, the features are changing, the sort of parts where the systems are good is changing all the time. And we humans need to constantly adapt to to that, right? And I think it's a big cognitive challenge. And I think we just all need to be aware that there are certain problems and certain challenges that we have to adapt to. I think the best way to do so is just interact with this technology as much as you can and maybe find new research ideas for out of that experience.

• 1:11:14Tim Scarfe?

  And how is AI scientist V two different to V one?

• 1:11:17Speaker 02

  In V one, we we use sort of a template-based approach. So we had like a base experiment. And then for that base experiment, we asked sort of an L L M to generate ideas sort of with semantic scholar calls and sort of literature search. And then it implemented sort of these ideas based on the template, right? It basically code this. And then it linearly executed like an experiment plan and wrote a paper in the end. And so what could happen was that there was an idea and that idea didn't work out, right? But then in the end, the paper, like the experiment. Were still executed linearly and you wrote a paper. And this was already impressive in the sense that it looked very much like like science. But if you think about human sort of science and like the scientific method,

• 1:12:02Speaker 02

  it's much more like tree search, like I said before, right? You sort of adapt what you're going to execute next. And you sort of refine based on evidence that you accumulated, right? So this is sort of the the notion of falsificationism from from Karl Popper, right? In the sense that we collect evidence for hypotheses and reject we reject others. And we do so in in a loop, basically, until we we want to publish or we find something. And we tried to take this notion and directly build it into the agentic scaffolding for the AI scientist V two. So now it's basically like an paralizable agentic tree search, where there's no longer a template experiment needed, but this is drafted up by the LLM itself. And thereby, the AI scientist V two can be applied to many more sort of settings, if you will.

• 1:12:48Speaker 02

  So at the core is sort of this new agentic tree search paradigm. And then we use sort of a couple of minor technical changes, like using a VLM reviewer for sort of figuring out if captions of a paper are aligned with the figures. And we we scale this up to many more sort of computational nodes and then write a paper in the end.

• 1:13:10Speaker 00

  This episode is brought to you by Indeed. You're ready to level up your business, but the pressure to find the right person to make that happen is intense. Indeed sponsored jobs can help carry the load. It boosts your job post to help it reach quality talent fast. Cut through the chaos with Indeed sponsored jobs. Get a $75 sponsored job credit at indeed.com/podcast. Terms and conditions apply. Hiring now? Then this is a job for Indeed sponsored jobs.

• 1:13:41Speaker 04

  Low energy, unfocused, foggy, you might be dehydrated. Gatorade Zero powders have been scientifically designed to help improve hydration balance in the body with a clinically proven electrolyte blend and no artificial colors or flavors. Great tasting hydration, no matter where you're going. Shop now online or at retailers nationwide. Gatorade hydrates better than water. Is it in you? Symptoms noted are signs of mild to moderate dehydration. Consult a healthcare professional if symptoms persist.

• 1:14:09Tim Scarfe?

  And again, so I'm trying to say this in the most polite way possible, but a critic might, I don't want to use the word slop, but a critic might say, we are producing papers which appear like papers, so they have figures and they have results and they have things written in a certain way, but they're not grounded deep down the epistemic phylogeny, which means that they have, you know, near the top of the tree, we're seeing some novelty in composition happening, but it doesn't reflect a deep understanding. What would you say to that charge?

• 1:14:41Speaker 02

  It's for sure that not every paper that comes out of the AI Scientist V2 is a nature-worthy publication, right? That's for sure the case. So definitely there is some amount of, let's say, slop or content that is not like a scientific big discovery being written up by the AI Scientist V2. But ultimately, like, we showed that it was possible to obtain a workshop-level paper. I do think this is sort of the first time, basically, where we can see that at least now we're able to fully autonomously spend compute, spend API calls to obtain some amount of scientific insights. And for me, at least right now, it's a good way to sort of prototype ideas or to investigate

• 1:15:22Speaker 02

  a certain field, get like initial starting point, initial results, and then to work on top of it. But for sure, more work needs to be done to make this entire process more robust, more efficient, and essentially produce many more sort of true positives, if you will.

• 1:15:40Tim Scarfe?

  Yeah, and it might be one of these things, you know, like when we moved from GPT-3 to GPT-4, there was just a massive increase in fidelity, because the thing is with slop, to me... It simply means lack of deep grounded understanding. And there's no reason in principle why these things couldn't have a deep grounded understanding, they just don't have it yet.

• 1:15:58Speaker 02

  Yeah.

• 1:15:58Tim Scarfe?

  So it's something that could improve over time, but it's likely to improve quite slowly. And then at some point, we might just think, oh, my God, we've got an AI scientist.

• 1:16:07Speaker 02

  Yeah, I mean, like, to me, this this kind of comes back to what we were discussing about before. So first off, there is a verifier in the loop, right? Or in the sense that experiments are actually executed on a computer, right? So the numerical results can be fed back or are fed back into the system to come up with the next thing to explore. But like, we haven't made like a, let's say, discovery, like a residual connections or something that have diffused into everything in machine learning. And I think what we really need is to make these systems be much better at sort of integrating knowledge over multiple experiments and sort of become better at sort of formulating the next hypothesis based on previous insights.

• 1:16:51Speaker 02

  And, yeah, this might require some amount of post-training on sort of these traces, basically, but I'm pretty positive that we might also get there with just diversity and scaling these systems up in an efficient but a scaled up way.

• 1:17:07Tim Scarfe?

  I'm just thinking that the first breakthrough discovery, would it resemble the AI Scientist paper, or would it resemble Shinka Evolve? So, for example, we could do like a massively scaled up Shinka Evolve, and we could say, I want to discover a new architectural design.

• 1:17:21Speaker 02

  Yeah.

• 1:17:22Tim Scarfe?

  And would that happen? And then we would get the AI Scientist paper to kind of write it up and do ablations and stuff. Maybe that would be the the pattern of it.

• 1:17:31Speaker 02

  To a certain degree, I've been thinking a lot about how you can potentially even combine these two paradigms, right? The AI Scientist and Chinkar or Alpha Evolve style optimization algorithms. And I do think there is some amount of work to be done on sort of this auto verification sort of aspect to it, on the sort of problem formulation aspect to it. The paper writing part is actually the least important about the AI Scientist, right? It's a form factor that we humans are. Sort of used to, and it helps anchor our mental model of like a scientific discovery. But ultimately, I'm not sure if the paper is going to be the the knowledge transmission medium in, let's say, twenty years, right?

• 1:18:13Speaker 02

  Something else I've been thinking of a lot is whether or not we can make papers much easier agentially accessible, right? In the sense that right now it's it's it's a LaTeX document, but you could imagine sort of equipping every paper with sort of several model context protocols so that every figure is reproducible, data is accessible, and essentially make it much easier for L L M agents to essentially either replicate work or to work off of them afterwards, right? Doing sort of absolute improvements, ablations yourself through that interface to a paper.

• 1:18:51Speaker 02

  But to be entirely honest, I'm not sure if it's going to happen because there have been many great ideas for improving sort of, let's say, the the format of scientific artifacts out there. And people still seem to to like the paper format, which has existed for, let's say, hundreds of years, right? So I think it's a question of incentives again, and really showing that if something like that would exist, it would enable much faster progress of A I agents for scientific discovery.

• 1:19:21Tim Scarfe?

  Yeah, paper is a great human interface. It's a similar thing with automated driving, right? That we could revolutionize the road network to have sensors and we could dramatically improve the the monitoring and observability and optimization. But I'm fascinated by that idea. So so you're saying not just reproducibility of the experiments, but also the way that the figures are designed and and the code and and so on. Because then we could create this huge playground where agents can repurpose, recombine, re-study work that has been published by other scientists. And it also made me think, does like having an automated scientist, does that make peer review more or less important?

• 1:20:01Speaker 02

  I do think it actually makes it more important, at least for now, right? In the sense that we now have a mechanism or could have a mechanism that generates. Many, many papers, right? And it first increases like the workload on on on human reviewers and we need some effective way for filtering and then essentially only taking the cream of the crop for human verification afterwards, right? So I think for now, like the ultimate verification is still like the human and the diffusion of the result to the community. And we need better tools for doing this automatic filtering and verification. Like we have the AI reviewer that comes with sort of the AI scientist.

• 1:20:42Speaker 02

  But you actually probably need some form of experiment execution for actually verifying everything. Yeah. But there is, for example, work by OpenAI on on paper bench and trying to go into that direction using sort of LLM soft verification and these types of things. So I I'm hopeful that we're going to figure this out in the next years.

• 1:21:02Tim Scarfe?

  Yeah, and I think one of the Rubicon moments is when the the new transformers architecture or something massive is discovered by AI and we're all using it. My worry, I suppose, is that probably folks like Google who have enough compute power, they're going to be running AI scientists and they're going to own many of these discoveries, which is why it's so important to have work which can efficiently discover new things in science.

• 1:21:26Speaker 02

  And it's important to have work that's openly available, right? I think like with the AI scientist and Shinka, we're really trying to to make sure that we can sort of apply the collective intelligence of all of us to to shape how this might look in the future.

• 1:21:40Tim Scarfe?

  Major, well, Rob, this has been so fantastic to have you on the show. Sakana is hiring amazing engineers, by the way. So if if this sounds like, and it is an amazing opportunity, get in touch with Rob and and the guys. And I trust you're working on some exciting new things that are coming up.

• 1:21:56Speaker 02

  Yes, and I I hope to be able to talk to you in the future again about some of this.

• 1:21:59Tim Scarfe?

  Absolutely. Rob, thank you so much for coming.

• 1:22:00Speaker 02

  Thank you so much, Tim.